The on-call alert threshold calibration decision record: why the calibration model you chose determines your false positive accumulation rate and your on-call engineer burnout surface

A 27-person B2B SaaS building a contract management platform launched their first microservice architecture in the second year of operation, decomposing from a monolith to seven services over three months. Each service was instrumented with CloudWatch metrics and alerts configured by the engineer who built the service, using thresholds chosen by judgment against the service's expected behavior at launch traffic volumes. The team had no alert threshold calibration process — no review cadence, no evidence requirement before setting a threshold, no mechanism for identifying alerts that were firing without requiring human action. The on-call rotation was two engineers sharing a PagerDuty schedule, each carrying a week at a time on top of their regular engineering work.

Within four months of the decomposition, both on-call engineers were being paged between two and six times per night on alerts that auto-resolved within thirty to ninety seconds without any intervention. The pattern was consistent: a metric would spike, cross the threshold, trigger a page, the on-call engineer would wake up, open their laptop, load the dashboard, and find the metric had already returned to the normal range. The alerts were technically accurate — the metric had exceeded the threshold — but the threshold values had been set against expected behavior at launch traffic volumes with no buffer for normal operating variance, and the services had grown to handle three to four times their launch traffic with corresponding increases in the legitimate variance of the metrics being monitored. A queue depth alert configured to fire at 50 items had been appropriate when the queue processed fifty transactions per minute at launch; at 150 transactions per minute, a queue depth of 50 represented a normal processing interval, not an anomaly. The threshold had not been updated because there was no process that required reviewing thresholds as operating conditions changed.

After three months of the paging pattern, the more senior of the two on-call engineers gave two weeks notice. His exit interview cited the on-call schedule as a primary reason. Two months later, the second gave notice citing the same reason. The engineering director, now managing an on-call coverage crisis, ran a retrospective analysis against ninety days of PagerDuty data and discovered that sixty-seven percent of the alerts that had fired during that period had resolved without any human action — the on-call engineer had not needed to take any step beyond acknowledging the page for two-thirds of all overnight interruptions. The analysis also found that eight alerts in the set had never once required human action in ninety days; they had fired eighty-three times combined and produced eighty-three interrupted sleep events across the rotation. The engineering director ran a three-day threshold calibration exercise, reviewing each alert against the operating data for the prior ninety days and adjusting thresholds to represent genuine anomalies rather than normal operating variance. The overnight paging frequency dropped from an average of 3.8 pages per night to 0.4. The calibration exercise had taken twelve engineering-hours. The cost of not having done it earlier was two experienced engineers and a hiring cycle. Connect this to the on-call escalation policy decision record: the escalation policy governs what happens after an alert fires and an on-call engineer determines they need help; the threshold calibration model governs what fires an alert in the first place; the two models operate at different layers of the incident response stack and must be designed together because the escalation policy's effectiveness depends on a signal environment where the first-responder can make a reliable triage decision within the mandatory hold time the escalation policy specifies; an escalation policy calibrated against a clean alert environment degrades predictably when alert noise rises because the hold time that was appropriate when 80% of firings required intervention becomes inadequate when 67% of firings do not, and the escalation model has no mechanism to adapt to the signal environment's degradation.

A 44-person developer tools company building a CI/CD orchestration platform had a threshold calibration process — one of the few companies at their size that had invested in formalizing how alerts were reviewed after initial configuration. The process ran quarterly and was facilitated by the site reliability engineer on the team. The review meeting examined each alert in the inventory against the prior quarter's firing data and made adjustments based on the team's assessment of which alerts were producing too much noise. The calibration input was predominantly alert firing frequency: alerts that had fired more than a defined threshold of times per week were candidates for threshold adjustment; alerts that had not fired in the quarter were reviewed for continued relevance. The process was genuine, well-attended, and consistently executed for two years.

The failure was in the calibration target. The process optimized for reducing alert volume — the number of times alerts fired — without measuring false positive rate or miss rate as independent inputs. In the third quarter, the team decided to tighten calibration more aggressively after the on-call rotation had produced feedback that alert noise was still high. The calibration review identified six alerts that had fired more than forty times in the prior quarter and recommended raising their thresholds. The SRE facilitated the adjustment without a systematic check against incident data: for each of the six alerts, did the prior quarter's incidents include incidents that this alert had detected before customer impact? The team did not have a lightweight process for that cross-reference; incident records were in PagerDuty and the threshold review was in a spreadsheet; the cross-reference would have required pulling incident data manually for each alert candidate, which was not part of the review protocol. The adjustments were made based on volume alone. Alert volume for the quarter dropped thirty-one percent. The team reported the calibration as successful.

One of the six adjusted alerts monitored API service memory growth rate — a metric that had fired forty-four times in the quarter, nearly all of which were benign accelerations in memory usage that resolved within minutes as the garbage collector completed a cycle. The threshold had been raised from a 15% memory growth rate per five minutes to a 25% growth rate per five minutes, which eliminated the benign GC-cycle firings. Eleven weeks after the calibration, the API service developed a memory leak introduced in a dependency version bump — an object reference in a request middleware component that prevented the garbage collector from reclaiming request-scoped objects after request completion. The growth rate the leak produced was 18% per five minutes, above the original threshold and below the adjusted one. The alert that would have fired at the two-hour mark when the leak had consumed 400MB did not fire. The alert that would have fired at the four-hour mark when the leak had consumed 800MB did not fire. The API service degraded over three days as available memory contracted, causing increasing request latency as the service began competing with its own GC cycles for CPU time. No alert fired. A customer reported degraded performance on day three. The root cause analysis identified the leak and the memory growth rate alert's adjusted threshold as the detection gap within two hours of the customer report. The post-incident review found that the quarterly calibration review had no mechanism for cross-referencing threshold candidates against recent incident data to verify that raising the threshold did not remove detection coverage for a real failure mode. Connect this to the incident postmortem action item accountability decision record: the post-incident review produced an action item to add an incident cross-reference step to the quarterly calibration process; the action item was assigned to the SRE with a target completion date of the next quarterly review; at the next quarterly review, the SRE had rotated off the team and the action item had not been transferred to the new on-call lead; the calibration process ran again with the same protocol; the incident cross-reference step had not been added; the threshold calibration failure mode — volume-optimized calibration silencing real detection coverage — was still present; the action item accountability model, not the calibration process, was the proximate failure; without a completion verification model that was independent of the assigned individual remaining on the team, the calibration improvement the post-incident review specified was lost at the next personnel transition.

A 61-person enterprise SaaS building a financial data aggregation platform for wealth management firms had invested significantly in alert quality. The team maintained an alert inventory spreadsheet with 247 configured alerts across fourteen services, each with a documented owner, a threshold rationale, a last-review date, a false positive rate tracking column populated quarterly, and a runbook link required before the alert was active. The calibration cadence ran every six weeks — more frequent than most organizations at their size — and was taken seriously: alerts that had produced false positive rates above twenty percent over the review period were reviewed for threshold adjustment at each meeting, with the owning engineer required to bring the operating data that justified the proposed new threshold. The calibration culture was genuine. The false positive rate across the inventory had been driven down to eleven percent at the aggregate level over eighteen months of sustained effort. The on-call rotation of six engineers reported the alert environment as manageable and the noise-to-signal ratio as substantially better than their experience at prior companies.

The calibration model had a gap that the per-alert false positive rate metric did not surface: it modeled each alert in isolation and had no model of how alerts behaved in combination during cascading failures. In the twenty-third month of operation, a deployment of the data normalization service introduced a schema validation regression that caused the service to fail silently — returning 200-status responses with empty data payloads — for a specific subset of account type codes that represented seventeen percent of the customer base. The normalization service's failure produced a cascade: the aggregation service, receiving valid-format but empty-content responses, began logging error counts that triggered the aggregation error rate alert; the downstream enrichment service, receiving empty aggregation outputs, produced an enrichment failure rate that triggered the enrichment error alert; the API gateway, serving requests for affected accounts, began producing elevated response times that triggered the gateway latency alert; the notification service, detecting delayed enrichment, produced a notification processing lag that triggered the notification queue depth alert; the caching layer, receiving empty content to cache, produced a cache hit rate anomaly that triggered the cache performance alert. Within three minutes of the normalization service's deployment completing, thirty-four alerts had fired across six services. Each alert was technically accurate. Every single one was a symptom of the same root cause deployment.

The on-call engineer, paged by PagerDuty, opened the alert stream and found thirty-four active firings across six services with a complex cross-service dependency pattern that was not immediately apparent from the individual alert descriptions. The first seven minutes of the incident were spent triaging the alert set — acknowledging alerts, determining which were symptoms and which were causes, attempting to identify the root cause service from the alert topology. The normalization service itself had not directly triggered an alert in the first three minutes because its failure mode was a silent response corruption, not an error rate spike; the normalization service's alerts were downstream of the symptom propagation, not the source of it. At the seven-minute mark the on-call engineer identified the pattern as a cascading failure with a single upstream source. At the nine-minute mark they identified the normalization service as the probable source and began the rollback procedure. The incident was resolved at the twenty-two-minute mark. The P1 MTTR SLA for the platform was twenty minutes. The incident breached the SLA. Post-incident analysis found that a correlated alert model covering the dependency graph between the normalization, aggregation, enrichment, API gateway, and notification services would have produced a single aggregate alert — "normalization service failure, downstream cascade in 5 services" — within thirty seconds of the root cause service's deployment completion, eliminating the seven-minute noise triage phase and placing the incident well inside the twenty-minute SLA. The alert correlation model had never been built because the calibration process governed individual alert quality and had no mechanism for modeling inter-alert behavior during correlated failure scenarios. The calibration work that produced an eleven percent false positive rate on individual alerts and a seven-minute triage phase on a thirty-four-alert cascade were not contradictory outcomes — they were produced by the same gap: a calibration model that optimized for per-alert quality in isolation and had no model of the system-level signal the alert set produced during multi-service incidents. Connect this to the sprint planning format decision record: the alert correlation model the post-incident review specified as a required capability was estimated at three weeks of engineering work; at the next quarterly planning session it competed against a roadmap of product feature commitments and a reliability backlog with eight other items; it was ranked fourth in the reliability backlog on the grounds that the individual alert quality metrics were strong; the sprint planning format had no mechanism for prioritizing reliability investments against their expected reduction in incident response time, which would have made the correlation model's three-week investment obviously justified against the pattern of SLA breaches it would prevent; it was instead evaluated as a reliability backlog item competing on organizational priority against feature work and other reliability items at the same level of abstraction; it was not prioritized into a sprint for six months, in which period two additional cascading failures produced four SLA breaches.

Structural properties set by the calibration decision

Two structural properties are determined by the founding decisions about alert threshold calibration model, threshold review cadence and evidence requirements, false positive tracking and resolution workflow, alert correlation and suppression policy, and on-call experience governance. Neither is visible in deployment frequency, incident rate, or engineering health metrics until they accumulate to a production detection failure or an on-call rotation collapse — at which point both properties have been actively degrading for months, and the accumulation is not reversible by the interventions organizations typically reach for first: compensation adjustments, shift schedule changes, on-call load reduction through staffing additions. These interventions address the consequences of the calibration model's outputs; they do not address the calibration model.

Property 1: False positive accumulation rate and the signal degradation threshold. The false positive accumulation rate is the rate at which non-actionable alert firings compound into a signal environment where engineers can no longer reliably distinguish incidents requiring intervention from normal operating variance that will resolve without human action. It does not cause burnout linearly — it accumulates below the visible surface and passes a threshold at which engineer behavior adapts to the noise. The adaptations take three forms. The first is response delay: engineers who have learned through experience that a high fraction of their alert notifications are non-actionable begin introducing a delay before responding — checking their phone rather than opening a laptop, waiting to see whether the alert auto-resolves — which is a rational individual response to a high false positive rate and a structural increase in MTTR for the incidents that are real. The second is reduced diagnostic effort: engineers who have been paged twelve times in a week on non-actionable alerts allocate less diagnostic energy to the thirteenth alert, which produces faster clearance of non-actionable pages and slower, lower-quality diagnosis of real incidents. The third is unilateral silencing: engineers with access to alert configuration begin disabling or raising thresholds on the most irritating alerts without a calibration review, which reduces their personal paging burden and removes detection coverage from the alerting layer without any record of the removal. Each adaptation reduces the individual engineer's exposure to alert noise at the cost of degrading the organization's incident detection reliability; the degradation is individually rational and organizationally harmful; and once the adaptations have been established as behavioral norms on the rotation, they are not reversed by fixing the threshold calibration — the behavioral adaptation persists after the noise source is corrected, and re-establishing the prior response discipline requires explicitly addressing the trust deficit the noise period produced. The false positive accumulation rate is controlled by a single mechanism: whether the threshold calibration model requires empirical evidence of what normal operating variance looks like for each metric in each service before any threshold value is set or modified. A threshold set against operating data that represents the full distribution of normal behavior — including variance from deployment activity, traffic spikes, and infrastructure events — will produce a low false positive rate because it represents 'abnormal' correctly. A threshold set against expected behavior at launch or against an intuition about what the value should be will produce whatever false positive rate the gap between the expected and the actual operating distribution produces, and that gap grows as the service's behavior evolves away from the launch configuration the threshold was set against. Connect this to the on-call escalation policy decision record: the escalation policy's mandatory hold time — the period during which the first responder is expected to diagnose the incident before paging a secondary — is calibrated against a signal environment where most pages require intervention; if the false positive accumulation rate has reached a level where thirty percent or more of pages do not require action, the mandatory hold time represents thirty percent of pages spent waiting for an alert to auto-resolve rather than investigating a real incident; the two policies must be reviewed together at each calibration review cycle, because the signal environment that the threshold calibration model produces determines whether the escalation policy's parameters are well-calibrated for the current alert inventory.

Property 2: On-call engineer burnout surface and the interruption compounding model. The on-call engineer burnout surface is the cumulative cognitive and physiological load produced by the interruption pattern generated by the alert environment, and it is a function not of incident volume but of false positive rate and interruption timing. An on-call engineer who handles three real incidents in a week, each requiring thirty minutes of focused diagnosis and remediation, experiences a fundamentally different cognitive and physiological load than an engineer who handles three real incidents in a week and twelve false positive interruptions — even if the twelve false positive interruptions each clear in under two minutes. The physiological cost of an interrupted sleep event is not proportional to the duration of the interruption; the cost is in the interruption itself, the transition through the arousal-to-alert state required to evaluate whether the page is actionable, and the degraded sleep quality that follows a partial awakening even when the engineer returns to sleep within minutes. An engineer experiencing four overnight interruptions that clear without action experiences compounded sleep disruption at a cost that cannot be distinguished from four real incidents in cortisol and cognitive recovery terms, which means the engineering director's assessment that the alert environment is "manageable because most pages are quick" is measuring the wrong variable. The burnout surface is determined by interruption frequency, not by incident complexity or resolution time. It is also compounding: the chronic sleep disruption that an elevated false positive rate produces accumulates into a cognitive load deficit that affects diagnostic quality during real incidents — on-call engineers with elevated false positive exposure perform worse on the diagnostic phases of real incidents than engineers with the same training and experience who have not been chronically sleep-disrupted, an effect that compounds over the duration of the rotation rather than resetting at the start of each on-call week. The burnout surface is not managed by reducing shift frequency, offering recovery days, or improving compensation at the rates those interventions are typically applied — a compensation adjustment or a recovery day offsets a fraction of the cost of one on-call week at elevated false positive rates, while the next week's rotation reproduces the same cost. The only intervention that reduces the burnout surface is reducing the false positive rate. Connect this to the engineering manager span of control decision record: the span model determines how much bandwidth each engineering manager has for proactive reliability work beyond their direct team's sprint execution; running threshold calibration reviews on a six-week cadence, maintaining an alert inventory with false positive rate tracking, designing an alert correlation model for the service dependency graph, and governing the on-call experience as a calibration quality outcome all require manager bandwidth that competes with sprint logistics at span levels above the team's effective capacity; an engineering manager who is operationally saturated will manage the on-call burnout surface through the interventions that require no calendar time — schedule adjustments, compensation conversations, morale work — and will not find the time for the calibration review cadence that is the only intervention that addresses the structural source; the contract management company's engineering director took three months to run the twelve-hour calibration exercise that would have prevented the on-call burnout and both departures; the delay was not negligence, it was bandwidth — the calibration was known to be needed and not scheduled because nothing in the manager's capacity model carved out time for reliability maintenance work independent of sprint operations.

The on-call alert threshold calibration ADR: five sections

Section 1: Alert threshold calibration model. The alert threshold calibration model specifies what evidence base is required before any alert threshold value can be set or modified, who has authority to set a threshold without the evidence requirement, and what the default behavior is for new services that do not yet have an operating data history. The model exists because the most common calibration failure — thresholds set by intuition against expected behavior rather than observed behavior — produces a predictable false positive accumulation rate from the moment of deployment, and the accumulation is not visible in the alert environment until it has already begun degrading the signal reliability the on-call rotation depends on. Three components. Evidence requirement for new alerts: before any alert threshold is set at service launch or introduction of a new alert for an existing service, the team must observe the metric's distribution during a baseline observation period — minimum two weeks for metrics on services receiving production traffic, minimum four weeks for metrics with strong seasonal or traffic-driven variance — under operating conditions that include at least one normal deployment, at least one traffic spike, and any recurring operating events (batch processing windows, backup windows, scheduled maintenance) that affect the metric's value; the threshold value must be set above the 99th percentile of the observed distribution for alerts that should not fire during normal operation, or above the 95th percentile for alerts where some false positive rate is acceptable in exchange for reduced miss rate; the observation data must be retained as the baseline record that future reviews compare against to detect operating behavior drift. Evidence requirement for threshold modification: a threshold modification — raising or lowering a threshold for an existing alert — requires a review of the false positive rate for the prior review period, the incident data for the prior review period to verify that the proposed modification does not remove detection coverage for incident patterns that occurred, and a written justification specifying which of the following conditions the modification addresses: the service's operating behavior has drifted above the baseline that justified the original threshold; the threshold was set incorrectly against insufficient baseline data; a false positive root cause has been eliminated and the threshold can be tightened; an incident analysis revealed that the threshold value was too conservative and missed the onset of a real incident at its correct severity level. Default for services without observation history: when a new service must have alert coverage before sufficient observation data is available — for example, a critical service launched under time pressure — thresholds are explicitly tagged as provisional, set conservatively to prefer false positives over misses, and scheduled for a calibration review at the four-week mark using the first four weeks of operating data; provisional thresholds are reviewed at the next scheduled calibration meeting regardless of whether they have fired or not; a provisional threshold that has not fired in four weeks is a candidate for tightening, not a confirmation that the threshold is correctly set. Connect this to the engineering team offboarding decision record: the threshold calibration model must specify the knowledge transfer requirement when the engineer who set a threshold departs; a threshold set against a four-week observation baseline represents a calibration judgment — the engineer's interpretation of what the operating data showed about normal behavior — that the setting value alone does not communicate to the next engineer who reviews it; the threshold review spreadsheet must retain not only the threshold value and last-review date but the operating data summary that justified the current value, so the reviewing engineer can evaluate whether the operating behavior has drifted beyond the baseline the threshold was set against rather than treating the current value as an authoritative specification of normal behavior.

Section 2: Threshold review cadence and evidence requirements. The threshold review cadence specifies how frequently each alert's threshold is reviewed against current operating data, what evidence the review requires at each cycle, and what conditions trigger an out-of-cycle review. The cadence must be frequent enough to catch threshold drift before it produces a false positive rate that degrades the signal environment, but not so frequent that review fatigue produces cursory reviews that modify thresholds without adequate evidence. Four specifications. Scheduled review frequency: for services whose operating behavior changes substantially with traffic growth, dependency changes, or deployment frequency (typically product-facing services and shared infrastructure services), the review cadence is six to eight weeks; for services whose operating behavior is stable and changes primarily through planned configuration changes (typically internal tooling, scheduled batch services, monitoring infrastructure itself), the review cadence is twelve to sixteen weeks; for all alerts in the inventory, the last-review date must be tracked and alerts more than two cycles past their scheduled review date are automatically flagged at the next calibration meeting as overdue regardless of their false positive rate history. Evidence requirements at each review cycle: the reviewing engineer must produce a false positive rate for the review period — the count of alert firings that did not precede or correlate with an acknowledged incident divided by total firings — using the incident and alert data for the period; a false positive rate above fifteen percent for an alert in a full review cycle triggers a mandatory threshold adjustment or root cause investigation; additionally, the reviewing engineer must cross-reference the alert against the incidents in the review period to verify that the alert fired ahead of customer impact for each incident it was intended to detect; an alert with a zero false positive rate and a zero detection rate — it has not fired but also has not detected any incidents — is not confirmed as well-calibrated; it is confirmed as quiet; the distinction requires the incident cross-reference. Out-of-cycle triggers: three conditions trigger a mandatory out-of-cycle threshold review independent of the scheduled cadence: a post-incident analysis finds that the alert did not fire for an incident it was designed to detect; a service deployment changes the metric's expected range by more than twenty percent (documented as a required step in the deployment runbook for services with calibrated alerts); or an on-call engineer submits a false positive report — a brief structured form specifying the alert, the firing time, and the reason the firing was non-actionable — that exceeds the three-reports-per-engineer-per-week threshold for the same alert. The out-of-cycle trigger mechanism converts individual false positive reports from passive complaints into structured calibration input, which is the behavioral change required to convert the on-call rotation's operational knowledge about alert quality into systematic threshold adjustments rather than unilateral silencing.

Section 3: False positive tracking and resolution workflow. The false positive tracking workflow specifies how false positives are classified and attributed to their root cause, who owns each root cause category, what the resolution time target is, and how chronic false positive alerts — alerts that continue producing false positives across multiple review cycles despite threshold adjustments — are escalated to a root cause investigation rather than a threshold adjustment cycle. The tracking model exists because most organizations treat high-false-positive alerts as a threshold calibration problem when the root cause is often a different failure: an alert that is monitoring the wrong metric for the condition it is intended to detect; an alert that is correctly designed but whose root cause is a known benign event that should be suppressed during its recurrence; an alert that is correctly designed and correctly configured but whose underlying service behavior includes a variance source that must be addressed in the service rather than in the threshold. Three root cause categories and their resolution owners. Threshold miscalibration: the metric correctly represents the condition the alert is designed to detect, and the threshold value is incorrectly set relative to the distribution of normal operating behavior; resolution is a threshold adjustment using the evidence model in Section 1; owned by the alert's assigned owner with the on-call lead as reviewer; time-to-resolution target is the next scheduled review cycle unless the false positive rate exceeds thirty percent in a review period, in which case the target is within five business days. Monitoring design mismatch: the metric does not correctly represent the condition the alert is designed to detect — the alert is measuring a proxy that produces noise because the proxy's relationship to the actual condition is unstable; resolution is a redesign of the alert to monitor a more direct indicator, which may require instrumentation changes in the service; owned by the alert owner in coordination with the service owner; time-to-resolution target is one sprint after the design mismatch is identified. Recurring benign event suppression: the metric correctly represents the condition and the threshold is correctly calibrated for steady-state operation, but a recurring benign event — a backup window, a scheduled batch process, a recurring deployment pattern — produces metric behavior that crosses the threshold predictably and without requiring human action; resolution is a suppression rule tied to the known event rather than a threshold change; owned by the alert owner; the suppression rule must be reviewed at the next scheduled calibration cycle to verify it has not outlived the benign event it was created to suppress, and suppression rules that have been active for more than twelve months without re-verification are automatically flagged for review. Chronic false positive escalation: an alert that has produced a false positive rate above fifteen percent in three consecutive review cycles despite threshold adjustments is escalated from the threshold adjustment workflow to a root cause investigation — the alert is moved to a monitoring-only state (no paging) pending the investigation, which must produce either a monitoring design change or a documented justification for re-enabling paging at a revised threshold within two review cycles; no alert remains in paging state with a false positive rate above fifteen percent for more than two calibration cycles without a scheduled root cause investigation. Connect this to the incident postmortem action item accountability decision record: every incident post-mortem that involves a detection gap — an incident that was not detected by the alert layer before customer impact — must produce a threshold review action item assigned to the owning engineer with a completion date within one review cycle; and every incident post-mortem that involves a noisy alert environment during the incident — excessive alert volume that complicated triage — must produce a correlation model review action item assigned to the on-call lead; these two action item categories must be tracked separately from the incident's operational action items because they address the alerting layer's structural quality, not the incident's immediate root cause; the distinction matters because operational action items are typically prioritized against the incident's urgency and the calibration action items are displaced; a tracking model that separates the two categories maintains the calibration action items in the calibration review queue where they belong rather than competing against incident remediation work on the same backlog.

Section 4: Alert correlation and suppression policy. The alert correlation and suppression policy specifies how alerts that fire simultaneously in causally related services are modeled to produce aggregate signals rather than individual per-service notifications, how the correlation model is designed and validated against historical incident data, and how suppression rules are governed to prevent them from outliving the benign events that justified them. The policy exists because per-alert calibration quality does not determine the signal quality the on-call engineer experiences during cascading failures; a set of individually well-calibrated alerts with zero false positives in isolation can produce a thirty-four-alert noise environment during a four-service cascade if the alert model has no correlation layer, and the on-call triage experience in that environment is functionally equivalent to a poorly calibrated alert environment regardless of the individual false positive rate metrics. Three components. Alert correlation model design: for each group of services with a documented causal dependency relationship — service A's failure mode produces downstream metric changes in services B and C — a correlation rule is designed that groups alerts firing within a correlation window from the same causal chain into a single aggregate notification; the aggregate notification specifies the root cause service candidate, the number and identity of affected downstream services, and the severity of the combined impact; the correlation window is set against the typical propagation time for the service dependency graph, which must be empirically measured from historical incident data rather than assumed; the correlation model must cover all first-order dependency relationships in the service graph at minimum, and second-order relationships for service groups where second-order cascades have occurred historically. Correlation model validation: before any correlation rule is deployed, it must be validated against the last twelve months of incident data — specifically, for each incident that involved multi-service alert involvement, would the proposed correlation rule have produced a single aggregate alert or multiple individual alerts? The validation must verify that the correlation rule would have correctly identified the root cause service as the aggregate anchor, not a downstream symptom service; correlation rules that produce incorrect root cause candidates in historical validation are redesigned before deployment; correlation rules that correctly aggregate the alert but mask relevant diagnostic information — the individual service alert details that the on-call engineer needs to diagnose the root cause — must include an expansion view that surfaces the individual alerts after the on-call engineer acknowledges the aggregate. Suppression governance: every active suppression rule must have a documented owner, a suppression justification specifying the benign event it suppresses, a defined expiration condition (either a date or an event condition whose occurrence ends the justification), and a last-review date updated at each calibration cycle; suppression rules without an expiration condition are assigned a twelve-month review date at creation; at the twelve-month review, the rule is re-evaluated against the current alert behavior and either renewed with a new review date, modified to a time-scoped suppression, or removed; no suppression rule remains active for more than twenty-four months without a documented renewal decision; a suppression rule inventory is maintained alongside the alert threshold inventory and reviewed in parallel at each calibration meeting. Connect this to the on-call escalation policy decision record: the alert correlation model must be tested against the escalation policy's mandatory hold time to verify that the aggregate alert provides sufficient diagnostic information for the first responder to make a correct escalation decision within the hold time window; a correlated alert that fires for a thirty-four-service cascade and provides no information about which service is the root cause candidate forces the first responder to either escalate immediately — violating the hold time model — or spend the hold time doing the cross-service correlation that the alert model should have done; the escalation policy's hold times must be recalibrated after the alert correlation model is deployed, because the signal the first responder is working against has changed substantially and the hold times that were appropriate before correlation are too long or too short after it.

Section 5: On-call experience governance. The on-call experience governance specifies how the on-call experience is monitored as an outcome of calibration quality — not as a personnel satisfaction metric — and what thresholds of false positive rate or interruption frequency trigger mandatory calibration reviews independent of the scheduled cadence. The governance model exists because most organizations manage on-call burnout as a personnel problem — schedule adjustments, compensation changes, rotation expansions — rather than as a calibration quality outcome, which produces interventions that partially offset the burnout cost without addressing the calibration failure that is generating it. Managing the on-call experience as a calibration outcome requires measuring it as a calibration outcome: the primary governance metrics are not satisfaction surveys or attrition rates but structured per-engineer, per-rotation paging data that connects the on-call experience to the alert environment that produced it. Four governance mechanisms. Weekly paging data review: after each on-call week, the on-call engineer submits a structured report specifying the total number of pages received, the count of pages that required human action, the count of pages that resolved without action, the count of off-hours pages (defined as pages received between 10pm and 7am local time) broken down by actionable and non-actionable, and any alerts that fired more than three times during the week without requiring action; the on-call lead reviews the reports and flags any week where the non-actionable page count exceeded thirty percent of total pages, the off-hours non-actionable page count exceeded one per night, or any single alert produced more than three non-actionable pages, for threshold calibration review within five business days rather than at the next scheduled cycle. Experience threshold triggers: three numeric thresholds trigger a mandatory full inventory calibration review independent of the scheduled cadence: the aggregate false positive rate across all paging alerts exceeds twenty percent over any two-week period; any individual on-call engineer's non-actionable page rate exceeds forty percent over a one-week rotation; the off-hours non-actionable page frequency exceeds an average of two per night for any on-call engineer in any rotation week; when any threshold is breached, the calibration review is prioritized as a reliability incident — not as a backlog item competing against feature work — with a completion target of five business days from the breach date. On-call load budgeting: on-call load — defined as hours per week consumed by alert triage, incident response, and non-actionable page acknowledgments — is measured and tracked per rotation, and a budget of no more than four hours per on-call week is established as the load ceiling above which the calibration model is considered out of compliance; a four-hour-per-week ceiling represents approximately 10% of a 40-hour work week, which is the upper bound of on-call overhead at which most engineers report the work as sustainable; load above the ceiling is not a signal that the team needs more on-call engineers — it is a signal that the calibration model needs adjustment, and the intervention is calibration before staffing. Calibration ownership and accountability: the on-call experience governance metrics are owned by the engineering manager, not the on-call lead or the SRE; the engineering manager presents the false positive rate, the paging load data, and the on-call experience threshold compliance status at each engineering team meeting; the ownership assignment is explicit because on-call burnout is a management accountability issue — the decision that produces the alert environment is a management decision about how to prioritize calibration work relative to feature delivery, and the person who made the prioritization decision must be the person accountable for its effects on the on-call rotation; assigning governance to the SRE or on-call lead produces a situation where the individual who monitors the burnout surface has no authority to direct the calibration work that would address it, which is the condition that produced the six-month delay in the enterprise SaaS's correlation model prioritization. Connect this to the engineering manager span of control decision record: on-call experience governance — running weekly paging data reviews, monitoring false positive rate trends, triggering calibration reviews when thresholds are breached, and presenting calibration compliance status at team meetings — requires a consistent allocation of manager bandwidth that the span model must treat as a non-negotiable overhead item rather than a discretionary activity; an engineering manager at or near their effective span ceiling will consistently defer the governance activities to the next sprint, the next quarterly review, or the SRE who "handles on-call," and the on-call burnout surface will accumulate until the first on-call engineer departure makes the deferral visible; the span model and the on-call governance model must be designed together so that the governance activities are schedulable within the coordination bandwidth the span allows, or the span must be reduced until the governance activities fit.

FAQ

Further reading

• On-call escalation policy decision record — the escalation policy governs what happens after an alert fires and the first responder determines they need help; the threshold calibration model governs what fires an alert in the first place; the two policies operate at different layers of the incident response stack and must be designed together because the escalation policy's mandatory hold time and escalation target breadth are calibrated against a signal environment where most pages require intervention — when the false positive accumulation rate rises above thirty percent, the hold time model breaks down because the period the first responder is expected to diagnose becomes partly a period of waiting to see whether the alert auto-resolves, and the escalation target breadth model becomes irrelevant if the first responder cannot determine from the aggregate alert which team owns the root cause service.
• Incident postmortem action item accountability decision record — every incident post-mortem involving a detection gap produces a threshold calibration action item; every incident post-mortem involving a noisy alert environment during triage produces a correlation model review action item; these two action item categories must be tracked separately from the incident's operational action items in the accountability model, because they address the alerting layer's structural quality and belong in the threshold calibration review queue where they can be completed at the six-to-eight week cadence rather than on the incident remediation timeline where they compete against urgency and are displaced; the accountability model that produced the developer tools company's calibration improvement being lost at the SRE's departure would not have produced that outcome if the calibration action item had been tracked in the calibration review queue rather than assigned to an individual in the incident tracking system.
• Sprint planning format decision record — alert correlation model development and threshold calibration review cadence both consume engineering bandwidth that competes with feature delivery at planning time; the sprint planning format must have a mechanism for evaluating reliability investments against their expected reduction in incident response time and on-call load, not only against their feature delivery opportunity cost; a three-week alert correlation model that would eliminate seven-minute triage phases and prevent SLA breaches in cascading failure scenarios must be evaluated against the quantified cost of those SLA breaches, not against the abstract priority of reliability work relative to product features; planning formats that do not have this mechanism will consistently underprioritize calibration investments relative to their actual organizational value because the value is expressed in incidents that did not happen and time not spent in triage, neither of which appears in the feature delivery metrics that most sprint planning formats optimize against.
• Engineering manager span of control decision record — on-call experience governance — weekly paging data reviews, false positive rate trend monitoring, calibration review triggering, and calibration compliance reporting — requires consistent manager bandwidth; at span levels above the effective threshold, engineering managers defer calibration governance to the SRE or on-call lead, who typically has no authority to direct the engineering work required to address the calibration failures the governance activities identify; the contract management company's twelve-hour calibration exercise was known to be needed three months before it was run; the delay was bandwidth, not knowledge; the span model must treat on-call governance as a non-negotiable weekly overhead item rather than a discretionary reliability activity so that the time exists in the manager's schedule before burnout and departure force the priority.
• Open-source extractor — alert threshold calibration decisions are made in the same AI-assisted design conversations where service architecture decisions are made — the threshold value, the evidence base it was set against, the alternatives that were rejected, and the failure modes the threshold was designed to catch are often discussed in the design conversation and never surfaced in the alert configuration comment or runbook; running the archived design conversation through the extractor surfaces the calibration rationale that the threshold value alone does not communicate — the same mechanism that surfaces 'why we chose the inverted permission evaluation order' surfaces 'why we set the memory growth rate threshold at 15% rather than 25% — the benign GC pattern reaches 18%, and the leak pattern we're worried about starts at 20%, so the threshold must be between 18% and 20% to distinguish them'; the extractor makes the design reasoning available to the engineer who encounters the threshold six months later and finds it miscalibrated, rather than leaving them to reconstruct it from the threshold value and the metric name alone.