The engineering on-call design decision record: why the on-call model you chose determines your incident response latency and your engineer sustainability surface

A 15-person developer tools startup had been running its production systems for eighteen months on an informal on-call model: if something broke at 2am, the engineer who happened to be awake would see it in the monitoring Slack channel and respond. For the first year, the model worked well enough — the team was small, every engineer had read most of the codebase, and the monitoring Slack channel generated enough ambient awareness that significant incidents rarely went unnoticed for long. The implicit assumption behind the model was that someone would always be watching; the model had never been tested against a scenario in which nobody was.

At 2:17am on a Tuesday in September, the primary database server hit its connection pool limit following an undetected leak introduced in a deploy six hours earlier. The monitoring Slack channel emitted four alerts over the following twelve minutes, each noting that API response times had exceeded the error threshold. All fifteen engineers were asleep. By 6:30am, when the first engineer opened Slack, the connection pool had been exhausted for four hours and fourteen minutes, the API had been returning 503s to every request, and two enterprise customers — both on the US West Coast, both in morning standup sessions — had noticed the outage and sent support tickets. The CTO's post-incident message to the team that afternoon was honest: "We got lucky this didn't happen on a Wednesday, when we have a demo at 10am. We need to fix this before it happens again." The fix implemented that week was a PagerDuty account, a 3-person primary rotation (the three most senior engineers), and a policy that all monitoring alerts route to PagerDuty with a 5-minute acknowledge-or-escalate window.

The PagerDuty implementation solved the paging problem. Over the following six months, no significant incident went unacknowledged for more than eight minutes. What it did not solve — and what the team did not recognize as a distinct problem until it became an attrition problem — was the sustainability of the rotation it had implemented. The three-person rotation meant that each engineer was on primary duty approximately every third week, with a typical week generating two to four pages, of which approximately one occurred outside working hours. The on-call compensation model had not been discussed at the time of the PagerDuty implementation because the three engineers had volunteered for the rotation and the assumption was that they would raise the issue if it became a problem. Nine months after the implementation, two of the three rotation members resigned within six weeks of each other. Both exit interviews cited on-call fatigue as a primary factor; one engineer noted specifically that the combination of 2am pages every ten to fourteen days, a recovery time policy that consisted of sleeping in if possible, and a company culture that had never explicitly acknowledged on-call as a burden that warranted compensation or structural protection had produced a sustained level of sleep disruption and background vigilance that she did not want to continue. The third rotation member, who had not resigned, described in the retrospective that he had been searching for other positions for three months before the other two left and had stopped only because the departures had left him as the only engineer who understood the billing system in sufficient depth to keep it running. The CTO's post-mortem on the attrition found that the on-call rotation had produced a free-rider problem in reverse: the three engineers who had volunteered to carry the company's availability had subsidized the availability of twelve engineers who had never been on-call, and the subsidy had never been named, measured, or compensated.

A 48-person B2B SaaS had built what its VP of Engineering described in an engineering blog post as a mature on-call process: a formal rotation with eight engineers across two teams, a severity classification system (P0 through P3), response time SLAs for each severity level (P0: acknowledge within 5 minutes, P1: 15 minutes, P2: 60 minutes), documented escalation paths from primary to secondary to engineering leadership, and a weekly incident review meeting where P0 and P1 incidents were presented to the full engineering team. The process had taken four months to design and implement and had produced measurable improvements in response times: the mean time to acknowledge had fallen from 23 minutes to 6 minutes, and the mean time to resolve had fallen from 4.2 hours to 2.1 hours. The engineering blog post was published at the 12-month mark of the new process.

Three months after the blog post, the VP of Engineering noticed a pattern in the weekly incident reviews that the aggregate metrics had obscured. She began tracking which engineer had ultimately resolved each P0 and P1 incident — not who had acknowledged the page but who had made the diagnostic breakthrough that unblocked the resolution. Over six weeks she reviewed 23 P0 and P1 incidents and found that in 17 of the 23, the resolving engineer was not the on-call primary who had acknowledged the page; it was a secondary escalation target, specifically one of three engineers who had built the most complex services in the stack: the payments service, the core data pipeline, and the real-time notification system. Of those 17 incidents, 12 had involved the on-call primary acknowledging the page, conducting a diagnostic session of between 35 and 90 minutes using the available runbooks and observability dashboards, reaching a point where the available diagnostic tools had been exhausted, and then escalating to the relevant secondary — waking a non-on-call engineer in the middle of the night. The remaining five had involved the on-call primary escalating within the first fifteen minutes because the error message pattern did not match any runbook entry and the primary did not have sufficient knowledge of the affected system to attempt a diagnosis.

The pattern had a name that the VP of Engineering recognized from a conference talk she had attended: shadow on-call. The three engineers who built the complex services were effectively on-call for every incident affecting those services, regardless of whether they appeared in the rotation schedule, because the formal rotation was staffed with engineers whose knowledge was insufficient to resolve incidents in those areas without escalation. The formal rotation's eight engineers were distributed correctly by headcount and seniority; they were not distributed correctly by system knowledge relative to the incident distribution. The payments service, the data pipeline, and the notification system generated 71% of P0 and P1 incidents; the three engineers who had built those systems were escalated to in 74% of those incidents. The VP of Engineering's analysis for the engineering leadership team was direct: "We built a rotation that routes pages to people who can acknowledge them and built an escalation model that routes hard problems to the people who built them. What we actually have is a 3-person shadow rotation carrying most of the incident resolution burden, three people who are de facto on-call every week regardless of the schedule, and an 8-person formal rotation that functions primarily as a first-response triage layer." The remediation required two simultaneous investments: a 12-week runbook expansion project in which the three shadow on-call engineers wrote structured diagnostic guides for the failure modes they had been resolving informally, and a rotation membership gate that required every engineer to complete a system design review and runbook walkthrough for each service area before being added to that service's rotation — meaning the rotation grew only as knowledge grew, not as the team grew.

A 75-person enterprise SaaS had, by any conventional measure, a sophisticated on-call operation. A four-person SRE team owned the on-call process, maintained the PagerDuty configuration, ran a quarterly rotation review, and wrote a 47-page on-call handbook that covered alerting architecture, severity definitions, escalation protocols, incident communication templates, and the post-incident review process. The organization had invested heavily in observability: custom dashboards for each service, anomaly-detection-based alerting that had reduced alert fatigue by 60% relative to the threshold-based system it replaced, and a distributed tracing setup that allowed on-call engineers to identify the upstream cause of cascading failures in most cases without reading service logs. Mean time to detect had fallen to 4.1 minutes. The CTO had presented the on-call architecture at two engineering conferences.

Two years into the mature on-call process, the VP of Engineering commissioned an analysis of on-call burden distribution across the engineering organization. The analysis was prompted by a comment from a senior engineer during a performance review: "I feel like I'm on-call every week even when I'm not on-call." The analysis looked at three metrics across all 75 engineers over the prior 12 months: formal on-call hours (time designated as primary or secondary rotation duty), escalation receipt rate (number of incidents for which the engineer was the escalation target regardless of rotation status), and post-incident resolution involvement (number of incidents where the engineer's participation was required for resolution). The results confirmed the senior engineer's intuition in a way the VP of Engineering had not anticipated. Formal on-call hours were distributed uniformly across the rotation, as expected. Escalation receipt rate was concentrated: the top 15% of engineers by escalation receipt accounted for 64% of all escalations. Post-incident resolution involvement was even more concentrated: the top 10% of engineers accounted for 71% of all post-incident resolution sessions.

The concentration was not distributed randomly across the organization; it was concentrated in the teams that had invested most heavily in the systems that generated the most complex incidents — the two teams that owned the data pipeline and the core authentication and permissions system. Both teams had written comprehensive runbooks, invested in observability tooling, and were considered the organizational exemplars of good on-call practice. The VP of Engineering's analysis identified the mechanism: the teams that invested most in runbooks and observability had reduced the volume of simple incidents that the formal rotation could resolve without escalation; what remained in their incident distribution was the complex incidents that required the engineers with the deepest knowledge of those systems. The teams that had not invested in runbooks and observability still generated simple incidents that the formal rotation could resolve, which meant their knowledge-concentrated engineers were not escalated to as frequently because the formal rotation was absorbing more of their incident volume. The investment in on-call quality had paradoxically produced a higher effective on-call burden for the engineers who had built the most reliable and well-documented systems, because good documentation and observability reduced the noise in the incident distribution and exposed the signal — the complex incidents that only the engineers with the deepest knowledge could resolve. The VP of Engineering's conclusion to the CTO was that the escalation model needed to be redesigned to treat complex incidents not as escalation failures requiring a more capable individual but as knowledge gaps requiring a knowledge transfer investment — specifically, a structured incident pairing model in which escalation targets were required to conduct the resolution session alongside the on-call primary rather than instead of them, with an explicit knowledge transfer objective for each session that fed directly into a runbook update before the incident was closed.

Structural properties set by the engineering on-call design decision

Two structural properties are determined by the founding decisions about on-call scope, alerting model, escalation architecture, rotation design, and the knowledge prerequisites for rotation membership. Both are invisible when the engineering team is small enough that system knowledge is uniformly distributed and on-call is a shared collective obligation, and visible when the team has grown large enough that knowledge concentration is significant and the on-call process either works against or with the knowledge distribution to determine who actually carries production availability.

Property 1: Incident response latency. The incident response latency is the elapsed time between when a production incident begins and when an engineer with sufficient context to resolve it is actively working on it — bounded by three factors. First, alerting model: the signals that trigger pages, the thresholds at which each signal triggers, and the routing model that determines which rotation or individual receives the page. An alerting model that is calibrated too sensitively — paging on conditions that resolve without intervention — produces alert fatigue, a pattern in which on-call engineers delay their response while they assess whether the current page is the genuine incident or the recurring noise, increasing the response window for genuine incidents. An alerting model that is calibrated too conservatively — paging only after a condition has been severe for long enough to exceed a threshold — delays detection relative to what earlier-signal alerting would have provided. An alerting model that routes pages to a rotation whose members do not have the knowledge to diagnose the alerted system produces a first responder who must escalate before substantive diagnostic work begins; the escalation time is added to the response window before the engineer who can actually work the incident is engaged. The alerting model requires separate calibration from the rotation design: you can have a well-calibrated alerting model and a rotation whose knowledge distribution makes the alerting routing ineffective, or a well-designed rotation and an alerting model that produces so much noise the rotation cannot maintain response quality. Connect this to the observability strategy decision record: the alerting model is the operational layer of the observability strategy; an organization whose observability strategy specifies that anomaly-based alerting is the primary detection mechanism needs an alerting model calibrated to the false positive rate of the anomaly detection system at the target sensitivity, and an on-call rotation whose members can distinguish anomaly-detection pages from threshold-based pages and apply different response urgency to each — a rotation that was designed before the anomaly detection system existed will have been calibrated to threshold-based urgency and may not have been recalibrated to the anomaly-based system's different false positive characteristics. Second, escalation model: the defined path from first responder to engineers with deeper knowledge, including the time thresholds that trigger escalation, the secondary contacts for each system area, and the protocol for incidents that affect multiple systems simultaneously and require simultaneous escalation to multiple secondaries. Escalation models that are under-specified — where escalation is possible but the threshold at which it is appropriate, the contact to escalate to for each failure type, and the expected response time of escalation targets are not documented — produce the two failure modes the second story exhibited: escalation too early, when first responders escalate on uncertainty rather than after a diagnostic attempt, adding unnecessary interruptions to engineers who would not have been needed if the first responder had attempted the runbook; and escalation too late, when first responders delay escalation out of reluctance to wake up colleagues, consuming the productive response window while working a diagnosis they lack the knowledge to complete. Third, context at incident time: the quality and availability of the information an on-call engineer needs to diagnose and respond to an incident — runbooks specifying the diagnostic steps for known failure modes, observability dashboards that surface the affected system's current state, and documented system knowledge that allows the on-call engineer to reason about failure modes the runbooks do not cover. On-call processes that invest heavily in alerting infrastructure without investing in context infrastructure produce on-call engineers who know that something is wrong — the page told them that — and have no structured path to understanding what is wrong; the response becomes trial-and-error remediation, which is slower than runbook-guided diagnosis and more likely to produce secondary incidents when remediation attempts affect adjacent systems.

Property 2: Engineer sustainability surface. The engineer sustainability surface is the set of conditions that determine whether on-call is a manageable responsibility engineers can sustain over multi-year careers at the organization or a burden that produces burnout, attrition, and active avoidance of systems whose ownership implies heavy on-call load — bounded by three factors. First, rotation model: the size of the on-call rotation, the frequency with which each engineer is on primary duty, the definition of on-call hours and obligations, and the compensation or protected time model that acknowledges the cost of availability outside working hours. Rotation models that are sized to engineering headcount rather than calibrated to measured incident load produce rotations whose sustainability is determined by whatever incident volume the system happens to generate at the time of sizing. The first story's three-person rotation was not inappropriate for a system with two to four pages per week at the time it was designed; it became unsustainable when the incident frequency was higher than that, and the organization had not built the feedback loop between incident load measurement and rotation size that would have produced a rotation review before the attrition occurred. Second, incident load distribution: the distribution of on-call burden across the rotation members, including the variance between the lightest and heaviest effective burden and the degree to which burden is concentrated on the engineers with the deepest system knowledge. The critical insight from the second and third stories is that the formal on-call schedule is a poor proxy for actual on-call burden when the rotation includes engineers with materially different levels of system knowledge: the engineers who must resolve the incidents rather than just acknowledge them carry a higher effective burden than their rotation frequency implies, and the engineers who acknowledge incidents but escalate most of them carry a lower effective burden than their rotation frequency implies. An on-call sustainability model that measures only rotation frequency and ignores escalation receipt rate and post-incident resolution involvement will consistently underestimate the burden carried by the engineers who built the most complex systems and overestimate the burden carried by the engineers who are formally on-call most frequently but escalate most of what they receive. Connect this to the engineering technical strategy decision record: the investment decisions in the technical strategy — which systems to build in-house versus using managed services, which systems to invest in observability and runbooks versus treating as low-risk — are implicit decisions about future on-call burden; a technical strategy that builds complex custom infrastructure without an accompanying investment model for the on-call burden that infrastructure will generate is a strategy that is trading short-term velocity for long-term sustainability cost that will be paid by the engineers who inherit the systems. Third, recovery time architecture: the protected time after on-call shifts for recovery, including the explicit policies about workday obligations on the day after an overnight incident, the compensation mechanism for on-call weeks with high incident load, and the organizational norms about context-switching from incident response back to development work. Recovery time policies that are stated but not enforced produce the same outcome as no policy: engineers who sleep in after a 3am incident and then feel guilty about being late, who take a compensation day after a high-incident week and feel that they are behind, and who absorb the cognitive cost of sustained on-call vigilance without acknowledgment that the cost is real and organizationally recognized. The enforcement of recovery time policies is a signal to engineers about whether the organization treats on-call as a real cost or an invisible overhead, and engineers read that signal accurately.

The engineering on-call design ADR: five sections

Section 1: On-call scope and coverage model. The scope and coverage model specifies which systems are on-call protected — meaning that incidents in those systems trigger pages outside working hours — which systems are monitored but not paged outside working hours, and the criteria by which a system moves from unprotected to protected coverage. This is the most foundational section and the one most likely to be determined by default — organizations that implement PagerDuty for the first time typically add all their monitoring alerts to the paging configuration because the cost of doing so is low and the consequence of not doing so is undetected incidents; the result is on-call coverage that is designed by accretion rather than by deliberate decision about which systems warrant the on-call burden that comes with paging coverage. The coverage model should specify the business impact threshold that justifies on-call coverage — the customer-facing availability or data integrity condition whose failure produces a business impact that exceeds the cost of the on-call burden required to respond to it — and should explicitly exclude from paging coverage the internal systems, staging environments, and non-critical services whose failures do not meet that threshold but that produce noise in an undiscriminating paging configuration. The model should also specify how the coverage scope is reviewed over time: as the system grows, new services are added to paging coverage by default; without a review process that periodically evaluates whether each covered service still meets the coverage threshold and whether the paging configuration for each service is calibrated correctly, the on-call scope expands without bound and the rotation's per-engineer incident rate increases without the team making an explicit decision that the increased rate is warranted. Connect this to the engineering technical strategy decision record: the on-call scope model is one of the downstream commitments of the technical strategy — a strategy that specifies that the product's reliability is a competitive differentiator is implicitly committing to on-call investment in the services that deliver that reliability; a technical strategy that does not include an explicit model of which services warrant on-call coverage is leaving the on-call scope to be determined by whoever implements the monitoring, which is a different person than the one who made the strategic commitment about reliability.

Section 2: Alerting model and severity classification. The alerting model specifies what signals trigger a page, at what thresholds, with what severity classification, and to whom. The severity classification is the most consequential structural decision in this section because it determines the response time SLA for each incident, the escalation threshold within each severity level, and the expectation for on-call engineers about what "acknowledge" means — whether it means "I saw this and am assessing" or "I am actively working this right now." Organizations that have a single severity level — or that have multiple severity levels but treat all of them with the same urgency norm — produce on-call engineers who cannot prioritize across simultaneous pages or communicate accurately to stakeholders about expected resolution time. The severity classification should be defined in terms of customer impact rather than engineering symptom: a severity classification that is based on which service generated the alert rather than what the customer experience consequence of the failure is produces a classification that reflects the monitoring infrastructure rather than the incident's actual urgency. The alerting model should also specify the signal type for each service: threshold-based alerting (page when metric exceeds a fixed value) is appropriate for well-understood failure modes with a known threshold relationship between metric value and customer impact; anomaly-based alerting is appropriate for systems whose normal operating range varies significantly with load or time-of-day patterns and where threshold-based alerting produces either false positives at peak load or detection latency during off-peak hours. An organization that deploys anomaly-based alerting for all services without distinguishing the failure mode characteristics of each service will find that anomaly detection's higher false positive rate at low-confidence thresholds produces alert fatigue in service areas where the signal-to-noise ratio is poor, and that the alert fatigue in those areas has secondary effects on how seriously on-call engineers treat anomaly-based pages across all services. Connect this to the engineering incident retrospective decision record: the alerting model should be updated after each post-incident review to reflect the failure modes the incident revealed — the detection latency, false positive rate, and signal clarity of the alerting configuration that was active during the incident — because the retrospective's root cause analysis will often identify alerting deficiencies (signals that were available but not configured to page, thresholds that were calibrated to avoid noise and missed the early signal of the incident) that are directly actionable in the alerting model update process.

Section 3: Escalation and handoff model. The escalation model specifies the path from first responder to engineers with deeper knowledge, including the time thresholds that trigger escalation, the secondary and tertiary contacts for each system area, the protocol for initiating escalation, and the expectations for the escalating engineer and the escalation target during the handoff. The most consequential structural decision in this section is the escalation trigger: whether escalation is based on time elapsed (escalate after 30 minutes without resolution progress), on diagnostic state (escalate when the first responder has exhausted the available runbooks and observability tools without identifying the cause), or on the on-call engineer's judgment (escalate when the on-call engineer believes escalation is warranted). Time-based escalation is the most reliable trigger because it is unambiguous and does not require the on-call engineer to make a quality judgment about their own diagnostic progress under the cognitive pressure of a live incident; the disadvantage is that it produces escalations for incidents the first responder was on track to resolve, which creates false-positive interruptions for escalation targets. Diagnostic-state escalation is the most efficient trigger — it escalates exactly when escalation adds value — but requires the on-call engineer to have an accurate model of the boundary between "runbook-exhausted and genuinely stuck" and "runbook-exhausted and close to finding the cause," a judgment that is difficult to make under incident pressure and that on-call engineers typically resolve by setting the bar for "genuinely stuck" too high, producing delayed escalations. The escalation model should also specify the handoff protocol: what information the escalating engineer provides to the escalation target (current system state, diagnostic steps attempted, hypotheses ruled in and out), what the escalation target is expected to do when they receive the handoff (confirm acknowledgment, estimated time to join, whether they need the on-call engineer to stay on), and what happens to the on-call engineer's obligations when the escalation target takes over the incident. A handoff protocol that does not specify these elements produces escalations in which the on-call engineer briefs the escalation target via a message that either over-explains (providing more context than the target needs to start) or under-explains (requiring the target to re-run diagnostics the on-call engineer already ran), and in which the on-call engineer's role after the handoff is ambiguous — they may stay engaged, drop off, or check back in, depending on individual judgment rather than a defined protocol. Connect this to the engineering documentation standards decision record: the escalation handoff protocol is a documentation standard applied to the incident response context; an organization that has invested in documentation standards for design decisions and architecture will find that applying the same documentation discipline to incident handoffs — specifying the required fields of a handoff note, the format in which diagnostic state is communicated, and the expectations for the escalation target — produces a material improvement in handoff quality that reduces the time from escalation acknowledgment to productive diagnosis.

Section 4: Rotation model and sustainability architecture. The rotation model specifies the size of the rotation, the rotation frequency for each engineer, the knowledge prerequisites for rotation membership in each system area, and the compensation and protected time model that acknowledges the on-call obligation's cost. The most consequential decision in this section — and the one most likely to be deferred until it becomes an attrition problem — is the knowledge prerequisite for rotation membership. Organizations that treat rotation membership as a function of seniority or headcount produce rotations whose members have varying levels of system knowledge that produce inconsistent incident response quality and concentrations of effective on-call burden on the engineers with the deepest knowledge. A knowledge prerequisite model specifies what an engineer must demonstrate before being added to the rotation for a given system area: typically a combination of a system design review (the engineer can explain the system's architecture, critical paths, and known failure modes), a runbook walkthrough (the engineer can execute the system's runbooks against a simulated incident), and a shadowing period (the engineer has responded to a specified number of real incidents alongside a senior rotation member and handled the diagnostic process with guidance). The knowledge prerequisite model serves two functions: it ensures that rotation members can actually respond to incidents in their rotation area, which reduces escalation rate and mean time to resolve; and it creates a structured mechanism for knowledge diffusion — the shadowing requirement ensures that deep system knowledge propagates from the engineers who built the systems to the engineers who will maintain them, using the on-call process as the knowledge transfer vehicle. The sustainability architecture should also include an explicit incident load model: the organization should measure its historical incident frequency, resolution time distribution, and out-of-hours page rate, and use those measurements to determine the minimum rotation size that keeps the per-engineer on-call burden below a defined sustainability threshold. The threshold should be stated explicitly — a maximum out-of-hours page rate, a maximum consecutive on-call weeks, a minimum days-off-call between rotation cycles — and the rotation size should be updated when the incident load measurements change. Connect this to the engineering technical strategy decision record: the rotation size and sustainability architecture are downstream commitments of the technical strategy; a strategy that specifies that the organization will build and maintain complex custom infrastructure needs a rotation sustainability model that accounts for the on-call burden that complexity generates, and the sustainability model should be an input to the investment decision rather than a consequence discovered after the investment is made.

Section 5: Learning and runbook model. The learning and runbook model specifies how the on-call process produces knowledge artifacts that improve future incident response — specifically, how incidents generate runbook updates, how runbook coverage is measured against the incident distribution, and how the runbook system is maintained as the systems it documents evolve. The most consequential structural decision in this section is the runbook update requirement in the post-incident process. Organizations that treat runbook creation as a pre-launch activity — create runbooks when a service enters production, update them when engineers remember to — produce runbook libraries that accurately reflect the failure modes known at launch and drift from the actual failure modes as the system evolves, the incident history accumulates, and the engineers who wrote the original runbooks move to other teams or leave the organization. The post-incident review process should include an explicit step that asks: "Does the runbook for this failure mode exist? Is it accurate to what we did in this incident? Does it include the diagnostic insight that allowed us to find the cause?" An affirmative answer to all three requires updating the runbook before the incident is closed; a negative answer to any of the three requires creating or updating the runbook as a required action item, not an optional one. The learning model should also specify how runbook coverage is measured: the fraction of historical P0 and P1 incidents that have a runbook that would have allowed the on-call primary to resolve the incident without escalation is a measurable proxy for on-call self-sufficiency; organizations that track this metric have a quantitative target for the runbook investment that allows them to make explicit decisions about when runbook quality is sufficient to support rotation expansion versus when it is not. Connect this to the open-source extractor: the reasoning behind on-call design decisions — why the rotation is sized the way it is, what incident history motivated the current alerting thresholds, what knowledge prerequisites were established and why, and what escalation architecture was chosen over the alternatives — is reasoning conducted in AI chat sessions when engineering leads work through how to design or redesign the on-call process after an attrition event, a high-severity incident, or a significant change in team size or system complexity; running those sessions through the extractor produces the on-call design decision record that allows the next engineering lead to inherit not just the current PagerDuty configuration but the analysis that produced it — which is the analysis they need to evaluate whether the current design should be updated as the team, systems, and incident history evolve.

FAQ

Further reading

• Engineering incident retrospective decision record — the retrospective process is the primary feedback loop for improving on-call quality over time; an organization whose retrospective model produces accurate root cause analysis and actionable runbook updates will find that each incident makes the next incident in the same failure mode easier to resolve; an organization whose retrospective model produces correct diagnoses but does not require runbook updates as a closed-loop action item will find that the same incidents recur with the same response latency indefinitely, because the knowledge produced by each resolution stays with the on-call engineer who resolved it rather than propagating into the runbook that the next on-call engineer will consult.
• Observability strategy decision record — the observability strategy determines the quality of the context available to on-call engineers at incident time; an organization with strong observability and weak runbooks produces on-call engineers who can see that something is wrong and have excellent data about what the affected system's current state is, but lack the structured diagnostic path that converts that data into a resolution action; an organization with strong runbooks and weak observability produces on-call engineers who have structured diagnostic paths for known failure modes but cannot verify which failure mode is active; the two investments are complementary and should be designed as a coherent system.
• Engineering technical strategy decision record — on-call design is one of the downstream commitments of the technical strategy; a strategy that specifies that the product's reliability is a competitive differentiator is implicitly committing to on-call investment in the services that deliver that reliability; a technical strategy that builds complex custom infrastructure without an accompanying model for the on-call burden that complexity will generate is making an investment whose operational cost will be paid by the engineers who inherit the systems, and the sustainability of the engineering organization is determined in part by whether the technical strategy's investment decisions were made with that cost made explicit or not.
• Engineering documentation standards decision record — runbooks are a specialized form of operational documentation, and the documentation standards decision determines whether runbooks are first-class artifacts with defined quality standards and ownership or informal documents that drift from the systems they describe; an organization that has invested in documentation standards for design and architecture decisions will find that applying those standards to runbooks — specifying the required structure, the review process, and the update trigger — produces a runbook library that is reliably useful at incident time rather than one that is comprehensive at launch and outdated six months later.
• Open-source extractor — the reasoning behind on-call design decisions — why the rotation is sized the way it is, what incident history motivated the current alerting thresholds, what knowledge prerequisites were established and why, and what escalation model was chosen over the alternatives — is reasoning conducted in AI chat sessions when engineering leads work through how to design or redesign the on-call process after an attrition event, a high-severity incident, or a significant change in team size or system complexity; running those sessions through the extractor produces the on-call design decision record that allows the next engineering lead to inherit the analysis that produced the current design, not just the configuration that resulted from it.