The on-call rotation design decision record: why the rotation schedule and handoff model you chose determines your team's toil ceiling and burnout risk

Published 2026-07-11 · WhyChose

On-call rotation design decisions are made once, in an early sprint, under the particular pressure of having just launched a product to production. The team is small, the system is still simple enough for every engineer to understand end-to-end, and the rotation setup conversation takes twenty minutes. Someone configures PagerDuty, assigns alerts to the "primary" schedule, and sets a one-week rotation. The conversation ends. The decision is not written down. The system grows.

Three years later, the alert routing configuration set in that twenty-minute conversation has never been revisited. The team has grown from five engineers to forty-five. The system has grown from one service to thirty-one. But the alert ownership model reflects the team that existed at launch — the engineers who were there in year one are the primary owners of the services they built, and PagerDuty still routes to them preferentially when those services generate alerts. Three of those engineers have the highest alert volumes on the team. One of them is on-call every sixth week and handles pages from systems she wrote four years ago that nine other engineers also know. The other two are on-call every fourth week and handle pages from systems that only they understand. The rotation design decision that was made in twenty minutes at launch is now extracting a cost that is visible only in attrition data.

Two ways on-call rotation design produces the wrong outcome

The accumulated alert routing problem

A 45-person B2B SaaS company builds a customer engagement platform. In the spring of year one, the three founding engineers set up on-call: a one-week rotation, everyone primary, all alerts routing to PagerDuty with no secondary tier. The system runs one service. There is nothing to configure. Three years later, the platform runs 27 microservices, the engineering team is 44 people, and the PagerDuty configuration has been extended incrementally with each new service: new service goes live, new service's alerts get added to the schedule, the engineer who built the service gets listed as a responder. The routing model has never been audited as a whole.

In Q3 of year three, the data platform team tracks a 14-week rolling average of pages per on-call shift and discovers that the distribution is not uniform. The median shift receives 7 pages per week. Three engineers receive an average of 22, 19, and 31 pages per shift. The engineer with 31 pages per shift is the original author of the legacy queue processing system — a service that generates high alert volume because its SLI thresholds were set conservatively in year one and never updated as traffic patterns stabilized. She is also the on-call owner for three other services that she wrote and that no one else has deep familiarity with. In Q3 alone she handles 23 weekend pages — pages that fire between 10pm and 6am local time. She submits her notice four weeks into Q4.

The exit interview produces a finding the team should have had from routing data: the alert ownership model had never been reviewed as a product team scale decision. Individual engineers had been added to individual alert routing rules one at a time over three years. The aggregate effect — that some engineers were carrying eight times the on-call load of their peers — was invisible in any single routing rule. It was only visible in the aggregate, and nobody had looked at the aggregate. The twenty-minute launch-day rotation design decision had compounded, undisturbed, for three years.

The handoff model failure under pressure

A 35-person API platform company builds a developer tool with infrastructure running across three geographic regions. In year two they implement a follow-the-sun on-call model: a US West engineer handles daytime PST hours, an EU engineer handles Central European daytime hours, and an APAC engineer handles Singapore daytime hours. The intent is to ensure that no engineer is woken at 3am for a problem that falls within another region's working hours. On paper, the coverage model is correct.

In November, a multi-region latency issue begins at 3pm PST. The US West on-call engineer starts investigating. Two hours later, at 5pm PST — 10pm CET — the EU handoff window opens. The US engineer is mid-investigation, has formed a hypothesis about CDN edge configuration that has not been confirmed, has ruled out three other hypotheses (database read replica lag, upstream DNS resolution, certificate renewal collision), and has made four configuration changes to observe their effects. The handoff to the EU engineer is a Slack message: "latency still elevated in eu-central-1, been paging for about 2 hours, not sure what's causing it, might be the CDN. good luck."

The EU engineer receives the handoff at 10pm local time. She has the Slack message and the alert history in PagerDuty. The US engineer is now offline. She does not know which hypotheses have been ruled out, what the four configuration changes were, or what the CDN hypothesis is based on. She begins the investigation from the alert history, re-examines the database read replica lag hypothesis (already ruled out, 45 minutes of her time), re-examines the DNS hypothesis (already ruled out, 30 minutes), and eventually reaches the CDN edge configuration path the US engineer had been investigating when the handoff happened. The total investigation time from her handoff is 3 hours and 40 minutes, of which approximately 1 hour and 45 minutes was re-doing work already completed. The incident is resolved at 1:40am CET. The follow-the-sun model that was intended to prevent late-night pages produced a late-night resolution for the EU engineer anyway — not because the coverage model was wrong, but because the handoff model had never been designed.

Three structural properties that determine on-call rotation outcomes

The alert routing architecture and its toil production surface

On-call toil is not randomly distributed. It is produced by three distinct alert categories that require different interventions. Actionable pages are alerts that fire for conditions requiring genuine engineering judgment: a novel failure mode that requires diagnosis, a degraded state where the recovery action depends on current context, a resource constraint that requires a non-trivial trade-off decision. These are the pages that justify having an engineer on-call. Automatable toil pages are alerts that fire for conditions where the correct action is always the same: restart this specific service when it crosses a memory threshold, clear this queue when it backs up beyond a defined depth, acknowledge this maintenance event that fires every Sunday between 2am and 4am. These pages interrupt an engineer, consume interrupt capacity, and fragment deep work — but they could be handled by an automated runbook or suppressed entirely. Noise pages are alerts that auto-resolve without action, that represent known false positives that have accumulated in the alert configuration but have never been suppressed, or that fire for conditions that monitoring was misconfigured to detect. The first measurement most teams do reveals that between 40 and 60 percent of pages fall into the automatable toil or noise categories.

The observability strategy decision record determines the instrumentation foundation from which alerts are derived. Alert quality is downstream of instrumentation quality: a team that instructs its monitoring stack to alert on CPU utilization will receive CPU alerts; a team that has designed SLI-based alerting against user-experience properties will receive SLI alerts. The two alert types have radically different toil profiles. CPU at 85% may or may not indicate a user-facing problem; the engineer who receives the alert at 2am must determine whether it does, which requires diagnosis. An SLI alert that says "request success rate dropped below 99.5% in the past 15 minutes" is actionable without diagnosis time: the condition is user-visible, the severity is known, and the runbook for that SLI is the starting point. Alert routing architecture that is not derived from the observability strategy produces a toil distribution that cannot be improved without changing the instrumentation model first — because the alerts being routed are the wrong alerts.

The toil classification baseline should be established at rotation design time and measured on a per-shift basis. The measurement is simple: each page resolution should include a tag — actionable, automatable, or noise — applied by the resolving engineer. After four to six shifts, the aggregate classification reveals the current toil profile: what fraction of pages require genuine judgment versus routine action versus no action. The on-call rotation design decision record should document the baseline classification at design time and a target trajectory: within two quarters, automatable toil pages should fall below 20 percent of total page volume as runbooks are automated, and noise pages should fall below 10 percent as alert thresholds are calibrated. Without a baseline measurement, there is no standard against which to evaluate whether toil is improving or worsening — and no trigger for reviewing the rotation design when it is worsening.

The SLO and error budget decision record provides the technical foundation for alert threshold calibration. Burn rate alerting derived from the error budget model replaces threshold-based alerting on individual metrics: instead of alerting when CPU exceeds 85%, alert when the error budget is burning at 14x the allowable rate over a one-hour window. This shift from metric-threshold alerting to budget-consumption alerting reduces noise pages dramatically — because budget burn rate alerts only when a user-visible degradation is occurring at a rate that will exhaust the reliability budget, not when an infrastructure metric crosses an arbitrary threshold. A team that migrates from metric-threshold alerting to budget-burn alerting typically reduces total page volume by 30 to 50 percent, primarily by eliminating noise pages and consolidating redundant threshold alerts into a single SLI burn rate signal. The on-call rotation design decision record should reference the SLO decision record's alert architecture directly: the rotation is designed around the expected page volume that the alerting model produces, not around a hypothetical future page volume after alerting is improved.

The handoff model and the incident state transfer surface

An on-call handoff is an incident state transfer. The incoming engineer needs to know not just what happened but what is currently unresolved, what has already been ruled out, what changes were made to the production environment during the shift that could affect the next shift's incident investigations, and what the current hypothesis is on any open issue. The quality of this transfer determines whether the incoming engineer starts with context or starts from scratch. Starting from scratch after a follow-the-sun handoff means that the coverage model designed to distribute the incident response load across time zones is partially negated by the context loss at each handoff boundary.

Two failure modes appear in handoff model design. The first is no structure: the handoff is an informal Slack message, a verbal summary at the end of a call, or nothing at all if the shift was quiet. The informal handoff works when nothing is happening and everything is resolved. It fails precisely when it matters most — when there is an active, unresolved, complex investigation at handoff time — because the outgoing engineer is exhausted and under pressure, and the structure required to communicate the investigation state clearly is not built into the handoff process. The second failure mode is excessive structure: a 40-field handoff document template that takes 45 minutes to complete and is therefore systematically skipped during busy shifts. Engineers who have experienced a difficult incident do not have 45 minutes of structured documentation time at shift end. A handoff template that takes 10 to 15 minutes to complete under normal conditions and 20 minutes after a difficult shift is completed; a template that requires 45 minutes is abandoned.

The effective handoff model has five sections and no more. Open incidents: for each in-flight issue, the symptom, the systems involved, the current hypothesis, the actions taken and their outcomes, and the specific next action the incoming engineer should prioritize. Recent production changes: deployments, configuration changes, infrastructure modifications, and feature flag changes made during the shift. Alert landscape changes: alerts suppressed, thresholds modified, or monitoring rules adjusted during the shift and why. Known issues under management: production conditions that are degraded but not being actively worked, escalations pending, customer-facing issues communicated to customer success. Shift notes: anything unusual that does not fit the above categories. Each section should be fillable in two to three minutes under normal conditions. The open incidents section takes longer during complex incidents — but that is the section that matters, and the time is justified.

The handoff model also determines what happens to institutional knowledge about recurring issues. A team that uses structured handoffs over 12 months accumulates a record of every recurring issue: what triggered it, what the standard resolution was, and whether the resolution is the same every time. This record is the input to the runbook automation that reduces automatable toil. Without structured handoffs, the knowledge that "this particular queue backup alert always resolves with command X, run by engineer Y, takes three minutes" lives only in individual engineers' heads and is lost when they are not the responder. The incident response playbook decision record captures the alert routing, runbook structure, and escalation path. The on-call rotation design record is its complement: the rotation design determines who responds when, and the handoff model determines what context they have when they respond. The two decisions are made together or they fail to reinforce each other.

The rotation scope and the primary/secondary coverage model

A single-tier rotation assigns one engineer to be the on-call responder for all systems during their shift. This model works at small scale when the system is simple enough for any engineer on the rotation to triage any alert, and when the team has comprehensive overlap in system knowledge. It fails at scale in two ways. First, when the system has grown complex enough that no single engineer has the depth to diagnose all components, the single-tier on-call engineer must escalate to off-rotation engineers for issues outside their expertise — which means those off-rotation engineers are effectively on-call without the on-call designation and compensation. Second, when a single-tier engineer is unavailable during their shift (illness, emergency, connectivity loss), there is no systematic coverage path. The team improvises, which means whoever checks Slack first handles the incident.

The two-tier coverage model separates the primary responder (who acknowledges the alert and begins initial triage) from the secondary responder (who escalates to when the primary needs domain expertise or is unavailable). The secondary is typically not expected to be woken up — they are available by text message for escalation, not by PagerDuty page. This model extends the on-call participant list: more engineers are in the secondary tier, which reduces the frequency of each engineer's secondary rotation. It also creates a structured escalation path that prevents the "whoever checks Slack first" improvisation. The tradeoff is rotation management complexity: two schedules, two escalation paths, and the question of what "secondary available by text message" means on a Saturday morning. The rotation design decision should specify this explicitly: does "secondary available" mean acknowledged within 30 minutes, within 15 minutes, or within 5 minutes? Does it mean no recreational travel over the rotation weekend, or just no international travel? Making this expectation explicit at design time prevents the disagreement about what was agreed that surfaces when the secondary doesn't respond in what the primary considers a reasonable time.

Service domain tiering is a third model: different engineers are primary for different system areas, and the on-call rotation is by domain rather than by calendar week. A team with four major domains — data pipeline, API tier, notification system, payment processing — might run four separate rotations with two to three engineers per domain, each rotation covering their domain's alerts. This model maximizes responder expertise at the cost of coordination overhead: the on-call schedule is more complex, incidents that cross domain boundaries require coordination between on-call engineers, and engineers who are not on-call may receive calls because their domain is involved in a cross-domain incident. Domain tiering is appropriate when the system has grown complex enough that generalist on-call is no longer producing good incident outcomes — when the primary on-call engineer is routinely unable to diagnose the issue and is spending the majority of their incident time looking for the right subject matter expert rather than resolving the problem. The rotation design record should document the point at which the single-tier model is expected to break down — the system complexity threshold, the team size threshold, or the specific incident pattern that would trigger a rotation model redesign.

Five sections the on-call rotation design decision record should address

1. Rotation schedule and participant breadth

Document the rotation length, the participant list, and the reasoning for each. The rotation length decision should be made against a current per-shift page count: how many pages does the team receive on average per week, and what is the variability? A one-week rotation with 30 average pages per shift is a different decision context than a one-week rotation with 7 average pages. If the per-shift page count is not measured, establish a baseline for the first four weeks of the rotation before committing to a length. Document the participant breadth — which engineers are in the rotation, why, and which engineers are explicitly not in the rotation and why. In a team of 20 engineers where 5 engineers are in the rotation, the decision about which 5 is as important as the rotation length. The participant selection should be documented against system knowledge coverage: each engineer in the rotation should have sufficient familiarity with the full system scope to triage the majority of alerts independently, with escalation paths for the remainder.

Document the compensation model: is on-call compensated separately from base salary, and if so, how? Whether this is a company policy decision or an engineering team decision depends on the organization. But the compensation model — even if it is "no separate compensation, expected as part of the engineering role" — should be written down so that engineers who join the team understand what they are agreeing to. The on-call expectation that is assumed rather than documented produces misalignment when an engineer discovers that "on-call" means 31 weekend pages per quarter, not the 7 they expected based on team norms they observed but were never explicitly told. The new-CTO onboarding problem applies specifically to on-call design: a new engineering lead who inherits a team without documented on-call expectations cannot evaluate whether the current rotation is sustainable, whether it is equitable, or whether the compensation model matches industry norms for the team's geography and stage — because none of this is written down anywhere they can find it.

2. Alert routing model and toil classification baseline

Document the alert routing architecture: which alert sources feed into the on-call rotation, how alerts are routed to specific engineers or teams within the rotation, and the escalation path when the primary does not acknowledge within the escalation window. For each major alert category, document the expected resolution path: which runbook applies, what system knowledge the resolving engineer needs, and what the escalation target is if the runbook is insufficient. This documentation does not need to be a complete runbook for every alert — it is a routing map that makes the alert coverage model auditable.

Document the toil baseline. Run the toil classification exercise (actionable, automatable, noise) for the first six to eight shifts of the rotation and record the aggregate split. The baseline is not a target — it is a starting measurement that defines the gap between current state and the toil profile that the rotation design is intended to produce. Commit to a toil reduction target for the next two quarters and document the mechanism: which categories of automatable toil will be addressed, who owns the runbook automation work, and what the success criterion is. A toil reduction target without an owner and a mechanism is a measurement without a response. The baseline should also include a per-engineer page distribution: are pages being handled uniformly across the rotation, or are specific engineers handling disproportionate volumes due to alert routing configurations that have not been reviewed? The alert routing model that routes alerts to the original service author long after other engineers have gained equivalent system knowledge is one of the most common sources of inequitable toil distribution.

3. Handoff model and incident state transfer protocol

Document the handoff protocol: what structured information is transferred at each rotation boundary, how it is recorded, where it lives, and what "complete handoff" means. The handoff template should be a living document, not a one-time creation: the first version of the template should be reviewed after the first four handoffs and updated to remove sections that are never filled in and add sections that engineers are filling in ad-hoc outside the template. The goal is a template that is completed consistently, not one that is aspirationally comprehensive. Document the handoff communication channel: is the handoff document written in a shared document, in a designated Slack channel, in the incident management tool, or in the on-call platform? Consistency of channel matters — if the incoming engineer has to search three places to find the handoff, the search time is overhead that reduces the model's value.

For follow-the-sun rotations, document the overlap window: the period when both the outgoing and incoming engineer are available simultaneously. Even a 15-minute overlap window — a brief synchronous call at handoff time — dramatically improves handoff quality for complex active incidents, because the incoming engineer can ask two or three clarifying questions that would otherwise require an asynchronous thread to resolve. Document who initiates the overlap call (outgoing engineer contacts incoming at the overlap window start), what the call's maximum duration is (15 minutes for routine handoffs, 30 minutes for active incidents), and under what conditions the call is skipped (all incidents resolved, nothing to transfer, incoming engineer unavailable). The overlap window is a cost — one engineer's time at the boundary of their shift — but for rotations that regularly produce active-incident handoffs, the cost of the overlap call is small compared to the re-investigation time it eliminates.

4. Primary/secondary coverage model and escalation chain

Document whether the rotation uses a single-tier or two-tier coverage model and the rationale for the choice. If single-tier, document the system knowledge requirement for rotation participants and the informal escalation path when the primary needs domain expertise. If two-tier, document the secondary's availability expectation explicitly: response time, travel constraints, and the conditions under which a secondary escalation is appropriate. Document the escalation chain beyond the secondary: if both primary and secondary are unavailable or unable to resolve an incident, who is the tertiary contact? For most teams at 15-50 engineers, the tertiary is an engineering manager or VP with production access who can make decisions and coordinate, not necessarily resolve the incident technically. Document this person and the conditions under which they are called — "30 minutes of primary+secondary contact attempts with no response" is a clear trigger; "when the primary thinks they need help" is not.

Document the cross-service escalation model for incidents that span multiple system domains. When a data pipeline failure triggers downstream failures in the API tier and the notification system simultaneously, three alerts may fire. If those three systems have separate alert owners, who coordinates? The on-call rotation design should specify a single incident commander for cross-domain incidents — typically the primary on-call engineer who received the first page — and an escalation path for pulling in domain experts when needed. Without a named incident commander, cross-domain incidents produce coordination failures: three engineers working in parallel on different symptoms of the same root cause, making conflicting configuration changes, and spending most of the incident time figuring out who is in charge. The incident response playbook decision record covers the full escalation protocol and incident management structure; the rotation design record should reference it and document the specific rotation-level decisions it relies on — which engineer is primary, how cross-domain escalations are triggered, and how incident command is assigned when the incident spans the primary's domain.

5. On-call health review cadence and rotation modification criteria

Document the on-call health review cadence and the specific metrics reviewed. At minimum, review the following data quarterly: per-shift page count over the quarter (median, 90th percentile, maximum), per-engineer page count over the quarter (distribution, outlier identification), per-category toil classification over the quarter (actionable vs automatable vs noise), after-hours page rate (pages outside business hours as a fraction of total), and time-to-acknowledge distribution (are primary engineers acknowledging within the defined window?). The health review should produce two outputs: a current-state assessment against the targets established in the rotation design record, and a list of specific remediation actions for any metric outside target, with owners and due dates.

Document the rotation modification criteria — the specific conditions under which the rotation design itself is revisited, rather than just tuning individual alert thresholds or runbooks. The rotation design should be revisited when: the per-shift average page count exceeds the established toil ceiling for two consecutive quarters; the per-engineer page distribution shows one engineer handling more than 2x the median over a rolling quarter; the team size changes by more than 50 percent since the last rotation design review; the system architecture changes significantly (addition of a new major service tier, migration of a critical system, adoption of a new infrastructure model); or an on-call attrition event occurs where the exit interview cites on-call load as a contributing factor. The last criterion is the most important and the most frequently ignored: attrition is the most expensive feedback signal, and by the time an engineer cites on-call load in their exit interview, the condition that drove the load has typically been present for 18 to 24 months. The decisions never written down pattern is most damaging here — the on-call design decision that was made at launch, never documented, and never reviewed is the decision that was accumulating cost invisibly the entire time. If the decision had been documented with explicit review criteria, the review criteria would have triggered a redesign before the cost became an attrition event.

The compliance automation decision record intersects with on-call rotation design in one specific way: SOC 2 Availability criteria require documented on-call coverage and documented incident response procedures. If the audit scope includes the Availability criterion, the on-call rotation design record is audit evidence — specifically for the A1 availability monitoring and capacity management controls and the CC7 system monitoring controls. The coverage model, the escalation chain, and the response time commitments in the rotation design record should be consistent with the availability commitments in the system description. An enterprise contract that commits to 99.9% uptime with a 30-minute response time SLA requires a rotation design that can actually achieve 30-minute response times — which is a constraint on the coverage model, the escalation chain, and the after-hours availability expectations. If these are not designed together, the on-call rotation will fail the SLA at the worst possible moment: during an enterprise customer's first production incident.

The ADR template provides the structure for the on-call rotation design decision record: decision (the rotation length, participant list, coverage model), rationale (the toil baseline, the system knowledge requirements, the coverage constraints), consequences (the per-shift page expectation, the handoff overhead, the secondary availability cost), and review date (the first health review date, typically one quarter after the rotation goes live). The record does not prevent the rotation from needing to be changed — it makes the change legible. When the quarterly health review shows that per-engineer page distribution has drifted to 5:1 between the highest and lowest rotation participants, the record provides the context to understand why: the alert routing model was designed for a team of five and has accumulated three years of incremental additions without a structural review. The fix is not to tweak individual alert thresholds. The fix is to re-run the alert routing audit against the current team and system topology and redesign the routing model as the team it has become. The WhyChose extractor can surface the original on-call design conversations from AI chat history — the ChatGPT session where the founding team debated rotation length, the Claude conversation where the escalation chain was worked out, the discussion about whether to use PagerDuty or Opsgenie. Those sessions contain the original constraints and tradeoffs that the rotation was designed around. When the rotation no longer fits the system or the team, recovering those constraints makes the redesign faster — because the team does not have to reconstruct why the original decisions were made before they can evaluate whether those reasons still apply.

Further reading