2026-09-24 · ~22 min read · Decision records

The incident command system decision record: why the authority model you configured determines your parallel mitigation failure mode and your context fragmentation surface

The incident command authority model, the incident coordination protocol, and the escalation detection mechanism are decisions that are almost never made explicitly — they emerge from informal seniority conventions, a Slack channel someone created, and alert thresholds calibrated at launch that nobody revisited. Three failure patterns: the engineering team whose two most senior responders pursued contradictory mitigations for 47 minutes because no one had unambiguous authority to make the binding call; the company whose war room produced a 28-minute context gap when the critical diagnostic insight emerged in a private DM and the war room participants kept working the wrong hypothesis; and the company whose incident process was invoked 14 days after a database degradation began, because the detection model recognized incidents by alert-type trigger rather than customer impact signal.

A 31-person B2B SaaS company built a project management and resource planning platform for professional services firms — consulting agencies, engineering firms, and architecture practices that needed to track project hours, staffing capacity, and client delivery milestones across teams of twenty to two hundred people. The company had a well-established on-call rotation and a PagerDuty alerting setup that covered database performance, API error rates, and queue processing latency. What the company did not have was a documented incident command structure. The informal convention was that whoever was paged handled the incident; if it was complex, they'd pull in whoever was most relevant to the affected system; and the most senior person in the room would naturally take the lead. For most incidents — the kind that a single engineer could diagnose and resolve in twenty to forty minutes — this convention worked well enough that it was never examined.

The exception arrived on a Tuesday afternoon when an infrastructure configuration change caused cascading latency across the company's job scheduling service. The on-call engineer, a mid-level backend engineer who had been with the company for eighteen months, was paged. He correctly identified that the job scheduler was affected, diagnosed that the issue was likely related to the configuration change that had just been deployed, and began the rollback process. Simultaneously, the company's most senior infrastructure engineer — who had built the job scheduling service and received the alert via a secondary notification — joined the incident Slack channel and, seeing the degradation pattern, concluded that rolling back the configuration change would not be sufficient because the job scheduler had already written corrupted state to the queue. His assessment was that a rollback would restore the configuration but leave the corrupted queue entries, and the right approach was to deploy a targeted hotfix that cleared the corrupted queue before restoring the configuration. He began preparing the hotfix.

The on-call engineer was four minutes into the rollback when the senior infrastructure engineer posted his assessment in the channel. Both believed their approach was correct. Both were partially right: the rollback was necessary, and the queue corruption was real. But the two approaches were in direct conflict in their sequencing — executing the rollback while the hotfix was being prepared meant that the queue would be restored in a partially consistent state before the hotfix cleared it, potentially compounding the corruption. The senior infrastructure engineer flagged this in the channel. The on-call engineer paused. The engineering manager joined the channel at this point, saw two engineers in apparent disagreement, and attempted to mediate by asking each to explain their reasoning. The mediation conversation lasted twenty-three minutes. During that period, neither the rollback nor the hotfix was executing. The service was fully degraded, customers were receiving timeout errors, and the two most technically capable people on the incident were waiting for a decision that no process had designated anyone to make. The engineering manager eventually called for the senior infrastructure engineer's approach, which was executed correctly and resolved the incident. Total duration from first alert to resolution: 54 minutes. Estimated resolution time if either approach had been executed without the authority conflict: 7 to 12 minutes.

The post-mortem reviewed the incident timeline in detail. The root cause of the extended duration was not technical — it was organizational. The incident response process specified escalation triggers, alert thresholds, and on-call rotation schedules. It did not specify who had binding decision authority when two responders reached different conclusions about the right mitigation approach. The "most senior person in the room" convention broke down the moment that the on-call engineer and the senior infrastructure engineer were simultaneously in the room with conflicting assessments. The convention had been designed for situations where the most senior person was clearly identifiable and unambiguously correct. It had no mechanism for the situation where two experienced engineers, each with deep knowledge of the affected system, reached different conclusions about the right path forward.

A 38-person developer tooling SaaS company built a continuous integration and deployment platform for mid-market software teams — a product that managed build pipelines, test orchestration, deployment approval workflows, and release gating across the engineering organizations of its customers. The company had invested meaningfully in incident response infrastructure: a dedicated #incidents Slack channel served as the war room for all P1 and P2 incidents, a formal incident template was posted at the start of each incident (severity, affected systems, IC, customer impact estimate), and a runbook library covered the most common failure patterns. The company's CTO had described the war room process as "our coordination guarantee — everything happens in the channel, nothing in DMs."

On a Thursday morning a database replica fell behind its primary by forty-seven seconds, causing reads from the replica to return stale data. For customers running deployment approval workflows — which queried deployment status to gate the next pipeline stage — the stale reads produced a specific failure pattern: a deployment that had been approved and executed successfully would appear in the read replica as still pending approval, causing the approval gate to block the next stage. Customers began opening support tickets describing deployments that appeared "stuck" despite showing as completed in other parts of the UI.

The incident was declared at 9:14am. The IC opened the war room channel, posted the incident template, and began the investigation. Three engineers joined the channel. The investigation initially focused on the deployment approval workflow service — the symptom appeared there first and the runbook had a section covering approval gate failures. The IC and two engineers worked the workflow service hypothesis in the channel, running diagnostic queries and checking recent deploys to the approval service. At 9:31am, one of the engineers in the channel opened a DM with the database infrastructure engineer who was not yet in the incident channel, asking whether there were any known replica issues. The database engineer checked the replica lag monitor and saw the 47-second lag. He replied to the DM: "yeah the replica is 47 seconds behind, that would explain the stale reads on the approval queries." The engineer in the channel replied to the DM: "ah okay I'll tell the IC," then got distracted by a query the IC had just posted in the channel asking for the results of a specific diagnostic. The DM context sat unshared.

At 9:44am — twenty-eight minutes after the war room opened, thirteen minutes after the replica lag was identified in the DM — the database engineer joined the war room channel and posted the replica lag status directly. The IC immediately recognized the root cause, switched the approval workflow reads to the primary database, and the incident resolved within four minutes of the database engineer's post. The full resolution time from incident declaration to resolution was 35 minutes. Based on the diagnostic path, a team with the replica lag information available at 9:31am would have resolved the incident in approximately 10 minutes. The war room had a 28-minute gap between when the root cause was identified and when the team working the incident had access to that information — not because the information was unavailable, but because the coordination model had not established that findings from outside the war room channel had to be brought into the channel before any action was taken on them. The war room guarantee — "everything happens in the channel, nothing in DMs" — described the intended protocol but not an enforced one. Under incident pressure, the friction of context-switching between DMs and the channel was high enough that the handoff step was deferred and then forgotten.

A 43-person SaaS company built an analytics and reporting platform for e-commerce operations teams — daily revenue tracking, inventory forecasting, customer segment cohort analysis, and automated anomaly detection for sales pattern shifts. The company operated a PostgreSQL database cluster serving both the production application and the analytics workloads, with a read replica dedicated to the heavier analytical queries. The company's incident response process was alert-triggered: PagerDuty was configured with alert rules covering database CPU above 85%, query latency p95 above 2,000ms, error rate above 2%, and queue depth above 500 items. An alert in any of these categories triggered the on-call engineer's pager and opened the incident process.

In early 2026, a partial index on the analytics read replica began to corrupt silently. The corruption affected a composite index used by the long-running cohort analysis queries — queries that joined the customer_events table across date ranges of sixty to ninety days. As the corruption progressed, the index's effectiveness degraded: queries that had completed in 40 to 80 milliseconds began requiring full or partial table scans, extending to 200ms, then 400ms, then over a second for the most date-range-intensive cohort queries. The degradation was gradual enough that it never caused a single daily session to exceed the p95 latency alert threshold of 2,000ms — the median query remained well within acceptable range, and the tail of the distribution rose slowly enough that the p95 moved from 95ms to 340ms over fourteen days without crossing the static threshold. Database CPU remained below 85% throughout, as the degraded queries consumed more CPU per query but were infrequent enough that the aggregate remained bounded.

The customers noticed first. The analytics platform's largest customer — a 200-person e-commerce operations team — began receiving reports from their inventory forecasting analysts on day 11 that the 90-day cohort reports were taking "a couple of minutes" rather than the usual "a few seconds." They opened a support ticket on day 12 describing the slowdown. The support ticket was categorized as a "performance question" and entered the standard support queue, where the first response time SLA was four business hours. The support engineer who reviewed the ticket on day 12 ran a test query and observed a 1.4-second completion time, noted that this was within the 2,000ms threshold, and responded that the query performance was within the product's defined parameters. Two more customers opened similar tickets on days 13 and 14.

On day 14, the company's customer success manager for the largest account escalated the ticket directly to the head of engineering, describing the customer's frustration with the response. The head of engineering ran the cohort query themselves, saw the 1.4-second completion time, and checked the query plan. The index corruption was visible in the EXPLAIN output: the query planner had abandoned the composite index and was performing a sequential scan on the customer_events table's date range. The incident process was invoked at that point, fourteen days after the corruption had begun producing customer-visible degradation. The resolution — rebuilding the index — took forty minutes. The incident process recorded a forty-minute incident. The customer impact record showed fourteen days of degrading performance affecting three customers, with total slow-query time accumulated across those fourteen days representing over 340 hours of degraded production operation for the affected customers' analyst teams. The incident detection model had measured a forty-minute event. The customer impact had been a two-week accumulation.

Structural properties set by the incident command system decision

Three structural properties are determined when a team establishes — or fails to establish — an incident command system decision record: what the IC authority model determines about the parallel mitigation failure mode when responders with equal informal status disagree, what the incident coordination protocol determines about the context fragmentation surface when working context is distributed across multiple threads, and what the incident detection model determines about the slow-degradation detection gap when the escalation process only recognizes alert-triggered incident starts. None of these are labeled as decisions when the team designates whoever-is-most-senior as the natural incident lead, describes the Slack channel as the war room without enforcing a single-thread-of-record requirement, and configures alert thresholds without establishing a customer complaint escalation path. They emerge as operational failures when two senior engineers are simultaneously in the channel with conflicting mitigation plans and no one has the authority to resolve the conflict, when a critical diagnostic insight lives in a DM thread while the war room pursues the wrong hypothesis, and when the post-mortem records a forty-minute incident that customers experienced as a two-week degradation.

Property 1: The incident command authority model and the parallel mitigation failure mode. The parallel mitigation failure mode occurs when two responders with equal informal authority pursue contradictory recovery strategies simultaneously, because the incident process has not established who can make a binding decision when the team disagrees. The failure mode is predictable from the authority structure: the "whoever is most senior takes the lead" convention produces the right outcome when the most senior person is unambiguously identified and their assessment is unambiguously correct. It breaks down in three specific configurations — when the incident crosses system boundaries and two people are each most senior in their own domain; when the most senior people in the room reach different conclusions about the right mitigation; and when the most senior person is the on-call engineer whose informal status is lower than others who join the incident. All three configurations are common in incidents of moderate to high severity, because those incidents tend to span systems and attract more senior responders. The IC authority model that prevents this failure has four required properties: the IC role is assigned at incident declaration (not claimed organically), the assignment follows a defined default rule that produces an unambiguous answer before the incident begins, the IC's authority explicitly covers mitigation approach decisions (not just communication and coordination), and the model specifies the mechanism for IC role transfer when the incident expands beyond the initial IC's expertise. An IC authority model that leaves any of these four properties undefined has a gap that will surface as a parallel mitigation conflict when the conditions are right. The incident escalation policy decision record connects at the threshold layer: the escalation decisions that determine who is paged and at what severity level determine who is in the room when the IC assignment decision must be made; the IC authority model should be specified in the same document as the escalation policy, not as a separate process that someone must remember to invoke when the escalation policy fires.

Property 2: The incident coordination protocol and the context fragmentation surface. The context fragmentation surface is the set of observations, hypotheses, test results, and conclusions that exist somewhere in the incident record — DMs, sidebar threads, ad-hoc video calls, notes in someone's scratch buffer — but are not in the active working context of the person making the next diagnosis decision. The fragmentation is not a coordination failure in the sense of people being negligent or careless; it is a predictable consequence of the communication friction dynamics under incident pressure. Under incident pressure, the friction of switching to the canonical channel and writing up a finding before acting on it is higher than the friction of acting immediately on a finding and reporting it to the channel afterward — or not reporting it at all, because the incident moved on before the report happened. The war room model reduces fragmentation by concentrating activity in one channel, but it does not eliminate fragmentation unless the single-thread-of-record requirement is explicit and enforced: every hypothesis, every test run and its result, every conclusion must be posted to the canonical thread before any action is taken on that information. The "before any action is taken" framing is the critical constraint — it converts the thread from a documentation medium (write down what happened) to a coordination medium (the thread is what enables the next action). The two-minute rule operationalizes this: if a finding is actionable, it is in the canonical thread within two minutes or the finder is not acting on it yet. Teams that implement the two-minute rule in practice report that the enforcement friction — the small delay between discovery and posting — is the mechanism that surfaces fragmentation before it accumulates. The on-call rotation compensation decision record connects at the responder model layer: the IC role structure and the coordination protocol both assume a responder who has the cognitive bandwidth to enforce the protocol under pressure; on-call rotation design decisions that produce responders operating on minimal sleep or under excessive incident frequency reduce the bandwidth available for protocol adherence and increase the probability of coordination failures.

Property 3: The incident detection model and the slow-degradation detection gap. The slow-degradation detection gap is the period between when a degradation begins producing customer-visible impact and when the incident response process is invoked. For alert-triggered incident processes, this gap is determined by two parameters: the alert threshold (below which the degradation is invisible to the detection system) and the degradation rate (which determines how long the degradation takes to cross the threshold). A degradation that develops gradually enough — gradually enough that no individual metric crosses its threshold while the aggregate customer impact accumulates — can produce a detection gap that is orders of magnitude longer than the resolution time. The 43-person analytics company's incident was resolved in forty minutes; the detection gap was fourteen days. The structural mismatch is between the incident detection model's definition of an incident (an alert fires) and the customer's experience of an incident (their workflows are degrading). Customer complaint signals are the early detection mechanism that the alert-triggered model misses, because individual customers' experiences of a degradation are not subject to the same aggregation thresholds as system-level metrics. A customer opening a support ticket describing "queries that used to take a few seconds now take a couple of minutes" is describing an incident from their perspective regardless of whether any monitoring alert has fired. The structural fix — establishing a customer complaint escalation path as a first-class incident trigger — requires two things: a support ticket triage protocol that routes performance degradation descriptions to the on-call engineer with a defined time window (not a standard four-hour SLA queue), and an on-call response protocol that treats a customer complaint escalation with the same urgency as an alert-triggered page. The trend-based monitoring complement — rate-of-change alerts on key metrics rather than absolute-value thresholds — provides a technical detection layer for gradual degradations that the customer complaint path may not surface until later. Together, they close the detection gap that alert-triggered processes leave open for the gradual failures that are often the most expensive to recover from because of the accumulated impact. The alerting threshold decision record connects at the threshold design layer: the specific alert threshold decisions — absolute-value versus rate-of-change, per-customer versus aggregate, static versus rolling-baseline calibrated — determine the precise shape and size of the slow-degradation detection gap; rate-of-change thresholds and per-customer SLA-aligned thresholds are the two alert design choices that narrow the gap most effectively for the gradual failures that absolute-value thresholds miss. The WhyChose extractor finds the incident command decisions buried in your AI session history — the incident process design session where escalation thresholds were specified but IC authority was not, the war room setup session where the Slack channel was created without a single-thread-of-record requirement, and the alert configuration session where the detection model was scoped to alert-triggered events without establishing a customer complaint escalation path.

The incident command system ADR: five sections

Section 1: Incident command authority model. Specify who holds IC authority during an active incident, how the IC is assigned at incident declaration, and how the IC role transfers when the incident scope expands. IC assignment: the default IC is the on-call engineer who declares the incident. The default applies unless a more senior on-call role is explicitly designated as IC in the escalation policy for the incident's severity tier. The assignment is stated in the first message of the canonical incident thread, as a named person, not a role description. IC authority scope: the IC has binding decision authority for mitigation approach during the incident. When two or more responders disagree about the right recovery action, the IC makes the call. The IC's decision is communicated in the canonical thread as a decision — "we are doing X, not Y" — and the team executes the chosen approach. Advisory input before the decision and retrospective discussion after the incident are appropriate; parallel execution of contradictory approaches during the incident is not. IC transfer: the IC model must specify the transfer mechanism for two scenarios. First: the incident expands to cover systems outside the IC's domain expertise, and a more senior engineer or the relevant domain expert is better positioned to hold the IC role. Transfer requires an explicit handoff in the canonical thread, using the minimum viable handoff document format specified in Section 5. Second: the IC has been running the incident for more than four hours and requires relief. Transfer follows the same handoff protocol. An IC transfer without a documented handoff in the canonical thread is not a valid transfer — the incoming IC does not hold binding authority until the handoff is complete. Connect this section to the incident escalation policy decision record for the escalation tier definitions that determine the starting severity level, the responder set that is paged, and the context in which the IC assignment default applies.

Section 2: Incident coordination protocol and single-thread-of-record requirement. Specify the canonical coordination venue, the single-thread-of-record requirement, and the two-minute rule. Canonical venue: when an incident is declared, one thread is designated as the canonical incident record. For teams using Slack, this is typically a dedicated incident channel or a thread in the ops channel. The venue is stated in the incident declaration. Single-thread-of-record requirement: every finding, hypothesis, test result, decision, and action taken during the incident is recorded in the canonical thread before the next action is taken. This is not a documentation-after-the-fact requirement; it is a coordination-before-the-next-action requirement. The reason is structural: the canonical thread is the mechanism by which subsequent responders — including a fresh IC taking over after four hours, a domain expert who joins mid-incident, and the engineer who picks up the incident the next morning if it runs overnight — have access to the complete diagnostic context without reconstructing it. Two-minute rule: if a finding is actionable, it is in the canonical thread within two minutes or the action is not taken yet. This applies to findings that originate outside the canonical thread — DM conversations, sidebar video calls, direct platform checks — and must be brought into the thread before they affect the investigation. DM conversations and sidebar channels are permitted for efficiency in gathering information; they are not permitted as substitutes for the canonical thread when the information gathered affects the incident's direction. Enforce the two-minute rule operationally by making it the IC's specific responsibility to pull outside findings into the thread before the investigation moves forward on them. Connect this section to the release process decision record for the rollback and emergency deploy protocols that determine the options available to the IC when making mitigation approach decisions; the coordination protocol must specify how rollback and hotfix decisions are logged in the canonical thread, including who authorized the action and what the expected outcome is.

Section 3: Incident detection model and customer complaint escalation path. Specify the incident trigger types, the customer complaint escalation path, and the trend-based monitoring requirements that supplement alert-triggered detection. Trigger types: document each trigger type that can open an incident — alert-triggered (PagerDuty or equivalent, specific alert IDs and severity mappings), customer complaint escalation (support ticket or CSM escalation, time window before on-call notification, classification criteria for routing), and self-declared (engineering team member observes a degradation pattern without a corresponding alert). The customer complaint escalation path must be a first-class trigger, not an informal fallback. Specify: which support ticket classifications qualify for on-call escalation (performance degradation, unexplained error patterns, availability concerns), the maximum time between ticket opening and on-call notification for each classification (two hours maximum for revenue-critical workflow descriptions), and the on-call response requirement for an escalated ticket (same urgency as a pager alert — acknowledge within the on-call acknowledgment window, investigate immediately, open an incident if the investigation confirms a real degradation). Trend-based monitoring: supplement absolute-value alert thresholds with rate-of-change alerts on the metrics most vulnerable to gradual degradation — p95 latency increasing more than 20% over a rolling 24-hour window, error rate trending upward for 48 consecutive hours, database query plan cost increasing more than 15% week-over-week. Rate-of-change thresholds detect gradual failures that stay below absolute-value thresholds; the combination of both types closes the detection gap for both sudden failures and gradual accumulation patterns. Connect this section to the alerting threshold decision record for the specific threshold design decisions — absolute-value versus rate-of-change, per-customer versus aggregate, static versus rolling-baseline calibrated — that determine the alert-triggered detection surface.

Section 4: Incident role definitions and minimum viable role coverage. Specify the roles available in the incident command structure and the minimum viable coverage requirements for each severity tier. Roles: Incident Commander (IC) — binding decision authority for mitigation approach and escalation decisions; Operations Lead — executes technical actions directed by the IC, including diagnostic queries, configuration changes, rollbacks, and hotfixes; Communications Lead — manages customer-facing status communication and internal stakeholder updates; Scribe — maintains the canonical incident thread as a structured record, not just a conversation log. For small teams (five to fifteen engineers), the IC and Operations Lead roles are often combined in a single person for lower-severity incidents. Minimum viable coverage by severity: P3 (minor degradation, no customer impact) — IC only; P2 (customer-visible degradation, SLA not yet breached) — IC and Operations Lead; P1 (customer-visible degradation, SLA breached or at risk, revenue impact) — IC, Operations Lead, and Communications Lead; P0 (major outage, multiple customers affected, revenue at significant risk) — all four roles. The minimum viable coverage requirements define when additional people need to be pulled into the incident and in which role. Pulling people in without role assignments produces the parallel mitigation failure mode; the role assignment must happen at join time, not organically. Connect this section to the on-call rotation compensation decision record for the rotation design that determines which roles can be staffed from the on-call pool and which require waking additional people outside normal rotation.

Section 5: Incident handoff procedure and authority transfer protocol. Specify the minimum viable handoff document and the authority transfer protocol. Minimum viable handoff document (six elements): current customer impact — which customers are affected, what they are experiencing, how long the impact has been running, and whether the impact is improving, stable, or worsening; current working hypothesis — the team's best current model of the root cause, with the evidence supporting it and the evidence that has ruled out alternatives; tests run and results — a list of the specific diagnostic steps taken, the specific results from each, and what each result confirmed or ruled out; changes made — every configuration change, restart, rollback, or hotfix deployed, with timestamps and the name of the person who executed each; active mitigations — anything currently in place that is partially addressing the impact, including stability assessment; open questions — the specific things the team does not know, cannot explain, or has not been able to investigate. The handoff document is written by the outgoing IC and posted to the canonical incident thread. Authority transfer: the incoming IC reads the handoff document and posts a two-to-three sentence summary of their understanding of the current state in the canonical thread. If the outgoing IC is still available, they confirm or correct the summary before transferring authority. If the outgoing IC is not available, the incoming IC proceeds from their own reading of the thread. The handoff document is posted; authority is transferred; the incoming IC's name replaces the outgoing IC's name in the incident thread header. No further handoff steps are required. The six-element document is the minimum — additional context is welcome but the six elements are the floor for a valid handoff. Connect this section to the release process decision record for the emergency deploy and rollback authorization protocols that determine which actions the incoming IC is authorized to take immediately upon assuming the IC role without additional approval steps.

FAQ

How should teams structure IC authority to avoid the parallel mitigation problem when two senior engineers disagree during an active incident?

The parallel mitigation problem occurs when two responders with equal informal authority reach different conclusions about the right recovery action and both begin pursuing their approach simultaneously. The structural cause is an IC model where authority is implicit — assigned informally to whoever is most knowledgeable about the affected system — rather than explicit, where a specific person holds binding decision authority from the moment the incident is declared. The fix requires two things: first, the IC role must be assigned at incident declaration, not claimed organically during the incident. The assignment can follow a simple default rule — the on-call engineer who opened the incident is the IC unless they explicitly hand off — but the default must be documented and known before the incident begins. Second, the IC's authority must be explicitly scoped to include binding decisions on mitigation approach. This means that when two responders disagree about whether to roll back or deploy a hotfix, the IC makes the call, communicates it in the canonical channel as a decision rather than a suggestion, and the team executes the chosen approach. The IC is not necessarily the most technically senior person in the room. The IC is the person with binding decision authority for the duration of the incident. Technical expertise advises the IC's decision; it does not substitute for the IC role. For incidents that cross system boundaries, where the initial IC is the expert on one system but not another, the IC authority model must specify whether the IC role transfers to the most relevant expert or whether a more senior person assumes the IC role as the incident expands in scope. Both models are viable; the failure mode is leaving the transfer mechanism undefined and discovering the ambiguity under pressure.

What is the minimum viable incident coordination model for a small team without dedicated incident management tooling?

For teams of five to fifteen engineers without dedicated incident management tooling, the minimum viable coordination model requires three things and nothing more. First: a single canonical thread. When an incident is declared, one thread — a Slack channel, a dedicated incident channel, a thread in the ops channel — is designated as the canonical record. Everything relevant to the incident goes in that thread: the initial declaration, every hypothesis, every test run and its result, every decision made and who made it, every action taken and when. Not a summary of what happened — the working record, written as it happens. The requirement is that nothing that affects the incident's diagnosis or resolution happens outside that thread without being reported back to it within five minutes. Second: a named IC. One person holds the IC role. Their name is stated in the first message in the canonical thread. They have binding decision authority for mitigation approach and escalation decisions. They can be the on-call engineer, and often will be for smaller teams. Third: an explicit declaration and an explicit resolution. The incident starts when someone declares it — names it, assigns an IC, opens the canonical thread. The incident ends when the IC declares resolution and the thread records the resolution time and a one-line description of what was done. These three requirements can be implemented with no tooling beyond a chat application. They prevent the two most expensive incident failure modes — parallel mitigation due to unclear authority, and context fragmentation due to distributed working threads — without requiring a process overhead that is disproportionate for a small team's incident frequency.

How should teams detect slow degradations that accumulate customer impact without crossing individual alert thresholds?

Slow degradations that stay below individual alert thresholds have two reliable early detection signals that alert-triggered incident processes miss: customer complaint signals and trend-based monitoring. Customer complaint signals — support tickets, CSM escalations, customer-facing status inquiries, direct emails — often appear before any monitoring alert fires for gradual degradations. The structural fix is a support ticket escalation path that triggers the incident response process: when a support ticket is opened that describes a performance degradation or unexplained error pattern, there is a defined process for the support team or CSM to escalate to the on-call engineer within a specified time window, and the on-call engineer's review of the ticket is the incident detection event, not a monitoring alert. The time window for escalation should be short — two hours maximum — for tickets that describe behavior affecting revenue-critical customer workflows. Trend-based monitoring is the technical complement: rate-of-change alerts on key metrics (p95 latency increasing more than 20% over a rolling 24-hour window, error rate trending upward over a 48-hour period) detect gradual degradation that absolute-value thresholds miss. The combination — customer complaint escalation path plus trend-based monitoring — closes the slow-degradation detection gap that alert-triggered processes leave open. The key design requirement is that the customer complaint escalation path must be a first-class trigger for the incident response process, documented in the on-call runbook alongside alert-triggered triggers, with a defined response time SLA and a defined IC assignment path. Without this, the customer complaint signals reach a support queue that is not part of the incident detection infrastructure, and the degradation accumulates until an alert finally fires.

What should a minimum viable incident handoff document contain when the IC needs to transfer authority mid-incident?

An incident handoff document transfers not just the incident but the diagnostic context — the working model of what is failing, what has been tried, and what the receiving IC needs to know to continue without reconstructing the investigation from scratch. Six elements are required. One: current customer impact, described specifically — which customers are affected, what they are experiencing, how long the impact has been running, and whether the impact is getting worse, stable, or improving. Two: the current working hypothesis — the team's best current model of what is causing the failure, with the evidence that supports it and the evidence that has ruled out alternative hypotheses. Three: tests run and results — a list of the specific diagnostic steps taken, the specific results, and what each result confirmed or ruled out. This is the element most commonly omitted and most costly to reconstruct. Four: changes made — every configuration change, restart, rollback, or hotfix deployed during the incident, with timestamps. Five: active mitigations — anything currently in place that is partially addressing the impact, including whether those mitigations are stable or degrading. Six: open questions — the specific things the team does not yet know, does not understand, or has not been able to investigate. The handoff document is written by the outgoing IC and read back by the incoming IC in the canonical incident thread before the incoming IC takes any action. The read-back step surfaces misunderstandings before they affect the investigation, not after. The handoff document is not a post-mortem; it is a working brief for the next responder, written under incident conditions and intended to be read in two minutes.

Structural properties set by the incident command system decision

The incident command system ADR: five sections

FAQ

Further reading