The observability strategy decision record: why the observability model you chose determines your cross-service debugging surface and your mean time to root cause isolation ceiling

A 24-person developer tools SaaS building a code review automation platform had invested carefully in their monitoring infrastructure. Prometheus scraped metrics from every service — HTTP request rate, error rate, and latency percentiles at p50, p95, and p99. Grafana dashboards showed each service's health at a glance. PagerDuty alerted the on-call engineer when p99 latency at the API gateway exceeded two seconds for more than three consecutive minutes. The team was proud of their monitoring setup; it had correctly paged on every significant production incident in the previous twelve months and had never produced a false alarm.

The platform's API gateway fronted six downstream microservices: authentication, repository indexing, review-session management, enrichment, comment threading, and storage. Each service had its own Prometheus metrics. The enrichment service was responsible for annotating review sessions with computed metadata — language detection, complexity scoring, author context derived from the repository index. It used a Redis cache to store enrichment results per commit hash, with a twenty-four-hour TTL. On normal traffic, the cache hit rate was above 95%; enrichment calls were typically resolved from cache in under two milliseconds.

On a Tuesday afternoon, the API gateway's p99 latency climbed from 310 milliseconds to 4.4 seconds over a thirty-five-minute window. The on-call engineer was paged at 2:47 PM. She opened the Grafana dashboard and checked each downstream service's metrics in sequence. Authentication: error rate 0.0%, p99 latency 18ms, nominal. Repository indexing: error rate 0.0%, p99 latency 82ms, nominal. Review-session management: error rate 0.0%, p99 latency 44ms, nominal. Enrichment: error rate 0.0%, p99 latency 67ms — slightly elevated from the usual 12ms but within a range she had seen during normal batch operations. Comment threading: nominal. Storage: nominal. The downstream services all reported healthy. The API gateway's own CPU and memory were within normal operating ranges. No deployment had been made in the past six hours. The on-call engineer restarted the API gateway pod. The latency remained elevated. She restarted the enrichment service pod. The latency remained elevated.

The actual problem was a specific interaction between the enrichment cache TTL and a customer's batch upload workflow that the metrics model had no mechanism to reveal. One of the platform's enterprise customers had a CI pipeline that submitted code review sessions in batches of 120, triggered by their nightly build process. This customer's repository had last been reviewed three days prior — well outside the twenty-four-hour enrichment cache TTL. The batch upload triggered 120 concurrent API requests, each of which called the enrichment service, each of which encountered a cold cache, each of which fell through to a database query that JOINed the repository index across five tables. The query was correct and executed in approximately 40ms per call at low concurrency. At 120 concurrent calls, the database's connection pool saturated and queue depth climbed, pushing per-query execution time to between 380 and 720 milliseconds. The enrichment service's p99 latency metric showed 67ms — the weighted average of the 95% of requests served from cache at 2ms and the 5% served via database at 600ms, diluted into a number that looked elevated but not alarming on its own.

The on-call engineer spent ninety minutes diagnosing this incident. She grepped API gateway request logs, correlating timestamps with the batch upload's arrival. She identified that the latency elevation had started at 2:09 PM — thirty-eight minutes before she was paged. She matched the 2:09 PM timestamp against enrichment service logs and found elevated response times concentrated on requests with commit hashes from the same customer's repository namespace. She identified the cold cache pattern from the enrichment service logs. She paged the enrichment service owner at 4:18 PM with her hypothesis. The enrichment service owner confirmed the cold cache N+1 pattern in four minutes. Total time from incident start to root cause identification: two hours and thirteen minutes, of which ninety minutes was the on-call engineer's log correlation work and four minutes was the service expert's confirmation.

What the monitoring model could answer: the API gateway latency is elevated. What it could not answer: which downstream service or interaction is producing the latency, because no downstream service was individually unhealthy when measured in isolation. What a distributed trace would have shown within five minutes of the page: the API gateway's request spans fanning out to the enrichment service, with enrichment service spans showing 400–700ms response times instead of the expected 2ms cache-hit times, with a span attribute indicating a cache miss for each slow request in the batch. The enrichment service would have been named as the latency source before the on-call engineer had finished checking the fourth downstream service's dashboard. The observability model the team had invested in was correct at detecting that something was wrong; it could not, by design, answer what was wrong at the level of granularity needed to reduce mean time to root cause below the manual log correlation threshold. Connect this to the incident severity classification decision record: the incident was classified as Severity 3 based on the impact scope; a classification framework that included a diagnosability dimension — whether the failure is diagnosable using available tooling or requires service owner escalation — would have triggered a parallel escalation protocol that reached the enrichment service owner within twenty minutes rather than ninety, because the classification would have surfaced the structural observation that this class of cross-service interaction failure is not diagnosable from per-service aggregated metrics without trace context to attribute the latency to a specific service in the call chain.

A 47-person fintech building a consumer lending platform had adopted distributed tracing eighteen months before the incident described here. The adoption had been driven by a painful prior incident — a cross-service failure that had taken four hours to diagnose using only per-service logs — and the team had committed to never repeating that experience. They instrumented Jaeger across all seven microservices in their loan application processing pipeline: intake, credit-check, income-verification, collateral-assessment, risk-scoring, approval-routing, and disbursement-preparation. Each service propagated the Jaeger trace context through all synchronous HTTP calls. The sampling rate was set at 10% — the operations team had argued that 100% sampling would produce unacceptable storage costs. Each service had its own Jaeger instance, because the platform had been built incrementally across three different engineering subteams and the observability infrastructure had grown to match the team structure rather than the query patterns of incident response.

The loan processing pipeline was orchestrated by a saga coordinator service that tracked each loan application through state transitions: INTAKE_RECEIVED, CREDIT_CHECK_INITIATED, CREDIT_CHECK_COMPLETE, INCOME_VERIFICATION_INITIATED, INCOME_VERIFICATION_COMPLETE, COLLATERAL_ASSESSMENT_INITIATED, COLLATERAL_ASSESSMENT_COMPLETE, RISK_SCORING_COMPLETE, APPROVAL_ROUTING_COMPLETE, DISBURSEMENT_PREPARED. The saga coordinator logged each state transition to its application log and used a Postgres table to persist saga state. The state transitions were not emitted as observable events into the Jaeger trace or into any event stream that could be queried independently of the saga coordinator's application log.

An income-verification service deployment at 10:14 AM introduced a bug in the error handling path for external credit bureau API responses. When the bureau API returned an unexpected response format — a condition that had occurred zero times in the previous six months but occurred that morning due to a bureau-side format change — the service's error handler correctly caught the exception but then returned HTTP 200 with a response body containing a system error indicator rather than the HTTP 500 the upstream saga coordinator expected from a service error. The saga coordinator received the 200 OK, interpreted it as a success, and advanced the affected loan applications to INCOME_VERIFICATION_COMPLETE. The collateral-assessment service expected specific income verification fields in the saga payload that the income-verification service had not populated because it hit the error path. The collateral-assessment service returned HTTP 422 on the missing required fields — which the saga coordinator correctly interpreted as a failure — and paused eight loan applications in COLLATERAL_ASSESSMENT_FAILED state, queuing an alert to the operations team.

The on-call engineer received the alert at 11:47 AM: eight loan applications stalled in COLLATERAL_ASSESSMENT_FAILED. She needed to determine whether the failure was in the collateral-assessment service itself, in the upstream income-verification service, or in the loan application data. She looked up the eight saga IDs and searched the collateral-assessment service's Jaeger instance for traces corresponding to those saga IDs. None of the eight had a stored trace — all eight were mortgage refinance applications, a product that represented approximately 11% of daily volume, and with 10% trace sampling, the probability of a specific application having a stored trace was 10%. Zero of eight had been sampled. She searched the income-verification service's Jaeger instance. Two of eight had stored traces. Both showed the income-verification service returning HTTP 200. Nothing in the trace indicated an error; the error had been caught and suppressed before the span was closed.

She looked at the saga coordinator's application log, searching for the eight saga IDs. She found the state transition records and traced each saga through its transitions. The income-verification step showed INCOME_VERIFICATION_COMPLETE for all eight, which was inconsistent with a collateral assessment failure caused by missing income verification data. She checked whether the income-verification service had written its output to the expected storage location. It had not. She paged the income-verification service owner at 12:41 PM, fifty-four minutes after the alert. The service owner identified the error handling bug within eight minutes of being paged: the HTTP 200 return on the error path was the cause, the external bureau API format change was the trigger, and the downstream collateral assessment failure was the symptom. Total time from the first failed loan application at 10:15 AM to root cause identification at 12:49 PM: two hours and thirty-four minutes. Of that, eight minutes was the service expert's diagnosis and the remainder was the on-call engineer's cross-service log correlation work.

A distributed trace would have shown the income-verification service's internal span producing an error event while returning a 200 status code — this is a canonical distributed trace diagnostic pattern, the span status (error) contradicting the HTTP response code (success). The cross-service trace would have named the income-verification service as the anomaly within five minutes rather than requiring fifty-four minutes of log correlation to reach the service owner hypothesis. The distributed tracing investment the team had made was real. But two structural gaps prevented it from reducing the MTTR below one hour. First, the 10% sampling rate meant that low-volume loan types had near-zero probability of having stored traces for any specific incident affecting them; the traces were unavailable exactly in the incident patterns most likely to involve edge cases — low-volume transaction types that hit uncommon code paths. Second, the per-service Jaeger topology required the on-call engineer to know which service to query before querying it, rather than starting from a cross-service query that showed the full call chain. A unified trace query surface — even with the same 10% sampling rate — would have reduced the work from manually querying seven service instances to a single query returning the full saga trace for the few applications that were sampled, surfacing the income-verification service as the first service to show an anomaly in the call chain order within the trace. Connect this to the data pipeline reliability decision record: the income-verification service returning success status on an error path is the same pattern as the fintech in that record whose JSON parsing library introduced silent float64 precision truncation — in both cases the detection gap was produced by a service returning a success status for an operation that had not succeeded, and in both cases the gap was structurally extended by the absence of observable signals that would have shown the output as invalid independent of the response status code.

A 71-person enterprise SaaS building a product analytics platform for retail enterprise customers had invested substantially in observability infrastructure. Every service was instrumented with Datadog metrics. Application logs were aggregated in Elasticsearch. Distributed traces ran at 100% sampling on the critical API and data processing paths. PagerDuty was configured with SLO-based alerting on error rate, latency, and availability SLOs on each critical service, with alert conditions calibrated to produce pages only on genuine service degradation. The engineering team's on-call rotation had operated without a false-alarm wake-up in eleven consecutive weeks. The observability infrastructure was the right investment for the infrastructure layer of the platform.

The platform's nightly data processing pipeline ran at 1:00 AM. The pipeline ingested product usage records uploaded by enterprise customers, classified each record using a category model, aggregated the classified records into customer-specific analytics, and wrote the results to a reporting database that scheduled report generation jobs read at 6:00 AM. The pipeline used a finite state machine for each processing batch: RECEIVED → VALIDATED → CLASSIFIED → AGGREGATED → COMPLETE, with a FAILED state reachable from any step. The state machine was implemented in the pipeline coordinator service and persisted its state in the reporting database. Each step was processed by a separate downstream service — validation, classification, aggregation — that the coordinator called in sequence, advancing the state machine on success or setting the batch to FAILED on error.

A new enterprise customer had onboarded three weeks earlier. Their product usage records followed the platform's standard format with one difference: their "product_variant" field, which all other customers populated as a non-empty string or omitted entirely, was present in their records but set to null on approximately 23% of records — records where their product configuration did not use variants. The classification service handled the product_variant field with a conditional: if the field was present and non-empty, route the record to a variant-specific classification branch; otherwise, route to the default branch. The condition was written as if record.product_variant — a truthiness check that treated null as falsy and routed null-valued records to the default branch, which was the intended behavior. The default branch was written assuming product_variant was simply absent rather than present-but-null. It had a code path that read record.product_variant.lower() to normalize case variation — a line that was unreachable for records without a product_variant field but that executed on the null-valued records, raising a TypeError. The TypeError was caught by the service's error handler. The error handler logged the exception at the WARNING level, incremented a warning counter, and returned an empty classification result rather than a failed result. The coordinator received the empty result, interpreted it as a successful classification with no applicable categories, and advanced the batch to CLASSIFIED state.

The classification service's Datadog metrics throughout the night: error rate 0.0%, warning rate 0.03% (too low to trigger any alert threshold), p99 latency 142ms, CPU 34%, memory nominal. The coordinator service's metrics: state machine transitions completing at expected rate, no FAILED states, all batches reaching COMPLETE state. The Elasticsearch logs contained 847 WARNING entries from the classification service across a three-hour processing window, each logging the TypeError with the affected record ID. No alert was configured on WARNING log volume — alerts were configured on ERROR log volume and on service error rate, because those were the signals that corresponded to actionable failures in the infrastructure model.

At 6:00 AM, the report generation jobs began querying the reporting database. Four enterprise customers whose reports depended on the new customer's analytics found their data for the new customer returned zero values across all product categories. The customers received their scheduled reports at 6:14 AM with every chart showing zero product usage. The platform's support team received four escalation emails before 7:00 AM. The SRE on call investigated starting at 7:23 AM. She checked the Datadog dashboards: all services healthy. She checked Elasticsearch: no ERROR entries in the classification service from the previous night's run. She checked the distributed traces for the nightly pipeline coordinator: all traces showed successful completions. The infrastructure observability model showed a healthy pipeline run. She queried the reporting database directly: 847 COMPLETE batches for the new customer, each with an analytics record containing zero classified events. She identified that the classification step had produced empty output for all 847 batches. She paged the classification service owner at 8:01 AM. The classification service owner read the WARNING log entries in Elasticsearch in eleven minutes and identified the null product_variant TypeError as the root cause.

Total incident timeline: failure began at 1:23 AM. Detection by the engineering team: 7:23 AM — a six-hour detection gap. Root cause identification: 8:12 AM, forty-nine minutes after investigation began. Resolution required reprocessing all 847 affected batches, taking four hours and fifteen minutes. Reports for the four affected customers were reissued at 12:47 PM — nearly seven hours after the scheduled delivery time. Connect this to the SLA design decision record: the platform's enterprise SLA committed to report delivery by 8:00 AM; the actual delivery was 12:47 PM, a four-hour-forty-seven-minute breach; the SLA breach triggered contract credit clauses for two of the four affected customers; the total incident cost was approximately $12,200 in engineering time and contract credits — all attributable to a six-hour detection gap that business event instrumentation on the state machine would have closed to under fifteen minutes by alerting on the condition "batch transitioned to CLASSIFIED with zero classified events, expected minimum N based on input record count." The infrastructure observability infrastructure the team had invested in was correct for the infrastructure layer — services were healthy, which was accurate. The platform needed an additional instrumentation layer: business event checkpoints at state machine transitions carrying outcome measurements, where a threshold alert on zero-output batches would have detected the failure within fifteen minutes of the first batch completing classification with empty results rather than eighteen hours later when a downstream consumer reported the incorrect output.

Structural properties set by the observability strategy decision

Two structural properties are determined by the founding decisions about what to instrument, at what layer, and how to correlate and query the resulting signals during incident response. Neither is visible during normal operations; both become visible only when an incident occurs in a failure class the observability model was not designed to detect — at which point the structural property is expressed as either a rapid diagnosis or an extended manual investigation, depending on whether the model covers that failure class.

Property 1: Cross-service debugging surface. The cross-service debugging surface is the set of failure modes across service boundaries that can be diagnosed by an on-call engineer using only the available observability tooling, without requiring escalation to the engineers who originally built the involved services. The surface is bounded by three factors that interact. The first is trace context propagation completeness: whether distributed trace context propagates through all communication channels — synchronous HTTP calls, async message queue publishes, scheduled job triggers, saga coordinator state transitions. A trace context that propagates through HTTP calls but drops across an async message queue publish creates a debuggability gap at every queue boundary, because the on-call engineer cannot follow the call chain from the producer service to the consumer service using the trace surface alone. The second is business event instrumentation at the application layer: whether the observability model includes events at the business process level — state machine transitions, saga completion checkpoints, batch processing outcomes with measurement dimensions — in addition to infrastructure events. A model that instruments only the infrastructure layer has a debugging surface bounded by what infrastructure signals can reveal about application-layer behavior, which excludes the class of failure that produces correct infrastructure behavior but incorrect business process output. The third is query surface design for incident response: whether the signal correlation model allows an on-call engineer who did not build the failing service to determine the failing service in under five minutes, or whether doing so requires manual log correlation across multiple dashboards and knowledge of where each service's logs and traces are stored. A cross-service debugging surface that requires service-specific knowledge to query converts the debugging surface into a surface available only to engineers who built each component — eliminating the primary benefit of an observability model, which is to distribute debugging capability across the on-call rotation. Connect this to the incident severity classification decision record: the incident severity classification framework should include a diagnosability dimension — whether the failure is diagnosable by the on-call engineer using available tooling or requires escalation to the service owner — because diagnosability determines the appropriate escalation protocol; a failure that is Severity 2 by impact but undiagnosable without service owner escalation should trigger a parallel escalation protocol rather than waiting for the on-call engineer to exhaust manual investigation options first.

Property 2: Mean time to root cause isolation ceiling. The mean time to root cause isolation ceiling is the minimum achievable time from incident detection to root cause identification given the current observability model, under optimal conditions — the structural lower bound that the instrumentation and query model sets regardless of engineer skill or team velocity. This is a ceiling, not an average, because it represents what is achievable by the best possible execution of the current model's diagnostic process even when an experienced engineer is executing the diagnosis optimally with all relevant signals available. The actual MTTR varies above the ceiling based on execution factors, but no amount of process improvement or skill development pushes MTTR below the ceiling the observability model sets. A service-level metric model without distributed tracing has a ceiling of "however long it takes to identify the failing component from per-service aggregate metrics and confirm the hypothesis through log correlation" — which ranges from twenty minutes for failures that produce distinctive metric signatures to several hours for interaction failures between individually-healthy services. A distributed trace model without a unified query surface has a ceiling lower than the metric-only ceiling for failures where traces are available, but still measured in tens of minutes for complex cross-service sagas, and undefined for failures where the sampling rate means traces are unavailable for the specific affected requests. A model with infrastructure metrics, distributed traces, a unified trace query surface, and business event checkpoints has a ceiling measurable in minutes when the query surface is designed for incident response, because the business event stream directly answers "which step in this process failed and what was the outcome measurement at that step" without requiring the on-call engineer to know which service to investigate first. Connect this to the capacity planning decision record: the engineering capacity invested in incident response — on-call rotations, postmortem processes, escalation coordination — produces returns directly proportional to the MTTR ceiling the observability model sets; a team that invests heavily in incident response process improvements while operating a metric-only observability model is optimizing around a structural constraint rather than addressing it; the highest-return reliability investment for most teams past a certain service count is closing the observability model gap rather than improving the manual diagnostic process the current model requires.

The observability strategy ADR: five sections

Section 1: Observability model selection and tier definition. The observability model selection is a decision about which layers of the system are instrumented and at what fidelity. The decision is not about which tools to adopt — tool selection follows model selection — but about what questions the observability model is required to be able to answer. The requirement set should be derived from the failure classes most likely to produce the most expensive incidents in this specific system, not from a generic "we need metrics and logs" assumption. For most systems with cross-service dependencies, the minimum viable model that reduces mean time to root cause below thirty minutes requires three layers: infrastructure metrics (request rate, error rate, latency at the service boundary — answers "is this service operating normally?"), distributed traces with a unified query surface (span chains across all service boundaries — answers "which service in this call chain is the anomaly source?"), and business event checkpoints at all stateful business process transitions (state machine transitions with outcome measurements — answers "did this business process complete with the expected output?"). The case for operating below this minimum — typically a resource constraint argument — should be evaluated against the cost of the incidents that the missing layer will fail to diagnose quickly. The developer tools example operated at the first layer only; the resulting MTTR ceiling for cross-service interaction failures was bounded by log correlation time. The fintech operated at the first and second layers but with a fragmented query surface; the resulting MTTR ceiling for failures affecting low-volume transaction types was bounded by manual trace lookup time across seven service-specific dashboards. The enterprise SaaS operated at the first and second layers with high trace coverage but no third layer; the resulting MTTR ceiling for application-layer business process failures was bounded by how long a downstream consumer takes to notice incorrect output. Each of these ceilings is the direct consequence of the model selection, not of the team's execution quality. Connect this to the SLA design decision record: the MTTR ceiling that the observability model sets is a direct input to the SLA commitments the team can credibly make; a platform whose SLA commits to data delivery by 8:00 AM and whose observability model cannot detect a business-process failure until a downstream job fails at 6:00 AM has a structural misalignment between the SLA commitment and the detection capability that should be resolved before the SLA is made rather than after it is breached.

Section 2: Instrumentation strategy and coverage model per layer. The instrumentation strategy specifies what to instrument at each layer, what constitutes sufficient coverage for each layer, and how coverage gaps are identified before incidents reveal them. For the infrastructure metrics layer, sufficiency is defined by the four golden signals — latency, traffic, errors, saturation — measured at every service boundary and at the outbound call paths between services. The critical coverage gap in infrastructure metrics is at service-to-service boundaries: a service's outbound call latency to its downstream dependencies is a distinct signal from the downstream service's inbound request latency, and the difference between them measures the network and serialization overhead that can itself be a latency source. The developer tools example: the enrichment service's inbound latency metric showed 67ms — a blend of cache hits and cache misses — while the API gateway's outbound call to the enrichment service would have shown the true distribution including the 600ms cache miss tail. Instrumenting only inbound request metrics at each service creates a blind spot for interaction latency that is visible only in the caller's outbound metrics. For the distributed trace layer, sufficiency is defined by whether trace context propagates across every communication boundary the system uses: synchronous HTTP, asynchronous message queues, scheduled jobs, batch processing triggers, and saga coordinator state transitions. The critical coverage gap in distributed traces is at async boundaries: a trace that drops context when a message is placed on a queue produces a broken chain at every queue boundary that requires manual timestamp correlation to reconnect. For the business event layer, sufficiency is defined by whether every stateful business process transition emits an observable event with an identity dimension, an outcome dimension, and a measurement dimension. The measurement dimension is the critical component: an event that records "batch X transitioned to CLASSIFIED" without recording how many records were classified is a debugging tool but not a detection mechanism. An event that records "batch X transitioned to CLASSIFIED with N classified records, expected minimum M based on input count" is a detection mechanism that can trigger a threshold alert when N is below M.

Section 3: Signal correlation and query surface design. The signal correlation model specifies how signals from different layers are correlated during incident response and what the query surface looks like for an on-call engineer who has received a page and needs to identify the root cause within a defined time target. The design principle is that the query surface must be operable without prior knowledge of which service is the root cause — the engineer starts from the symptom (latency elevated at the API gateway, batch in a failed state, downstream consumer returning empty data) and the query surface should produce the answer without requiring the engineer to know where to look first. This requires a unified correlation entry point: a dashboard or query surface that takes an incident identifier — a request trace ID, a saga ID, a batch ID, a time range — and returns the relevant signals from all instrumented layers for that identifier in a single view. The trace topology for the fintech's seven-service saga should have been queryable as "show me all spans and state transitions for saga ID X across all services" rather than "query service 1's Jaeger instance, then service 2's, then service 3's." Building this unified query surface is an engineering investment distinct from the instrumentation investment: instrumenting seven services produces signals in seven places; building a unified query surface consolidates those signals into a single diagnostic entry point. The investment is often deferred because the signals feel "available" once the instrumentation is in place. The fintech's incident demonstrates why availability of signals and queryability of signals under incident time pressure are not the same thing. Connect this to the data pipeline reliability decision record: a data pipeline that emits business event checkpoints at each stage transition produces a unified query surface for the pipeline's own execution history; querying "show me all stage transitions for pipeline batch X" from a business event store is structurally the same operation as querying "show me all service spans for trace ID Y" from a distributed trace store — both are unified entry points that eliminate the need for per-service manual lookup; the pipeline reliability record describes this as the failure propagation containment model; the observability record describes it as the signal correlation model; they are the same architectural pattern applied to different problem surfaces.

Section 4: Alerting model and SLI/SLO definition. The alerting model specifies what conditions produce alerts, what alert conditions correspond to which incident response protocols, and how the alert model is calibrated to produce a signal-to-noise ratio that keeps the on-call rotation alert to real failures rather than habituated to noise. The SLI/SLO framework is the foundation: defining the service-level indicators that measure the user-visible properties of each service and the objectives that specify the acceptable operating range for each indicator. The critical design decision is whether SLIs are defined at the infrastructure layer only (error rate, latency) or also at the business process layer (classification rate above expected minimum, saga completion within expected duration, batch output above expected record count). Infrastructure-layer SLIs detect infrastructure failures correctly and promptly; they produce no signal on the class of failures that produce correct infrastructure behavior and incorrect business process output. Business-process SLIs produce signal on that failure class — the classification service returning empty results, the income-verification service returning success status on an error path, the state machine completing with zero output — but require the business event instrumentation layer to exist before they can be defined. The alert calibration for business-process SLIs requires establishing the expected baseline for each measurement dimension from historical data before setting alert thresholds: a classification service that classifies an average of 78% of records should alert when a batch's classification rate drops below 50%, not when it drops below 80%, to avoid false alarms on legitimate low-classification batches. The threshold should be computed relative to the input characteristics of each batch — the customer tier and record type distribution that predict the expected classification rate — rather than as a global constant. An alert model calibrated this way is specific, low-noise, and actionable; it triggers when a batch's output is anomalous relative to its own expected output, not when a global threshold is crossed.

Section 5: Observability review cadence and coverage auditing. The observability model is not a one-time infrastructure decision; it is a living model that must be maintained as services are added, communication patterns change, and new failure classes are discovered through incidents and near-misses. The review cadence should include three components. A post-incident observability gap review: every incident postmortem should include a structured question — "was the root cause diagnosable using available observability tooling within the MTTR target, and if not, what instrumentation would have enabled a faster diagnosis?" The developer tools incident would have produced a backlog item for "add distributed tracing with cache hit/miss span attributes to enrichment service calls from API gateway," because the gap was in the observability model rather than in the service behavior — the cache behavior was correct; what was missing was observability of it. A quarterly coverage audit: for each critical service and each cross-service communication boundary, verify that the instrumentation coverage matches the current model specification — that traces propagate across async boundaries added since the last audit, that business event checkpoints exist for state machine transitions added to new services, and that the unified query surface covers all services in the current critical path. Coverage audits are specifically important after service additions because a new service that joins the critical path may not propagate trace context if it was built before the trace propagation standard was adopted. An annual observability model review: evaluate whether the model's tier definition and sufficiency criteria still match the current failure class distribution and SLA commitments. The AI chat session history for the architecture and observability design sessions that produced the current model should be reviewed at this point. Running those sessions through the open-source extractor before the annual review surfaces the deferred instrumentation decisions with their original deferral rationale — specifically the revisit conditions that were defined at decision time and that the current review should evaluate against: "deferred business event instrumentation on the classification service because the state machine schema was not yet stable; revisit when the classification pipeline is in maintenance mode and the state machine schema is locked." If the state machine is now locked but the business event instrumentation was never added because the revisit condition was never reviewed, the extractor surfaces that gap as a recoverable decision rather than a lost one.

FAQ

Further reading

• Incident severity classification decision record — the severity classification framework and the observability model are jointly responsible for the incident response protocol; the severity framework determines the escalation protocol triggered by an alert, while the observability model determines whether the alert contains enough diagnostic context to execute the right escalation protocol without manual investigation first; a classification framework that includes diagnosability as a dimension — distinguishing between failures that are immediately diagnosable using available tooling and failures that require service owner escalation — produces escalation protocols calibrated to the actual diagnostic complexity of each incident class rather than to its impact severity alone; the developer tools incident was Severity 3 by impact and undiagnosable without service owner escalation by diagnostic complexity, a combination the standard severity framework did not distinguish and that added ninety minutes to root cause identification time.
• SLA design decision record — the SLA commitments a platform can credibly make are bounded by the detection capability of its observability model; a platform that commits to data delivery by 8:00 AM and whose observability model cannot detect a business process failure until a downstream job fails at 6:00 AM has a structural gap between the SLA commitment and the earliest possible detection time; the SLA design decision should include an explicit assessment of the detection gap for each class of failure that would produce an SLA breach, and the observability model should be designed to close detection gaps that would produce SLA breaches before the SLA commitment is made; the enterprise SaaS example: a business event checkpoint on the classification state machine transition with a threshold alert on zero-output batches would have detected the failure within fifteen minutes of the first batch completing with empty results rather than at 7:23 AM when the SRE began investigating customer escalations, converting a guaranteed four-hour SLA breach into a potential same-night recovery.
• Data pipeline reliability decision record — the data pipeline reliability model and the observability strategy are complementary: the reliability model defines what correct pipeline behavior looks like (freshness, completeness, correctness, correction commitments per consumer); the observability model defines what signals are available to detect deviations from correct pipeline behavior before those deviations propagate downstream; a pipeline with a reliability model but no business event instrumentation has defined correctness commitments without detection mechanisms; a pipeline with business event instrumentation but no reliability model has detection signals without defined thresholds; the joint design — reliability SLIs that specify the expected measurement dimensions for each pipeline stage transition, combined with business event checkpoints that emit those measurements — produces a detection model that is both threshold-alertable and interpretable in the context of the reliability commitments the pipeline has made to its downstream consumers.
• Capacity planning decision record — the engineering capacity invested in incident response — on-call rotations, postmortem processes, escalation coordination, service owner pages outside business hours — is a direct function of the MTTR ceiling the observability model sets; a team operating a metric-only observability model invests on-call capacity in log correlation and manual investigation for every cross-service interaction failure; the same on-call capacity invested in building a unified distributed trace query surface and business event instrumentation produces a durable reduction in the log correlation work required per incident; the capacity planning decision for reliability engineering should include an explicit evaluation of the observability model improvement investment versus the on-call capacity cost of the current MTTR ceiling, specifically for the failure classes that produce the highest on-call burden per quarter; in most systems past a certain service count, the observability model improvement is the higher-return investment.
• Open-source extractor — the architecture and design sessions in ChatGPT or Claude where engineers worked through the observability instrumentation plan, sampling rate tradeoffs, alert threshold reasoning, and trace topology choices are the primary source for the deferral reasoning that the ADR will compress into "not instrumented"; running those sessions through the extractor before writing the observability ADR and before conducting annual observability model reviews surfaces the specific revisit conditions that were defined at decision time — the service stability threshold at which business event instrumentation was planned, the traffic volume at which the sampling rate was expected to be revisited, the incident frequency at which alert threshold calibration was expected to be data-driven rather than estimated; these revisit conditions are the operational intelligence that determines whether the next team who inherits the observability model is inheriting a known gap with a defined closure condition or is inheriting what appears to be a complete model with an invisible structural gap.