The service observability ownership decision record: why the telemetry ownership boundary you chose determines your cross-team debugging friction and your monitoring coverage gap

A 52-person B2B SaaS company built a project resource planning platform for professional services firms — billable hour tracking, project margin analysis, resource forecasting, and client reporting. The engineering organization of 24 engineers was structured around a product organization of five squads (Projects, Resources, Billing, Reporting, Integrations) and a platform organization of three platform squads (Infrastructure, Developer Experience, and Observability). The Observability squad — three engineers plus a tech lead — was responsible for the full instrumentation lifecycle for all production services: designing the metrics schema, writing the dashboard and alert configurations, instrumenting new services as they were deployed, and maintaining the alert quality as services evolved.

The centralized model had been chosen deliberately. Eighteen months earlier, when the engineering team was 11 engineers and the service portfolio was 6 microservices, the observability quality had been inconsistent — some services had excellent latency dashboards written by engineers who cared about observability, others had nothing beyond the default Kubernetes container metrics. After a 3-hour incident in the Billing service where the on-call engineer spent the first 90 minutes determining that the problem was in Billing and not in the downstream payment gateway — both had similar latency characteristics from the upstream caller's perspective, and there were no per-service error rate dashboards — the VP of Engineering decided to centralize instrumentation ownership. The rationale was sound: a dedicated team with deep observability expertise would produce consistently higher quality instrumentation than product engineers instrumenting their own services as an afterthought, and it would create a single standard rather than the divergent practices that had emerged organically.

The Observability squad succeeded at its stated objective. The six existing services were instrumented to a high standard within six weeks of the squad's formation: consistent metric naming conventions, detailed latency histograms at p50/p95/p99, per-endpoint error rate dashboards with automated alerting at 1% and 5% error rate thresholds, distributed tracing with full span coverage for all cross-service calls, and structured logs with trace IDs in every log line. The quality of the instrumentation improved measurably. The on-call rotation's mean time to identify the failing service in an incident dropped from 47 minutes to 9 minutes over the first quarter after the squad was established.

The problem emerged 14 months later, when the service portfolio had grown from 6 to 23 microservices. The Integrations squad had shipped 8 new services for third-party integration support. The Billing squad had decomposed its original monolith into 4 separate services. The Infrastructure squad had shipped a new internal authentication service and a configuration management service. The Observability squad had instrumented 16 of the 23 services. The remaining 7 were in the instrumentation backlog — a Notion document where the squad tracked upcoming instrumentation work. The backlog had been present for the entire 14-month period; it had been as short as 2 items at one point and as long as 9. The current depth of 7 items had accumulated because the squad's three engineers were managing a combination of new service instrumentation, alert tuning for existing services, and a migration of the metrics storage backend from a self-hosted Prometheus cluster to a managed time-series database. The 7 uninstrumented services were not in active on-call rotation coverage — they were newer, lower-traffic services that had not yet had production incidents. The gap between services deployed and services instrumented was not tracked anywhere visible to engineering leadership.

At month 14, an incident occurred in the new Payments Reconciliation service — one of the 7 uninstrumented services — that had been deployed 8 weeks earlier as part of the Billing squad's decomposition project. The service was responsible for reconciling daily payment batches against the external payment processor's settlement reports. A silent data processing bug caused payment batch reconciliation to fail for 3 enterprise customers over 11 consecutive days, with the failed reconciliations silently written to an error table rather than triggering an alert. No alert fired because the Payments Reconciliation service had no alerting. The error table had grown to 127 failed reconciliation events before one of the affected customers reported a discrepancy in their monthly billing summary. The investigating engineer opened the observability dashboard for Payments Reconciliation and found nothing — no metrics, no traces, no structured logs. The service was emitting Kubernetes container metrics (CPU, memory, restart count) and application stdout logs in an unstructured format written by the original developer. The investigation required 4.7 hours — 1.2 hours identifying that the service was the source of the discrepancy (the upstream caller had no per-destination error rate data), and 3.5 hours reading raw unstructured application logs to trace the pattern of failed reconciliations and identify the processing condition that caused them. Connect this pattern to the alerting threshold decision record: the alerting threshold decision determines which signals trigger alerts and at what sensitivity; the observability ownership decision determines whether those signals exist at all; an alerting strategy that has been designed and validated for the instrumented portion of the service portfolio creates a false safety guarantee when applied to an engineering organization where 30 percent of production services are uninstrumented, because the alert coverage and the service coverage are not the same metric and are not tracked together.

The post-incident review identified three findings. First, the coverage gap: 7 of 23 production services were uninstrumented at the time of the incident, and none of the three engineers in the Observability squad, none of the squad leads, and none of the engineering leadership had visibility into this number. Second, the deployment-to-instrumentation gap: the Payments Reconciliation service had been deployed to production 8 weeks before the incident, but instrumentation was still in the backlog. The deployment process had no gate or check that required instrumentation before or within a defined window after deployment. Third, the gap detection mechanism: there was no mechanism that compared the set of services in production against the set of services with active observability coverage. The Observability squad maintained a Notion backlog of upcoming work but did not maintain a registry of services with their instrumentation status. Engineering leadership received a weekly uptime report but not a weekly coverage report. The gap between services deployed and services instrumented was structurally invisible under the current model.

The VP of Engineering made two changes. The deployment pipeline was updated to include an instrumentation gate: new services could be deployed to production, but the on-call rotation inclusion (the point at which a service was added to the on-call alert routing) required a completed instrumentation checklist signed off by the Observability squad. The Observability squad began maintaining a service registry with instrumentation status, published weekly to the engineering-observability Slack channel. The Payments Reconciliation service was instrumented within 6 days of the incident; the remaining 6 uninstrumented services were instrumented over the following 3 weeks. The centralized ownership model was retained — the quality benefit was real and the squad's backlog management was the variable to fix, not the ownership structure itself.

A 41-person cloud infrastructure company built a developer-facing API for automated cloud resource provisioning — infrastructure-as-code execution, cost estimation, policy enforcement, and change audit logging for teams managing cloud infrastructure through code. The engineering team of 19 engineers was organized into 8 product and platform squads ranging from 2 to 4 engineers each: API Gateway, Identity and Access, Resource Planner, Execution Engine, Policy Engine, Cost Estimator, Audit Service, and Platform Infrastructure. Each squad owned their service end-to-end — architecture, implementation, deployment, and observability. The VP of Engineering had explicitly designed the model this way: the engineering culture valued deep ownership, the squads were staffed with senior engineers, and the argument for centralized observability tooling had been made in the founding team's architecture discussion and rejected on the grounds that centralized tooling creates bottlenecks and that senior engineers should be trusted to instrument their own services to the standards they knew were appropriate for them.

Within-service observability at this company was genuinely excellent. Each squad had built exactly the instrumentation their service needed: the Execution Engine had detailed per-provider latency histograms (AWS, GCP, Azure provider latency tracked separately at p50/p95/p99), a resource provisioning success rate by resource type, and a queue depth alert that fired at 80% of maximum concurrent execution capacity. The Cost Estimator had a per-estimation-request latency histogram, a cache hit rate metric, and a staleness alert that fired when the pricing data cache exceeded 6 hours without a successful refresh. Each service's on-call engineer knew the dashboards intimately and could diagnose within-service incidents quickly. The single-service on-call MTTR for the 12 months under the federated ownership model was 7 minutes — outstanding.

The cross-service incident MTTR was different. An incident at month 14 started with 4 enterprise customers reporting that infrastructure provisioning requests were silently timing out — requests submitted through the API would return a 202 Accepted, but the provisioning job would never complete and no error would be surfaced to the customer. The full request path for a provisioning operation was: API Gateway → Identity and Access (authorization check) → Resource Planner (capacity planning check) → Execution Engine (the actual provisioning job). The on-call engineer was from the API Gateway squad. Her initial investigation confirmed that API Gateway was accepting requests and handing them off to the Identity and Access service — all metrics nominal from the Gateway's perspective. She opened the Identity and Access dashboard in a separate browser tab.

The first incompatibility surfaced immediately. The Identity and Access service had been built by the Singapore-based two-engineer squad, who had set the log timestamp format to UTC+8 (Asia/Singapore timezone) because that was the local timezone of both engineers. The API Gateway used UTC. A sequence of events that spanned 14:23 UTC in the API Gateway logs corresponded to 22:23 in the Identity and Access logs. The on-call engineer spent 8 minutes identifying the timezone discrepancy and manually converting log timestamps to correlate the two services' event sequences. The Identity and Access metrics were clean — authorization checks were completing within expected latency. She moved to the Resource Planner.

The Resource Planner and the Execution Engine had been instrumented using a different distributed tracing library than the API Gateway and Identity and Access services. The API Gateway and Identity and Access used the OpenTelemetry SDK with W3C traceparent headers for trace context propagation. The Resource Planner squad had selected a vendor-specific tracing library for their service that used B3 headers. Trace context was not propagated across the Identity and Access → Resource Planner boundary — the B3 header format was rejected by the Resource Planner's library as an unrecognized header, and the trace that began at the API Gateway terminated at the Identity and Access → Resource Planner handoff. There were no distributed traces connecting the four services in the provisioning call path. The on-call engineer had latency metrics for each service independently but could not trace a specific failing request through the system — she could see aggregate latency percentiles but not the specific execution path of a timing-out request. Connect this pattern to the distributed tracing decision record: the distributed tracing decision specifies the tracing library, the context propagation format, and the span coverage requirements for the organization; when adjacent service teams make independent tracing decisions and choose incompatible libraries or propagation formats, the result is a fragmented trace graph where traces begin at the originating service and terminate at the first incompatibility boundary; the tracing infrastructure exists but provides no cross-service visibility for incidents that cross the incompatibility boundary, which is precisely the set of incidents where distributed tracing provides the most diagnostic value.

The on-call engineer called the Resource Planner squad lead — it was 22:40 UTC, 06:40 in Singapore — to review the Resource Planner logs together. The Resource Planner logs confirmed that provisioning requests were entering the service and being handed off to the Execution Engine. The on-call engineer opened the Execution Engine metrics dashboard. The Execution Engine squad had built their metrics using a library that stored duration values in milliseconds; every other squad used seconds. The Execution Engine dashboard showed a queue processing latency of "4200" — the on-call engineer read this as 4200 seconds (70 minutes) and concluded there was a catastrophic queue processing regression. She began escalating to the Execution Engine squad lead and drafting a severity-1 incident notification. Twenty-three minutes into the escalation, the Execution Engine squad lead — looking at the same dashboard from his own perspective — clarified that the value was 4200 milliseconds, not seconds — 4.2 seconds, which was within the normal range for the current queue depth. The actual incident cause was a configuration change to the Execution Engine's maximum concurrent job limit that had been deployed 4 hours earlier and reduced the limit from 50 to 15, causing jobs to queue rather than execute immediately. The queue depth metric — which would have shown 3× the normal job backlog immediately after the configuration change — was named queue_depth_gauge in the Execution Engine's metrics, a non-standard convention that the on-call engineer had not thought to query during the investigation because every other service used queue_depth or jobs_queued. The incident was resolved 6.3 hours after the first customer report.

The post-incident retrospective identified the instrumentation incompatibilities as contributing factors in the investigation delay. The engineering team agreed on three immediate standards: UTC-only timestamps for all log output, W3C traceparent format for all distributed trace propagation, and milliseconds as the unit for all duration metrics across all services. Adopting the standards did not require replacing any tracing library — the vendor-specific library used by Resource Planner and Execution Engine supported both B3 and W3C formats; switching the propagation format required a one-line configuration change. The log timestamp standard required 2-3 hours of migration work per service. The metric unit standard required a migration for the Execution Engine and Cost Estimator services, which had both used milliseconds for durations. The standards were documented and added to the new-service engineering checklist. The federated ownership model was retained, with the addition of a schema contract document specifying the three standards and the onboarding checklist for new services. The cross-service incident MTTR for the next 12 months was 41 minutes — still above the within-service MTTR of 7 minutes (because cross-service incidents inherently require coordination across squads) but no longer inflated by schema incompatibility overhead.

A 48-person developer tooling company built a code quality platform — automated static analysis, dependency vulnerability scanning, test coverage enforcement, and license compliance checking — distributed as a CLI tool and a CI/CD integration. The engineering team of 22 engineers had been operating for 3.5 years. The founding team of 5 engineers had all come from infrastructure and SRE backgrounds and had built the original 3 services — Analysis Engine, Scanner Service, and Report Generator — with instrumentation standards that reflected their shared professional background. Every service emitted structured JSON logs with trace IDs, correlation IDs, and request IDs on every line. Every service emitted custom metrics for the operations that mattered: per-analysis-type latency, vulnerability scan throughput, report generation queue depth. Distributed tracing using OpenTelemetry was set up from day 1 and covered all inbound and outbound operations. The founding engineers did not write down an instrumentation standard; they reviewed each other's code and the standard was transmitted through code review feedback and informal expectation.

The team grew from 5 to 22 engineers over 18 months. The hiring was primarily in the product engineering organization — the company's growth was in the product surface area, and the new engineers were predominantly mid-level and senior product engineers from companies that had varying observability cultures. The founding engineers moved into engineering management and technical leadership roles; three had left the company by month 30. The informal observability standard lived in the shared knowledge of the founding cohort and in the codebase of the original three services. New engineers onboarding into the codebase would see the observability patterns in the existing services and might adopt them — or might not, depending on whether their code reviewer pointed it out and whether the reviewer had a strong enough opinion to require it. Connect this pattern to the runbook quality decision record: runbook quality and observability coverage are both subject to the same cultural transmission failure mode; both depend on institutional knowledge held by senior engineers who may not be available when the runbook is needed or the service is investigated; both decay when the institutional knowledge is not formalized into a written standard with a verification mechanism; the observability coverage failure and the runbook quality failure often co-occur in the same service, because the engineer who skipped instrumentation also skipped writing the runbook, and the engineer who wrote neither was not required to by any formal process.

Over the 18-month growth period, the engineering team shipped a new microservice cluster: License Validator, Dependency Graph Builder, Policy Enforcement Engine, Compliance Report Generator, and an Integration Orchestrator. These 5 services were built primarily by engineers hired in the 12-24 month window — engineers who had seen the original 3 services' observability and had varying responses to it. Three of the 5 new services had been built by engineers who instrumentated them thoroughly, following the patterns in the original services. Two had been built by engineers who did not prioritize observability during initial development and had not received feedback in code review requiring them to instrument the services to the founding standard. The gap was not visible because there was no coverage registry, no deployment gate for instrumentation, and no periodic coverage audit. The two underinstrumented services were in production, handling real traffic, and returning correct results — there had been no production incidents in either service that required investigation.

At month 31, a billing anomaly was reported by an enterprise customer. The customer's monthly invoice showed 47 dependency vulnerability scans billed at the Pro tier rate, but the customer's records showed only 12 scans initiated during the billing period. A 35-scan discrepancy at $0.40 per scan ($14 overcharge) triggered the report through the customer success team. The billing investigation required tracing from the billing event records (where the 47 scans appeared) backward through the License Validator and Dependency Graph Builder services (the services in the processing path for dependency scans) to identify the point at which the count diverged from the customer's 12 initiated requests. The Dependency Graph Builder and License Validator were two of the 5 new services. The Dependency Graph Builder had no distributed tracing configured — it emitted unstructured print-style logs with no request IDs, correlation IDs, or trace IDs. The License Validator had distributed tracing configured but no custom metrics for the per-scan counting operations — only framework-level HTTP request counts, which aggregated all scan types together and could not distinguish between scan types or account for scan retries. There was no way to trace a specific customer's scan requests through the Dependency Graph Builder using the existing telemetry. The investigation required a forensic reconstruction approach: a staging environment was set up, the customer's API requests were replayed with added instrumentation in the Dependency Graph Builder, and the output was compared to the billing records. The forensic reconstruction required 3 engineer-weeks of effort. The root cause — a retry logic bug in the Dependency Graph Builder that was retrying failed scans with a new billing event rather than reusing the original event's billing context — was found on day 11 of the investigation.

The company's post-incident process added two requirements to the engineering onboarding checklist: all new services must add distributed tracing with W3C traceparent propagation before the first production deployment, and all services that participate in billable operations must emit per-operation custom metrics with sufficient cardinality to trace individual customer requests (capped at a specified cardinality limit to prevent metric explosion). Both requirements were added to the code review checklist used during service creation. The instrumentation standard — which had previously existed only in the institutional knowledge of the founding engineers — was documented for the first time as a written specification that new engineers could read, reference, and be held to in code review. The three well-instrumented services in the new cluster needed no changes. The two underinstrumented services, plus a review of the 3 original services to verify they still met the standard, required approximately 80 engineer-hours of work over 3 weeks.

Structural properties set by the observability ownership decision

Three structural properties are determined when an engineering organization decides — or fails to explicitly decide — who owns instrumentation, what the minimum standard is, and how compliance is verified. The properties are not visible in the on-call dashboard or the service uptime metric. They are established by whether the ownership model creates a gap between service deployment and service instrumentation, whether independently-owned observability stacks remain interoperable enough to support cross-service incident investigation, and whether the instrumentation standard is formal enough to survive the departure of the engineers who originally held it.

Property 1: The monitoring coverage ceiling. Every observability ownership model creates a monitoring coverage ceiling — a maximum fraction of the production service portfolio that will have adequate instrumentation under the model given the constraints of who owns the instrumentation work and what mechanism ensures it gets done. For a centralized platform team model, the coverage ceiling is determined by the ratio of the platform team's instrumentation throughput to the rate at which new services enter production: if the team can instrument 2 services per engineer per month and the service portfolio is growing by 3 services per month with a 3-engineer team, the uninstrumented backlog grows by 3 services per month regardless of how efficiently the platform team operates. For a federated service-team model, the coverage ceiling is determined by the consistency with which service teams prioritize instrumentation against other competing development work — feature deadlines, bug backlogs, technical debt — and the coverage ceiling is not a function of throughput but of organizational culture and enforcement. For a cultural transmission model, the coverage ceiling decays over time as the cultural transmission path becomes less reliable with team growth and turnover. The common factor across all three models is that the coverage ceiling is invisible unless it is measured — there is no self-reporting mechanism in the production environment that distinguishes "services with adequate instrumentation" from "services with inadequate instrumentation"; the gap exists in the space between the set of services in production and the set of services with active observability coverage, and that gap is invisible without a service registry that tracks both sets explicitly. Connect this property to the incident response playbook decision record: the incident response playbook specifies the investigation procedure for production incidents; the observability ownership decision specifies whether the data required to execute that procedure exists; an incident response playbook that specifies "check the service's latency dashboard and error rate metrics for the failing component" is only executable for the fraction of services that have latency dashboards and error rate metrics; for uninstrumented services, the playbook's investigation steps are not executable and the on-call engineer must fall back to reading raw logs or adding instrumentation during the incident, which is the equivalent of installing smoke detectors while the fire is burning.

Property 2: The cross-team debugging friction coefficient. Every federated observability model — one where multiple teams make independent instrumentation choices — produces a cross-team debugging friction coefficient: the time overhead per service boundary crossed during incident investigation that results from schema incompatibilities, format mismatches, and missing interoperability standards. The friction coefficient is determined by the number of independent choices made about trace context propagation format (W3C vs. B3 vs. custom), log timestamp format (UTC vs. local timezone, seconds vs. milliseconds, ISO 8601 vs. epoch), metric duration unit (seconds vs. milliseconds vs. microseconds), and metric naming conventions (camelCase vs. snake_case, service name prefix vs. no prefix, operation name included vs. excluded). Each incompatibility adds time to incident investigation: time spent identifying that an incompatibility exists, time spent converting between formats, time spent verifying that the converted data is correctly correlated. The friction coefficient compounds with the number of services in the incident's call path — an incident spanning 4 services with 3 incompatibility boundaries takes longer to investigate than an incident spanning 4 services with no incompatibility boundaries by the sum of the overhead at each boundary. The friction coefficient can be reduced to near zero without centralizing ownership by adopting a shared schema contract — a document specifying the minimum set of cross-service conventions that all services must adopt — and enforcing the contract at deployment time or through code review. The schema contract does not require all services to use the same tracing library or the same metrics library; it requires only the conventions that affect cross-service telemetry correlation: trace context propagation format, timestamp format, and metric unit convention for durations. Connect this property to the on-call rotation design decision record: the on-call rotation design specifies who is responsible for investigating a production incident; the observability ownership decision specifies whether that engineer will be able to trace the incident across service boundaries without escalating to additional squads; a rotation model that pages the on-call engineer and expects them to diagnose the root cause independently is undermined when cross-service trace propagation is broken and the on-call engineer must escalate to 3 squad leads to correlate telemetry that a working distributed trace would have surfaced in minutes; the on-call rotation's MTTR is a function of both the rotation design and the observability interoperability, and improving MTTR requires understanding which of the two is the bottleneck in the specific incident type that is driving the MTTR up.

Property 3: The coverage decay rate. Every observability model that relies on informal standards or cultural transmission decays over time as the population of engineers who hold the standard turns over and new engineers who do not hold the standard build new services. The decay rate is a function of the ratio of engineers who know the standard to engineers who do not, the effectiveness of the mechanism by which the standard is transmitted (code review, pairing, documentation, automated enforcement), and the priority that code reviewers assign to observability compliance relative to other review criteria. The decay is invisible because there is no coverage metric that tracks compliance over time — each new service's instrumentation status is determined at the time it is built and is not periodically re-evaluated unless an incident exposes it. The decay becomes visible only when an incident occurs in a recently-built service and the investigating engineer discovers that the required data was never captured; at that point, the decay has been accumulating since the service was deployed, but the specific accumulation is only revealed at the incident. The coverage decay rate can be reduced to near zero by converting the instrumentation standard from cultural knowledge to a deployment gate — a check in the deployment pipeline that verifies the minimum instrumentation requirements before a service can receive production traffic. A deployment gate does not depend on the transmission effectiveness of cultural knowledge or the consistency of code reviewer enforcement; it converts observability compliance from a soft expectation to a hard requirement with a mechanical enforcement path. Connect this property to the logging strategy decision record: the logging strategy decision specifies the format, structure, and content of application logs; the observability ownership decision specifies who is responsible for ensuring those logging standards are applied to each production service; a logging strategy that specifies structured JSON logs with trace IDs and correlation IDs on every line is only realized when the ownership model includes a verification mechanism that checks compliance before or shortly after deployment; the logging strategy and the observability ownership decision must be written together, because the logging strategy is the specification of what good looks like and the observability ownership decision is the specification of how that standard is enforced — without the enforcement mechanism, the logging strategy is an aspiration rather than a requirement.

The service observability ownership ADR: five sections

Section 1: The ownership boundary specification with coverage accountability. Begin the service observability ownership decision record by specifying, for each production service, which team or role is responsible for ensuring the service meets the minimum instrumentation standard. The ownership specification has three components. First, the ownership model: the choice between centralized platform team ownership, federated service-team ownership, a hybrid model (platform team owns the shared infrastructure and schema contract; service teams own per-service instrumentation within the contract), or individual engineer ownership. Document the rationale for the chosen model and explicitly state what the model does not solve — a centralized model solves schema consistency but creates a coverage throughput bottleneck; a federated model solves coverage throughput but creates a schema interoperability risk; both must be stated. Second, the coverage accountability mechanism: the specific process by which the gap between services in production and services with adequate instrumentation is measured and surfaced. At minimum, this is a service registry with instrumentation status fields (instrumented, partially-instrumented, uninstrumented) that is maintained as part of the service deployment lifecycle, not as a periodic audit. The registry does not need to be elaborate — a spreadsheet or a Notion database with one row per service and a status column is sufficient to make the coverage gap visible. Third, the deployment-to-instrumentation deadline: the maximum time between a service's first production deployment and the service achieving "instrumented" status in the registry. For a centralized model, this deadline bounds the instrumentation backlog; for a federated model, it specifies when the service team must complete their instrumentation work. The deadline converts the instrumentation requirement from "eventually" to a specific obligation with a due date that can be tracked and reported. Connect this section to the incident response playbook decision record: the incident response playbook's investigation procedure should reference the service registry as the authoritative source for identifying the owning team and the instrumentation status of services in the incident's call path; when an incident occurs in an uninstrumented service, the playbook should specify the fallback investigation path for uninstrumented services explicitly, not leave the on-call engineer to discover mid-incident that the expected dashboards do not exist.

Section 2: The minimum observability standard with verifiable specifications. Specify the minimum instrumentation requirements that every production service must meet before it can be added to the on-call alert routing. The minimum standard must be stated in verifiable terms — specifications that can be checked mechanically, not qualitative descriptions. Five required categories. First, latency measurement: specify the required latency metrics by name (the naming convention), unit (seconds or milliseconds, not "the appropriate unit"), and percentile coverage (p50, p95, p99 minimum, with p999 recommended for services with SLA commitments). The metric names must be specified in the decision record so that they can be queried in a coverage check. Second, error rate measurement: specify the required error rate metrics by name and the label dimensions required (at minimum, an error class label distinguishing user errors from system errors from dependency errors), with a cardinality budget (maximum number of distinct label value combinations) to prevent metric cardinality explosion. Third, distributed trace propagation: specify the required trace context propagation format (W3C traceparent is recommended for new deployments), the required span coverage (all inbound HTTP/gRPC operations and all outbound calls to external services), and the requirement that trace IDs and span IDs appear in all structured log output for correlating logs with traces. Fourth, structured logging: specify the required log format (JSON or the organization's standard), the required fields (timestamp in UTC with millisecond precision, log level, service name, trace ID, span ID, request ID), and the requirement that all log output use the structured format rather than mixing structured and unstructured output. Fifth, health endpoints: specify the required health check interface — the endpoint path, the required response schema for healthy and unhealthy states, and whether the health check must include dependency health or only the service's own health. The minimum standard document becomes the reference for the deployment gate check and the code review checklist. Connect this section to the logging strategy decision record: the logging strategy decision record and the observability minimum standard overlap in their coverage of structured log format and required log fields; if both documents exist, they should reference each other explicitly and the logging standard in the observability minimum standard should be a reference to the logging strategy decision record rather than a re-statement, to avoid the documents diverging over time as each is updated independently.

Section 3: The schema contract with adoption and drift management. Specify the cross-service observability conventions that all services must adopt to ensure cross-service telemetry interoperability. The schema contract is distinct from the minimum observability standard: the standard specifies what each service must emit; the schema contract specifies the conventions that must be consistent across all services to enable cross-service correlation. Four required conventions. First, trace context propagation format: specify a single format (W3C traceparent recommended for new greenfield deployments; B3 if the existing service portfolio already uses B3 uniformly) and specify the requirement that all services must accept and propagate incoming trace context headers in this format regardless of which tracing library they use. Document the migration path for services that currently use a different format — whether the migration is required immediately, by a specified deadline, or only for new services. Second, log timestamp format: specify UTC with millisecond precision in ISO 8601 format as the required timestamp format for all log output, with no exceptions for local timezone. The local-timezone exception is the most common source of timestamp correlation failures and the most easily avoided with a single line in the schema contract. Third, metric duration unit: specify seconds or milliseconds (seconds is preferred because it aligns with Prometheus conventions, but milliseconds is acceptable if the existing portfolio uses it uniformly) and specify the requirement that the unit be consistent across all services. Include in the contract a requirement that the metric name includes the unit as a suffix (latency_seconds or latency_ms, not latency) so that the unit is explicit to engineers reading the metric name rather than requiring knowledge of each service's convention. Fourth, service name convention in telemetry: specify how the service name is included in metric names, log fields, and trace span names — as a prefix (service_name.operation_name.metric_name), a label (service="service_name"), or both — and require this convention to be consistent across all services so that filtering telemetry by service in incident investigation uses the same query syntax for all services. The drift management mechanism is as important as the initial contract: specify the process by which the schema contract is updated (who has authority to propose changes, how changes are communicated to service teams, and the migration window for existing services to adopt contract changes) and the process by which contract violations in existing services are identified and remediated.

Section 4: The cross-team incident investigation protocol. Specify the process for investigating production incidents that span service boundaries and require telemetry correlation across teams. The cross-team investigation protocol has four components. First, the escalation path: for an incident involving services owned by multiple teams, specify the process for establishing a joint investigation — who pages which squad leads, at what severity threshold joint investigation is required rather than sequential handoffs between squads, and what the expected response time is for squad leads joining a cross-service incident during and outside business hours. The escalation model for cross-service incidents is the responsibility of the on-call rotation design; the observability ownership decision record should reference the on-call rotation decision record's escalation section explicitly and add the observability-specific expectation that each squad lead joining the investigation has access to their service's dashboards and can correlate their service's telemetry with the shared incident timeline within 10 minutes of joining the investigation. Second, the shared incident timeline standard: specify the format in which each squad documents their service's telemetry findings during a cross-service incident — the timestamps in UTC, the metric and log queries used, and the conclusions drawn — so that findings from multiple squads can be correlated into a shared timeline without time zone conversion or unit reconciliation overhead. Third, the instrumentation gap filing process: specify the process by which instrumentation gaps discovered during incident investigation are filed as work items with a priority and an owner — which team owns the gap, which backlog the work item goes into, what the expected resolution timeline is (the same deployment-to-instrumentation deadline specified in section 1), and who verifies closure. Fourth, the post-incident coverage audit: for incidents where an uninstrumented service was in the call path, specify that the post-incident review must include an audit of the current coverage status for all services in the incident's call path, not just the uninstrumented service that was identified during investigation. Connect this section to the incident response playbook decision record: the incident response playbook and the cross-team investigation protocol are two documents that must be written in reference to each other; the incident response playbook specifies the investigation procedure from the on-call engineer's perspective; the cross-team investigation protocol specifies the coordination model when the investigation requires multiple teams; both documents should reference each other at the point where their scope overlaps — the escalation section of the playbook should link to the cross-team investigation protocol, and the escalation path in the cross-team investigation protocol should reference the playbook's severity classification system.

Section 5: The coverage verification mechanism and review cadence. Specify how instrumentation coverage compliance is verified at deployment time and monitored over time, and the cadence at which the coverage status is reviewed and acted on. The coverage verification mechanism has two components. First, the deployment-time check: the specific check added to the deployment pipeline that verifies the minimum observability standard before a service can be deployed to production or added to the on-call alert routing. The check should be mechanical — a query against the metrics backend confirming that the required metric names are being emitted, a query against the tracing backend confirming that spans are being recorded, a query against the logging infrastructure confirming that structured logs with required fields are being emitted — rather than a manual checklist sign-off that depends on human attention during each deployment. The mechanical check converts observability compliance from a soft expectation to a hard deployment requirement. Second, the ongoing coverage monitoring: the mechanism by which coverage status is tracked after deployment and surfaced to engineering leadership. The service registry with instrumentation status fields (described in section 1) is the data source; the mechanism that makes it actionable is a periodic report — weekly is the recommended cadence — that calculates the current coverage percentage, shows the trend over the prior 4 weeks, and lists the services below the minimum standard with the owner, the deployment date, and the days since deployment. The coverage review cadence is the specification for when engineering leadership reviews the coverage report and what action is triggered when coverage falls below a specified threshold (90% is a reasonable minimum standard for teams with more than 10 services). Connect this section to the incident postmortem culture decision record: postmortem findings about missing telemetry — "the service had no distributed tracing configured," "the error rate metric did not distinguish between user errors and dependency errors" — are the primary signal that the coverage verification mechanism has a gap; postmortem culture that uses systems attribution for these findings will produce action items aimed at the coverage mechanism (add the service to the deployment gate coverage check; update the minimum standard to include the missing signal) rather than individual attribution findings that assign blame to the engineer who built the uninstrumented service; the coverage verification mechanism's review cadence should include a review of postmortem telemetry gap findings from the prior period to identify patterns in coverage gaps that indicate systematic failures in the minimum standard specification or the deployment gate check.

FAQ

Further reading

• Distributed tracing decision record — the distributed tracing decision specifies the tracing library, the trace context propagation format, and the span coverage requirements for the organization; it is the technical foundation of the cross-service investigation capability that the observability ownership decision either enables or undermines; when the distributed tracing decision specifies W3C traceparent as the propagation format but the observability ownership model allows each service team to choose its own tracing library without verifying that it supports W3C propagation, the result is trace graph fragmentation at the incompatibility boundary; the distributed tracing decision record should be written in reference to the observability ownership decision record, explicitly specifying that the chosen propagation format must be adopted by all services and that the schema contract enforcement mechanism is the mechanism by which compliance is verified.
• Alerting threshold decision record — the alerting threshold decision specifies which signals trigger alerts and at what sensitivity thresholds; it is only executable for signals that exist; an alerting strategy built on the assumption that all production services emit error rate metrics and latency percentiles is only as comprehensive as the fraction of services that actually meet the minimum observability standard; the coverage gap and the alerting gap are the same gap seen from two different angles — the services that are uninstrumented are also the services with no alerts, and the alerting threshold tuning work done for the instrumented portion of the portfolio creates a false sense of comprehensive alert coverage when 30 percent of services are producing no alertable signals at all; the observability ownership decision record and the alerting threshold decision record must cross-reference each other to make explicit that the alerting strategy applies to the instrumented service population and that the coverage percentage is the scaling factor for its actual effectiveness.
• Runbook quality decision record — runbook quality and observability coverage are both subject to the same cultural transmission failure; both depend on institutional knowledge transmitted through code review and informal mentorship; both decay as the senior engineers who hold the standard move into management or leave the company; both are invisible in their decay — the runbook quality failure and the observability coverage failure are only discovered when an incident occurs in the relevant service and the on-call engineer discovers mid-incident that the expected resource does not exist; the runbook quality decision record and the observability ownership decision record should specify complementary enforcement mechanisms — a deployment gate for instrumentation compliance and a deployment gate for runbook existence — so that new services cannot enter the on-call rotation without both the observability data required to diagnose incidents and the runbooks required to guide the investigation.
• Incident postmortem culture decision record — postmortem findings about missing telemetry are the most reliable signal that the observability coverage mechanism has a gap; a postmortem culture that uses systems attribution will produce action items targeting the coverage mechanism — add the service to the deployment gate, update the minimum standard to require the missing signal — rather than individual attribution findings that assign blame to the engineer who built the uninstrumented service; the observability coverage review cadence should include a review of postmortem telemetry gap findings from the prior period, because the pattern of which services and which signal types appear in postmortem findings reveals where the minimum standard is underspecified or where the deployment gate is failing to catch non-compliant services; systems attribution for telemetry gap findings and a formal postmortem review cycle are the two mechanisms that drive continuous improvement in observability coverage without requiring a manual audit of every service on a fixed schedule.
• Open-source extractor — the observability ownership decision is one of the most consistently buried decisions in AI chat history; the founding engineers who chose the original ownership model, debated the tradeoffs between centralized and federated ownership, and set the initial minimum standard almost certainly had those conversations in ChatGPT or Claude — the architecture session where the VP of Engineering and the platform lead worked through the pros and cons of each model, the review of the distributed tracing library options, the discussion of the schema contract requirements — because these are exactly the kind of complex, open-ended design decisions that benefit from a thinking partner; the result is that the rationale behind the ownership model is buried in AI chat exports while the architecture document records only the conclusion; when a new engineer joins the platform team and asks why the company chose federated ownership with a schema contract rather than centralized ownership, or why W3C traceparent was chosen over B3, the answer is in the original decision conversation and is unretrievable from the architecture document alone; the extractor surfaces those buried decisions from the AI chat export, making the observability ownership rationale findable by the engineer who inherits the model and must decide whether to change it.