2026-09-24 · ~21 min read · Decision records

The observability cost governance decision record: why the telemetry cardinality model you chose determines your metric series explosion surface and your incident retrospective coverage gap

The telemetry cardinality governance model, the signal retention tier specification, and the cost anomaly detection policy are decisions that are almost never made explicitly — they emerge from the first observability SDK added to a service, the default retention settings accepted without asking what incident retrospective coverage requires, and the implicit assumption that cloud cost alerting will catch observability spend anomalies alongside compute. Three failure patterns: the engineering team whose Prometheus-to-Datadog migration produced a cardinality explosion because no governance policy constrained which label dimensions engineers could add; the platform team whose retention defaults created a 35-day coverage gap that made a customer performance regression undiagnosable; and the data engineering team whose logging misconfiguration ran for 18 days before anyone noticed because no cost anomaly detection existed for the observability cost center.

A 32-person B2B SaaS company built a workflow automation platform for operations teams — process templating, approval routing, SLA monitoring, and integration with Salesforce and ServiceNow for enterprise accounts. The company had grown from nine to thirty-two people over three years. The engineering team of fourteen had instrumented their services with Prometheus from early on, running a self-hosted Prometheus stack on a dedicated EC2 instance in their AWS account. By the beginning of year three, the Prometheus stack was showing signs of capacity strain — the engineering team's oncall rotation had fielded three incidents in four months where the Prometheus instance had run out of disk or memory, causing a gap in metric collection and generating false-positive alerts. After the third incident, the platform team evaluated alternatives and decided to migrate to Datadog. The migration decision was made over two weeks of evaluation, and the rationale was documented in a brief ADR: Datadog's hosted model eliminated the Prometheus instance capacity management problem, the Datadog agent was easier to configure than Prometheus exporters for a team without dedicated SRE expertise, and the annual cost at current metric volume was estimated at approximately $8,000 — manageable for an engineering budget at the company's stage.

The migration itself proceeded cleanly over six weeks. Each service team configured the Datadog agent, converted their existing Prometheus metrics to Datadog metrics, and added the instrumentation code that the Datadog libraries encouraged. The existing Prometheus metrics used a disciplined set of label dimensions — service name, environment, endpoint path, and HTTP status code class. These produced a known and manageable series count. As service teams configured Datadog, several engineers added additional labels that the Datadog documentation highlighted as useful for filtering and correlation in the Datadog UI: user_id on user-facing API latency metrics, request_id on internal service call metrics, and customer_id on tenant-specific processing time metrics. Each of these additions seemed individually useful — user-level latency percentiles would help identify per-user performance degradation, request-level correlation would make distributed trace queries more actionable, and customer-level processing times would help diagnose tenant-specific slowdowns.

At the end of month one on Datadog, the platform team received an estimated usage notification from Datadog flagging that the account's metric series count was tracking toward the next pricing tier. The platform team reviewed the notification and dismissed it, assuming the series count would stabilize as the migration finished and the usage patterns settled. At the end of month two, the Datadog bill arrived: $31,400 for the month, nearly four times the estimated annual cost divided by twelve. The platform team pulled the Datadog usage dashboard and found 4.8 million active metric series, compared to the approximately 12,000 series the Prometheus stack had produced for the same service fleet. The series explosion was traced to the three high-cardinality labels: the user_id label produced one series per active user per metric per endpoint per environment — at 40,000 monthly active users across sixteen instrumented endpoints, this label alone produced 640,000 series per metric. The request_id label produced one series per request for the services that had adopted it — at the company's request volume, this was approximately 3.2 million unique series per day, cycling through as requests aged out of Datadog's retention window and new requests created new series. The customer_id label produced a smaller but still significant number of series for the enterprise accounts.

The root cause was structural: the cardinality governance model had not been specified as part of the Datadog migration decision record, and the migration ADR's cost estimate had been based on the existing Prometheus label dimensions without examining whether the migration process would introduce new dimensions. The Datadog agent libraries and documentation provided examples showing high-cardinality labels as useful filtering mechanisms, and engineers implementing the new instrumentation had followed those examples without a governance constraint that defined which dimensions were appropriate for metrics versus traces and logs. The three engineers who added the high-cardinality labels had each made a locally reasonable decision — user-level filtering in metrics was genuinely useful for debugging performance issues — without a policy that explained why the cost of that usefulness at the observed user volume was unjustifiable relative to achieving the same filtering capability through trace-level context rather than metric-level dimensions. The engineering team spent three weeks removing the high-cardinality labels, updating their debugging workflows to rely on trace correlation rather than metric filtering for per-entity queries, and implementing a CI lint step that checked new metric declarations against an allowed dimension list. The remediated monthly bill stabilized at $9,200 — close to the original estimate. The three months of excess spend totaled approximately $66,000.

A 41-person platform engineering SaaS company built infrastructure tooling for mid-market engineering teams — managed Kubernetes environments, secret rotation automation, certificate lifecycle management, and a developer CLI that abstracted away cloud provider-specific configuration. The company had been running for four years and had an engineering team of twenty across three product squads and a two-person SRE function. The SRE team was responsible for the company's own observability infrastructure: a Datadog account for metrics, a Grafana Cloud stack for logs and traces, and a PagerDuty account for alerting. When the SRE team had set up the Grafana Cloud stack in year two, they had accepted the default retention settings for each signal type: logs retained for 30 days, traces retained for 15 days. These defaults had been accepted because they were the tier the company was paying for, and the assumption was that 30-day log retention and 15-day trace retention were sufficient for the incidents the company had experienced to date — outages and configuration failures that were typically identified and diagnosed within hours to days of occurrence.

In the company's fourth year, a large enterprise customer — a 200-person financial services SaaS using the company's managed Kubernetes product — opened a support ticket reporting intermittent latency spikes in their deployment pipeline. The reported symptom was that the kubectl apply calls their CI system made against their managed cluster occasionally took three to seven times longer than baseline, causing deployment timeouts. The support team asked for examples and the customer provided two recent occurrences — one from the previous day and one from nine days ago. The SRE team pulled the traces for the nine-day-old occurrence and found them available. They confirmed an anomalous latency spike in the cluster's API server response times, correlated with a specific control plane operation. The SRE team began investigating the cluster's configuration history to find when the anomalous latency had first appeared.

The customer's follow-up report, submitted five days later, described the issue as having started "approximately five to six weeks ago" based on their internal deployment time series data, which the customer maintained in their own Grafana instance using webhook events from the company's platform. A latency spike visible in their internal data corresponded to a timestamp 37 days before the support ticket was opened. The SRE team pulled traces for the 37-day-old timestamp. The Grafana Cloud retention window was 15 days. The traces for that date no longer existed. The SRE team pulled logs for the same timestamp. The Grafana Cloud log retention was 30 days. The logs for that date no longer existed either. The team had seven days of traces and no logs for the period when the anomaly had first appeared. The SRE team's investigation was limited to what could be inferred from the customer's external deployment time series data and from the patterns in the current trace data, which showed the anomalous behavior still active but had no historical baseline context to identify what configuration or deployment event had triggered the initial degradation.

The investigation took three weeks, compared to the SRE team's estimate of two to three days if the original trace and log data had been available. The root cause was eventually identified as an interaction between the cluster's API server audit log configuration and a specific control plane operation's retry behavior — a configuration that had been in the company's cluster template since year three and whose interaction with the customer's workload pattern had only surfaced once the customer's deployment frequency exceeded a threshold that had not been present when the configuration was written. The fix was a two-line configuration change. The three-week investigation delay had been driven entirely by the absence of the historical signals that would have immediately shown when the problematic configuration state first appeared. The SRE team's post-incident review updated the Grafana Cloud retention configuration to 90 days for logs and 30 days for traces. The monthly cost increase was $1,100 — an increase the team's retrospective noted could have been adopted in year two for less than the cost of the extended three-week investigation, which had consumed approximately 120 engineer-hours across the SRE team and the support team.

A 47-person developer tooling SaaS company built a continuous integration and deployment analytics platform — pipeline performance profiling, build time regression detection, and a test failure correlation tool that surfaced patterns across CI failures for enterprise engineering teams. The company had grown to forty-seven people over three years and had an engineering team of twenty-one across four product squads and a small data engineering team of three. The company's observability stack was Datadog for application performance monitoring, with instrumentation in the Python backend services and the Go data pipeline services. The monthly Datadog bill had been stable at approximately $8,500 for the six months prior to the incident, with minor variation from month to month as the service fleet grew. No one person was designated as the budget owner for the Datadog account, and the bill was reviewed monthly by the head of engineering as part of the overall infrastructure cost review during the finance team's monthly reporting cycle.

In the fourth week of the company's Q3, the data engineering team deployed a new pipeline service — a log aggregation preprocessor that consumed raw CI log streams from customer integrations and performed normalization before the records entered the main data processing pipeline. The service was new code, written over four weeks, and had not been deployed to production before. In the development environment, the service had been configured with LOG_LEVEL=DEBUG to support debugging during development. Before the production deployment, the deployment configuration had been reviewed for environment variables and secrets, but the review had not caught the debug logging configuration because the deployment template for new services populated environment variables from a development defaults file, and the LOG_LEVEL=DEBUG value from the development defaults had been carried forward without being overridden for production. The service deployed to production emitting debug logs at a rate of approximately 8,000 log lines per second per instance, compared to the expected 200 to 400 log lines per second at INFO level for a service processing the same input volume. The service ran three instances. At 24,000 log lines per second across three instances, the Datadog log ingestion cost for the new service alone was approximately $1,100 per day.

The head of engineering's monthly infrastructure cost review occurred eighteen days after the service deployment. At that review, the Datadog bill for the month-to-date showed $23,000 already consumed, compared to the $8,500 monthly baseline — a 170% cost increase. The finance team flagged the variance as requiring explanation. The head of engineering queried the Datadog usage dashboard and found that log ingestion volume had increased by a factor of 12 on the day the new pipeline service had been deployed. The root cause was identified within two hours of the finance team's flag. The fix — changing LOG_LEVEL=DEBUG to LOG_LEVEL=INFO in the production deployment configuration and redeploying — took thirty minutes. The total excess spend for the eighteen-day period was approximately $20,400. The remediation was straightforward. The detection was the problem.

The post-incident analysis identified three structural gaps. First, there was no cost attribution model that mapped Datadog costs to specific services or teams — the monthly bill was a single aggregate number, and identifying which service was responsible for a cost spike required manually querying the Datadog usage API rather than reading a cost allocation dashboard. Second, there was no cost anomaly alert configured in the Datadog account — the usage metrics API provided daily estimated spend, but no alert had been configured to fire when daily spend exceeded a threshold. Third, there was no deployment checklist requirement to verify that production-bound services were configured with production-appropriate log levels — the development defaults file was the source for new service environment configurations, and there was no CI step that validated that LOG_LEVEL was not set to DEBUG in production deployment artifacts. The eighteen-day detection window was entirely attributable to the absence of cost attribution and anomaly detection. The head of engineering's next monthly review would have caught the issue even without these controls; the eighteen-day wait for the monthly review was the cost governance gap. A daily cost attribution query with an anomaly alert would have detected the spike within 24 hours of deployment and reduced the excess spend from $20,400 to approximately $1,100.

Structural properties set by the observability cost governance decision

Three structural properties are determined when a team builds — or fails to specify — an observability cost governance decision record: what the cardinality governance model determines about the metric series explosion surface as engineers instrument new services with high-cardinality dimensions, what the retention tier specification determines about the incident retrospective coverage gap as customer-reported issues surface outside the retention window, and what the cost anomaly detection policy determines about the spend runaway surface as misconfigured services emit telemetry at volumes that only become visible at the next billing review. None of these are labeled as decisions when the team signs up for Datadog, accepts the default retention settings, and assumes cloud cost alerting covers observability spend. They emerge as operational failures when a Prometheus-to-Datadog migration produces a $31,000 monthly bill instead of the $700 estimated because user_id and request_id were added as metric labels without a governance policy, when a customer regression investigation is extended by three weeks because the trace and log retention window expired 7 days before the investigation began, and when a logging misconfiguration runs for 18 days consuming $20,000 in excess observability spend because the detection mechanism was the monthly finance review rather than a same-day engineering alert.

Property 1: The telemetry cardinality governance model and the metric series explosion surface. Observability platforms that price by metric series count (Datadog, Grafana Cloud's metrics tier, New Relic) have a cost structure that scales multiplicatively with label cardinality: one series per unique combination of label values per metric per service. A metric with three low-cardinality labels — service (five values), environment (three values), status code class (five values) — produces 75 series. The same metric with an additional user_id label at 40,000 active users produces 3,000,000 series. The cost difference between these two configurations at typical hosted metrics pricing is approximately $2,850 per month per metric, and the series explosion is proportional to user growth — doubling users doubles the series count and the corresponding cost. The cardinality governance model that prevents this explosion is not a prohibition on useful high-cardinality analysis — it is a policy that routes high-cardinality analysis to the signal type designed for it. Traces are the appropriate signal for user-level, request-level, and session-level correlation: a trace carries the user_id, request_id, and session_id as trace attributes (span tags) without producing one series per user per trace — traces are stored as individual events, not as time-series aggregations. Logs carry entity-level context as log fields for the same reason. The cardinality governance model specifies which dimensions are appropriate for metrics (bounded, enumerable, low-cardinality dimensions: service name, environment, region, endpoint path for enumerated endpoints, status code class, error type, deployment version) and which dimensions are appropriate for traces and logs (high-cardinality, per-entity identifiers: user_id, tenant_id, request_id, session_id, order_id). The enforcement mechanism is a CI lint step that validates new metric declarations against the allowed dimension list and rejects declarations that include prohibited high-cardinality dimensions. Connect this property to the observability strategy decision record: the signal type selection decisions — which scenarios are served by metrics versus traces versus logs — are the architectural context in which the cardinality governance model operates; a governance model that prohibits high-cardinality dimensions from metrics only achieves its intent if the high-cardinality analysis needs are served by traces and logs; the observability strategy must specify which use cases each signal type addresses and the cardinality governance model must be consistent with that specification.

Property 2: The retention tier specification and the incident retrospective coverage gap. Signal retention determines how far back in time an engineering team can look when investigating a problem that has been identified. The incident retrospective coverage gap is the set of investigations that cannot be completed because the signals that would answer the diagnostic questions are no longer in the retention window. The gap is driven by the lag between when an incident begins and when an engineering team has enough evidence to initiate a full investigation — a lag that, for customer-reported performance regressions, can be weeks to months. The retention tier specification closes this gap by answering, before the retention setting is configured, the question: what is the longest plausible lag between an incident beginning and an engineering team needing historical signals to diagnose it? For most production systems, this lag is driven by the slowest customer feedback loop: enterprise customers with monthly business reviews who report a performance degradation that has been present since "sometime last month" require a minimum of 30 to 45 days of signal retention to support a retrospective investigation. Compliance requirements (HIPAA audit logs, PCI cardholder environment access logs, SOC 2 audit trails) frequently require 12 months of log retention regardless of incident investigation needs. The cost of upgrading from a 15-day to a 30-day trace retention and from a 30-day to a 90-day log retention in most hosted observability platforms is 20 to 40 percent of the base observability cost — a cost increase that is typically smaller than the engineer-hours consumed by a single investigation extended by missing historical data. The retention specification must document the rationale for each retention window per signal type: not "30 days logs because that's the default" but "90 days logs because enterprise customer SLAs allow up to 60 days for performance issues to be reported, and a 30-day buffer is added for investigation initiation lag." Connect this property to the incident command system decision record: the incident detection model decisions that determine how incidents are identified — alert-triggered versus customer-reported versus proactive monitoring — directly determine the retention window requirement; an incident response process that relies on customer reports as the primary detection signal for slow-burn degradations requires significantly longer retention than a process with comprehensive SLO monitoring that catches degradations within minutes; the incident detection model and the retention tier specification should be specified together, because the detection model determines the investigation initiation lag that the retention window must cover.

Property 3: The cost anomaly detection model and the spend runaway surface. Observability costs are driven by telemetry volume — log lines ingested, metric series reported, trace spans submitted — and telemetry volume can increase suddenly from a configuration change, a new high-volume service, a debug logging flag, or a cardinality explosion. The spend runaway surface is the observability cost that accumulates between when a cost-producing anomaly begins and when it is detected and resolved. The surface is bounded below by the anomaly detection latency: an anomaly detected within 24 hours produces a spend runaway surface proportional to one day's cost at the anomalous rate; an anomaly detected at the monthly billing review produces a spend runaway surface proportional to the entire month's cost at the anomalous rate. The detection latency is entirely determined by the cost anomaly detection model: whether a cost attribution model maps costs to services and teams, whether an alert threshold triggers a same-day engineering alert for anomalous spend, and whether the engineering team has the operational tooling to query the observability platform's usage API during an investigation. The cost anomaly detection model is not a monitoring problem in the same category as application performance monitoring — it is a governance problem that requires designating a budget owner for the observability cost center and giving that owner both the attribution model (which service or team is producing which cost) and the detection mechanism (daily cost variance alert against a rolling baseline) to act on cost anomalies before they compound into billing-cycle-scale surprises. Connect this property to the cloud cost chargeback decision record: the cost attribution and chargeback model decisions for the overall infrastructure cost center determine whether observability costs are tracked separately from compute costs or aggregated together; observability cost anomalies are often masked by compute cost variance when both are tracked in the same cost center, because a 170% increase in observability cost (from $8,500 to $23,000 per month) appears as a 12% increase in total infrastructure cost when compute costs are $100,000 per month — below the threshold that triggers a cost review in many budget models; separating the observability cost center from the compute cost center, with its own baseline and anomaly threshold, makes observability cost anomalies visible at the granularity at which they can be diagnosed and resolved. The WhyChose extractor finds the observability cost governance decisions buried in your AI chat history — the Datadog evaluation session where the cost estimate was based on existing metric dimensions without examining whether the migration would introduce new ones, the infrastructure setup session where the default retention settings were accepted without evaluating the incident retrospective window, and the monthly infrastructure review session where the Datadog bill was noted as "higher than expected" without triggering a same-day investigation into which service had changed its telemetry volume.

The observability cost governance ADR: five sections

Section 1: Telemetry cardinality governance policy and prohibited dimensions. Specify the allowed and prohibited label dimensions per signal type before any observability platform migration or new service instrumentation begins. Allowed metric dimensions (must be low-cardinality and bounded): service name, environment (production/staging/canary/development — four values), AWS region or GCP region (enumerated), HTTP status code class (2xx/3xx/4xx/5xx — five values), API endpoint path (only for endpoints with enumerable paths; paths containing resource IDs must be parameterized to the route pattern before use as a dimension), error type (bounded to a defined enumeration of error categories maintained by the platform team), and deployment version (with a documented policy for pruning version label values after they are no longer deployed, to prevent unbounded series accumulation as version count grows). Prohibited metric dimensions (high-cardinality, use traces or logs instead): user_id, session_id, request_id, transaction_id, order_id, customer_id, tenant_id, account_id, and any other per-entity identifier whose unique value count scales with user, customer, or request volume. Enforcement: a CI lint step checks new metric declarations against the allowed dimension list; any metric declaration that includes a prohibited dimension fails the CI check with a specific error message explaining why the dimension must be moved to trace attributes or log fields. Governance exception process: if a team believes a prohibited dimension is justified for a specific metric, they must submit a cardinality impact assessment (estimated series count at current and projected user volume, estimated monthly cost at the observability platform's current pricing tier, alternative analysis showing why the required analysis cannot be served by traces or logs) to the platform team for review. Connect this section to the observability strategy decision record for the signal type selection decisions that determine which use cases are served by metrics versus traces versus logs — the cardinality governance model is only enforceable if the high-cardinality analysis needs that would otherwise drive engineers toward high-cardinality metrics have an alternative signal type pathway that serves the same use case.

Section 2: Signal retention tier specification per signal type. Specify the retention window for each signal type, with the rationale for each window anchored to the incident retrospective coverage requirement. For each signal type (logs, traces, metrics, events), document: the retention window selected, the longest plausible lag between an incident beginning and an engineering team needing historical signals to diagnose it (the coverage requirement), any compliance or regulatory requirement that supersedes the coverage-driven minimum, and the incremental monthly cost of the selected retention relative to the platform's default. Minimum retention windows based on incident retrospective coverage: logs at 90 days (covers enterprise customer reporting lags of up to 60 days plus 30 days for investigation initiation), traces at 30 days (covers most performance regression retrospectives; extend to 60 days if the customer base includes enterprise accounts with quarterly review cycles), metrics at full resolution for 30 days and 1-hour resolution for 13 months (covers year-over-year traffic seasonality comparisons for capacity planning). Compliance overrides: HIPAA-regulated systems require 6 years of audit log retention; PCI-regulated systems require 12 months of access log retention; SOC 2 Type II requires retention sufficient to support the audit period plus evidence of continuous monitoring. Document the cost delta between the platform's default retention and the retention required by the coverage and compliance specifications — this delta is the cost of closing the coverage gap, and it should be presented as the cost of not having historical signals for a specific incident investigation rather than as an infrastructure cost increase. Connect this section to the incident command system decision record: the incident detection model decisions determine the retention window requirement; a detection model that relies primarily on customer reports as the signal for slow-burn degradations requires the longest retention windows; a detection model with comprehensive SLO monitoring and latency percentile alerts can justify shorter retention because the detection lag is measured in minutes rather than weeks.

Section 3: Observability cost attribution model and budget ownership. Specify who owns the observability cost budget, how costs are attributed to services and teams, and what the review cadence is. Budget owner: designate a specific role (typically the platform engineering team lead or the SRE team lead) as the observability cost budget owner with responsibility for reviewing the monthly bill, investigating anomalies, and approving exceptions to the cardinality governance policy. The budget owner must have read access to the observability platform's usage API and must be the default recipient of cost anomaly alerts. Cost attribution model: configure the observability platform's usage API to report cost by service name, by team (using service-to-team mapping maintained by the platform team), and by signal type (logs vs. traces vs. metrics). Most hosted observability platforms expose cost by source tag or service tag in their usage dashboards and APIs — cost attribution is a configuration step, not a custom instrumentation requirement. Report cost attribution in a weekly cost breakdown posted to the platform team's Slack channel and in the monthly infrastructure cost review. Budget model: set a monthly observability budget per team based on the prior quarter's average cost per team, with a 20% growth allowance for teams actively adding new services. Teams that exceed their budget allocation by more than 25% in a month receive a cardinality and log volume audit in the following week. Connect this section to the infrastructure cost allocation decision record: the infrastructure cost center organization and chargeback model decisions determine whether observability is tracked as a shared platform cost or as a per-team cost; observability cost governance is most effective when cost attribution reaches service and team granularity, because cost anomalies are caused by specific services and teams and are resolved by the engineers who own those services.

Section 4: Cost anomaly detection policy and alert thresholds. Specify the anomaly detection thresholds, the alert routing, and the investigation protocol. Alert conditions: (1) daily observability spend exceeds 150% of the 7-day rolling average daily spend for any service — fires to the on-call engineer for that service; (2) total daily observability spend exceeds 130% of the monthly budget divided by 30 — fires to the budget owner and the head of engineering; (3) metric series count for any service exceeds 150% of the prior-day series count — fires to the on-call engineer and the budget owner as a potential cardinality incident. Alert routing: cost anomaly alerts must route to the on-call engineering rotation, not to a finance email list — observability cost spikes require code or configuration changes to resolve, which requires engineering availability, not financial approval. Alert timing: observability platform usage APIs provide sub-daily granularity for most cost signals (Datadog's estimated monthly cost endpoint updates every hour; Grafana Cloud's usage API provides daily metrics by source); the cost anomaly detection query should run daily at 8am UTC, comparing the prior day's cost per service against the 7-day rolling average and firing alerts for any service above threshold. Investigation protocol: when a cost anomaly alert fires, the on-call engineer queries the observability platform's usage breakdown to identify the specific signal type (logs, traces, metrics) and specific source (service name, hostname) driving the cost increase, then correlates the increase with recent deployments in the CI/CD deployment log. Most cost anomalies are caused by configuration changes deployed within the past 48 hours and are resolvable by reverting or correcting the deployment. Connect this section to the observability sampling decision record: the sampling rate decisions for traces and logs determine the baseline telemetry volume the cost anomaly detection model's rolling average is built on; a sampling rate change — either intentional (new sampling policy) or unintentional (sampler misconfiguration) — produces a cost change that the anomaly detection model must correctly classify as an intentional rate change versus an unintentional volume increase; the observability ADR should specify whether sampling rate changes require a budget owner approval step so that intentional cost increases are explicitly authorized rather than detected as anomalies.

Section 5: New service instrumentation checklist and cardinality review gate. Specify the cardinality review requirement for new service instrumentation and for migrations to new observability platforms. New service instrumentation checklist: before any new service is instrumented and deployed to production, the service's telemetry configuration must pass four checks. First, cardinality check: every custom metric defined in the service must declare its label dimensions in a metrics manifest file; the CI lint step runs against this manifest and rejects any metric with a prohibited dimension. Second, log level check: the deployment configuration must explicitly set LOG_LEVEL=INFO (or an equivalent production-appropriate level) as a required field, not as a default inherited from a development configuration; a CI check validates that no production deployment artifact contains LOG_LEVEL=DEBUG or LOG_LEVEL=TRACE. Third, cost estimate: the metrics manifest and the expected log volume (lines per second per instance at peak load) must be submitted with a cost estimate at the observability platform's current pricing, reviewed by the budget owner before the service's first production deployment. Fourth, attribution tag: the service must include the owning team's attribution tag in all telemetry (Datadog service tag, Grafana Cloud `team` label) so that cost attribution is available from the service's first deployment. Platform migration review: any migration to a new observability platform or to a new pricing tier of an existing platform must include a cardinality audit of the existing metric inventory against the new platform's pricing model; the cost estimate for the migration must reflect the actual cardinality of the existing label dimensions under the new platform's series pricing, not an estimate based on metric count alone. Connect this section to the deployment strategy decision record: the deployment process decisions — what constitutes a complete deployment, what checklist items are required before a deployment is approved for production — determine whether the instrumentation checklist is enforced as part of the deployment gate or exists as an advisory document; the cardinality check, log level check, cost estimate, and attribution tag requirement are each most effective when they are blocking conditions on the production deployment approval, not post-deployment review items.

FAQ

What dimensions should be prohibited from metric labels to prevent cardinality explosions when teams instrument new services?

Any dimension whose cardinality is proportional to a per-entity count that is expected to grow without bound must be prohibited from metric labels: user_id (one series per user), session_id (one series per session), request_id (one series per request), order_id, transaction_id, customer_id. These dimensions are appropriate for traces and logs — both of which are stored per-event rather than indexed across series — but they are inappropriate for metrics because metrics aggregate across time and require one series per unique combination of dimension values. The resulting series counts scale to the product of user volume times service count times metric count, producing a cost multiplier that typically exceeds the observability budget by two to three orders of magnitude before anyone explicitly evaluates whether the dimension was appropriate. Permitted label dimensions are those whose cardinality is bounded and small: service name, environment (production/staging/canary), region, HTTP status code class (2xx/4xx/5xx rather than individual codes), endpoint path (only for endpoints with enumerable paths), error type (bounded to a defined enumeration), and deployment version (with a policy for pruning old versions after they are no longer deployed). The governance policy is most effective when it is mechanically enforced: a CI lint step that checks new metric declarations against the allowed dimension list, a code review requirement for any new label that adds a dimension not on the allowed list, and a platform team review gate for any label whose cardinality cannot be bounded at instrumentation time. The enforcement is necessary because the cost consequence of a high-cardinality label is invisible to the engineer who adds it — the metric appears to work correctly at low traffic, the series count grows gradually as the dimension's unique values accumulate in production, and the problem is typically discovered at the next billing review rather than at the moment the label is added.

How should teams determine the minimum signal retention window that provides adequate incident retrospective coverage?

The minimum retention window is determined by the longest plausible lag between an incident beginning and an engineering team having enough evidence to know an investigation is warranted. This lag has three components: the detection lag (how long before a monitoring system or customer reports a problem), the escalation lag (how long before the report reaches an engineer who can initiate an investigation), and the investigation initiation lag (how long before the investigation consumes historical signals). For most production reliability issues, the detection lag drives the retention requirement. Customer-reported performance regressions — where the customer notices a slow or unavailable feature and opens a support ticket — have the longest detection lags: enterprise customers with monthly touchpoints may report an issue 20 to 40 days after it begins. The retention specification that closes the incident retrospective coverage gap for most production systems requires: logs retained for at least 90 days; traces retained for at least 30 days; metrics retained at full resolution for at least 30 days and at hourly resolution for at least 13 months. These are minimums; compliance-regulated systems typically require longer log retention that supersedes the incident coverage minimum. The cost difference between the platform's default retention and these minimums is typically 20 to 40 percent of the base observability cost — worth evaluating before accepting a default that produces a coverage gap that surfaces as a three-week investigation extension when a customer reports a 35-day-old performance regression.

How should observability cost anomaly detection be configured to catch cost spikes within hours rather than at the next billing cycle review?

Cost anomaly detection for observability requires three elements: a cost attribution model that maps cost to its source at service or team granularity, a baseline cost per unit time (daily or hourly), and a threshold above which an alert fires to the cost budget owner. The attribution model is the most important element and the most often missing: if all observability cost is aggregated at the account level, a cost spike from a single misconfigured service is only detectable against total account cost, which means the spike must be large relative to total spend before it triggers an alert. Most observability platforms expose per-service or per-source cost in their usage API, and configuring the cost attribution model means reading that API daily and assigning cost to the team that owns each service. The baseline is the rolling 7-day average daily cost per service. The threshold is a dollar amount or percentage increase that warrants immediate investigation — a practical threshold: alert when any service's daily cost exceeds 150% of its 7-day rolling average. The alert should fire to the on-call engineer for the relevant service, not to a finance email distribution list. The reason the alert must go to an engineer is that the root cause is typically a code change — a debug logging configuration, a new high-cardinality metric, a trace sampling rate misconfiguration — that requires a code deployment or configuration change to resolve; a finance escalation has a multi-day resolution timeline while an engineering alert has a same-shift resolution timeline.

At what scale does investing in a formal observability cost governance policy become worth the implementation overhead?

Observability cost governance is worth formalizing when the monthly observability bill is large enough that a 2x cost spike would be a budget-significant event — typically when the bill exceeds $3,000 to $5,000 per month. Below this threshold, the cost of a governance process (engineering time, tooling, review overhead) typically exceeds the cost of a cardinality incident or a retention misconfiguration. Above this threshold, the cost of a cardinality explosion (which can increase the monthly bill by 5x to 20x in less than a month) or an 18-day logging misconfiguration (which can add $10,000 to $30,000 in unexpected spend) exceeds the governance implementation cost. The threshold also depends on the team's instrumentation velocity: a team actively adding new services and new metrics every sprint is at higher risk of a cardinality incident than a team with a stable metric inventory. Three signals that indicate the governance investment is overdue regardless of current cost: the metric series count has increased more than 3x in the past quarter without a corresponding 3x increase in service count; the observability bill has surprised the engineering or finance team in any of the past three billing cycles; and any engineer on the team cannot accurately estimate within 2x what a new metric with a given label set would cost per month. A simple governance policy — a prohibited dimensions list enforced in CI, a retention specification documented in the observability ADR, and a daily cost alert configured in the observability platform's usage dashboard — requires approximately one engineer-day to implement and produces immediate detection capability for cardinality incidents and logging misconfigurations.

Structural properties set by the observability cost governance decision

The observability cost governance ADR: five sections

FAQ

Further reading