2026-09-19 · ~22 min read · Decision records

The observability sampling decision record: why the trace sampling rate you configured determines your incident diagnosis gap and your telemetry cost runaway surface

Trace sampling model, per-endpoint sampling configuration, and trace context propagation boundary are observability infrastructure decisions that are almost never made explicitly — they emerge from the default settings of the first tracing framework setup and accumulate their consequences as traffic volume, service count, and incident frequency grow. Three failure patterns: the team whose 1% head-based sampling rate was set as the Jaeger default and never revisited, leaving an entire class of low-frequency high-severity failures invisible during incident forensics; the team that fixed blind sampling by switching to 100% without per-path controls, and watched their observability bill triple in thirty days as health checks and background workers dominated span volume; and the team whose distributed traces have systematic completeness gaps because 30% of their inter-service calls bypass the API gateway where trace context is injected.

A 29-person SaaS company built a document review and contract management platform for mid-market legal teams — a system that parsed uploaded contracts, extracted clauses, flagged non-standard provisions, and routed documents through a multi-stage approval workflow. The platform processed approximately 4,000 contract uploads per day at peak. The engineering team had instrumented all services with distributed tracing using Jaeger when the platform was first put into production two years earlier. During the initial setup, a senior engineer had chosen a sampling rate of 1% — the Jaeger getting-started guide's recommended starting point for production deployments, and a rate that produced a manageable volume of stored traces at the team's traffic level. The sampling rate had never been revisited.

In the platform's second year, the team began receiving intermittent support tickets from customers reporting that contract clause extraction had returned incomplete results — documents where known non-standard provisions had not been flagged. The reports came in clusters: several tickets in a week, then silence for two weeks, then another cluster. The engineering team investigated each cluster as it arrived. They pulled application logs, queried the database for the affected document records, and looked at the tracing dashboard. The tracing dashboard showed the extraction pipeline completing successfully for every document in the window around each incident. The error rate charts showed no anomaly. The logs showed no exceptions. The affected documents had extraction run status marked as "completed" — but the clause extraction results were incomplete.

The investigation of the third cluster, six months after the first reports, finally identified the cause: a race condition in the clause extraction worker that was triggered by a specific combination of document length and concurrent upload volume. The race produced a silent truncation — the extraction job wrote a partial result set and marked the job complete without error. The failure only occurred when two conditions coincided: a document exceeding 180 pages and an upload queue depth above a threshold that occurred during morning peak load on approximately 2% of business days.

The trace data that would have identified this failure was a trace showing the extraction worker completing with an unusually short elapsed time for a long document — a timing anomaly that, combined with the document's page count and the queue depth at job start, would have pointed directly at the truncation condition. The team searched the Jaeger trace store for extraction traces from documents over 180 pages during the three identified incident clusters. They found four traces from over 180-page documents across all three clusters combined. The 1% sampling rate, applied uniformly to all extraction jobs, had captured approximately 4 of the estimated 400+ extraction jobs that had run on documents exceeding 180 pages during the incident windows. None of the four captured traces happened to coincide with an incident-triggering queue depth. The trace that would have shown the race condition had never been recorded.

The sampling rate configuration had been set once during the initial infrastructure setup, in a session where the engineering team was working through a list of observability stack items and the sampling rate was one of five configuration values set in thirty minutes. Nobody in the room had discussed what sampling rate was appropriate given the failure modes they were trying to detect. The Jaeger documentation noted that 1% was appropriate for "high-traffic production systems" — a characterization the team accepted without examining what fraction of their actual failure modes would be observable at 1% coverage. The decision that 1% head-based sampling was an appropriate sampling model for their workload — which implicitly accepted that any failure mode affecting fewer than a few percent of requests would have low forensic observability — was never written down, never reviewed against the team's incident history, and never questioned until six months of intermittent customer reports had been investigated without resolution because the critical trace data had been statistically excluded before the failure manifested.

A 36-person SaaS company built a developer workflow automation platform — a tool that ran CI/CD pipelines, executed integration test suites against pull requests, and generated automated code quality reports for engineering teams at Series A and B software companies. The platform ran distributed workloads across multiple worker pools: a pipeline execution pool that ran CI steps, a test runner pool that executed integration tests in ephemeral containers, and a reporting pool that aggregated results and generated PDF reports. The team had experienced several incidents where the tracing data collected at 1% head-based sampling had been insufficient to reproduce the failure path. After a particularly frustrating four-hour incident investigation where the root cause was eventually identified through log pattern analysis rather than trace inspection, the engineering manager proposed switching to 100% trace sampling.

The proposal was discussed in a thirty-minute team meeting. The objection raised was cost: at 100% sampling, the Tempo cluster would receive substantially more data. The engineering manager asked how much more. Nobody had the exact number. An engineer estimated "maybe 2x." The team decided to make the switch and monitor the cost for two weeks before evaluating whether the increase was acceptable. The sampling rate was updated from 1% to 100% across all services via a Helm values change and a cluster rollout. The deployment completed on a Thursday afternoon.

By the following Thursday, the observability team had received an alert from their cloud cost monitoring: the Tempo storage bucket had consumed three times the data volume of the previous equivalent week. The team pulled the Tempo span ingestion dashboard and looked at where the span volume was coming from. The breakdown was immediate and striking: 43% of all spans were from the container health check probes — Kubernetes liveness and readiness probes hitting the /healthz endpoint on every worker pod every 10 seconds, each probe producing a single span with an identical success result and a sub-millisecond duration. Another 19% were from an internal heartbeat service that pinged each worker pool every 30 seconds to verify pool availability — 8 worker pool endpoints, 2 pings per minute per endpoint, each ping producing two spans (the originating request span and the target service span). Another 32% were from the bulk report generation jobs: when a team had 400 pull requests in a repository, the reporting pool generated one span per pull request per reporting run, producing 400 spans per run per repository — a volume that scaled directly with the customer's repository size and pull request activity, with zero additional diagnostic value beyond what a single representative span would provide.

The three categories combined — health check probes, heartbeat pings, and bulk per-record spans — represented 94% of the total span volume. The user-facing request paths, pipeline execution spans, and integration test result spans that the team had switched to 100% sampling to better observe represented 6% of the total span volume. The switch to 100% sampling had produced a 3x cost increase, but had only meaningfully increased the diagnostic coverage of the 6% of spans that were already receiving adequate sampling at 1% (since 1% of 6% of volume was still a reasonable coverage of user-facing paths at the team's traffic level). The health check probes, which had been contributing 43% of all spans at 100% sampling, had contributed 43% × 1% = 0.43% of total spans at 1% sampling — an unmissable sampling gap of zero useful traces, since health check probes produce no diagnostic information.

The cost projection for maintaining 100% sampling indefinitely was four times the previous monthly Tempo infrastructure cost. The team reverted to 1% sampling within two weeks of the switch, having spent the intervening period paying for 94% of their span volume to be wasted on high-frequency zero-diagnostic-value probes and bulk record spans. The revert returned them to exactly the problem they had started with: insufficient trace coverage for low-frequency failure modes on user-facing paths. The fundamental error was treating the sampling decision as a single global configuration value — a rate applied uniformly to every endpoint, every job type, and every call pattern — rather than as a per-traffic-class configuration that distinguished between paths by their diagnostic value and their volume-to-signal ratio. That distinction had never been articulated as a decision. The initial 1% rate was set globally. The 100% switch was applied globally. Neither configuration reflected an understanding of what the sampling rate was supposed to accomplish for the different workload classes the platform ran.

A 43-person SaaS company built a supply chain visibility platform — a tool that aggregated shipping events from freight carriers, customs brokers, and warehouse management systems, and provided real-time shipment tracking and exception alerts to logistics teams at e-commerce companies. The platform's backend was structured as a set of microservices: an ingest service that received events from carriers via webhooks and message queues, a normalization service that mapped carrier-specific event schemas to the platform's canonical model, an enrichment service that added geographic and customs status context, an aggregation service that maintained current shipment state, and a query service that served the customer-facing API. The team had instrumented all five services with OpenTelemetry and was using a parent-based sampling strategy at 10% — the API gateway at the edge injected the W3C traceparent header on 10% of incoming requests, and each downstream service honored the sampling flag in the header to produce a fully correlated trace for the 10% that were sampled.

The parent-based strategy worked correctly for the customer-facing API path: requests entering through the API gateway received a traceparent header with a 10% probability of the sampled flag being set, and all five services honored the flag as the request propagated through the normalization, enrichment, and aggregation services before returning to the query service for the API response. Traces for these requests were complete — all participating services were represented in the trace and the causation chain from the original API request to each downstream service call was intact.

The platform also had a set of direct service-to-service gRPC calls that did not pass through the API gateway. The normalization service called the enrichment service directly via gRPC to resolve geographic context for carrier location codes — a call that happened on every event received from every carrier. The aggregation service called the normalization service directly via gRPC when a batch reconciliation job detected inconsistencies between the aggregated state and the raw event log. Both of these direct gRPC calls had been instrumented with the OpenTelemetry gRPC interceptor, but the interceptor extracted the trace context from the incoming gRPC metadata rather than from the HTTP headers — and the services that initiated these direct calls were not injecting the trace context into the gRPC metadata before making the call. The gRPC calls were traced, but each call started a new root trace with an independent sampling decision rather than continuing the trace of the originating request.

The team discovered the propagation gap during a latency incident investigation. A subset of customers was reporting shipment status delays — their dashboard was not reflecting carrier events for 30 to 90 minutes after the events had been ingested. The engineering team pulled traces for the affected customer shipments from Tempo and found complete traces for the API query paths showing that the query service was returning data correctly from the aggregation service. The traces showed no latency anomaly on the query path. But the investigators could not find traces that connected the carrier webhook ingest events to the downstream normalization and enrichment calls that should have processed those events. The ingest service traces showed events being received and published to the internal message queue. The normalization service traces showed events being consumed and normalization running. But the traces from the ingest service and the traces from the normalization service were separate root traces — the ingest trace and the normalization trace had different trace IDs and could not be correlated without manually joining them through the event ID that appeared in both trace's span attributes.

The manual correlation, once performed, revealed that the normalization service was producing a high rate of geographic enrichment failures on a specific carrier's location codes — a carrier that had changed its location code format two weeks earlier. The enrichment failures were not surfaced as errors in the normalization trace (the service treated unresolvable location codes as a non-fatal condition and continued processing with partial enrichment), but they were producing a significant increase in the duration of the direct gRPC call from normalization to enrichment, because the enrichment service was attempting multiple cache lookups and fallback resolution strategies before returning a partial result. The latency on the direct normalization-to-enrichment call had increased from an average of 8 milliseconds to an average of 340 milliseconds for events from the affected carrier. But because the normalization traces and the ingest traces were disconnected, the trace data did not show the full end-to-end impact of the enrichment latency on customer-visible shipment update delays. The investigation team had to reconstruct the causation chain manually from logs, span attributes, and event IDs — a process that took four hours and would have taken thirty minutes with complete distributed traces connecting the full event processing path from carrier webhook to customer dashboard update.

The propagation gap had existed since the platform was first deployed. The direct gRPC calls had been added before the OpenTelemetry instrumentation was fully rolled out and had been instrumented by a different engineer than the one who configured the API gateway trace context injection. Neither engineer had mapped all inter-service call paths against the trace context propagation boundary to verify that every call path was covered. The sampling decision record — defining which paths required trace context propagation, how context was carried across asynchronous message bus calls (which required explicit metadata injection), and what the audit process was for verifying propagation coverage across a growing service count — had never been written.

Structural properties set by the sampling decision

Three structural properties are determined when an engineering team establishes — or fails to establish — an observability sampling decision record: how completely the trace data covers low-frequency high-severity failure modes, how sustainable the telemetry storage cost is as traffic and service count grow, and how reliably distributed traces connect spans from all participating services into a coherent causation chain. None of these are labeled as decisions during the initial observability stack setup — they emerge from the default sampling rate chosen from a getting-started guide, the global sampling configuration applied uniformly without traffic class differentiation, and the instrumentation coverage gaps left by services that bypass the trace context propagation boundary.

Property 1: The trace sampling model and the incident diagnosis completeness guarantee. The sampling model determines what fraction of trace data from any given failure mode is available for forensic investigation. Head-based sampling at a fixed rate is a statistical expectation, not a coverage guarantee: a 1% sampling rate provides an expected 0 captured traces for a failure affecting 0.5% of requests, and an expected 1 captured trace for a failure affecting 1% of requests over a window of 100 occurrences — coverage that is adequate for aggregate latency percentile measurement but insufficient for the forensic investigation of low-frequency high-severity failures where the engineer needs the actual trace from the actual failing request. Tail-based sampling — which defers the sampling decision until after the trace is complete and applies different rates based on outcome (100% of error traces, 100% of traces exceeding a latency threshold, 5% of traces below threshold) — provides a guarantee that every error and every latency outlier is captured regardless of its frequency. The tradeoff is pipeline complexity: tail sampling requires a stateful buffering component (an OpenTelemetry Collector with a tail sampling processor configured with enough memory to buffer the trace completion window) that does not exist in a pure head-based sampling architecture. The decisions never written down in the sampling domain include not just the sampling rate value but the sampling model's coverage guarantee — what the team has implicitly accepted as the boundary of what trace data will and will not be available for incident forensics — and the review process for evaluating whether that guarantee remains appropriate as the team's failure mode experience accumulates. The observability strategy decision record connects at the instrumentation coverage layer: the strategy record defines what signals are collected (metrics, logs, traces) and at what service boundaries — the sampling decision record defines the rate at which the trace signal is retained after collection, and the two must be considered together because a tracing strategy that instruments every service at 100% but samples 99% of the resulting spans provides less forensic coverage than a strategy that instruments only critical paths but retains 100% of spans from those paths.

Property 2: The per-endpoint sampling configuration and the telemetry cost model. A uniform sampling rate applied across all endpoints conflates traffic paths with fundamentally different diagnostic value-to-volume ratios. Health check probes arrive at sub-second intervals on every containerized service and produce identical successful spans that contribute zero diagnostic information proportional to their volume. Bulk processing operations — batch imports, report generation, queue consumers processing high-volume message streams — produce a span count that scales with record count rather than with request count, generating orders of magnitude more span volume than the user-facing interaction that initiated the batch job. Background workers running on a fixed schedule produce continuous span volume regardless of user activity. None of these high-volume low-diagnostic-value paths require the same sampling rate as user-facing write operations, payment flows, or authentication paths where every failed or slow trace carries diagnostic information about a user-visible failure. The cost model for telemetry storage scales with span volume, not request count — a team that switches from 1% to 100% sampling without differentiating health check probes from user-facing paths will see their storage cost increase in proportion to the volume of probes and bulk operations, not in proportion to the coverage improvement on the paths they actually needed to improve coverage on. The distributed tracing decision record connects at the instrumentation framework layer: the tracing framework's support for per-operation sampling rules — OpenTelemetry's rule-based sampler, Jaeger's remote sampling strategies, or custom sampler implementations — determines whether per-endpoint sampling is available without a per-service instrumentation change; the sampling decision record should specify the per-endpoint sampling configuration using the mechanisms available in the chosen framework, rather than accepting the uniform rate as the only option the framework offers. The observability platform decision record connects at the storage backend layer: the platform choice determines the cost model for stored telemetry — cloud-hosted managed tracing backends typically charge per span ingested or per GB stored; self-hosted Jaeger or Tempo clusters have storage cost tied to object storage consumption; the sampling rate directly controls this cost and changes to the sampling configuration have immediate financial impact that should be modeled before deployment rather than measured after.

Property 3: The trace context propagation boundary and the trace completeness requirement. A distributed trace is only as complete as its weakest propagation link: every service in a call graph that either fails to read incoming trace context or fails to write outgoing trace context for its downstream calls produces a break in the trace that disconnects upstream spans from downstream spans, making the resulting trace fragment usable for local performance analysis but unusable for cross-service causation analysis. The propagation boundary is the set of service entry points where trace context is injected — at an API gateway, at a service mesh sidecar proxy, or at each individual service's instrumented HTTP/gRPC client. Services that call each other directly through paths that bypass the propagation boundary start new root traces with independent sampling decisions, producing disconnected trace fragments for the calls that matter most during latency investigations: direct service-to-service calls that occur on hot paths are precisely the calls where cross-service causation tracing is most valuable for identifying which service is contributing to end-to-end latency. The logging strategy decision record connects at the correlation identifier layer: when distributed traces have propagation gaps, log-based investigation requires a consistent correlation identifier — a request ID, event ID, or operation ID — that appears in every log line from every service that participated in the request, allowing manual correlation across service boundaries where trace context is absent; the logging strategy should specify what correlation identifier is used and how it is propagated across service boundaries, as a fallback for the cases where trace context propagation is unavailable; ideally the correlation ID and the trace ID are the same value so that logs and traces can be joined in the observability platform. The CI/CD pipeline decision record connects at the propagation verification layer: the pipeline can enforce propagation coverage by running an integration test that generates a known cross-service request path and validates that the resulting trace in the tracing backend contains spans from every expected participating service; this test fails when a new service-to-service call is added without propagation coverage, catching the gap at deploy time rather than during an incident. The incident response playbook decision record connects at the operational response layer: the playbook should specify what sampling override procedure is available during an active incident — how to temporarily increase the sampling rate for a degraded service path without a full service redeployment — because the standard sampling rate that is cost-appropriate for steady state may be insufficient for capturing the trace data needed to diagnose a low-frequency failure mode during a time-pressured incident investigation. The WhyChose extractor finds the sampling configuration decisions in your AI session history — the initial instrumentation session where a senior engineer asked about sampling rate and accepted the getting-started guide's 1% default, the post-incident retrospective where someone noted "we didn't have a trace for the failing request" and the follow-up was captured as a backlog item that became stale before any action was taken, and the cost review session where the observability bill was discussed and someone said "we should look at what's generating all those spans" without examining the per-endpoint breakdown that would have identified the health check probe volume.

The sampling ADR: five sections

Section 1: Sampling model selection and the diagnosis completeness guarantee. Specify whether the team uses head-based, tail-based, or a hybrid sampling model for each service tier, and state explicitly what the chosen model's coverage guarantee is for low-frequency high-severity failures. If head-based sampling is chosen, document the accepted limitation: at a given sampling rate R, a failure mode affecting fraction F of requests has an expected trace capture count of R × F × total_requests per unit time, and the team accepts that failures where R × F × total_requests < 1 per investigation window will have no trace data available. If tail-based sampling is chosen, specify the buffering window, the memory requirements for the sampling processor, and the fallback behavior when the tail sampling buffer is exhausted (typically a fallback to head-based sampling at a specified rate). For hybrid configurations — tail-based sampling for user-facing synchronous request paths, head-based sampling for background jobs and event-driven paths — specify the demarcation clearly so that new services and new call paths are assigned to the correct sampling tier at instrumentation time. Connect this section to the alerting threshold decision record: alerts derived from trace-based metrics (error rate alerts that count error spans, latency alerts that measure p99 span duration) have a statistical sensitivity floor proportional to the sampling rate; at 1% head-based sampling, an error rate alert based on sampled span counts has a minimum detectable error rate of approximately 1% before the expected sampled span count drops below the alert threshold's minimum signal; this sensitivity limit must be taken into account when configuring the alert threshold and documented in the alert definition so that responders understand the confidence interval of a sampled-metrics-based alert.

Section 2: Per-endpoint sampling rate configuration and traffic class taxonomy. Define the traffic class taxonomy that determines which sampling rate applies to each endpoint or operation type, and specify the sampling rate for each class. A minimum practical taxonomy has four classes: (1) infrastructure health endpoints — liveness, readiness, metrics scrape, internal heartbeat calls — sampled at 0% or excluded from instrumentation entirely; (2) bulk processing operations — batch jobs, queue consumers processing high-volume streams, bulk import/export operations — sampled at a per-batch rate (one representative span per N records, plus 100% of error-producing records) rather than a per-record rate; (3) background scheduled operations — cron jobs, periodic reconciliation jobs, scheduled notifications — sampled at a low rate (1-5%) sufficient to detect systematic failures without producing continuous span volume from successful runs; (4) user-facing synchronous operations — API requests initiated by users, payment processing, authentication flows, data mutation operations — sampled at 100% with tail-based sampling where the budget allows, or at a rate high enough that the expected sampled span count exceeds 10 per minute for the failure rate the team wants to be able to detect forensically. Connect this section to the SLO and error budget decision record: SLO compliance measurement from trace data requires that the sampled span count is statistically sufficient to estimate the error rate within the confidence interval required for the SLO measurement period; an SLO measured over a 30-day rolling window with a 5% error budget has different minimum sampling rate requirements than an SLO measured over a 5-minute burn rate window for paging; specify the minimum sampling rate that the chosen SLO measurement methodology requires.

Section 3: Trace context propagation boundary and coverage audit. Specify the trace context propagation boundary — the set of service entry points where trace context is injected into outgoing requests — and the coverage requirement that every inter-service call path must satisfy. The coverage requirement is: every call path that crosses a service boundary must carry the W3C traceparent header (or the team's chosen propagation format) from the calling service to the called service, and every called service must extract the trace context from incoming requests and propagate it to all outgoing requests made during the handling of that request. Specify the three call path categories that require explicit propagation verification: synchronous HTTP/gRPC calls (covered by the instrumented HTTP/gRPC client if the correct client is used), asynchronous message bus calls (require explicit metadata injection on publish and extraction on consume, typically via the message broker's instrumentation library), and background jobs that initiate root spans (require that the root span carries a correlation ID that connects it to the triggering event even when no upstream trace context is available). Specify the propagation audit procedure: a list of all service-to-service call paths is maintained in the decision record, annotated with the propagation mechanism used for each path; the list is reviewed whenever a new service-to-service call is added as part of the architectural review for the feature; a CI integration test validates end-to-end trace completeness for the critical path call graph. Connect this section to the service mesh decision record: a service mesh with automatic sidecar injection can provide trace context propagation at the infrastructure layer for all HTTP calls that pass through the sidecar, without requiring application-level instrumentation of each individual HTTP client call; where a service mesh is in use, the propagation boundary is at the sidecar layer rather than the application layer, and the coverage audit must verify that all service-to-service calls pass through the sidecar rather than bypassing it via direct IP calls or loopback connections.

Section 4: Telemetry retention policy and storage cost governance. Specify the retention period for stored traces and the cost governance mechanism that prevents sampling configuration changes from producing unplanned cost increases. The retention period should distinguish between the hot retention window — the period during which traces are available for immediate query in the tracing backend UI, typically 7-30 days — and the cold retention window — the period during which traces are archived to object storage for compliance or forensic purposes, typically 90-365 days depending on the team's compliance requirements. Specify the storage cost model for the chosen backend: per-span ingestion pricing (common in cloud-hosted managed tracing services), per-GB storage pricing (common in self-hosted Tempo or Jaeger with object storage), or a combination; calculate the expected monthly cost at the configured sampling rates using the per-traffic-class volume estimates from section 2, and establish a cost alert threshold that triggers a sampling configuration review when actual costs exceed the estimate by more than a defined percentage (20-30% above estimate is a reasonable trigger). Specify the change control procedure for sampling rate changes: any change that is projected to increase total span volume by more than a defined factor (2x is a reasonable threshold) requires an explicit cost projection and approval before deployment, to prevent the "switch to 100% and measure the impact" approach that produces unexpected cost spikes. Connect this section to the observability platform decision record for the specific cost model of the platform in use, and to the general infrastructure cost governance process for the team.

Section 5: Sampling policy review triggers and incident-time override procedure. Specify the conditions that trigger a review of the sampling policy and the procedure for temporarily increasing sampling rates during active incidents. Review triggers should include: a post-incident finding that relevant trace data was absent because of insufficient sampling coverage (the most common trigger in practice), a traffic volume change of more than 50% that alters the cost model calculation from section 4, the addition of a new high-volume traffic source (a new large customer, a new integration, a new background job) that was not accounted for in the original per-class volume estimates, and a quarterly cost review where actual telemetry costs are compared to the cost projection from section 4. The incident-time override procedure must specify: what mechanism exists for changing the sampling rate without a service deployment (remote sampling configuration endpoint, collector-side sampling override, or environment variable with hot-reload support); what the maximum safe temporary sampling rate is for each service given the telemetry backend's ingestion capacity; how long the override should remain in effect before being reverted; and who is authorized to invoke the override during an active incident. If no dynamic sampling rate mechanism is available, this section should document that limitation explicitly and include a task for implementing one as technical debt, because the absence of an incident-time override means that sampling-related investigation limitations discovered during an incident can only be resolved with a deployment cycle. Connect this section to the incident response playbook decision record for the runbook entry that covers the sampling override procedure, so that on-call responders can find the override mechanism quickly during an active incident without needing to know the observability stack architecture from memory.

FAQ

What is the difference between head-based, tail-based, and parent-based trace sampling?

Head-based sampling makes the sampling decision at the start of a trace, before the request has been processed and before the outcome is known. A probabilistic head-based sampler at 1% will sample approximately one in one hundred incoming requests, regardless of whether those requests succeed, fail, or time out. The decision is fast and requires no buffering because it is made on the first span before any downstream services are called. The limitation is that the sampling decision is blind to outcome: a critical failure affecting 0.5% of requests will be underrepresented (expected 0 traces per 200 occurrences at 1% sampling) even when the aggregate rate of the failure makes it a P1 incident. Tail-based sampling makes the sampling decision after a trace is complete or after a configurable timeout window. A tail-based sampler can inspect the completed trace and apply different sampling rates based on outcome: sample 100% of traces that contain an error span, sample 100% of traces that exceed a latency threshold, and sample 1% of successful traces under threshold. The tradeoff is buffering cost: tail-based sampling requires holding completed trace spans in memory until the full trace arrives and the sampling decision can be made, which requires a stateful component in the telemetry pipeline (an OpenTelemetry Collector with a tail sampling processor, for example) and enough memory to buffer trace volume between the completion window and the sampling decision. Parent-based sampling propagates the sampling decision from the originating service to all downstream services via trace context headers (W3C traceparent or B3 headers), so that all spans for a given trace share the same sampling decision. Parent-based sampling ensures trace completeness — a sampled trace includes spans from all participating services — but is only as reliable as the propagation mechanism: services that do not receive or honor the trace context header make independent sampling decisions, producing trace fragments that cannot be connected to the originating request.

What sampling rates should different endpoint classes use?

Endpoint class determines the appropriate sampling rate based on two factors: diagnostic value (how much information does a trace from this endpoint contribute to incident diagnosis?) and volume-to-signal ratio (how many spans does this endpoint generate relative to its diagnostic contribution?). High diagnostic value, low volume: user-facing API endpoints that handle writes, payment processing, authentication flows, and data transformation operations should be sampled at 100% or with tail sampling that captures 100% of errors and slow requests. The volume is typically bounded by human interaction rates and the diagnostic value is high — these are the paths where incidents surface and where forensic trace data is most needed. Low diagnostic value, high volume: health check probes, liveness and readiness endpoints, metrics scrape endpoints, and internal heartbeat calls should be sampled at 0% or 1%. Every health check probe arrives at a predictable interval, produces an identical successful span, and contributes no diagnostic information that could not be inferred from the absence of a health check failure. Background jobs and batch operations require per-operation sampling rather than per-request sampling: a batch import job that processes 10,000 records per run should sample representative spans from the batch (the first record, every 100th record, any record that produces an error) rather than applying a flat per-record sampling rate that produces 10,000 spans per batch run. The practical starting point for most teams is a three-tier configuration: 0% for infrastructure health endpoints (health checks, metrics scrape), 100% with tail-based error capture for user-facing write paths and payment flows, and 5-10% for read-heavy query paths where latency percentile data is needed but per-request forensics are rarely required.

How do you ensure trace context propagates correctly through all inter-service call paths?

Trace context propagation requires that every service participating in a distributed trace both reads incoming trace context headers and writes outgoing trace context headers on every inter-service call, using a consistent header format. The two most common formats are W3C Trace Context (traceparent and tracestate headers, the IETF standard supported natively by OpenTelemetry) and B3 (the Zipkin format, used by many older Jaeger and Zipkin deployments). Propagation breaks at three places. First, missing instrumentation: a service that has not been instrumented for tracing will neither read nor write trace context headers; calls passing through this service will have their trace context lost, and downstream services will start new traces disconnected from the originating request. Second, format mismatch: a service that reads W3C headers but receives B3 headers from an upstream service will not find its expected header and will start a new trace. Third, bypassed propagation boundary: when a service makes a direct inter-service call without using the instrumented client that injects trace context, the outgoing request does not carry the trace context header and the downstream service starts a new trace. The audit for propagation coverage requires mapping all inter-service call paths — including direct service-to-service calls, asynchronous message bus calls (where trace context is carried in message headers or metadata), and calls from background workers that initiate their own root spans rather than continuing a trace from an upstream request.

How do you increase sampling rate temporarily during an active incident without a deployment?

Temporary sampling rate increases during active incidents require a dynamic sampling configuration mechanism that does not require a service restart or deployment. Two approaches exist. The first is remote sampling configuration: OpenTelemetry SDKs and Jaeger clients support remote sampling strategies served from a sampling configuration endpoint. When a service fetches its sampling strategy from a remote endpoint at a configurable polling interval (typically 30-60 seconds), the operations team can update the remote configuration to change the per-operation sampling rate and services will pick up the new rate within one polling interval without restart. This requires that the remote sampling configuration service was set up at instrumentation time — it cannot be added retroactively during an incident. The second approach is collector-side sampling override: if the telemetry pipeline routes spans through an OpenTelemetry Collector with a tail-based sampling processor, the sampling configuration in the Collector can be updated and the Collector hot-reloaded without touching the instrumented services. For teams without either mechanism in place, the incident-time alternative is to deploy a configuration change — updating an environment variable or feature flag that controls the sampling rate — which requires a deployment cycle but not a code change. The incident response playbook should specify which mechanism is available, how to invoke it, and what the maximum safe sampling rate is for each service given the telemetry storage capacity of the backend.

Further reading