What is the difference between head-based sampling and tail-based sampling in distributed tracing, and why does it matter for capturing slow requests?

Head-based sampling makes the sampling decision at the root span — the first span in a trace, typically at the API gateway or the entry point service — before any downstream spans are produced. The decision (sample or drop) is propagated in the trace context header to all downstream services, which produce and export their spans only if the header indicates the trace was sampled. Head-based sampling implementations include AlwaysSampler (100% of traces are sampled), NeverSampler (0%, used for development), TraceIdRatioBased(p) (a fraction p of traces are sampled based on the trace ID, which is deterministic — the same trace ID will always produce the same sample/drop decision regardless of which service evaluates it), and ParentBased(TraceIdRatioBased(p)) (downstream services respect the sampling decision made by the root span rather than making independent decisions). The problem with head-based sampling for performance diagnosis is that it samples blindly: a 1% head-based sampling rate applied to a service handling 500 requests per second produces 5 sampled traces per second, which are statistically distributed across the request population — meaning that in a P99 latency spike where 1% of requests take 2,400ms, the expected number of sampled slow-request traces is 1% × 1% × 500 = 0.05 per second, or approximately 3 slow-request traces during a 60-second incident window. Tail-based sampling defers the sampling decision until all spans in a trace are collected. A tail-sampling processor (OpenTelemetry Collector tail sampling processor, Grafana Tempo's TraceQL-based filtering) buffers all incoming spans for a configurable decision wait window (30 seconds is a common default), then evaluates a policy against the complete trace: if the root span's duration exceeds a latency threshold (e.g., 1,000ms), or if any span in the trace has error=true, or if the HTTP status code is 5xx, the trace is kept; otherwise, the trace is sampled at the base rate (e.g., 1%). The result is that 100% of slow or erroring requests produce complete traces in the backend, while normal successful requests are sampled at the low base rate. The operational cost of tail-based sampling is that the processor must buffer all spans for the decision window, requiring significant memory proportional to spans_per_second × bytes_per_span × decision_window_seconds, and must be deployed in a stateful, horizontally-scaled configuration where all spans from the same trace are routed to the same processor instance — a requirement that is not met by the default OTEL Collector configuration without consistent hashing or sticky routing.

Why does trace context propagation format matter when deploying OpenTelemetry across multiple services, and what breaks when formats don't match?

Distributed tracing works by injecting a trace context into each outbound request — a header containing the trace ID (shared across all spans in a request chain), the span ID (the current span's identifier, which becomes the parent span ID for the next downstream span), and the sampling decision. When a downstream service receives the request, its SDK reads the trace context header, extracts the trace ID and parent span ID, and creates a new child span linked to the parent. This creates the span tree that the tracing backend assembles into a waterfall view. The trace context propagation format specifies the header names and the encoding of the trace ID, span ID, and sampling flag. W3C Trace Context (standardized in IETF RFC 9532) uses traceparent (e.g., 00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-01) and tracestate headers. Zipkin B3 multi-header uses X-B3-TraceId, X-B3-SpanId, X-B3-ParentSpanId, and X-B3-Sampled. Zipkin B3 single-header uses a b3 header encoding all fields in a single value. Jaeger native uses uber-trace-id. Datadog uses x-datadog-trace-id and x-datadog-parent-id with a different trace ID encoding (64-bit integers rather than 128-bit hex). When Service A injects a W3C traceparent header and Service B only reads X-B3-TraceId, the downstream service B creates a new root span with a fresh trace ID rather than a child span of Service A's span. The tracing backend receives two unlinked traces: a complete trace for Service A ending at the call to B, and a separate root trace for Service B with no parent. The service dependency graph in the UI (Jaeger's System Architecture view, Grafana Tempo's service graph) shows that A calls B (inferred from the service map), but a specific slow request cannot be followed through both services as a single waterfall. The fix requires updating every service to use the same propagation format and redeploying — a cross-service configuration change that is low-risk individually but requires coordinated deployment across all services to take effect simultaneously, or a transition period where each service reads multiple propagation formats (OTEL SDKs support multi-propagator configuration: composite propagators that read all supported formats and write the primary format) until all services are updated.

How do you calculate the storage cost of distributed tracing at production scale, and what are the cost levers?

The distributed tracing storage cost formula is: cost_per_month = requests_per_second × sampling_rate × spans_per_request × bytes_per_span × seconds_per_month × storage_cost_per_byte × retention_days / 30. For concrete numbers: a service handling 200 requests per second, with 100% head-based sampling, 15 spans per request (API gateway span + 3 service spans + 8 database query spans + 2 external API call spans + 2 middleware spans), 1,500 bytes per span (span name, service name, trace ID, span ID, parent span ID, start and end timestamp, 5 span attributes at 100 bytes each, status code), 30-day retention, and Cassandra storage at $0.023/GB/month produces: 200 × 1.0 × 15 × 1,500 × 2,592,000 × 0.023 / (1024^3) ≈ $270/month in raw storage before replication (Cassandra's default replication factor of 3 multiplies this to $810/month). The cost levers are: sampling rate (the most powerful lever — reducing from 100% to 1% reduces cost by 99% but reduces slow-request trace capture by the same factor), spans per request (reducing instrumentation depth — removing low-value middleware spans, skipping health check endpoint traces — reduces cost proportionally; health check endpoints called by Kubernetes probes at 10-second intervals produce 6 spans per minute per pod × the number of pods, totaling millions of low-value spans per day), bytes per span (reducing span attribute cardinality — avoiding high-cardinality attributes like user ID, request body content, or full SQL query text — is primarily a cardinality concern for index backends like Elasticsearch but also reduces raw storage), and retention period (30-day retention vs 15-day vs 7-day halves or quarters the storage cost; the retention period must be long enough to support post-incident forensic analysis of incidents discovered days after they occurred, which is the primary driver for retention periods longer than 7 days). The cost-efficient architecture for high-volume tracing is tail-based sampling with a low base rate for normal traffic (0.1%-1%), 100% capture for slow and erroring requests, and object storage backend (Grafana Tempo on S3 stores at $0.023/GB/month with no replication overhead because S3's durability is built in, versus Cassandra's $0.023 × replication factor).

How does distributed tracing integrate with structured logging and metrics to form a complete observability picture?

The three observability pillars — metrics, logs, and traces — are independently useful but most powerful when correlated. The correlation model requires three integration points. First, trace ID injection into structured logs: the OTEL SDK's trace context must be read by the application's logging library (Winston, Loguru, Zerolog, Logrus) and injected into each log event's structured fields as trace_id and span_id. When a 500 error occurs during a traced request, the application log entry that records the error carries the same trace ID as the trace in Jaeger, allowing the on-call engineer to navigate from the log entry to the full request waterfall with a single query. The integration requires an explicit bridge between the logging library and the OTEL SDK — the OTEL specification defines a Log Bridge API for this purpose, and most language SDKs provide it — but it is not configured automatically when the OTEL SDK is installed; it requires an explicit call to add a trace ID formatter or bridge to the logging configuration. Second, exemplars in Prometheus metrics: an exemplar is a sample point in a metric time series that carries an embedded trace ID for the specific request that produced the metric observation. Prometheus 2.26+ and OpenMetrics support exemplars for histogram metrics (http_request_duration_seconds_bucket with an exemplar pointing to the trace of the 2,400ms request that fell into the >=2000ms bucket). Grafana's Prometheus data source displays exemplar links inline in metric charts — clicking an exemplar opens the corresponding trace. Exemplars require explicit SDK configuration (the OTEL Prometheus exporter must be configured with exemplar sampling enabled) and are not generated automatically. Third, the log aggregation platform's trace ID indexing: the trace_id field in structured logs must be indexed by the log aggregation platform (Elasticsearch, Loki, Splunk, CloudWatch Logs Insights) for the trace-to-log correlation to be queryable in reverse — navigating from a trace in Jaeger to the associated application logs requires that the log platform supports querying by trace_id and that the ingestion pipeline preserves the trace_id field as an indexed attribute. The decision record must specify all three integration points, not just the tracing backend installation, because each integration point is an independent configuration step in the initial tracing setup session and in the logging infrastructure setup session, and none of the three is configured by default when the OTEL SDK is installed.

The distributed tracing decision record: why the sampling model you chose determines your request trace completeness ceiling and your observability cost floor

Published 2026-07-18 · WhyChose

Distributed tracing decisions are made when an observability consultant recommends adding traces alongside metrics and logs, when a performance incident reveals that logs alone don't show which service in a chain caused the latency spike, and when a Kubernetes migration makes it impractical to SSH into individual pods and grep log files for request identifiers. The AI session that handles the initial tracing setup is infrastructure-oriented and terminates at the working trace: it installs Jaeger or configures DataDog APM or adds the OpenTelemetry SDK to the application, configures the OTLP exporter, generates a test request, opens the Jaeger UI, confirms that the trace appears with the expected spans, and closes. The trace works. The session succeeds. The session closes.

What the AI session does not produce is the second half of the tracing decision. The session answers "how do I get traces into the tracing backend?" and delivers a working Jaeger instance with a first trace visible. It does not ask: what is the sampling model — is the OTEL SDK using AlwaysSampler (100% sampling, which generates 4.3 million spans per day at 50 requests per second with 10 spans per request, accumulating $340/month in Cassandra storage before the first production incident); or TraceIdRatioBased(0.01) (1% sampling, which generates 43,000 spans per day at manageable cost but means that a P99 latency spike affecting 1% of requests produces zero sampled slow-request traces during a 10-minute incident window because the sampled traces are the fast ones, not the anomalous ones); or tail-based sampling (decision deferred until all spans are collected, capturing 100% of slow and erroring requests while sampling normal traffic at 1% — the model that costs the same as head-based 1% sampling but captures the traces that matter, at the cost of a stateful sampling processor with consistent span routing); what is the trace context propagation format — is it W3C Trace Context (traceparent/tracestate, the IETF standard that OTEL SDKs default to since 2021), or Zipkin B3 (X-B3-TraceId multi-header, or b3 single-header), or Jaeger native (uber-trace-id), or Datadog's proprietary headers — and does the format match across every service that participates in a distributed request, including the API gateway, every microservice, the message queue consumers, and the background job workers, because a format mismatch at any service boundary silently breaks span correlation and produces fragmented disconnected traces; what is the trace retention period and the storage backend, and does the retention period match the typical discovery lag for performance incidents (incidents discovered from customer reports 48 hours after they occurred require 48 hours of retention to be diagnosable from traces); how does the tracing system integrate with the structured logging platform so that a log entry for an error includes the trace ID that links the log event to the full request waterfall; what is the instrumentation scope — which external calls, database queries, cache operations, and background jobs produce spans, and what attributes are attached to each span type. Each of these questions has an answer that is not derivable from "a trace appeared in the Jaeger UI" as the session completion criterion. Each answer determines whether a P99 latency incident produces actionable traces of the slow requests or empty search results because the 1% sampling rate captured only the fast requests, whether a cross-service request produces a unified trace waterfall or disconnected fragments because two teams independently chose different propagation formats, whether the tracing storage costs $50/month or $800/month at production scale because the sampling rate was never documented and nobody revised it as request volume grew, and whether a log entry for a critical error can be correlated to the full request trace or remains an isolated data point. The answers are in the AI sessions. They are almost never written down.

Two ways distributed tracing decisions produce the wrong outcome in production

The sampling rate that made slow-request traces invisible during a P99 incident

A developer-tools SaaS starts its observability journey with a Claude session six months after launch. The application is running in Kubernetes, the team has structured logging via Winston to CloudWatch, Prometheus metrics via prom-client, and now they want distributed tracing. The session installs Jaeger via Helm into the observability namespace, configures the Jaeger Cassandra backend with a single-node Cassandra StatefulSet, adds @opentelemetry/sdk-node to the Node.js API service, configures the OtlpGrpcExporter pointed at the Jaeger collector, creates a sample endpoint with manual instrumentation showing a root HTTP span and a child database query span, generates a curl request, opens the Jaeger UI at the port-forwarded address, confirms the trace appears with the two spans and their timing, marks the Jira ticket as done. The session took 90 minutes. The OTEL SDK is using its default sampler, which is ParentBased(AlwaysSampler) — 100% of traces are sampled. The session does not document this. It does not document the expected span volume at production scale or the projected storage cost.

Six months later, the service is handling 50 requests per second across three API pods. Each request produces approximately 12 spans: the root HTTP span at the Express middleware layer, a span for each database query (typically 4-6 per request due to an ORM that issues separate queries for related entities), a span for a Redis cache check, a span for an outbound call to the email validation service, and a span for the response serialization. 50 requests/second × 12 spans/request × 86,400 seconds/day = 51.8 million spans per day. Each span in the Cassandra backend occupies approximately 800 bytes (serialized Thrift, including indexed fields for trace ID, service name, operation name, and duration, and blob storage for tags and logs). 51.8 million spans × 800 bytes = 40.7 GB per day raw, before Cassandra's replication factor of 3 applies — producing 122 GB per day of actual disk writes. The Cassandra cluster, which was sized for the development load visible during the session (approximately 200 spans per day from manual test runs), is using 94% of its 300 GB PVC. Cassandra's write performance degrades at high disk utilization. The observability engineer notices in the Grafana dashboard that Cassandra's write latency has increased from 2ms to 340ms average over the past 3 weeks as disk utilization grew. He opens a new Claude session to fix the cost problem. The session identifies the sampling rate as the lever, changes ParentBased(AlwaysSampler) to ParentBased(TraceIdRatioBased(0.01)) in the OTEL SDK configuration — 1% sampling — and restarts the application pods. Jaeger's span ingestion rate drops from 600,000 spans per minute to 6,000, Cassandra write latency recovers to 4ms within 20 minutes, and disk utilization stops growing. The session does not document the sampling rate change. The change is visible in the application's environment variable configuration in the Kubernetes deployment manifest but there is no ADR, no comment in the configuration file, and no JIRA ticket. The configuration change is merged as part of a larger "observability cost optimization" PR with 40 other changes.

Three weeks later, the API service experiences a P99 latency spike. The p50 request duration is 180ms (normal). The P99 request duration, visible in a Prometheus histogram, spikes from 420ms to 2,400ms for a 40-minute window between 14:22 and 15:02 UTC on a Thursday. The on-call engineer queries Jaeger for traces of the affected endpoint (/api/projects) during the incident window. The Jaeger search returns 8 traces for the endpoint in the 40-minute window. The engineer examines all 8 traces. Every trace shows a request duration between 140ms and 230ms — normal request durations. None of the 8 traces shows the 2,400ms anomalous latency. The engineer checks whether the Jaeger UI is filtering by max duration, confirms it is not, re-runs the query, gets the same 8 traces. The engineer opens a Slack thread: "Jaeger is not showing the slow requests — is tracing broken?" Two engineers investigate. Neither knows the current sampling rate. One checks the OTEL documentation and confirms that "1% sampling means 1 in 100 requests is traced." The engineer calculates: 1% sampling × 200 requests per minute for the affected endpoint × 40 minutes = 80 sampled traces expected. The discrepancy (8 found vs 80 expected) is unresolved. The engineer discovers the TraceIdRatioBased(0.01) configuration 45 minutes into the investigation by grepping the Kubernetes deployment manifests. The second finding is that 1% sampling on slow-tail requests produces approximately 0.01 × 1% × 200 rpm × 40 minutes = 0.08 traces of the specific slow-tail requests (P99 is 1% of the distribution, and 1% sampling samples 1% of that, for a 0.01% capture rate of the worst-performing requests). The 40-minute P99 spike produced zero sampled traces of the slow requests. The incident cannot be root-caused from traces. The post-mortem action item is "implement tail-based sampling to capture all slow and erroring requests regardless of the base sampling rate" — a two-day engineering task that involves deploying an OpenTelemetry Collector with the tail_sampling processor, configuring consistent span routing via the OTEL Collector's loadbalancing exporter, and tuning the decision_wait window. The task would have been the initial design if the sampling model had been documented as a decision with its operational implications at the time the tracing configuration was established.

The propagation format conflict that fragmented traces across a microservices migration

A B2B SaaS platform migrates from a Rails monolith to microservices over 18 months. The platform team establishes distributed tracing early in the migration as a prerequisite for observability across service boundaries. The first service to be extracted, the Payments service, is written in Go. A Claude session configures tracing: installs go.opentelemetry.io/otel, configures the OTLP gRPC exporter to the Jaeger collector, instruments the service with a trace middleware that creates a root HTTP span for each incoming request and injects W3C Trace Context headers (traceparent) into outbound HTTP calls. The session verifies that a test payment flow produces a trace in Jaeger showing the root span at the Payments service and a child span for the downstream call to the legacy monolith's billing endpoint. The W3C traceparent header is the OTEL SDK's default propagation format. The session does not document this as a platform decision. It does not note that all future services must use the same propagation format to maintain trace continuity.

The second service to be extracted, the Notifications service, is written in Python. The engineer responsible for the extraction migrates an existing Zipkin setup that was deployed during an earlier observability effort before the platform standardized on Jaeger. The engineer configures tracing with opentelemetry-sdk-python, adds the B3MultiFormat propagator (which reads and writes X-B3-TraceId, X-B3-SpanId, X-B3-Sampled headers) because that is what the existing Zipkin configuration used, installs the OTLP exporter pointed at the Jaeger collector, and confirms that traces from the Notifications service appear in Jaeger. The engineer does not check whether the B3 format matches what the Payments service uses. Neither the Payments service's decision to use W3C nor the Notifications service's decision to use B3 is documented in a shared decision record.

The third service, the Inventory service, is written in Java by an engineer who previously worked at a DataDog customer. She configures the DataDog Java agent for tracing, which injects DataDog proprietary headers (x-datadog-trace-id, x-datadog-parent-id, x-datadog-sampling-priority) into outbound calls and reads those headers from incoming requests. The DataDog agent also exports spans to the OTLP endpoint at the Jaeger collector via the DataDog agent's OTLP pipeline. The Inventory service's traces appear in Jaeger. The engineer confirms that the service graph in Kiali shows the Inventory service calling the Payments service. She does not test whether a trace that originates in the API gateway passes through the Inventory service and the Payments service as a single unified trace, or whether it produces separate traces for each service.

Eighteen months into the migration, the platform has 11 services. Three use W3C Trace Context (the Payments, Analytics, and User services), three use Zipkin B3 multi-header (the Notifications, Reporting, and Audit services), two use B3 single-header (the Search and Recommendation services, which used an older OTEL Python SDK version that defaulted to B3 single-header), one uses DataDog proprietary headers (the Inventory service), and two use no trace context propagation at all (the legacy Rails monolith endpoints that have not been instrumented, which receive incoming requests with trace context headers but do not propagate them to downstream calls). A payment request that flows through the API gateway (W3C) → Inventory service (DataDog) → Payments service (W3C) → Notifications service (B3 multi-header) produces four separate unlinked traces in Jaeger: a trace for the API gateway ending at the Inventory call, a trace for the Inventory service starting as a root span (DataDog headers not read by the W3C-configured API gateway), a trace for the Payments service starting as a root span (W3C traceparent from the API gateway is present but the Inventory service did not propagate it because DataDog's agent does not read W3C traceparent by default), and a trace for the Notifications service starting as a root span. The engineering team's estimate for unifying the propagation format is 6 weeks: updating every service's SDK configuration, testing that each service correctly reads the upstream service's headers and propagates the correct downstream headers, coordinating deployment across 11 services, and validating that end-to-end traces assemble correctly in Jaeger. The estimate would have been 0 hours if a trace context propagation standard had been specified as a platform decision at the time the first service was extracted and documented as a requirement that all future services must implement.

Three structural properties that distributed tracing decisions determine

The sampling model and the trace completeness ceiling

The sampling model specifies the strategy for deciding which requests produce complete traces, the sampling rate per service and per operation type, the mechanism for communicating the sampling decision across service boundaries, and the cost at production request volume. Without a documented sampling model, the team's mental model is "we have tracing with Jaeger" — a model that is accurate but does not answer whether the tracing system captures the requests that matter for incident diagnosis or whether it captures a random sample that is statistically representative of the fast-request majority but excludes the slow-request minority where root cause lives. The sampling model must specify six properties. First, the sampler type per deployment phase: during development and staging, AlwaysSampler (100% sampling) is appropriate because trace volume is low and complete trace coverage is valuable for debugging; in production at scale, head-based ratio sampling or tail-based sampling is required to manage cost; the sampler type must be configured per deployment environment and the production sampler must be specified at design time, not discovered when Cassandra runs out of disk. Second, the sampling rate and its basis: for head-based ratio sampling, the rate must be justified against the request volume and the minimum viable trace count for statistical significance; a 1% sampling rate that produces 500 traces per hour at 50,000 requests per hour is adequate for trend analysis of normal request behavior but produces 5 traces per hour of requests in the 99th percentile latency bucket, which is insufficient for P99 incident diagnosis; the rate must be specified with the request volume and span volume it implies, not as a bare percentage. Third, the tail-based sampling policy for operations that require complete coverage: health check endpoints, high-frequency cache operations, and routine CRUD endpoints on stable-behavior paths can be sampled aggressively or dropped entirely; API endpoints handling financial transactions, authentication, and user-visible operations with variable latency distributions should be captured at 100% for slow and erroring requests regardless of the base rate; the tail-based sampling policy must specify the latency threshold (traces with root span duration >= 1,000ms are always kept), the error policy (traces with any span bearing error=true are always kept), the status code policy (traces with HTTP 4xx and 5xx root spans are always kept), and the base rate for non-matching traces (1% or 0.1%). Fourth, the sampling decision propagation model: the trace context header that carries the sampling flag must be respected by all downstream services — a service that makes an independent sampling decision regardless of the upstream service's propagated decision will break trace continuity for 99% of sampled traces (the upstream service decides to sample a trace, the downstream service independently decides not to sample, and the trace in the backend is missing all downstream spans); the ParentBased sampler in OTEL SDKs implements correct propagation, but must be explicitly configured. Fifth, the high-cardinality operation exclusion list: operations that are called extremely frequently and produce low-diagnostic-value traces — Kubernetes liveness probes at 10-second intervals, Redis PING health checks, load balancer health check endpoints, metrics scrape requests — must be explicitly excluded from tracing to prevent them from consuming sampling budget and storage; the exclusion list must be maintained in the sampling configuration alongside the sampling rate, not in individual service configurations where it will diverge as new services are added. Sixth, the cost model at target scale: the sampling model must include a cost projection at the target production request volume (requests_per_second × sampling_rate × spans_per_request × bytes_per_span × retention_days × storage_cost_per_gb), updated when request volume grows by a factor of 5x or the span count per request increases by 3 or more, so that storage cost surprises are detected before the Cassandra PVC fills. The SLO and error budget decision record specifies the latency SLO; the tail-based sampling latency threshold must be set below the P99 SLO target so that all requests that would breach the SLO are captured with 100% completeness, not sampled at the base rate.

The trace context propagation model and the service boundary correlation surface

The trace context propagation model specifies the header format used to transmit the trace ID, span ID, and sampling decision across service boundaries, the list of services and infrastructure components (API gateways, load balancers, message queue consumers, scheduled jobs) that participate in trace context propagation, the multi-propagator configuration for backward compatibility during format transitions, and the async context propagation model for spans that cross thread or process boundaries via queues, workers, and batch jobs. Without a documented propagation model, trace context format is whatever each team selected when they first configured tracing — a default that reflects each team's language, SDK version, and prior experience rather than a platform decision that enables end-to-end trace assembly. The propagation model must specify seven properties. First, the canonical propagation format for the platform: W3C Trace Context (traceparent/tracestate, IETF RFC 9532) is the standard for new systems after 2021, supported natively by all major OTEL SDK versions and by most API gateways, service meshes, and CDN providers; Zipkin B3 multi-header remains in wide use in Java ecosystems (Spring Boot's Micrometer Tracing defaults to B3) and in systems migrating from Zipkin to Jaeger; the choice must be made once and applied consistently across all services, with a migration plan if the platform is consolidating from multiple formats. Second, the API gateway propagation configuration: the API gateway (Kong, Traefik, NGINX, AWS API Gateway) is the entry point where trace context is first injected into a request arriving from an external client that does not carry a trace context header; the gateway must inject a trace context header in the canonical format with a fresh trace ID and a sampling decision based on the gateway's sampler configuration; if the gateway does not inject trace context, the first internal service creates a root span rather than a child span, and the client-to-gateway leg of the request is invisible in traces. Third, the service mesh propagation model: Istio's default propagation format is Zipkin B3 multi-header (as of Istio 1.14); Linkerd uses OTEL W3C Trace Context; if the service mesh and application-layer tracing use different formats, the service mesh's sidecar spans (Envoy spans for HTTP requests between services) and the application's spans (database query spans, business logic spans) may belong to different traces if the service mesh's context is not read by the application SDK, producing a split between infrastructure-layer traces and application-layer traces that cannot be correlated. Fourth, the async trace context propagation model: when a request handler publishes a message to a Kafka topic or an SQS queue, and a separate consumer process reads the message and performs work on behalf of the original request, the trace context must be serialized into the message metadata and deserialized by the consumer to create a linked child span; OTEL provides a text map propagator interface for this purpose, but the message schema must reserve fields for trace context metadata, and the consumer service must be instrumented to read and activate the context from the message metadata rather than starting a fresh root span; the schema for trace context in message metadata must be specified as a platform contract alongside the HTTP header format, because a Kafka message schema change to add trace context metadata fields requires coordination between producer and consumer teams. Fifth, the baggage propagation policy: W3C Trace Context's tracestate header and OTEL's Baggage API allow arbitrary key-value pairs to propagate across service boundaries alongside the trace context; baggage is useful for propagating request-level context (user ID, tenant ID, feature flag state) for use in downstream spans and logs, but must be governed — unbounded baggage propagation on high-throughput requests increases per-request header size and introduces a data leak vector if sensitive values are propagated to external or third-party services; the baggage propagation policy must specify which keys are permitted in baggage, the maximum baggage payload size, and whether baggage is stripped before requests are forwarded to external services. Sixth, the b3 single-header versus multi-header distinction: B3 single-header (b3) and B3 multi-header (X-B3-TraceId) are different wire formats for the same logical content; OTEL SDKs that default to B3 may use either format depending on the SDK version (older Spring Micrometer defaults to multi-header; some Go B3 libraries default to single-header); the canonical format specification must indicate which B3 variant is used, because a service that only reads multi-header will not read single-header context from an upstream service, and the Jaeger UI will show a broken trace without an explicit error. Seventh, the propagation validation test: the end-to-end trace assembly for the platform's most critical request path (user login → API call → multiple service calls → database queries → async notification) must be verified as a test in the integration test suite or as a scheduled synthetic trace in the production environment, with an alert that fires if the trace assembly produces disconnected root spans rather than a unified waterfall; trace fragmentation is a silent failure — the Jaeger UI continues working and showing traces, but the traces are not end-to-end, and the failure is only noticed when an engineer tries to follow a slow request through multiple services and cannot. The API gateway decision record specifies the gateway's routing configuration; the trace context injection model must be specified alongside the gateway configuration, because the gateway is the entry point where the trace is originated for all inbound requests.

The trace retention, storage backend, and incident diagnosis latency floor

The trace retention and storage model specifies the tracing backend technology and its operational requirements, the retention period per trace category, the query model for trace search and correlation with logs and metrics, and the trace data model for span attributes. Without a documented retention model, trace data is retained until storage fills, queried with whatever latency the backend produces at current data volume, and correlated with logs and metrics by manual trace ID lookup in separate tools. The retention model must specify five properties. First, the storage backend selection and its operational characteristics: Jaeger with Cassandra (high write throughput, Cassandra operational complexity, replication factor multiplies cost); Jaeger with Elasticsearch or OpenSearch (rich indexing and query capabilities, Elasticsearch operational complexity and cost at scale, useful for attribute-based trace search); Grafana Tempo with object storage (S3/GCS/Azure Blob, lowest cost per span at scale, TraceQL query language, slow search performance for ad-hoc queries without a Tempo backend cache); Zipkin with Elasticsearch or MySQL (legacy choice, limited query capability with MySQL); DataDog APM (managed, no operational overhead, high cost at scale, 15-day default retention, deep integration with DataDog metrics and logs); Honeycomb (managed column store, sub-second ad-hoc query on high-cardinality attributes, dynamic sampling via Refinery, high cost at scale); AWS X-Ray (managed, native AWS integration, 30-day retention, limited query capability compared to Jaeger or Honeycomb); the selection must document the operational requirements (who manages Cassandra or Elasticsearch, what is the backup policy, what is the upgrade path, what is the SLA for trace query availability during a storage backend maintenance window). Second, the retention period per trace category: high-priority traces (P99 slow requests, error traces, traces flagged by the tail-based sampler) may warrant longer retention (60-90 days) for forensic analysis of slow incident discovery; normal sampled traces may be retained for 15-30 days, sufficient for post-incident analysis of incidents discovered within 2 weeks; the retention period must be set before the storage backend is provisioned, because reducing retention on Cassandra or Elasticsearch requires a rolling deletion process that can cause query performance degradation if not managed carefully. Third, the query model for incident diagnosis: the trace search interface must support the queries that are actually useful during incidents — "find all traces for endpoint /api/payment with duration > 1,000ms in the last 2 hours" (requires indexing of root span operation name and duration), "find all traces where any span has error=true" (requires indexing of error status on all spans, not just root spans), "find all traces with http.status_code=503" (requires indexing of span attribute values, not just span names); the Jaeger UI's search capability and TraceQL in Grafana Tempo provide these queries with different latency profiles; the query capability must be evaluated against the incident diagnosis workflow the team uses, not against the trace storage backend's benchmark performance on synthetic workloads. Fourth, the trace-to-log correlation model: the trace ID must be injected into structured log output (trace_id and span_id fields in every log event) during an active trace, the logging library must be integrated with the OTEL SDK's trace context API to read the current span's context, and the log aggregation platform must index the trace_id field for reverse correlation (searching logs by trace ID to find all log events from a specific request). The correlation model must specify the logging library integration mechanism (OTEL Log Bridge API, Winston middleware, Loguru processor, Zerolog hook), the log field names (trace_id, span_id, trace_flags — using vendor-specific field names from DataDog dd.trace_id or GCP trace is acceptable but must be consistent), and whether the log aggregation platform (Elasticsearch, Loki, Splunk, CloudWatch Logs Insights) natively supports navigation from a log entry to the corresponding Jaeger or Tempo trace. Fifth, the metrics exemplars model: Prometheus histogram metrics (http_request_duration_seconds_bucket) can carry exemplars — sample points embedding the trace ID of the request that produced the metric observation — that enable navigation from a Grafana metrics panel to the specific trace of the anomalous request that fell into the high-latency bucket; exemplar support requires the OTEL Prometheus exporter to be configured with exemplar sampling and the Prometheus scrape configuration to enable the OpenMetrics format required for exemplar transmission; the exemplar model must be specified alongside the metrics decision record as an integration point between the metrics and tracing pillars. The logging strategy decision record specifies the structured log format and the log aggregation platform; the distributed tracing integration with logging must be specified in both the logging decision record and the tracing decision record, because neither record alone captures the full integration contract (the logging record specifies the field schema, the tracing record specifies the OTEL integration mechanism that produces the trace_id field value).

Three AI session types that embed tracing decisions without documenting them

The initial tracing setup session is where the sampling model and propagation format are established without being named as decisions. The session is infrastructure-oriented and terminates at the working trace: install the tracing backend, configure the SDK exporter, instrument a single endpoint, confirm a trace appears in the UI, and close. The sampler type is whatever the OTEL SDK defaults to — AlwaysSampler in OTEL SDKs configured with a non-parent-based sampler, ParentBased(AlwaysSampler) in the default SDK configuration — which is appropriate for development but will produce millions of spans per day at production scale. The propagation format is whatever the SDK defaults to — W3C Trace Context in OTEL SDKs after 2021, B3 multi-header in older Spring Boot configurations, Jaeger native in Jaeger client libraries. The storage backend is whatever the session recommended — Jaeger all-in-one (in-memory, data lost on pod restart, appropriate for local development), Jaeger with Cassandra (production-grade but operationally complex), Grafana Tempo (cost-efficient but requires query tuning). None of these defaults are the result of a policy decision about the team's production requirements; they are the tools' default behavior applied to a development verification task. The session result is "a trace is visible in the Jaeger UI" — not "sampling model is ParentBased(TraceIdRatioBased(0.01)) with tail-based sampling for slow and error traces deployed via OTEL Collector tail_sampling processor, propagation format is W3C Trace Context with B3 multi-header as fallback during the 6-week Notifications service migration, storage backend is Grafana Tempo on S3 with 30-day retention for normal traces and 90-day retention for error traces, projected cost at 100 req/s is $45/month." The second result requires treating each configuration choice as a production decision with cost and operational consequences rather than as a development verification step with a binary working/not-working completion criterion. The observability strategy decision record specifies the three-pillar observability model (metrics, logs, traces) and the platform selection; the distributed tracing decision record must be consistent with the observability platform selection — if the observability platform is DataDog, the tracing backend is DataDog APM and the propagation format must include DataDog's proprietary headers for full APM integration; if the platform is Grafana, the tracing backend is Grafana Tempo and the query model is TraceQL.

The performance debugging session embeds tracing configuration changes as temporary adjustments that become permanent without documentation. The session is triggered by a specific performance problem: a slow endpoint, an unexplained latency spike, a customer complaint about API response time. The engineer or AI session uses distributed traces as the primary investigation tool, which requires that the traces are available for the affected endpoint during the investigation window. If the sampling rate is too low (1%, producing 5 traces per minute at 500 rpm), the session increases the sampling rate for the affected service to 100% to ensure that the slow requests are captured. The session may use AlwaysSampler in the service configuration temporarily, or increase the TraceIdRatioBased fraction to 1.0, or disable the sampling flag in the service's environment variables. The investigation completes: the session identifies the slow span (a database query without an index that was producing a full table scan), implements the fix (adds the missing index), confirms that the P99 latency recovers. The session closes. The sampling rate modification from 1% to 100% may or may not be reverted. If the service was previously at 1% and the session reverts it after the investigation, no documentation is needed and no harm is done. If the session does not revert it — because reverting was not the primary task and was not mentioned in the session transcript — the service continues at 100% sampling, generating 100x the span volume, and storage costs increase 100-fold before anyone notices. Alternatively, the session may add instrumentation for the specific slow operation that was investigated (database query spans) without extending that instrumentation to other operations (Redis cache operations, external API calls) that were not relevant to the current investigation, leaving the tracing coverage in a partial state where database spans are present but cache and external call spans are absent — a coverage gap that is invisible until the next incident involves a cache operation and the engineer cannot find the cache spans in the trace. The new CTO onboarding problem with distributed tracing is that the incoming technical leader cannot determine the intended sampling model from the deployed configuration: the OTEL SDK's TraceIdRatioBased(0.1) environment variable may be the design decision made at system initialization, or the result of three rounds of cost optimization that each halved the rate, or the residual state after a performance debugging session that changed the rate and was not fully reverted. Without a decision record, the configuration is indistinguishable from the intent.

The service mesh or Kubernetes migration session embeds tracing decisions across two independent layers without specifying how they interact. The session deploys Istio for traffic management and enables Istio's distributed tracing integration by setting the meshConfig.enableTracing: true flag and configuring the tracing.zipkin.address to the Jaeger collector. The session verifies in Kiali that the service graph is populated and that traces appear for inter-service calls. The session closes. Two tracing layers are now active: Istio's Envoy sidecar layer (which generates spans for every HTTP/gRPC request passing through the sidecar, using Zipkin B3 multi-header propagation) and the application layer (which generates spans for database queries, cache operations, and business logic, using whatever propagation format was configured in the initial tracing setup). The session does not document whether these two layers produce spans that belong to the same trace or separate traces. The answer depends on whether the application-layer OTEL SDK is reading the B3 multi-header that Istio's sidecar injects, or only reading W3C Trace Context. If the application uses W3C Trace Context (OTEL SDK default after 2021) and Istio uses B3 (Istio default), the sidecar span and the application spans belong to separate traces: Jaeger shows a trace with two sidecar spans (client sidecar for the outbound call and server sidecar for the inbound call) and a separate trace with the application's database query spans, with no link between them. The fix — configuring Istio to use W3C Trace Context, or configuring the application's OTEL SDK with a composite propagator that reads both W3C and B3 — requires a change to either the Istio MeshConfig (a cluster-level change affecting all services) or each service's OTEL SDK configuration, and a redeployment. The session also does not document the interaction between Istio's sampling configuration (meshConfig.defaultConfig.tracing.sampling = 1.0 by default, meaning 1% of requests are traced at the mesh layer) and the application's OTEL SDK sampling configuration: if Istio samples 1% of requests and the application uses AlwaysSampler, the application produces spans for 100% of requests but Istio only provides the sidecar context for 1% of them — the other 99% of application traces are missing the sidecar spans and appear as root spans without the inter-service call context that shows which upstream service initiated the request. The service mesh decision record specifies the service mesh selection and its traffic management configuration; the tracing integration model must be specified in both the service mesh decision record and the distributed tracing decision record, because the two decisions interact at the propagation format and sampling configuration layers, and a change to one without reviewing the other will produce the trace fragmentation scenario that appears in Kiali as a working service graph but in Jaeger as disconnected traces.

The five sections of a distributed tracing decision record

The first section documents the tracing backend selection and SDK configuration. The tracing backend choice determines the operational requirements (managed service vs self-hosted, required infrastructure), the query capability and latency at scale, the integration with the existing observability stack (DataDog APM integrates natively with DataDog metrics and logs; Grafana Tempo integrates natively with Grafana's dashboards and Loki logs), and the cost structure (per-span ingestion pricing for DataDog and Honeycomb, storage-cost-only for Jaeger with object storage backends, flat licensing for self-hosted Jaeger). The rationale for the tracing backend must be recorded because migration between backends (from Jaeger to DataDog APM, from self-hosted Zipkin to Grafana Tempo) is a significant undertaking that affects SDK configuration across every service, storage migration or data loss, and team training on a new query interface. The SDK configuration must specify the OTEL SDK package and version per language runtime, the exporter type and endpoint (OTLP gRPC vs OTLP HTTP, the collector address, whether the exporter sends directly to the backend or through an OTEL Collector pipeline), the processor type (BatchSpanProcessor with batch size and export timeout vs SimpleSpanProcessor for development), and the resource attributes attached to every span (service.name, service.version, deployment.environment, cloud.region, k8s.namespace.name — the attributes that appear in every span for filtering and grouping in the Jaeger UI and TraceQL queries). The OTEL Collector deployment model must be specified if a collector is deployed: whether it is deployed as a sidecar alongside each service pod, as a DaemonSet on each node, or as a centralized deployment (required for tail-based sampling, where all spans from a trace must arrive at the same processor instance), and the collector pipeline stages (receivers, processors, exporters) that transform spans between the application SDK and the backend. The container orchestration decision record specifies the Kubernetes cluster configuration; the OTEL Collector deployment model must be consistent with the cluster's resource allocation — a DaemonSet collector that runs on every node consumes CPU and memory on every node, and the resource limits must be specified to prevent the collector from competing with application workloads during high-trace-volume periods.

The second section documents the sampling model and rate configuration. The sampling model specification must address six sub-decisions. Head-based versus tail-based sampling: head-based sampling is operationally simpler (no stateful collector, no span buffering, no consistent routing requirement) but cannot differentiate between fast-and-normal requests and slow-and-anomalous requests; tail-based sampling requires a stateful OTEL Collector with the tail_sampling processor, consistent span routing (all spans from a trace must reach the same processor instance, requiring a loadbalancing exporter in the upstream collector tier), and a decision_wait window that determines the maximum trace duration that can be evaluated (a decision_wait of 30 seconds means traces longer than 30 seconds are evaluated with incomplete spans, which affects long-running batch operations and streaming request handlers). The base sampling rate for head-based or for the non-matching policy in tail-based: the rate must be set against the storage cost projection at production volume, not as a default; a 1% rate at 1,000 requests/second × 15 spans/request × 1,000 bytes/span × 86,400 seconds = 12.96 GB/day before replication — a concrete number the team can evaluate against the storage budget. The tail-based sampling policy specification: the latency threshold for always-keeping traces (traces with root span duration >= N milliseconds are always sampled, where N is chosen below the P99 SLO target — if the P99 SLO is 500ms, a threshold of 400ms captures the traces most likely to breach the SLO while allowing fast traces to be sampled at the base rate); the error policy (any span with status.code=ERROR triggers always-sample); the status code policy (HTTP 5xx root spans trigger always-sample, HTTP 4xx may or may not depending on whether client errors are diagnostic); the operation exclusion policy (health check, readiness, liveness, and metrics scrape operations are explicitly excluded from the always-sample policies and sampled at 0% or 0.1%). The operation-level rate overrides: high-frequency, low-value operations may be sampled at 0.01% or excluded entirely, while low-frequency, high-value operations (payment processing, user authentication, administrative operations) may be sampled at 10% or 100% regardless of the base rate. The sampling configuration storage location: the sampling configuration must be stored as a versioned artifact (a Kubernetes ConfigMap for the OTEL Collector, an environment variable in the service deployment manifest, or a remote sampling configuration via Jaeger's remote sampling endpoint) and its change history must be visible in git — a sampling rate change that was made by editing an environment variable without a configuration change review is invisible to the team and impossible to correlate with a change in storage costs. The sampling rate review trigger: the sampling configuration must be reviewed when request volume grows by 5x, when span-per-request count increases by 3+, or when storage costs exceed the monthly budget threshold, with the projected cost at the new volume documented in the review. The SLO and error budget decision record specifies the P99 latency target; the tail-based sampling latency threshold is a derived parameter of the latency SLO, and the two must be updated together when the SLO target changes.

The third section documents the trace context propagation format and the inter-service contract. The propagation format specification must be a platform-level decision, not a per-service choice: the canonical format (W3C Trace Context for new platforms, B3 multi-header for platforms migrating from Zipkin) must be specified, all new services must use the canonical format, and the exception process for services that cannot use the canonical format must require approval and a documented timeline for migration. The API gateway propagation configuration must specify how the gateway injects trace context for inbound requests that do not carry a trace context header, which propagation format the gateway uses, and whether the gateway reads and validates incoming trace context headers from external clients (which should be treated as untrusted — a malicious client that injects a crafted trace context header can attempt to manipulate sampling decisions or inject trace IDs that collide with legitimate traces). The service mesh interaction must specify the mesh's default propagation format, whether it matches the application-layer canonical format, and the configuration change required to align them (Istio's meshConfig.defaultConfig.tracing.customTags and extensionProviders configuration for W3C Trace Context; Linkerd's traceCollector configuration). The async propagation specification must define the message metadata fields for trace context in each queue technology in use (Kafka message headers for W3C trace context: traceparent and tracestate as Kafka headers with String key type; SQS message attributes for W3C trace context: traceparent as a String message attribute), the producer instrumentation pattern (read the current OTEL context, inject into message metadata using a TextMapPropagator), and the consumer instrumentation pattern (extract context from message metadata, call span.setRemoteParent or activate the extracted context as the current context for the processing span). The propagation validation test specification must define a test that generates a request through the complete request path (API gateway → all services → all async consumers) and asserts that the Jaeger trace for the request contains spans from every service without fragmentation into multiple root traces; the test must be run after every service deployment that changes the OTEL SDK version, the propagation format configuration, or the collector routing configuration. The API gateway decision record specifies the gateway technology; the trace context injection configuration for the gateway is a joint concern of the API gateway decision record and the distributed tracing decision record, and both records must cross-reference the propagation format specification to prevent divergence during gateway configuration changes.

The fourth section documents the instrumentation scope and span attribute model. The instrumentation scope specifies which operations in each service produce spans, what attributes are attached to each span type, and what the cardinality constraints on span attributes are. The default auto-instrumentation provided by OTEL SDK instrumentation libraries covers HTTP server and client calls, gRPC server and client calls, and database queries via ORM auto-instrumentation — but does not cover cache operations (Redis commands require the redis instrumentation library or manual instrumentation), message queue publishing and consuming (Kafka instrumentation requires explicit SDK setup), background job execution (Celery, Sidekiq, BullMQ each require specific instrumentation libraries), and business-logic operations (payment processing steps, PDF generation steps, external API call retries) that require manual span creation. The instrumentation scope specification must list each service and the operations that produce spans, the instrumentation libraries used for each operation type, and the operations that are explicitly not instrumented (with the rationale — high-frequency health check operations that would inflate trace volume without diagnostic value, internal utility functions where the call site is already covered by a higher-level span). The span attribute model must specify which attributes are attached to each span type: HTTP server spans must carry http.method, http.route (not http.url, which is high-cardinality due to URL parameters), http.status_code, http.request_content_length, and http.response_content_length; database query spans must carry db.system, db.name, db.statement (with PII scrubbing applied to query parameter values, not query structure), db.operation, and db.sql.table; the cardinality constraint on span attributes must prohibit high-cardinality values as attribute keys or values — user.id, tenant.id, request.body, and session.token must not be attached as span attributes because they would create millions of unique attribute values in Elasticsearch or Jaeger's index, causing index bloat and degrading query performance; tenant and user context must be propagated via OTEL Baggage and attached to log events rather than to span attributes if low-cardinality grouping by user or tenant is required. The database query span attribute policy must explicitly address the PII and security risk of db.statement: database query spans that include the full SQL statement text (including WHERE clause values interpolated from user input) expose user data in the tracing backend if the query is not parameterized or if the parameter values are logged in the statement attribute; the policy must specify whether db.statement records the parameterized statement template (safe), the full statement with values (unsafe unless the database contains no PII), or is excluded from spans (safe but reduces diagnostic value for slow query analysis). The database connection pooling decision record specifies the ORM and connection pool configuration; the database query span instrumentation must be specified alongside the ORM configuration, because the ORM's instrumentation library generates db.statement attribute values using the ORM's query builder output format, and the PII risk of the statement format depends on whether the ORM generates parameterized queries (safe) or string-interpolated queries with literal values (unsafe).

The fifth section documents the trace retention, storage sizing, and operational model. The retention model must specify the retention period per trace category (error traces: 90 days; slow traces captured by tail-based sampler: 30 days; base-rate sampled normal traces: 15 days), the storage sizing calculation at target production scale with the specified sampling model (sampling_rate × requests_per_second × spans_per_request × bytes_per_span × seconds_per_day × retention_days = total_storage_bytes, divided by the backend's replication factor for replicated backends), the storage backend's high-availability configuration (Cassandra replication factor and consistency level, Elasticsearch replica count, Grafana Tempo's object storage backend redundancy), and the data lifecycle management configuration (Cassandra's TTL on span records, Elasticsearch's Index Lifecycle Management policy for trace indices, Grafana Tempo's block compaction and retention policy). The operational model must specify who is responsible for the tracing infrastructure (the platform team, the observability team, or a managed service vendor), the monitoring configuration for the tracing infrastructure itself (Jaeger's collector ingestion rate, Cassandra's write latency and disk utilization, the OTEL Collector's span drop rate when the backend is unavailable — if the backend is unavailable and the SDK's export queue is full, spans are dropped silently), the tracing backend SLA (whether a Jaeger backend outage means that traces are dropped immediately or buffered in the OTEL Collector's export queue for a retry window), and the backup and recovery model (whether trace data is backed up and whether it would be restored after a storage failure, or whether traces are considered ephemeral and a storage failure results in data loss for the retention window). The trace access control model must specify who can query traces in the tracing backend: traces may contain span attribute values that include partial user data (IP addresses, request paths, user agent strings), database query structures that reveal the application's data model, and error messages that expose implementation details; the tracing backend's access control must be consistent with the team's data handling policy for user data and must not provide read access to traces to users who do not have access to the equivalent log data. The incident response integration must specify how the on-call runbook references trace queries for each alert type: a P99 latency alert in PagerDuty should include a link to the Jaeger search query for slow traces of the affected endpoint in the time window of the alert, not a generic link to the Jaeger UI; the specific query must be pre-tested and maintained alongside the alert definition so that it returns results for the alert condition and not empty results due to a stale query or a trace index rotation. The secrets management decision record specifies the secrets storage model; the tracing backend API key (DataDog API key, Honeycomb API key, Grafana Cloud API key, Jaeger collector's authentication token if enabled) is a high-sensitivity secret whose compromise allows an attacker to inject false traces that mask malicious activity, or to query traces to enumerate the application's internal service topology and database schema structure from span attributes; the tracing backend credential must be rotated on the same schedule as database credentials and stored in the same secrets management system, not hardcoded in environment variables in Kubernetes deployment manifests.

The initial tracing setup session that installed Jaeger with AlwaysSampler, the performance debugging session that changed TraceIdRatioBased from 0.1 to 0.01 to fix storage costs and did not revert when the debugging was complete, and the service mesh configuration session that deployed Istio with B3 propagation without verifying alignment with the application's W3C Trace Context format — each produced distributed tracing decisions whose operational consequences (a P99 latency incident producing zero sampled traces of the slow requests because the 1% head-based rate captures the fast-request majority and not the slow-request minority; a cross-service request trace fragmented into four disconnected root spans because three teams independently chose different propagation formats; a Cassandra cluster at 94% disk utilization because the sampling rate was never documented and the storage growth at 5x request volume was never projected) exceed what a sampling model specifying tail-based capture of slow and error traces with a documented cost projection, a propagation format specification that every service references before configuring their OTEL SDK, and a retention policy with storage sizing at target volume would have cost to specify at the time the founding tracing decisions were made. The decisions are in the AI sessions: the sampler type accepted from the SDK default without documenting the cost at production request volume, the propagation format chosen based on the SDK's version-specific default without documenting it as a platform contract, the storage backend selected based on the session's recommendation without specifying the operational requirements or the retention period. WhyChose's open-source extractor surfaces these initial setup sessions, performance debugging sessions, and service mesh configuration sessions as structured decision records before a P99 latency incident produces empty Jaeger search results and the engineer cannot root-cause the problem in a 40-minute incident window, before three independently configured services produce four disconnected trace fragments that are visible in the Kiali service graph but unusable for cross-service request tracing, and before a sampling rate change made to fix Cassandra disk utilization becomes the undocumented reason that the next performance incident cannot be diagnosed from traces. The decisions never written down in a distributed tracing setup are the sampling model that determines whether slow-request traces are captured or statistically invisible, the propagation format that determines whether a multi-service request produces a unified trace or a disconnected scatter plot, and the storage cost projection that determines whether the tracing system is a $50/month diagnostic tool or an $800/month storage bill that someone halves with an undocumented configuration change that removes diagnostic visibility from the P99 of the next incident.