The timeout and deadline propagation decision record: why the timeout layer model you chose determines your zombie request accumulation surface and your retry amplification failure mode

A 33-person fintech company built a payment processing platform for e-commerce merchants — order management, payment orchestration, and real-time fraud scoring. The architecture had three tiers relevant to payment processing: an API gateway that received merchant payment requests, a payment orchestration service that managed the payment lifecycle and communicated with payment provider integrations, and a payment provider integration service that wrapped the external payment provider's API and translated responses into the platform's internal format.

Each tier had its own timeout configuration. The API gateway was configured with a 30-second request timeout — the infrastructure team had set this because the platform's SLA to merchants was 30 seconds for a complete payment response. The payment orchestration service had a 25-second timeout for calls to the integration service, set to be shorter than the gateway timeout so that the orchestration layer would have time to return an error response before the gateway forced a connection close. The payment provider integration service had a 20-second timeout for calls to the external payment provider API — the provider's documentation listed a 15-second average response time for approval calls, and the engineering team had added 5 seconds of margin.

The orchestration service also had a retry policy: if the integration service returned a timeout error or a 503, the orchestration service would retry the payment request once, immediately. The retry policy had been added in month four of operations after a period of intermittent integration service timeouts caused merchant-facing errors during a payment provider maintenance window. The retry was unconditional — it did not check whether the integration service had received and begun processing the original request before timing out.

In month eighteen, the external payment provider underwent a geographic expansion that temporarily degraded their approval API's response times in the merchant's primary region. For approximately 90 minutes, the provider's approval endpoint averaged 22 seconds per response — above the integration service's 20-second timeout but below the orchestration service's 25-second timeout. The integration service timed out on every payment that reached the approval stage. On timeout, it returned a 504 error to the orchestration service. The orchestration service, following its retry policy, issued an immediate retry request to the integration service.

The original requests, however, had not been cancelled at the provider. The integration service's timeout was a client-side timeout — it stopped waiting for the provider's response after 20 seconds, but the provider had received the payment request and begun processing it. The provider's approval process takes 22 seconds during the degraded period: the integration service times out at second 20, returns a 504 to orchestration, orchestration retries, integration service sends a second approval request to the provider — and the provider approves the original request at second 22, then receives and approves the retry request at approximately second 42. Both approvals succeed. Both are logged by the provider as distinct approved transactions.

Across the 90-minute window, 34 payment requests were processed twice. The provider charged the merchant's customer accounts twice for each. The platform's own payment records showed one approval per transaction — the retry's approval — with no record of the original. The customers' bank statements showed two charges. Support tickets arrived over the following 48 hours as customers checked their statements. Each required a manual refund initiated by the merchant's finance team.

The incident review identified the root cause as the combination of a retry policy that did not account for the in-flight state of the original request at the downstream provider. The 20-second timeout at the integration service was correct: the provider's documentation said 15-second average, and 20 seconds is a reasonable ceiling. The retry at the orchestration layer was correct in isolation: if the integration service is unavailable, retrying is the right behavior. The problem was that the two decisions were made independently without considering their interaction at the specific failure mode where the integration service's timeout fires while the provider is mid-processing. The payment request to an external provider is not idempotent unless the caller specifies an idempotency key. The platform had no idempotency key on payment requests. The timeout decision and the retry decision together created the idempotency requirement that the architecture did not satisfy.

A 41-person developer tools company built a platform for engineering teams — static analysis, dependency auditing, and security scanning integrated into CI/CD pipelines. The platform's architecture was a microservices design: an API gateway received analysis requests from CI integrations, an analysis orchestrator fanned out each request to seven specialized analysis services (dependency graph builder, CVE matcher, license checker, SAST runner, secret scanner, SBOM generator, and results aggregator), and each analysis service stored results in a shared PostgreSQL database.

The API gateway was configured with a 15-second response timeout. This was the primary timeout in the system — the CI integration expected a response within 15 seconds, and beyond that the CI pipeline would move on. The analysis orchestrator had no timeout configured on its outbound calls to the analysis services: the engineering team's view was that the orchestrator's own timeout would be inherited from the gateway's connection lifetime. The analysis services had no deadline propagation mechanism: they did not read any deadline header from the incoming request and did not apply any context cancellation to their database operations or their outbound calls to external vulnerability databases.

Under normal operating conditions, the full analysis pipeline completed in 8–12 seconds for typical repositories: the orchestrator dispatched all seven analysis services in parallel, waited for all seven to complete, aggregated the results, and returned the response to the gateway within the 15-second window. For large repositories (monorepos with 400+ packages or repositories with unusual dependency graphs), the pipeline took 16–22 seconds, exceeding the 15-second gateway timeout. At those sizes, the gateway closed the connection to the CI integration client at second 15 and logged a 504 error.

When the gateway closed the connection, it terminated its outbound connection to the analysis orchestrator. The orchestrator's HTTP server detected the closed connection and returned from its handler coroutine — but the orchestrator had already dispatched the seven analysis service calls as independent goroutines. Those goroutines had no mechanism to detect that the orchestrator's handler had returned. They continued executing. Each analysis service, receiving a well-formed request, processed it fully: the dependency graph builder continued parsing 400 packages, the CVE matcher continued querying its vulnerability database, the SAST runner continued executing its rule set across the repository's codebase. For a large repository, this downstream execution lasted 40–60 seconds after the gateway had closed the client connection at second 15. All seven analysis services wrote their results to the PostgreSQL database at the end of their execution — results that the gateway had already abandoned and that the client would never receive.

At launch, the analysis platform handled 40–80 requests per minute with a large-repository rate (those exceeding the 15-second threshold) of approximately 3%. At this rate, the zombie request population was small: 3% × 60s of downstream execution × 70 rps ≈ 126 zombie requests in-flight at any given moment, each holding 7 database connections (one per analysis service). The platform's database connection pool had 200 connections — 126 zombie connections consumed 63% of the pool at steady state, but real requests handled the remaining 37% without incident.

Over 14 months, as the platform onboarded larger enterprise customers with larger repositories and higher CI pipeline frequencies, both the large-repository rate and the requests-per-minute count increased. At month fourteen, the platform was handling 210 requests per minute with a large-repository rate of 11%. The zombie request population at steady state: 11% × 50s average downstream execution × 210 rps ≈ 1155 zombie requests, each holding 7 database connections = 8085 zombie database connections. The database connection pool had been increased to 500 connections to handle the growing load. The zombie requests consumed 8085 connections — 16× the pool capacity. The database connection pool was perpetually exhausted.

Real requests could not acquire database connections. The analysis services, waiting for a connection, held their handler threads. The analysis orchestrator, waiting for responses from the analysis services, accumulated in-flight requests. The gateway, receiving responses beyond 15 seconds, fired more timeouts. More timeouts created more zombie requests. At peak, the monitoring showed 23% of all in-flight requests across the analysis services as zombie requests — requests whose upstream caller had already given up. The database P99 connection wait time was 8.4 seconds. P99 analysis latency was 19.8 seconds. The CI integrations reported a 31% pipeline failure rate attributable to the platform.

The fix required implementing deadline propagation at every layer: the gateway passed a X-Request-Deadline header containing the absolute timestamp of the 15-second deadline; the orchestrator read the header, derived a Go context.WithDeadline, passed the context to all seven goroutine dispatches, and ensured that each goroutine's database and HTTP operations used the derived context; each analysis service read the deadline header, derived its own context, and used context-aware database drivers for all queries. After the fix, every gateway timeout cancelled the corresponding downstream work within 200ms. The zombie request population dropped to near zero.

A 47-person B2B SaaS company built a project management platform for software teams — task tracking, sprint planning, milestone reporting, and integration with code repositories. The platform's backend was a Django application backed by PostgreSQL, deployed on AWS with RDS. At launch, in month one, the team configured a 100-millisecond timeout for all database queries — a value that reflected the team's observation that their most complex queries completed in 12–40ms against the development database, and that 100ms provided a generous margin while still ensuring that a hung or runaway query would not hold a connection indefinitely.

The platform's core data model centered on a tasks table: every task, subtask, comment, status update, and history event was a row in this table, with a compound index on (project_id, status, updated_at) that supported the primary list views. At launch, the table had approximately 14 million rows spread across the 60 initial customer accounts. The 100ms query timeout was never reached in production during the first year of operations — the team monitored their Datadog dashboard monthly and saw consistent P99 query times of 8–22ms.

Over the following 23 months, the platform grew from 60 to 840 customer accounts. The tasks table grew from 14 million rows to 2.3 billion rows. The compound index on (project_id, status, updated_at) was never rebuilt: index rebuilds on the live table required a maintenance window that the team had repeatedly deferred because the platform had no planned downtime. The index's fragmentation factor reached 47% by month eighteen. The statistics on the status column, initially four values with roughly equal distribution, had grown to include a resolved status that represented 91% of all rows — but the query planner's statistics had not been refreshed since month four, and the planner continued to estimate equal distribution across statuses, producing filter order decisions that were wrong for 91% of queries.

By month twenty-three, the P99 query time for the primary task list query — a paginated list of tasks for a given project, ordered by update time, with a status filter — was 310ms for the platform's 40th-percentile accounts (accounts with 1–3 million tasks) and 840ms for the 90th-percentile accounts (accounts with 8–15 million tasks). Both values exceeded the 100ms timeout. At peak load — Mondays between 9:00 and 11:00 AM when users returned to review sprint progress — the 100ms timeout fired on 12% of all task list requests.

The Django application's retry policy, added in year one to handle transient RDS connection drops, retried any database OperationalError — which included the timeout error — up to three times with a 50ms backoff between attempts. Each retry reissued the original query. A request that encountered a 100ms timeout on its first attempt would receive three additional 100ms query attempts before the Django handler returned a 500 error to the user. A single user-facing task list request that hit the timeout path issued 4 database queries: the original, plus three retries, each running 100ms and failing. The total database time consumed by a single timed-out request: 400ms of active CPU time on RDS, across 4 queries that all produced no results for the user.

At 12% timeout rate, 1200 requests per minute peak, and 4 database queries per timed-out request, the retry amplification added: 12% × 1200 rps / 60 × 3 extra queries = 72 extra queries per second against the already-struggling database. The extra queries competed with the 1056 non-timed-out requests per minute for database CPU. RDS CPU utilization at peak rose from a pre-timeout baseline of 61% to 94%. At 94% CPU, queries that would normally complete in 12ms took 50–80ms — within range of the 100ms timeout. The timeout began firing on queries that should have been fast. By 9:45 AM on a typical Monday, the timeout rate had risen from 12% to 34%, and the extra retry load was causing RDS CPU to reach 99%, where query execution was nearly halted.

The team diagnosed the issue at 10:07 AM on the second Monday it occurred. The immediate fix — increasing the query timeout from 100ms to 500ms — stopped the retry cascade: at 500ms, the task list queries completed successfully for the 40th-percentile accounts (310ms), reducing the retry amplification and allowing RDS CPU to fall below the cascade threshold. The 90th-percentile accounts still timed out at 500ms (840ms queries), but at a much lower rate that did not trigger the cascade. The incident review noted that increasing the timeout had "fixed" the symptom by eliminating the cascade, but had not addressed the underlying causes — index fragmentation, stale query planner statistics, and a retry policy that amplified load on a slow database — and that the 500ms timeout would itself become a false positive generator as the table continued to grow.

Structural properties set by the timeout and deadline propagation decision

Three structural properties are determined when a team decides — or fails to explicitly decide — how to implement timeouts in a distributed system: what the timeout layer model determines about the zombie request accumulation surface when downstream work is not cancelled on upstream timeout, what the timeout value calibration surface determines about the retry amplification gap when stale timeout values fire on operations that should succeed, and what the idempotency coupling determines about the duplicate operation failure mode when retries occur at boundaries where the downstream operation may already be in-flight. None of these properties are labeled as decisions in the conversations that produce them. The timeout layer model emerges from per-service timeout constants set by individual engineers against their service's current performance profile — without specifying whether downstream services will be cancelled when the timeout fires. The timeout value calibration surface emerges from timeout constants that are set once and never reviewed — without a schedule tied to the system's growth metrics. The idempotency coupling emerges from retry policies added to handle transient errors — without specifying the state at the downstream boundary when a timeout occurs.

Property 1: The timeout layer model and the zombie request accumulation surface. A timeout in a distributed system is a unilateral decision by the calling layer to stop waiting for a response. It does not, by itself, cancel the work that the called layer has already begun. The zombie request accumulation surface is the sum of all in-flight work across downstream layers that continues executing after an upstream timeout, consuming CPU, memory, database connections, and external API rate limits for operations whose results will never reach the client. The surface's size at steady state is: upstream timeout rate × average downstream execution time beyond the cancellation point × requests per second. A system with a 5% gateway timeout rate, 40 seconds of average downstream execution beyond the gateway, and 200 requests per second has a steady-state zombie population of 400 requests. Each zombie request holds every resource it acquired during its lifecycle — database connections, thread pool slots, external API calls — until it either completes or encounters a non-context-aware error. The zombie accumulation surface is not bounded by the number of concurrent users or the gateway capacity: it is bounded by the downstream execution time × the timeout rate, and both can grow without a corresponding increase in user load. As demonstrated by the developer tools platform, the cascade is a positive feedback loop: more zombies consume more database connections, which slow down real requests, which produce more gateway timeouts, which produce more zombies. The prevention is deadline propagation — passing the remaining time budget from the upstream layer's timeout to every downstream call, so that downstream work is cancelled when the upstream deadline expires. Without deadline propagation implemented at every I/O boundary in every service on the request path, the zombie accumulation surface grows with the platform. Connect this property to the observability strategy decision record: detecting the zombie request accumulation requires a metric that counts requests cancelled by context cancellation, not requests that complete with errors; a service that has effective deadline propagation shows a context-cancellation counter that tracks closely with the upstream timeout rate; a service without effective deadline propagation shows a near-zero context-cancellation counter regardless of the upstream timeout rate, which is the signal that indicates the zombie accumulation surface is growing.

Property 2: The timeout value calibration surface and the retry amplification gap. A timeout value is accurate when it is set at a multiple of the P99 response time for the operation it bounds — a value that fires during genuine slowness (above-P99 latency) but does not fire during normal operation. The calibration surface degrades continuously as the system scales: index fragmentation, data volume growth, changed query patterns, and shifting cardinality distributions cause P99 response times to drift upward from their launch values, while the timeout value remains fixed unless explicitly updated. When P99 response time grows to within 2–3× of the timeout value, the timeout begins firing at an above-normal rate during peak load — not because the system is experiencing genuine slowness, but because the timeout value was calibrated against a system that no longer exists. The retry amplification gap is the additional load placed on the already-slow dependency by the retry policy: if a timeout fires because a database query takes 310ms against a timeout of 100ms, the retry policy issues an additional 100ms query for the same operation — a query that will also time out, since the underlying condition (slow queries due to index fragmentation) has not changed between the original and the retry. The amplification factor is (1 + retry count) × the timeout rate: a 12% timeout rate with 3 retries creates 36% additional query load on the database beyond the real request load. At the margin, this additional load drives the database toward CPU saturation, which slows all queries, which increases the timeout rate on queries that would previously have been fast — the cascade demonstrated by the B2B SaaS. The prevention requires two independent decisions: (1) a timeout value review cadence that ties timeout values to P99 metrics at current production scale, updating timeout values when P99 reaches 30% of the timeout value; and (2) a retry policy that distinguishes between safe and unsafe retry conditions — specifically, a retry on timeout should only occur for operations where the timeout guarantees the operation was not executed (read operations, idempotent write operations with idempotency keys verified as present in the request). Connect this property to the retry strategy decision record: the retry policy must specify the conditions under which a retry is safe, and timeout is not uniformly a safe retry condition; a timeout on a write operation is only safe to retry if the write operation carries an idempotency key and the downstream service enforces idempotency on that key; the timeout and retry decisions are interdependent and must be documented together, with the timeout conditions that trigger retries explicitly classified as safe or unsafe based on the idempotency properties of the operation at the retry boundary.

Property 3: The idempotency coupling and the duplicate operation failure mode. A timeout at a layer boundary does not reveal whether the downstream operation was received and processed by the downstream service before the timeout fired. The client-side timeout is a failure of response delivery, not a guarantee of operation non-execution. A payment approval request that reaches the payment provider at second 0 and receives no response by second 20 (the integration service's timeout) may have been approved at second 19: the approval was processed, the funds were moved, the record was created — but the response was delayed or lost in transit. When the orchestration layer retries the payment request after the integration service timeout, it sends a second approval request to the provider for the same payment. If the provider does not enforce idempotency on payment requests — returning the original result for duplicate requests rather than processing the request twice — the second request creates a second charge. The idempotency coupling is the structural relationship between the timeout model and the idempotency requirement: the timeout model determines which operations are retried and under what conditions; the idempotency requirement is the mechanism that makes those retries safe. The duplicate operation failure mode occurs when the retry boundary crosses a non-idempotent operation. The timeout and deadline propagation decision must enumerate every operation that may be retried when a timeout fires and classify each as: (a) read-only (always safe to retry), (b) idempotent write with idempotency key enforced at the downstream service (safe to retry), or (c) non-idempotent write or idempotent write without downstream enforcement (unsafe to retry without additional safeguards). For category (c), the decision must specify either a deadline propagation alternative (cancel the request rather than retry, forcing the client to restart from a clean state) or an idempotency implementation requirement (the downstream service must be upgraded to enforce idempotency before the retry policy is applied). Connect this property to the API idempotency decision record: the idempotency requirement at the retry boundary is not an API design preference — it is a correctness requirement imposed by the timeout model; if the timeout model allows retries on write operations, the API idempotency decision record must specify idempotency key enforcement for those operations, and the timeout decision record must reference the idempotency record as a dependent decision; together they form a correctness constraint that neither record can satisfy alone.

The timeout and deadline propagation decision ADR: five sections

Section 1: Timeout layer inventory and the layered timeout model. Begin the timeout and deadline propagation decision record by enumerating every layer in the system that has an independently configured timeout and specifying the relationship between those timeouts. The layer inventory must be exhaustive: API gateway timeout, load balancer idle timeout, service-to-service call timeout, database query timeout, external API call timeout, message queue consumer acknowledgment timeout, and any background job execution timeout. For each layer, document the current configured value, the P99 response time of the operation it bounds (measured at current production scale), and the ratio between the timeout value and P99 (a healthy ratio is 3–5×; a ratio below 2× indicates a miscalibration risk). Document the layering constraint: the timeout at each layer must be shorter than the timeout at the layer above it. The layering direction is: client timeout > gateway timeout > service-to-service timeout > database query timeout. This ordering ensures that inner-layer timeouts fire before outer-layer timeouts, giving the inner layer time to clean up, release resources, and return a graceful error response before the outer layer forcibly closes the connection. A common violation is configuring service-to-service timeouts equal to or longer than the gateway timeout: when the gateway fires before the service-to-service call times out, the gateway terminates the connection while the service-to-service call is still in-flight, producing a resource leak. Specify the layering margins explicitly: "the gateway timeout is 30 seconds; service-to-service timeouts must be no more than 25 seconds; database query timeouts must be no more than 20 seconds." These margins are not arbitrary — they are the recovery windows that each layer has to handle its timeout cleanly before the outer layer fires. Document the decisions that set each timeout value: what P99 data was used, at what data volume, at what traffic level, and on what date. This documentation is the calibration record that enables future review. Connect to the circuit breaker resilience decision record: the circuit breaker threshold at each service boundary should be calibrated against the timeout values at that boundary; if the service-to-service timeout is 5 seconds, the circuit breaker should open when more than 10–20% of calls exceed 5 seconds within a rolling 30-second window — this prevents the calling service from queueing up in-flight timeout-wait threads across many concurrent requests while the circuit is not yet open.

Section 2: Deadline propagation model and context cancellation. Specify the deadline propagation model that governs how the remaining time budget is passed from one layer to the next across every service boundary. The deadline propagation model must answer three questions: (1) how is the deadline passed in outbound requests; (2) how is the deadline consumed by the receiving service to bound its own outbound calls; and (3) how is context cancellation wired into every I/O operation so that downstream work is actually terminated when the deadline expires. For gRPC services, specify that the calling code derives a child context from the incoming request context with context.WithDeadline(incomingCtx, deadline) and passes the child context to all outbound gRPC calls — this propagates the remaining deadline through the gRPC framework automatically and enables server-side cancellation when the client's deadline expires. For HTTP services, specify the deadline header convention: X-Request-Deadline carrying an ISO 8601 timestamp, or X-Request-Timeout carrying the remaining milliseconds. Specify that the receiving service computes the effective timeout as min(service_configured_timeout, incoming_deadline - current_time) — the remaining budget, not the service's configured timeout — and applies it as a local context deadline. Without this subtraction, a downstream service with a 5-second configured timeout that receives a request with 2 seconds remaining will issue its own outbound calls with a 5-second deadline, allowing downstream work to continue for 3 seconds after the upstream caller has already abandoned the request. For context cancellation at I/O boundaries: specify that every database query must use a context-aware driver method (pgx.QueryContext, asyncpg.execute with a coroutine that is cancelled on context cancellation, JDBC's PreparedStatement.setQueryTimeout); every external HTTP call must use an HTTP client configured with context cancellation; every cache read/write must pass the request context. Specifying deadline propagation at the architecture level without also specifying context-aware I/O at the implementation level is incomplete: a service that reads the incoming deadline header, derives a local context, but issues its database queries without that context does not cancel its database queries when the deadline fires, and the zombie accumulation surface persists at the database layer even after the service layer correctly propagates the cancellation. Document the enforcement mechanism: a linter or static analysis rule that detects context-less database calls is the only scalable way to ensure that new code does not introduce zombie I/O paths. Connect to the health check design decision record: the service's readiness probe should verify that the deadline propagation middleware is loaded and functioning — a service that has deployed without its deadline propagation middleware is functionally equivalent to a service that never had deadline propagation, and the readiness probe is the earliest detection point for a missing middleware registration.

Section 3: Timeout value calibration methodology and review cadence. Specify the timeout value calibration methodology and the cadence at which timeout values are reviewed against current production metrics. The calibration methodology defines the initial value for a new timeout: measure the P99 and P99.9 response time for the operation at the current production scale, then set the timeout at 3–5× P99 (not P99.9 — P99 is the appropriate baseline because P99.9 includes genuine outliers that the system should time out, while P99 represents the upper bound of normal operation). Separate timeout values by operation class within each layer: a database query timeout should not be a single global constant; it should be at minimum two values — one for simple lookups (primary key, small index scans) and one for complex queries (aggregations, full-table-range scans, report queries). A single global query timeout will either be too short for complex queries or too long for simple queries: the right timeout for a primary-key lookup is measured in milliseconds; the right timeout for a report-generation query may be seconds. Specify the review cadence as a calendar trigger and a metric trigger. The calendar trigger: review timeout values quarterly and whenever a database is migrated to a new data volume tier (e.g., when the primary table crosses 100M rows, 1B rows, or 5B rows). The metric trigger: review timeout values whenever the measured timeout rate for an operation class exceeds 0.5% at normal traffic levels — this rate indicates that the current timeout value is tight relative to the current P99, and the system is one traffic spike away from the cascade demonstrated by the B2B SaaS. Specify the recalibration procedure: when a timeout rate exceeds 0.5%, measure the current P99 for the affected operation class, calculate the new timeout value at 3–5× P99, verify that the new value maintains the layering constraint (still shorter than the layer above), update the configuration, and document the recalibration in the ADR with the date, the triggering metric value, the old timeout, and the new timeout. Require that the recalibration also examine whether the underlying P99 growth reflects a genuine operational inefficiency (index fragmentation, missing statistics, query plan regression) that should be fixed rather than accommodated by a larger timeout — increasing a timeout is a mitigation, not a fix, and the ADR should track whether the recalibration has been accompanied by a root cause investigation. Connect to the database query optimization decision record: the database query timeout recalibration is closely coupled to the query optimization cycle; a query that requires a timeout increase is a query whose execution plan should be reviewed for optimization opportunities; the timeout recalibration event should trigger a query profile review for the affected operation class before the new timeout value is deployed, so that the optimization opportunity is not deferred indefinitely by a series of increasing timeout accommodations.

Section 4: Retry coupling and idempotency requirements at timeout boundaries. Specify the retry policy's relationship to the timeout model at each retry boundary, classifying each operation that may be retried on timeout as safe or unsafe. The classification criteria: a safe retry on timeout is an operation where the timeout guarantees that the downstream operation was not executed — the canonical example is a read-only database query where an execution timeout means the query was cancelled before completion, never producing a result; a read-only operation retried after a timeout repeats the same read, not a different write. An unsafe retry on timeout is an operation where the timeout does not guarantee non-execution at the downstream service — any write, state-change, external API call, or payment operation falls into this category unless the downstream service enforces idempotency. For each operation classified as unsafe for timeout-triggered retry, the decision must specify one of: (a) idempotency key enforcement — the caller includes an idempotency key in the request, the downstream service returns the original result for duplicate requests with the same key, and the timeout-triggered retry is safe because the duplicate will return the same result; (b) deadline propagation instead of retry — when the caller's deadline expires, propagate the cancellation upstream and allow the client to restart the operation from scratch with a new idempotency key; (c) compensating transaction — if neither idempotency enforcement nor deadline cancellation is feasible, specify the compensating transaction that detects and reverses duplicate operations (as used by payment processors who detect duplicate charges via transaction deduplication windows). Document the idempotency enforcement requirements for every upstream service that retries on timeout, naming the downstream service's idempotency mechanism — the Stripe idempotency key header convention, gRPC's request-level idempotency token, or a custom deduplication window — and the expiration policy for idempotency records. Specify the idempotency window duration: idempotency keys must be retained for at least 2× the maximum timeout at the retry boundary, so that a retry that arrives at the downstream service at any point within the timeout window will find the original result still cached. Connect to the API idempotency decision record: the idempotency key design and the idempotency record retention policy are downstream consequences of the timeout and retry model; the idempotency record must be written before the non-idempotent operation begins (not after completion, as a post-hoc record) so that a timeout that fires after the record is written but before the operation completes will return the in-progress state to the retry, not execute the operation a second time; the write-ahead idempotency record pattern is the specification that makes timeout-triggered retries safe for write operations.

Section 5: Timeout monitoring, alert model, and incident response coupling. Specify the monitoring and alerting model for timeouts across all layers, including the metrics, the alert thresholds, the escalation paths, and the on-call response playbook for timeout incidents. The metrics: a timeout counter per operation class per layer, a timeout rate (timeouts / total requests, rolling 5-minute window), a zombie request population estimate (calculated as: upstream timeout rate × downstream average execution time beyond the cancellation point × rps), and a retry amplification factor (total database queries / user-facing requests, which exceeds 1.0 when retries are firing). The alert thresholds: alert at P2 (page on-call engineer) when any layer's timeout rate exceeds 1% for more than 5 minutes at normal traffic; alert at P1 (page on-call + engineering lead) when timeout rate exceeds 5% or when the zombie population estimate exceeds 20% of the connection pool capacity. These thresholds are designed to catch the cascade early: the B2B SaaS's cascade began at 12% timeout rate; an alert at 1% would have surfaced the P99 drift 6–8 weeks before the cascade was visible in user-facing latency metrics. The alert at 1% also catches the initial miscalibration that creates the condition for a cascade: a newly deployed service with a too-tight timeout manifests as a 1–3% timeout rate immediately, not a 12% rate after months of growth. Specify the on-call response playbook for a timeout alert: (1) identify the operation class with the elevated timeout rate; (2) measure the current P99 for that operation class against the configured timeout value — if P99 is within 3× of the timeout, recalibrate the timeout; (3) check the retry amplification factor — if greater than 1.2, disable retries on timeout for the affected operation class while the recalibration is applied; (4) check the zombie population estimate — if above 10% of the connection pool, verify that deadline propagation is active on the affected service path; (5) check the circuit breaker state for the affected dependency — if the circuit is not open but the timeout rate is above 10%, evaluate whether the circuit threshold needs adjustment. The playbook must specify the rollback path for a timeout miscalibration: if a deployed timeout change increases the cascade (timeout rate rises after the change), the mitigation is to revert to the previous timeout value and investigate the underlying P99 growth, not to increase the timeout further. Connect to the alerting threshold decision record: the timeout rate alert thresholds belong in the same alerting threshold decision record as the error rate and latency alerts; the 1% threshold is a P2, not a P3, because a sustained 1% timeout rate indicates a system approaching a cascade boundary, not a transient anomaly; the escalation path from timeout alert to the engineer who owns the affected service's query performance is a handoff that must be specified in the alerting threshold decision to avoid the alert reaching the on-call generalist who has no context on the query optimization history.

FAQ

Further reading

• Retry strategy decision record — the retry conditions, the backoff model, and the maximum retry count that determine how many additional requests each timeout generates; retry-on-timeout is only safe for idempotent operations, and the retry strategy decision must classify each retried operation class by idempotency to prevent the amplification pattern where a stale timeout value plus a retry policy drives a database toward saturation.
• Circuit breaker resilience decision record — the circuit breaker threshold model, the recovery window, and the half-open state behavior that together prevent timeout accumulation cascades; the circuit breaker and timeout values are calibrated together — the circuit should open at a timeout rate that is 10–20× the normal baseline, not at an absolute error count that ignores the relationship between the threshold and the system's current load.
• API idempotency decision record — the idempotency key design, the deduplication window, and the write-ahead record pattern that make timeout-triggered retries safe for write operations; the idempotency requirement at each retry boundary is not an API preference but a correctness constraint imposed by the timeout model, and the two decisions must reference each other to be complete.
• Health check design decision record — the readiness probe behavior when a circuit breaker is open and the liveness probe behavior when a service is accumulating zombie requests; a service whose deadline propagation middleware is absent or misconfigured should fail its readiness probe to route traffic away before the zombie accumulation reaches the connection pool exhaustion threshold.
• Observability strategy decision record — the metrics instrumentation and the distributed trace sampling model that surface timeout rates, zombie population estimates, and retry amplification factors; the timeout and deadline propagation decision relies on the observability strategy's metrics for its calibration reviews and its cascade detection alerts — the two decisions share the same instrumentation dependency and should be reviewed together when the observability platform is updated.
• Open-source extractor — find the timeout and deadline propagation decisions buried in your AI chat history: the infrastructure session where gateway and service timeouts were set to "reasonable values" without specifying the layering constraint, the retry policy session that added retry-on-timeout without examining the idempotency requirements, and the performance optimization session where a timeout was increased as a mitigation without investigating the underlying P99 growth that made the original value a false positive generator.