The retry strategy decision record: why the retry policy you chose determines your thundering herd failure surface and your cascading overload behavior under partial outages

A 35-person API platform company built a data enrichment service that called three third-party geocoding and demographic data APIs on behalf of its customers. In the early months, each API call was a simple synchronous request with a 5-second timeout and no retry logic. The engineering team observed that API calls failed 1–3% of the time due to transient network errors or brief upstream unavailability, and these failures surfaced to customers as error responses. In a sprint review, the decision was made to add retry logic so that transient failures were handled transparently. A senior engineer added retries to the HTTP client library. The implementation: on any 5xx response or network exception, sleep 1 second and retry, up to 3 times. The code was reviewed, merged, and deployed. The retry interval was fixed at 1 second because that was the example in the HTTP client library's documentation. No one discussed jitter. The team had not previously encountered a thundering herd. The retry policy was not written down as an architectural decision — it was a code change in a single pull request.

Eight weeks after the retry logic was deployed, the primary geocoding API experienced a partial outage. The API's database had a slow disk causing query latencies to spike from 120 milliseconds to 4,500 milliseconds on 65% of requests. Requests that reached the 5-second timeout were failing. The enrichment service was handling 400 concurrent requests at the time the degradation began. Each of the 260 failing threads — 65% of 400 — waited for their 5-second timeout, received an error response, and then slept for exactly 1 second before retrying. The retry logic had no randomization. One second after the first wave of failures, all 260 retrying threads sent their first retry simultaneously. The geocoding API, which had just begun to recover as its database checkpoint completed, received a synchronized burst of 260 requests in under 100 milliseconds. The burst overwhelmed the recovering database, causing it to fall behind again. The process repeated at seconds 1, 2, and 3 — each retry wave prevented the recovery that would otherwise have completed in the interval before the next wave. The outage lasted 22 minutes instead of the 4 minutes it would have taken for the database to recover without the synchronized retry amplification. The enrichment service's customers saw errors for the full 22 minutes, including the 14 minutes that would have been available for transparent recovery if the retries had been desynchronized.

The post-incident review identified the fixed retry interval immediately. An engineer changed the sleep to 1 + random(0, 1) seconds. The technical fix took 15 minutes. The design discussion that followed took two hours: why had the retry policy been implemented without considering concurrent retry behavior, what was the correct backoff formula, was a retry budget needed, and was there a way to detect when retries were amplifying load rather than absorbing transient failures. None of these questions had been asked during the original implementation because the original session had been a code review thread about transient failure handling, not an architectural decision record about the retry policy, its failure modes, and its interaction with concurrent request load.

A 22-person fintech company built a B2B invoicing platform that charged customers via Stripe at the end of each billing cycle. The payment processing flow was: retrieve the invoice from the database, calculate the total, call the Stripe charge API, record the charge result. The engineering team decided to add retry logic to the Stripe API call to handle intermittent network failures — Stripe's API occasionally returned network timeouts during high-traffic periods, and a failed charge required a manual reconciliation step that took 20 minutes per incident. The implementation added retry on network timeout: if the Stripe call raised a timeout exception, retry up to 2 times with a 2-second backoff. The code was reviewed and merged. The review noted that Stripe supported idempotency keys but observed that the team's payment wrapper had been calling Stripe without them since the beginning, and that no duplicate charges had occurred in 14 months of operation. The session that added the retry did not produce a written analysis of whether the Stripe charge operation was safe to retry.

Three months after the retry was deployed, a customer reported that their credit card had been charged twice for the same invoice — two identical charges two seconds apart on the same day. Investigation revealed that the original Stripe API call had succeeded: Stripe had received the charge request, debited the card, and recorded the transaction. The response transmission had then failed due to a network partition between the enrichment platform's data center and Stripe's API endpoint, causing the caller to receive a timeout exception after 10 seconds. The retry logic, seeing the timeout exception, had treated the operation as failed and retried with the same parameters and no idempotency key. Stripe received a second charge request that was syntactically identical to the first but had no idempotency key to deduplicate against the already-completed transaction. Stripe processed it as a new charge and charged the card again. The customer discovered the duplicate 11 days later when their accounting team reconciled their statement. The refund took 3 days to process.

Audit of the payment log revealed 13 duplicate charges over the three months since retry was enabled, all from the same timeout-on-success failure mode. Twelve had been refunded proactively after the customer's accounting team noticed — the 13th was discovered only during the post-incident audit and the customer had not noticed, or had noticed and had not reported it. The fix required adding Stripe idempotency keys: a UUID generated per invoice at invoice creation time and stored in the invoice record, passed in every Stripe API call for that invoice. With an idempotency key, Stripe deduplicates the call on its side and returns the original charge response rather than processing a new charge. The fix took 4 hours to implement and required a backfill to add idempotency keys to existing invoices. The founding payment processing session had documented "charge customer via Stripe at billing cycle end." The session that added retry had not produced an idempotency analysis specifying which downstream operations were safe to retry, whether Stripe's API was idempotent, or what mechanism was required to make it safe to retry charge operations on timeout.

A 60-person e-commerce company built a recommendation engine that called the user profile service on every page load to fetch the user's purchase history, browse history, and preference signals for personalization. The user profile service was a shared dependency for six product teams. The recommendation engine team added a 30-second timeout to the user profile service calls — the reasoning was that a 30-second wait was the maximum a user would tolerate before the page load failed visibly — and 3 retries on 5xx responses and timeouts, configured in the recommendation engine's HTTP client settings. The thread pool for the recommendation engine's request handlers was set at 200 threads, sized for peak traffic of 40 requests per second with a normal latency of 450 milliseconds per user profile call. The timeout (30 seconds) was set by the backend team. The retry count (3) was set in a separate pull request by a different engineer two weeks later. The thread pool size (200) had been set by the DevOps team six months earlier and had not been revisited. None of the three numbers had been multiplied together and compared to the thread pool capacity.

Four months after the retry configuration was deployed, the user profile service experienced a storage degradation — a disk throughput contention between two data-intensive background jobs. Latency for 40% of user profile requests spiked to 28 seconds, just under the 30-second timeout. The requests that crossed the 30-second threshold timed out and were retried. The 40% that were slow but not timed out were holding threads for up to 28 seconds each instead of the normal 450 milliseconds. The threads that had timed out retried immediately, holding their threads for another 30-second window. The maximum thread hold-time was 30 seconds × (1 + 3) = 120 seconds. At 40 requests per second with 40% degradation, 16 requests per second were holding threads for up to 120 seconds each. At t=4 minutes, the 200-thread pool had been fully consumed: 16 requests per second × 60 seconds per minute × 4 minutes × 0.5 average occupancy ≈ 192 threads occupied, and the remaining 8 threads were consumed within another 30 seconds. New incoming requests found no available thread. The recommendation engine's load balancer marked its instances as unresponsive. All page loads that required personalization fell back to the default (non-personalized) ranking, which degraded the click-through rate. The degradation lasted 31 minutes — 9 minutes longer than the user profile service's storage contention itself, because the thread pool remained exhausted for 9 minutes after the user profile service recovered as the stuck threads worked through their timeout windows.

The fix required three coordinated changes: reducing the per-request timeout from 30 seconds to 3 seconds (acceptable because user profile service p99 latency was 1.2 seconds under normal conditions), reducing retry count from 3 to 1, and adding a retry budget that capped retrying threads at 10% of the pool. Post-fix max_hold_time dropped from 120 seconds to 6 seconds. The thread pool exhaustion calculation at the new parameters: 200 × 0.9 ÷ (40 × 0.4 × 3) = 180 ÷ 48 = 3.75 seconds before the non-retrying threads exhausted under 40% failure. The retry budget and the lower timeout together ensured that the circuit breaker, which opened after 50% failure rate over a 10-second window, would open in 10 seconds and shed load before pool exhaustion became complete. The founding recommendation engine session documented "personalization calls user profile service on each page load." The timeout, retry count, and thread pool configuration were set across three separate pull requests by three different engineers, none of whom had been asked to calculate the thread hold-time product or size it against the pool.

Structural properties set by the retry strategy decision

Three structural properties are determined when a team decides how to handle failures in service-to-service or service-to-external-API calls. None appear explicitly in the sessions that implement retry logic — they are the operational consequences of choices made under the pressure of improving reliability without reducing availability, a goal that retry achieves under low-failure-rate conditions and undermines under high-failure-rate conditions.

Property 1: The retry policy model and the thundering herd failure surface. A retry policy has five parameters: the retry count (maximum number of attempts after the first failure), the backoff algorithm (how long to wait between retries), the jitter model (how to desynchronize concurrent retriers), the retry budget (the maximum fraction of concurrent requests that may be active retries at any moment), and the retryable condition set (which failure modes trigger a retry versus an immediate failure). The first four parameters interact to determine the thundering herd surface.

Fixed-interval retry without jitter produces synchronized waves. If N concurrent requests fail simultaneously and all sleep for interval d before retrying, the downstream service receives N requests at d, 2d, and 3d — each wave is the full original load. A recovering service that can absorb 60% of normal load recovers between waves if the interval is long enough, but each wave risks resetting the recovery. Fixed-interval retry is correct only when a single caller is retrying in isolation, which is almost never the production condition.

Exponential backoff without jitter reduces the retry rate over time but does not desynchronize concurrent retriers. If 400 threads all fail at t=0 and all use sleep = base × 2^attempt with the same base, they all wake up at t=base, then at t=3×base, then at t=7×base. The waves are spaced farther apart than fixed-interval but each wave is still 400 simultaneous requests. Exponential backoff with full jitter uses sleep = random_between(0, min(cap, base × 2^attempt)). A population of N concurrent retriers using full jitter generates arrivals that are approximately uniformly distributed across the backoff window, reducing peak retry load from N per instant to approximately N / window_seconds per second — a rate that a recovering service can absorb if the window is sized against the recovery time.

The retry budget is the second-order protection. Even with full jitter, a prolonged downstream outage causes the retrying population to grow as new requests arrive and fail, each joining the retry queue. Without a budget, 100% of concurrent requests eventually become retries, and the cumulative retry load exceeds the original request load by the retry count factor. The retry budget caps the retrying fraction: when more than the budget fraction (typically 10–20%) of concurrent requests are active retries, new requests are failed immediately with a client-side error rather than queued for retry. This ensures that the retry load amplification is bounded by the budget fraction regardless of how long the outage lasts. The circuit breaker decision record documents the companion pattern: the circuit breaker opens when the downstream failure rate exceeds a threshold and stops all outbound requests (not just the retrying fraction) for a cooling-off period, allowing recovery without any retry load. The circuit breaker is the correct response to a sustained outage; retry is the correct response to a brief transient failure. The retry ADR must specify the threshold (failure rate or error count over a window) at which the circuit breaker replaces the retry policy as the failure response.

Property 2: The idempotency requirement and the safe retry surface. Retry is safe when the downstream operation produces the same observable outcome whether executed once or twice. The safe retry surface is the set of operations where this guarantee holds. Read operations are unconditionally in the safe surface — a database SELECT or HTTP GET that returns data produces the same data whether called once or ten times (subject to consistency model, but without the duplicate-execution problem). Write operations are in the safe surface only if the operation is idempotent by design or an idempotency key is in use.

Natural idempotency covers operations where the state transition is the same on the first and second execution. Setting a configuration value to a specific constant is naturally idempotent — executing it twice produces the same final state. Deleting a resource that returns 404 on a second attempt is naturally idempotent — both executions produce the intended absence of the resource. Inserting a row with a unique constraint is conditionally idempotent — the second insertion fails with a constraint violation rather than creating a duplicate, and the caller can treat the constraint violation as equivalent to success. These operations require error handling for the duplicate response but not an idempotency key.

Synthetic idempotency via an idempotency key covers operations that are not naturally idempotent — POST requests that create new resources with server-generated IDs, financial transactions, message sends. The idempotency key is a UUID generated by the caller at the start of the logical operation and passed in every retry of that operation. The server stores the operation result keyed on the idempotency key and returns the stored result on subsequent requests with the same key rather than re-executing the operation. The API idempotency decision record documents the server-side idempotency key implementation; the retry decision record must verify that the idempotency key is implemented for every downstream operation in the retryable set before retry is enabled for that operation. The idempotency analysis must answer three questions: does the downstream service implement idempotency key semantics for this operation, is the idempotency window (the retention period for stored results) longer than the maximum retry window (per_request_timeout × retry_count), and is the duplicate detection behavior of the server distinguishable from the original-failure behavior (a duplicate idempotent call should return the stored success response, not an error that the caller would interpret as a new failure requiring another retry).

The payment processor decision record documents the idempotency requirements for payment processing integrations specifically; payment APIs from Stripe, Adyen, Braintree, and most other providers implement idempotency keys, and the retry ADR for payment integrations must specify the idempotency key generation strategy (UUID per invoice, per order, or per charge attempt), the key storage location (in the calling service's database so that a service restart can retry the same logical operation with the same key), and the key scope (client-specific, to prevent key collisions between different callers of the same downstream API). For operations where idempotency keys are not available — legacy APIs, third-party services with no idempotency support — the safe retry approach is to use an async queue with at-most-once delivery semantics instead of synchronous retry. The queue and messaging decision record documents the infrastructure for async retry via message queue; guaranteed-delivery queues with dead-letter queues and at-least-once redelivery are the correct model for non-idempotent operations that must be retried, because the queue's redelivery mechanism can be combined with a server-side idempotency check in the consumer rather than requiring the idempotency guarantee from the downstream API.

Property 3: The timeout × retry count product and the thread pool hold-time. The timeout and the retry count are the two parameters that determine how long a single thread is held by a failing request. The maximum hold-time is per_request_timeout × (1 + retry_count): the initial attempt times out (duration: per_request_timeout), each subsequent retry times out (duration: per_request_timeout each), for a total of one initial attempt plus retry_count retries. This is the worst case — when every attempt times out rather than receiving an error response quickly. Backoff intervals add to this, but for resource consumption the timeout product dominates because the thread is occupied during the timeout wait, whereas during the backoff sleep the thread may be released (in non-blocking I/O frameworks) or held (in thread-per-request models).

Thread pool exhaustion occurs when the number of threads holding connections for the maximum hold-time equals the pool size. The time to exhaustion under a partial outage with failure fraction f is approximately pool_size ÷ (request_rate × f × max_hold_time). At a 30-second timeout, 3 retries, 40-request-per-second throughput, 40% failure fraction, and a 200-thread pool: 200 ÷ (40 × 0.4 × 120) = 200 ÷ 1920 = 0.1 seconds. The pool exhausts in one-tenth of a second — effectively instantaneously relative to any human reaction time. The correct design parameters for the retry policy must be set jointly with the timeout and the thread pool size, not independently.

The practical resolution is not to eliminate retries but to constrain the max_hold_time to a value that produces an acceptable pool exhaustion time and to use the retry budget to cap the retrying fraction of the pool. If the retry budget limits retrying threads to 10% of the pool (20 threads in a 200-thread pool), the remaining 90% of threads (180 threads) handle requests that succeed or fail quickly. The exhaustion calculation for the non-retrying pool fraction uses only the normal-operation timeout, not the max_hold_time: 180 threads handle requests at normal timeout (e.g., 3 seconds), so under the 40% failure rate with immediate failure (no retry, just fail fast), the pool handles 180 ÷ (40 × 0.4 × 3) = 180 ÷ 48 = 3.75 seconds before exhausting. With a 3.75-second exhaustion time for the non-retrying fraction, the circuit breaker (which should open within 10 seconds of detecting a 40% failure rate) will open before the non-retrying pool exhausts, shedding load. The database connection pooling decision record documents the same timeout × pool capacity product for database connections; the analysis is identical in structure — hold-time determines effective pool throughput, and the pool must be sized against the worst-case hold-time under the retry policy, not the normal-operation hold-time.

What the founding session records and what it omits

The founding retry session typically records the symptom (transient failures visible to customers, manual reconciliation required for timed-out calls) and the solution (add retry to the HTTP client, use a backoff interval). It may record the retry count and the backoff base interval if these were debated. What it does not record is the failure classification model distinguishing retryable from non-retryable conditions, the jitter formula and the reasoning for choosing it over fixed-interval or non-jittered exponential backoff, the retry budget and the load amplification calculation that justifies the budget fraction, the idempotency analysis for each operation in the retryable call set, or the max_hold_time calculation and the pool sizing check against the worst-case outage scenario.

The omissions are consequential for different failure modes. The absence of jitter produces synchronized thundering herds during partial outages — the failure mode is invisible during low-concurrency testing and appears only when the concurrent request count is high enough for the synchronized waves to overwhelm the recovering service. The absence of an idempotency analysis produces duplicate side effects — the failure mode appears only when the timeout-on-success case occurs, which is rare enough to go undetected for months and severe enough (duplicate financial transactions, duplicate emails sent, duplicate records created) to cause business harm when it occurs. The absence of the max_hold_time calculation produces thread pool exhaustion during partial outages — the failure mode is invisible in testing because tests typically don't simulate the product of timeout duration and concurrent thread count at the scale required to exhaust the pool in seconds.

The retry strategy decision record does not need to be exhaustive. It needs to answer five questions: which failure conditions trigger retry (the retryable condition set), what is the backoff algorithm including the jitter formula and the retry budget (the retry policy), which downstream operations are in the safe retry surface and what idempotency mechanism is required for each non-read operation (the idempotency analysis), what is the max_hold_time and the pool exhaustion time under the worst-case outage scenario (the thread budget calculation), and at what failure rate does the circuit breaker replace retry as the failure response (the circuit breaker threshold). Five answers written down in the founding session avoid the thundering herd, the duplicate charge, and the thread pool exhaustion that each trace back to a retry configuration parameter that was set without the surrounding calculation.

The WhyChose decision extractor finds the founding retry sessions in your ChatGPT and Claude export — the "should we add retry to this client?" thread, the "why did we set the timeout to 30 seconds?" conversation, the "what backoff interval should we use?" research session. It extracts the decision and the trade-off that was actually considered, not the surrounding debugging context that buries the parameter choice in twenty messages about whether the failure was the client's fault or the server's.

The five ADR sections for a retry strategy decision

Section 1: Retry policy specification — retry count, backoff algorithm, jitter formula, and retry budget. Specify the retry count: the maximum number of retry attempts after the first failure. The retry count is not a free parameter — it is bounded by the max_hold_time calculation in Section 4. A retry count of 3 with a 30-second timeout produces a 120-second max_hold_time; a retry count of 1 with a 3-second timeout produces a 6-second max_hold_time. Specify the backoff algorithm: exponential backoff with full jitter is the correct default for all services where multiple concurrent callers may fail simultaneously — this is the normal condition in production. The formula is sleep = random_between(0, min(cap, base × 2^attempt)), where base is the initial backoff (e.g., 100ms for internal services, 1s for external APIs), cap is the maximum sleep duration (e.g., 30s), and attempt is the zero-indexed attempt number. Specify the retry budget: the maximum fraction of concurrent requests that may be active retries at any moment. The budget is enforced by a token bucket or a counter shared across the request handler threads — when the retrying fraction exceeds the budget, new retry attempts are rejected immediately (not queued). The budget protects the downstream service from load amplification during prolonged outages; a 10–20% budget is appropriate for most services. Specify the retryable condition set in Section 2. Document the interaction between the retry policy and the circuit breaker: the circuit breaker threshold (failure rate over a window) at which the circuit opens and retry is superseded, the half-open probe interval, and the success threshold required to close the circuit. The circuit breaker decision record documents the circuit breaker parameters separately; the retry ADR must reference the circuit breaker configuration and specify the handoff: when the circuit is open, retry is not attempted — the circuit open state is the global signal that retrying individual requests is not useful and that load shedding is required.

Section 2: Failure classification model — retryable versus non-retryable conditions. Classify each failure mode that the caller may encounter as retryable (transient, likely to succeed on retry) or non-retryable (permanent, will not succeed on retry regardless of delay). Non-retryable conditions include: 4xx client errors except 429 (the request is malformed or unauthorized — retry will receive the same error); business logic errors that indicate the operation cannot proceed regardless of retry (insufficient funds, duplicate resource conflict, validation failure); and any error that indicates a problem with the caller's request rather than with the downstream service's availability. Retryable conditions include: network timeouts (the downstream service may have been temporarily overloaded or the network experienced a brief interruption); 429 Too Many Requests with a Retry-After header (the downstream service has rate-limited the caller — retry after the specified delay, which is the server's own backoff recommendation); 503 Service Unavailable (the downstream service is temporarily overloaded or in a rolling deployment); and connection errors that indicate the downstream service was not reached at all (not errors that indicate the service was reached and rejected the request). The error handling strategy decision record documents the error classification model for the broader application; the retry ADR's failure classification must be consistent with the application's error hierarchy and must specify the concrete HTTP status codes, exception types, and gRPC status codes that map to each retry category. Specify the timeout behavior: a network timeout is retryable only if the idempotency analysis in Section 3 confirms the operation is safe to retry — a timeout does not mean the operation failed, it means the caller did not receive a response, and the operation may have completed successfully on the server before the response was lost. Document the non-retryable-by-default policy: any operation where the idempotency analysis is incomplete or uncertain must be classified as non-retryable until the analysis is complete. The classification must be reviewed when the downstream API's behavior changes — a new error code, a changed meaning for an existing status code, or a new operation added to the retryable call set.

Section 3: Idempotency analysis — safe retry surface and idempotency key implementation. Enumerate every downstream operation that is in the retryable condition set from Section 2. For each operation, classify it as naturally idempotent (the same outcome on first and second execution without additional mechanism), conditionally idempotent (the duplicate execution fails with a detectable error that the caller handles as equivalent to success), or non-idempotent (the duplicate execution produces a second side effect — a second charge, a second row, a second email). For each non-idempotent operation, specify the idempotency mechanism required before retry is enabled. The primary mechanism is the idempotency key: a UUID generated by the caller at the start of the logical operation, stored in the caller's database with the operation's parameters, and passed in every attempt and retry. The idempotency key must be stable across service restarts (stored in the database, not in memory) and must persist for longer than the maximum retry window. The server must implement idempotency key deduplication: on receipt of a request with a key that matches a previously completed operation, the server returns the stored response without re-executing the operation. Document the idempotency window for each downstream API: the duration for which the server retains idempotency keys. If the idempotency window is shorter than the maximum retry window, the retry count or timeout must be reduced to fit within the window, or async retry via a durable queue must replace synchronous retry. The API idempotency decision record documents the server-side key implementation; the retry ADR documents the caller-side key lifecycle: generation, storage, transmission, and expiration. For operations where the downstream API does not support idempotency keys, specify the alternative: either async retry via a durable queue with consumer-side duplicate detection using the distributed locking pattern to serialize concurrent retriers on the same operation, or removal of the operation from the retryable set (the caller receives the timeout error and handles it as a non-retryable failure, triggering a compensating action or a manual review). Do not add retry to a non-idempotent operation with no idempotency mechanism — the apparent reliability improvement from retry is outweighed by the duplicate side-effect risk.

Section 4: Timeout and thread budget — max_hold_time calculation and pool sizing. Specify the per-request timeout for each downstream operation. The timeout is the maximum time the caller will wait for a response before treating the call as failed. The timeout must be calibrated against the downstream operation's p99 latency under normal conditions (the timeout should exceed the p99 to avoid false timeouts under momentary load spikes) and under degraded conditions (the timeout should not be so long that a degraded downstream service ties up threads for minutes rather than seconds). A common calibration: timeout = p99_normal × 3, rounded to the next second, with a maximum of 5 seconds for synchronous request-path calls and 30 seconds for background processing. Calculate the max_hold_time: per_request_timeout × (1 + retry_count) + sum_of_backoff_intervals. The backoff intervals are bounded by the exponential-with-cap formula; include the maximum possible backoff in the hold-time calculation for conservatism. Calculate the thread pool exhaustion time under the target worst-case outage scenario. Define the scenario: the outage failure fraction (e.g., 40% of requests fail), the request rate at peak traffic, and the thread pool size. The exhaustion time formula is pool_size ÷ (request_rate × failure_fraction × max_hold_time). If the exhaustion time is less than the circuit breaker open window (the time from first failure to circuit open), the service will exhaust its thread pool before the circuit breaker protects it. Resolution: reduce the timeout, reduce the retry count, increase the thread pool, or widen the retry budget to fail requests more aggressively when the retrying fraction exceeds the budget. The goal is an exhaustion time that exceeds the circuit breaker open window by a factor of at least 2, so that the circuit breaker opens before the pool exhausts. The database connection pooling decision record documents the same hold-time product for database connections; apply the same calculation to database calls if the retry policy covers database operations. Specify the non-blocking I/O model if used: in async frameworks where threads are not held during I/O waits, the pool exhaustion model does not apply in the same form — the relevant resource is the in-flight request count limit (the number of requests awaiting a response), not the thread count. Document which model applies.

Section 5: Retry observability — retry ratio monitoring, budget exhaustion alerting, and load amplification visibility. Specify the metrics required to make the retry policy observable. The primary metric is the retry ratio: the fraction of outbound requests for each downstream dependency that are retries rather than first attempts. The retry ratio is a leading indicator of downstream degradation — it rises before the overall error rate rises, because retries absorb transient failures that do not surface as errors to the caller. A retry ratio above 5% for a stable dependency should trigger an alert (not a page) to the on-call team, indicating that the downstream service is experiencing transient failures at a rate that warrants investigation. A retry ratio above the retry budget threshold (typically 15–20%) should trigger a page, because the budget is being saturated and the circuit breaker is about to open or should already be open. The observability strategy decision record documents the alerting infrastructure; retry metrics must be structured with the downstream service name as a dimension so that a single dashboard can show the retry ratio per dependency, enabling correlation between a retry spike and a downstream service incident. Specify the retry-induced load amplification metric: the total outbound request count per second to each downstream dependency, divided by the inbound request count to the caller. Under normal operation (retry ratio near zero), the ratio is approximately 1.0. During a partial outage with a 20% retry ratio and retry count 2, the ratio is approximately 1.2–1.4 — the caller is sending 20–40% more requests to the downstream service than it is receiving from its own callers. Document the threshold at which the amplification ratio triggers a scaling review: if the downstream service is running near capacity and the caller is amplifying its load by 40%, the downstream service needs capacity added before the next outage. Specify the duplicate detection metric for operations with idempotency keys: the server-side idempotency key hit rate (the fraction of requests that matched a stored key and returned a cached response rather than executing). A hit rate above 0.1% on a payment or other high-stakes operation should trigger a review — it indicates that callers are regularly retrying successfully completed operations, which is the signal that the timeout calibration or idempotency window is mismatched. The API rate limiting decision record documents the rate limit model for downstream APIs; the retry policy must respect 429 responses with Retry-After headers by sleeping for the server-specified duration rather than the backoff formula duration — the 429 response is the server's explicit instruction on when to retry, and ignoring it in favor of a shorter client-calculated backoff is the behavior that causes exponential retry amplification against rate-limited APIs.

Further reading

• The circuit breaker decision record — the companion pattern that supersedes retry during sustained outages; the circuit breaker threshold, half-open probe interval, and success threshold for circuit close must be specified jointly with the retry policy.
• The API rate limiting decision record — the rate limiting model that synchronized retries violate; 429 Retry-After semantics must be respected in the retry backoff implementation, not overridden by the client's own backoff calculation.
• The API idempotency decision record — the server-side idempotency key implementation that enables safe retry of non-naturally-idempotent operations; the idempotency window, key storage model, and duplicate response behavior must be verified before retry is added to any write operation.
• The queue and messaging decision record — async retry via durable message queue as the correct alternative to synchronous retry for non-idempotent operations without server-side idempotency key support; at-least-once redelivery combined with consumer-side duplicate detection.
• The payment processor decision record — idempotency key requirements specific to payment processing integrations; idempotency key generation per invoice, per order, and per charge attempt; duplicate charge detection and reversal procedures.
• The observability strategy decision record — alerting infrastructure for retry ratio spikes, retry budget exhaustion events, and load amplification above threshold; retry metrics must be structured with the downstream dependency name as a dimension for per-service visibility.
• The error handling strategy decision record — the error classification model that determines retryable versus non-retryable conditions; the retry failure classification must be consistent with the application's error hierarchy and must enumerate the specific HTTP status codes and exception types that map to each category.
• The database connection pooling decision record — the same timeout × hold-time product for database connections; the pool sizing calculation under partial database outage with retry enabled uses the same formula as for HTTP client thread pools.
• The distributed locking decision record — distributed lock as an idempotency mechanism for consumer-side duplicate detection in async retry queues; the lock serializes concurrent retriers on the same logical operation without requiring server-side idempotency key support.
• WhyChose decision extractor — finds the founding "should we add retry?" and "what timeout should we set?" sessions in your ChatGPT or Claude export and extracts the decision and the trade-offs that were actually weighed, without the surrounding debugging conversation that buries the parameter choice in thirty messages about whether the downstream service was at fault.