What is a circuit breaker pattern and how does it prevent cascading failures in distributed systems?

A circuit breaker is a resilience pattern that wraps calls to a downstream dependency and monitors the failure rate of those calls over a rolling time window. When the failure rate exceeds a configured threshold, the circuit breaker trips — entering an open state — and subsequent calls to the downstream dependency fail immediately without attempting the network call, returning a fast failure or a fallback response to the caller instead. The circuit breaker remains open for a configured reset timeout, after which it enters a half-open state and allows a small number of probe requests through. If the probe requests succeed, the circuit breaker transitions back to closed (normal operation). If the probes fail, it trips back to open and the reset timeout restarts. The pattern prevents cascading failures by transforming a slow failure into a fast failure: a service that calls a degraded downstream dependency without a circuit breaker will block its threads or async handlers waiting for responses that time out slowly. If each request takes the full timeout duration (commonly 30 to 60 seconds) before failing, a service handling 50 concurrent requests with a 30-second downstream timeout can exhaust its thread pool in seconds when all 50 requests are blocked waiting for a degraded dependency. New incoming requests queue up or are rejected. The service itself becomes unavailable not because of its own code but because it is blocking on a degraded downstream dependency. A circuit breaker prevents this by detecting the degradation quickly (after the failure threshold is crossed in the monitoring window) and then returning fast failures for all subsequent requests during the open state, allowing the calling service's threads to process other work rather than blocking indefinitely. The decision to adopt circuit breakers is not the question — the question is where to apply them, what failure thresholds to calibrate, and whether the circuit breaker implementation is consistent across the service fleet. A circuit breaker applied to one service's outbound calls but not another's means the fleet has uneven cascading failure protection: the unprotected service is the path through which a downstream degradation propagates into the wider platform during an incident.

How should retry policies be designed to avoid making a downstream outage worse?

A retry policy that increases load on a degraded downstream service is a pattern that turns a recoverable degradation into an extended outage. The most common failure mode is a retry storm: when a downstream service degrades, multiple upstream callers begin retrying failed requests. If each caller retries immediately (no backoff) or with a fixed short interval (1-second retry, 3 attempts), all callers simultaneously re-issue their failed requests at the same rate as the original request rate, doubling or tripling the load on the already-degraded service. The additional load prevents the service from recovering, which causes more failures, which causes more retries, which sustains the elevated load indefinitely. The service cannot recover until the retry pressure is removed — either by the callers giving up, by a circuit breaker opening and stopping the retries, or by human intervention. The three design principles that prevent a retry storm are: (1) exponential backoff with jitter — each retry waits for a duration that grows exponentially (first retry after 1 second, second after 2 seconds, third after 4 seconds) plus a random jitter value (a random fraction of the backoff interval, chosen independently per caller) that desynchronizes the retry waves across multiple callers. Without jitter, all callers that started their first retry at the same time will also start their second retry at the same time, producing synchronized retry waves that hit the downstream service in bursts rather than spreading evenly. With jitter, the retry waves spread out, reducing peak load on the recovering service. (2) Idempotency requirements — retries are safe only for operations that can be executed multiple times with the same effect as a single execution. Read operations are typically idempotent; write operations (creating an order, charging a card, sending an email) are not idempotent unless explicitly designed to be — using an idempotency key that the downstream service uses to deduplicate duplicate requests. A retry policy that retries non-idempotent write operations risks creating duplicate records, double-charging a card, or sending a notification twice. The idempotency requirement must be documented per operation type in the resilience decision record; a blanket retry policy that applies to all operations without idempotency verification is a double-execution risk. (3) Retry budget — a bounded number of retries (typically 2 to 3) with a total timeout that limits the maximum latency the caller will experience. Unbounded retries convert a transient downstream error into an indefinitely blocking call. The total timeout must be shorter than the caller's own timeout as seen by its upstream — a service that will retry for 90 seconds total must have an upstream timeout of more than 90 seconds, or the upstream will time out before the retry sequence completes, leaving the retry attempts as wasted load on the recovering service with no benefit to the end user. The retry policy, backoff configuration, and idempotency requirements should be documented in the resilience decision record with the rationale for the chosen values — specifically the expected recovery time of the downstream service, which determines the maximum useful retry interval, and the idempotency guarantee provided by each operation type.

What is a timeout hierarchy and why does getting it wrong cause cascading failures?

A timeout hierarchy is the set of timeout values configured at each layer of a service dependency chain, ordered so that inner timeouts (closer to the downstream dependency) are shorter than outer timeouts (at the request-handling boundary). The correctness property is that the timeout at each layer must be shorter than the timeout at the layer above it, with enough margin to allow for retry attempts. If the timeout at an inner layer is longer than or equal to the timeout at the outer layer, the outer layer will time out before the inner layer, which means the outer layer cancels the request and moves on while the inner layer is still waiting for the downstream response. The inner layer's eventual response — either success or failure — is then handled by code that has already returned to the caller, producing error log noise at best and incorrect state updates at worst if the inner call succeeds after the outer layer has already returned an error response. A concrete example: if the outer HTTP handler has a 30-second timeout for the full request, and the inner database call has a 45-second timeout, the HTTP handler will time out and return a 504 to the client while the database query is still running. The database query may complete 15 seconds later, update a record, and commit a transaction — leaving the system in a state where the client received a timeout error but the server-side state was successfully updated. This is a data consistency problem that is caused by an incorrect timeout hierarchy. The cascading failure amplification occurs in the other direction: if the inner timeout is too long relative to the recovery time of the downstream service, a degraded downstream service will hold inner connections open for the full timeout duration, exhausting the inner connection pool, which exhausts the service's capacity to process other requests, which makes the service unavailable to its own upstream callers. The correct timeout hierarchy for a three-layer chain (client → service A → service B → database) is: service B's database timeout (e.g., 5s) < service A's service-B call timeout (e.g., 10s, accounting for one retry at 5s timeout) < client's service-A call timeout (e.g., 25s, accounting for one retry at 10s timeout plus overhead). The timeout values and their rationale should be documented per service dependency pair in the resilience decision record: the expected P99 response time of the dependency in normal operation, the acceptable tail-latency budget for the calling service, the number of retry attempts the timeout must accommodate, and the alert threshold for when the actual P99 approaches the timeout limit.

When should a service return a fallback response instead of an error when a downstream dependency is unavailable?

A fallback response is a degraded but functional response that a service returns when a downstream dependency is unavailable, rather than propagating the downstream failure as an error to the calling service. Fallbacks are appropriate when a meaningful, useful response can be constructed without the downstream dependency and when the degraded response is preferable to an error from the user's perspective. The canonical cases are caching-based fallbacks (returning a cached version of data that would normally be fetched fresh), static fallbacks (returning a sensible default when personalized data is unavailable), and functional degradation fallbacks (returning a reduced-capability response when an enhancement service is unavailable). An e-commerce product page that shows product details from a local cache when the inventory service is unavailable is a caching-based fallback — the page renders, the user can see the product, and the inventory count may be stale but the absence of an error page is preferable to a 503. A recommendation engine that returns a static list of popular products when the personalization service is unavailable is a static fallback — the recommendations are not personalized but they are plausible and the page renders without error. The design decisions for fallback responses are: (1) what is the acceptable staleness for a cached fallback — a 5-minute-old inventory count is acceptable during a brief degradation; a 6-hour-old price is not acceptable for a checkout page. The cache TTL and the staleness threshold must be documented and must account for the expected duration of downstream outages in the worst-case scenario; (2) which pages or features have no acceptable fallback and must fail with an error — a checkout payment form has no acceptable fallback when the payment processor is unavailable, because displaying a payment form that cannot process payments misleads the user. The honest response is to hide the checkout button or show an explicit degraded-mode message; (3) the fallback must be tested. A fallback that works in production only when the downstream dependency is completely unreachable but fails when the dependency returns partial or malformed responses is not a reliable fallback. The circuit breaker's open state should trigger the fallback consistently, and the fallback must handle the case where its own data source (the cache) is also empty — the first-request-after-deployment cold-start case. The resilience decision record should enumerate fallback strategies per downstream dependency, the staleness threshold for each cached fallback, and the features that have no acceptable fallback and must propagate failures to the user.

The circuit breaker and resilience pattern decision record: why the failure handling strategy you chose determines your cascading failure surface and your recovery time

July 10, 2026 · WhyChose

Circuit breakers are added to distributed systems one service at a time, usually after a production incident makes the absence obvious. The first service to get a circuit breaker is the one whose downstream dependency went down and caused the most visible damage. The second is the one that failed next. The pattern propagates backwards through the incident history — each new circuit breaker is a scar from a previous outage. What does not exist is a decision record that describes the failure handling strategy for the fleet as a whole: which services require circuit breakers and which do not, what thresholds govern the transition between closed, open, and half-open states, how retry policies interact with the circuit breaker state, and what timeout values at each service boundary ensure that a slow downstream failure propagates as a fast failure rather than as a blocking accumulation of held threads. The fleet accumulates circuit breakers, but the gaps between them remain invisible until a new failure path is discovered during an incident.

The failure handling strategy is not one decision — it is a set of related decisions that must be consistent across the service fleet to prevent cascading failures. A circuit breaker on service A's calls to service B protects service A from service B's failures but does not protect service C from service A's failures if service C has no circuit breaker on its calls to service A. When service B degrades and service A's circuit breaker opens, service A starts returning fast failures to service C. If service C has no circuit breaker on its calls to service A and has a long timeout configured for those calls, service C's threads block waiting for service A's fast failures to arrive — which they do, quickly, but if service C's request handling is synchronous, blocking on service A, then service C's thread pool will fill with requests blocked on service A calls even though service A is returning fast failures. Fast failures from service A are not the same as no calls to service A: service C still makes the calls and still waits for responses, even if those responses come in milliseconds rather than seconds. If service C's concurrency is 200 requests and service A is returning fast failures in 2 milliseconds, the thread pool will not exhaust from latency, but if service A is degraded rather than tripped and responses are taking 800 milliseconds per call, a service C with 200 threads and 200 concurrent requests that each make one call to service A will saturate its thread pool in roughly 200 × 800ms / thread-switching-overhead seconds. The cascading failure surface is determined by the combination of circuit breaker coverage, timeout configuration, and retry policy across the full dependency graph — not by any single service's resilience configuration.

The resilience decision record is missing from most service architectures because resilience patterns are adopted reactively rather than proactively. The first production incident produces a circuit breaker on one service. A post-incident review recommends "add circuit breakers to all services" but the recommendation is vague — it does not specify which library to use, what the failure threshold should be, how the timeout hierarchy should be structured, or where the fallback responses should come from when a circuit breaker trips. Engineers implement circuit breakers independently, using different libraries, different threshold values, and different fallback strategies, producing a fleet where the resilience behavior is inconsistent and the cascading failure surface is unknowable without reading each service's implementation individually. A resilience decision record written before the fleet grows beyond three or four services — or written retroactively as a consolidation exercise after the first serious cascading failure — gives the fleet a shared vocabulary for failure handling, consistent thresholds that can be reasoned about at the fleet level, and a documented dependency graph that identifies the unprotected paths before they are discovered in production.

Two things that happen when the decision is not written down

The checkout cascade: a 40-minute payment API degradation that took down the entire e-commerce platform

A 120-person e-commerce company operated a service-oriented platform with seven backend services: a product catalog service, an inventory service, a pricing service, a checkout service, an order management service, a payment processing service, and a notification service. The checkout service called the payment processing service to initiate charges. The order management service called the checkout service to create orders. The product catalog service was called by the frontend directly and also by the order management service to validate that ordered items still existed. The services had been built over three years by three successive engineering teams, and the resilience configuration of each service reflected the priorities and experiences of the team that built it.

The payment processing service had a circuit breaker on its calls to the external payment gateway — the first team, which had suffered a payment gateway outage two years earlier that caused the payment service to hang while the gateway was unresponsive, had added the circuit breaker after that incident. The circuit breaker was configured with a 50% failure threshold over a 30-second window and a 60-second reset timeout. The checkout service, which called the payment processing service, had no circuit breaker. The second team that built the checkout service had not experienced a payment service outage and had not added one. The checkout service had a 45-second timeout on its calls to the payment processing service, configured to match the gateway's maximum response time plus a 5-second buffer. The order management service called the checkout service with a 60-second timeout. The frontend called the order management service via an API gateway with a 90-second timeout. All three timeouts were set independently, by different teams, without a timeout hierarchy document that established the required relationship between the layers.

On a Tuesday evening during a peak traffic period, the payment gateway began degrading: responses that normally completed in 200 to 800 milliseconds were taking 8 to 25 seconds, and roughly 35% of requests were timing out after 30 seconds. The payment processing service's circuit breaker detected the degradation: within 90 seconds of the degradation start, the failure rate in the 30-second window crossed 50% and the circuit breaker tripped to open. The payment processing service began returning fast failures (HTTP 503 with a circuit-open header) to the checkout service in under 10 milliseconds. This was the correct behavior — the payment processing service was protecting itself and its callers from the gateway's degradation.

The checkout service, receiving fast failures from the payment processing service, had no circuit breaker. It attempted to handle the failures by returning HTTP 422 errors to the order management service for each failed checkout attempt, but it also had retry logic: on a payment failure, it retried the payment call up to two additional times before giving up. The retry was implemented without exponential backoff — it retried immediately, with a fixed 500-millisecond wait between attempts. Each checkout attempt that hit a tripped payment circuit breaker generated three calls to the payment processing service: the original call (returning a fast failure in 10ms), a first retry after 500ms (another fast failure in 10ms), and a second retry after another 500ms (another fast failure in 10ms). The three calls took approximately 1.02 seconds total. With the payment circuit breaker tripped, every checkout attempt was guaranteed to fail in roughly one second.

Under normal traffic, the checkout service handled 180 requests per minute. During the degradation, all 180 requests per minute began failing at the checkout stage. The order management service, receiving 422 errors from checkout, also had retry logic: it retried failed checkout calls twice, with a 2-second backoff. Each order that hit a checkout failure generated three checkout attempts, each of which generated three payment calls. The order management service's retry logic increased the checkout service's effective request rate from 180 to 540 requests per minute — three times the normal rate — without any corresponding increase in user traffic. The checkout service handled the elevated request rate without issue (it was not thread-constrained at 540 rpm), but the payment processing service received 1,620 requests per minute — nine times the normal rate — of fast-failing circuit-breaker calls. The payment processing service handled these without issue as well, since the circuit was open and responses were immediate.

Twenty-two minutes into the degradation, the payment gateway began partially recovering: some requests started completing in 400 to 600 milliseconds, but roughly 20% were still failing. The payment processing service's circuit breaker entered half-open state and sent probe requests. The probe requests succeeded, and the circuit breaker transitioned back to closed. The payment processing service began routing real requests to the gateway again. Within 90 seconds, the failure rate in the monitoring window rose above 50% again (the gateway was still partially degraded), and the circuit breaker tripped back to open. This cycle repeated three times over the next 18 minutes: the circuit breaker closed on partial recovery, load hit the partially-recovered gateway, the gateway degraded again, the circuit breaker tripped. During each closed window, some checkout attempts succeeded — which made the incident harder to diagnose because error rates were intermittent rather than binary.

The incident became a platform-wide outage at the 35-minute mark not because of the payment service or the checkout service but because of the product catalog service. The product catalog service called an external content delivery API to fetch product descriptions. That API had a 30-second connection timeout and a 45-second read timeout. The product catalog service had no circuit breaker on its calls to the content API. During the incident, the content delivery API began experiencing elevated latency — likely due to the same upstream network condition that was affecting the payment gateway, though this was never confirmed. Product catalog requests began taking 30 to 40 seconds instead of the normal 200 to 500 milliseconds. The product catalog service had a thread pool of 40 concurrent request handlers. With each request taking 35 seconds instead of 0.4 seconds, the thread pool saturated: within 90 seconds of the content API degradation, all 40 threads were occupied waiting for slow content API responses. New incoming requests queued up. The queue filled. The product catalog service started returning HTTP 503 errors to the frontend and to the order management service that called it to validate ordered items.

The order management service, now receiving 503 errors from the product catalog service in addition to 422 errors from the checkout service, entered a failure state that cascaded to the frontend. The API gateway's 90-second timeout was exceeded by order management requests that were themselves waiting for responses from checkout (which was retrying) and product catalog (which was blocking on the content API). The frontend began timing out on all order-related requests. The 40-minute payment gateway degradation, which affected only payment processing, had expanded into a full platform outage through two unprotected paths: the checkout service's retry amplification that stressed the partial recovery, and the product catalog service's thread pool exhaustion from an unrelated dependency that happened to degrade at the same time. A resilience decision record that mapped the dependency graph, required circuit breakers on all outbound service calls, and enforced a timeout hierarchy would have contained the payment gateway degradation to the payment processing service and the product catalog degradation to the content-fetching path — two recoverable partial degradations rather than one platform-wide outage.

The retry storm that extended a 4-minute database slowdown into a 90-minute outage

A 45-person SaaS company ran a user-facing API backed by a PostgreSQL primary and two read replicas. The API service was horizontally scaled to six instances, each handling up to 80 concurrent requests for a maximum fleet concurrency of 480. The database connection pool per instance was configured at 20 connections — the standard configuration from the ORM documentation that the team had adopted without modification. Database queries in normal operation completed in 2 to 15 milliseconds for reads and 5 to 40 milliseconds for writes. The API service had no circuit breaker on its database calls — the database was considered an internal infrastructure component rather than a remote service, and the team's mental model of circuit breakers was that they were for external service dependencies only. The API service had a retry policy on database calls: it retried any database error up to three times with a 200-millisecond fixed interval before returning an error to the caller.

At 2:17am UTC, a routine database maintenance job ran a VACUUM ANALYZE on the platform's largest table — the events table, which had accumulated 14 months of data and had not been manually vacuumed since the team moved to a managed database service that handled autovacuum automatically. The autovacuum settings were the default PostgreSQL values, which scheduled autovacuum based on the percentage of dead tuples relative to total table rows. The events table's autovacuum had been running nightly and keeping up with the dead tuple accumulation during normal write rates. The maintenance engineer ran the manual VACUUM ANALYZE because a recent query plan review had identified that the table statistics were stale — the planner was choosing a sequential scan over an index scan on a frequently executed query, and a manual ANALYZE was expected to fix it without requiring a deployment. The manual VACUUM ANALYZE on a 1.8-billion-row table acquired an exclusive lock during its initial phase, blocking all concurrent writes to the table for approximately 90 seconds, then transitioned to a mode that ran concurrently with writes but at a significant I/O cost. During the concurrent phase, disk I/O on the primary reached 95% utilization, and queries that normally completed in 2 to 15 milliseconds began taking 800 milliseconds to 4.2 seconds. Some queries queued behind the lock acquisition phase were taking 15 to 30 seconds.

The API service, observing database queries taking 800 milliseconds to 4 seconds instead of 2 to 15 milliseconds, began hitting the ORM's default 5-second query timeout on 12% of requests. Each timed-out query triggered the retry policy: three retries at 200-millisecond intervals, with no exponential backoff and no jitter. All six API instances simultaneously began retrying failed database queries. At normal operation, the six instances used roughly 60 of their 120 pooled connections. During the retry phase, each retry attempt from a timed-out query immediately occupied a new connection from the pool — the ORM did not reuse the connection from the timed-out query, which was still being cancelled at the database level. Within 90 seconds of the VACUUM's I/O spike beginning, all 120 pooled connections were occupied: 60 with in-progress queries, 40 with first-retry attempts for queries that had timed out, and 20 with second-retry attempts. Incoming API requests queued, waiting for a connection from the pool. The pool-wait timeout was 10 seconds (the ORM default). Requests waiting for a pool connection for more than 10 seconds returned a database pool timeout error to the caller — not a database query error, but a pool saturation error that the API's error handler treated identically: triggering the same retry logic.

The retry loop was now feeding itself. Requests that failed due to pool exhaustion were retried, and those retried requests competed for pool connections with the original requests that were still in their retry sequences. The database connection count at the primary rose from 120 (the six API instances' pools) to 180 as partially-cancelled connections from timed-out queries were not immediately released by the database — PostgreSQL holds a connection until the client-side close is processed, and the ORM's cancellation path had a 3-second cleanup delay before releasing the connection to the pool. The additional connections competed for the already-stressed I/O budget, adding to the query latency. The VACUUM ANALYZE's I/O spike, which had been expected to last approximately 4 minutes based on the table size and the managed database service's I/O capacity, was sustained beyond 4 minutes by the additional I/O from 180 concurrent connections running retrying queries.

The VACUUM ANALYZE completed at 2:28am UTC — 11 minutes after it started. Query latency on the primary did not immediately return to normal. The 180 connections were all carrying retry-queued requests, and the backlog of queued requests on the API side meant that new requests continued to arrive at the pool faster than the pool could process them even as individual query latency decreased. The pool did not fully drain of retry-amplified backlog until 3:47am UTC — 90 minutes after the VACUUM started and 79 minutes after the VACUUM completed. The on-call engineer, woken by alerts at 2:22am UTC, had identified the VACUUM as the trigger within 12 minutes and confirmed the VACUUM had completed at 2:28am UTC but could not understand why the database was still under load 40 minutes later. The retry storm's self-sustaining dynamic — retries creating more connections, more connections increasing I/O, elevated I/O sustaining elevated latency, elevated latency causing more retries — was not documented anywhere. The post-incident review identified the retry policy as the proximate cause of the extension and established that database calls should be treated as remote calls subject to the same resilience patterns (circuit breaker, exponential backoff with jitter, retry budget) as external service dependencies. The resilience decision record that was written after the incident included the retry policy, backoff configuration, and the note that database calls require idempotency verification before retrying — a SELECT is safely retried, but an INSERT or UPDATE must use idempotency keys or must not be retried at all until the original transaction outcome is confirmed.

Three structural properties that are set at failure handling strategy selection time

The circuit breaker state machine and the failure detection surface

A circuit breaker's protective coverage is determined by where in the call stack it is applied, not by whether it exists somewhere in the system. A circuit breaker on service A's calls to service B protects service A from service B's failures. It does not protect service A from service C's failures (if service A also calls service C without a circuit breaker), and it does not protect service D from service A's failures if service D calls service A without a circuit breaker. The failure detection surface — the set of failure paths that the circuit breaker fleet can detect and contain — is the union of all individual circuit breakers' scopes. The gaps in the failure detection surface are the paths through which a localized failure can cascade into a broader outage. The failure detection surface is set when the resilience decision record specifies which service dependencies require circuit breakers. A decision that applies circuit breakers to all outbound calls at every service boundary produces a failure detection surface that covers the full dependency graph. A decision that applies circuit breakers only to calls to external third-party services (payment gateways, email providers, CDNs) leaves internal service-to-service calls unprotected — the checkout-to-payment and catalog-to-content-API gaps that produced the e-commerce cascade above.

The circuit breaker state machine has three parameters that must be calibrated per dependency: the failure threshold (the percentage of calls in the monitoring window that must fail before the circuit trips), the monitoring window duration (the time period over which the failure percentage is calculated), and the reset timeout (how long the circuit remains open before testing recovery with probe requests). These three parameters determine how quickly the circuit breaker responds to a degradation, how aggressively it opens, and how quickly it allows traffic to resume after recovery. A low failure threshold with a short monitoring window produces an aggressive circuit breaker that trips on brief transient errors — a momentary spike in errors that would resolve in 5 seconds trips the circuit and blocks traffic for the full reset timeout, which may be 60 seconds. A high failure threshold with a long monitoring window produces a slow circuit breaker that trips only on sustained, severe degradation — a 30% error rate over 2 minutes may not trip a 50% threshold circuit breaker, leaving 70% of traffic succeeding while 30% fails, which may be an acceptable operating point for a non-critical dependency but is not acceptable for a checkout payment dependency. The calibration of these parameters requires knowing the normal error rate of the dependency (the baseline below which circuit trips should not occur), the expected duration of transient errors (brief enough that the circuit should not trip, or sustained enough that it should), and the recovery time of the dependency after a failure (which determines the reset timeout). These values are only knowable from the dependency's SLA documentation and from historical incident data — both of which should be referenced in the resilience decision record alongside the chosen parameter values and their rationale.

The half-open state probe rate — how many requests the circuit breaker allows through during the recovery test period — is a parameter that is often omitted from circuit breaker configurations that use default settings. A circuit breaker that allows a single probe request in half-open state will re-trip if that one probe happens to fail, even if 95% of subsequent requests would have succeeded. A circuit breaker that allows 10 probe requests and requires 8 of 10 to succeed before transitioning back to closed is more robust to transient errors during recovery but allows 10 requests to potentially fail before the circuit re-trips. The probe rate and success threshold interact with the expected recovery latency of the downstream service — a downstream service that recovers gradually (latency decreasing from 5 seconds to 1 second over 30 seconds) will produce a different pattern of probe results than a service that recovers abruptly (latency dropping from 5 seconds to 200 milliseconds in one second). This connects to the observability platform decision record: the circuit breaker state transitions (closed to open, open to half-open, half-open to closed, half-open to open) should be emitted as metrics and logs, and the probe success rate during half-open should be tracked to validate that the threshold configuration is appropriate for the dependency's recovery pattern. A circuit breaker that re-trips on every recovery attempt is a circuit breaker whose reset timeout is too short relative to the dependency's recovery time — the service is probing before recovery is complete. Without the metrics, this calibration problem is invisible until it is noticed during a post-incident review.

The retry policy and the idempotency surface

A retry policy is only safe for operations where the retry cannot produce a worse outcome than a single failed attempt. The idempotency surface is the set of operations in the system that can be safely retried — where executing the operation twice with the same input produces the same final state as executing it once. The idempotency surface is set by the API design of each downstream dependency and by the idempotency guarantees that each operation either provides natively or requires explicit key-based deduplication to achieve. Read operations (GET requests, SELECT queries, read-only RPC calls) are typically idempotent — reading the same data twice returns the same result (assuming no concurrent writes), and the second read has no side effects. Write operations are not idempotent unless explicitly designed to be: creating a resource (POST /orders), charging a payment instrument, sending a notification, appending to an event log — all of these operations have side effects, and retrying a write operation that failed with an ambiguous status (a network timeout that may have occurred before or after the server processed the request) risks creating a duplicate resource, a double charge, or a duplicate notification. The idempotency key pattern — where the caller generates a unique key for each logical operation and the server uses the key to deduplicate — converts a non-idempotent operation into an idempotent one from the caller's perspective: the first call creates the resource; subsequent calls with the same key return the already-created resource without creating a duplicate. The idempotency key must be durable — the server must store the key and its associated result for at least the maximum retry window — and must be scoped to the logical operation, not the individual HTTP request. A caller that generates a new idempotency key per retry attempt defeats the purpose of the pattern.

The retry policy interacts with the circuit breaker state machine in a way that must be explicitly documented. A retry policy that retries on any error will retry on errors returned by an open circuit breaker — fast failures that carry a circuit-open indicator. Retrying on a circuit-open error is counterproductive: the circuit is open because the downstream is degraded, and retrying does not change the downstream state. The retry policy should distinguish between retriable errors (transient errors where the operation may succeed on retry — network timeouts, 503 Service Unavailable, 429 Too Many Requests with a Retry-After header) and non-retriable errors (permanent errors where retry cannot help — 4xx client errors, circuit-open fast failures, resource-not-found responses). The distinction must be implemented in the retry predicate — the function that determines whether a given error should be retried — and the predicate logic must be documented in the resilience decision record so that engineers adding new retry logic in new services know the correct predicate pattern. Without the documented pattern, each service implements its own predicate independently, producing a fleet where some services retry on circuit-open errors (amplifying load on recovering services), some services do not retry at all (producing unnecessary failures for retriable transient errors), and some services retry on 4xx errors (triggering duplicate side effects for operations that failed because of bad input, not transient infrastructure issues). This connects to the API versioning decision record: the error response format — specifically whether a 503 response includes a circuit-open indicator, a retry-after header, or a structured error body that distinguishes the failure type — must be consistent across services for the retry predicate to function correctly. A retry predicate that checks for a circuit-open header cannot function if only some services include that header.

Exponential backoff with jitter is the standard retry delay strategy for preventing retry storms, but the specific parameters — the initial delay, the maximum delay, the backoff multiplier, and the jitter range — must be calibrated against the expected recovery time of the downstream service and the acceptable tail latency for the end-to-end request. A service whose downstream dependency typically recovers from transient errors in 50 to 200 milliseconds benefits from a short initial delay (100 milliseconds) and a low maximum delay (2 seconds), because waiting longer does not increase the probability of success. A service whose downstream dependency recovers from degradation over 2 to 5 minutes benefits from a longer maximum delay (30 to 60 seconds) that allows the recovery to complete before the caller gives up. The total retry budget — the sum of all retry delays and response times — must be shorter than the caller's timeout as seen by its own upstream caller, establishing the outer bound of the timeout hierarchy. A retry budget that exceeds the upstream timeout wastes retry attempts that cannot complete before the upstream cancels the request. The resilience decision record should specify the retry parameters per dependency type (internal service calls, external API calls, database calls) with the rationale for the chosen values and the upstream timeout constraint that bounds the total retry budget.

The timeout hierarchy and the cascading failure amplification factor

The timeout hierarchy is the ordered set of timeout values across the full request-handling chain, where each inner timeout must be strictly shorter than the outer timeout that bounds it. The cascading failure amplification factor is the ratio by which a slow downstream failure is amplified into a broader service degradation, which is determined by the combination of timeout values and the concurrency of the service's request handlers. A service with 100 concurrent handlers and a 30-second downstream timeout will see its entire concurrency budget consumed in 30 seconds if the downstream becomes completely unresponsive — all 100 handlers will be blocked waiting for the 30-second timeout to expire, and incoming requests will queue or be rejected for 30 seconds. A service with 100 concurrent handlers and a 2-second downstream timeout will see its concurrency budget consumed in 2 seconds under the same conditions — a 15x improvement in the time to saturation and a proportional reduction in the impact on upstream callers, since the service will start returning fast errors to upstream callers 2 seconds after the downstream becomes unresponsive rather than 30 seconds later. The timeout value is the single largest lever on the cascading failure amplification factor, and it is set at deployment time in configuration that is typically not reviewed after initial setup. The residue of the timeout history — values chosen in one context (a development environment with a slow local database), carried into another (a production microservice fleet with a millisecond-latency cache layer) — determines the current amplification factor without anyone having made a recent decision about whether the factor is appropriate.

The correct approach to timeout hierarchy design starts from the end-to-end latency budget — the maximum acceptable response time for the top-level user-facing request, typically expressed as the P99 SLA for the API or web page. The end-to-end budget is divided across the service call chain with each layer receiving a fraction that accounts for its own processing time plus the time allocated to its outbound calls, including retries. A three-tier chain (frontend → API service → downstream service → database) with an end-to-end P99 budget of 2 seconds might allocate: database timeout 200ms (P99 database query time in normal operation is 15ms; 200ms provides headroom for normal query variability and one connection-pool-wait without retry), downstream service call timeout 600ms (the downstream service makes one database call with 200ms timeout, has 200ms of its own processing time, and the 200ms margin covers network round-trip and queueing), API service call timeout 1.5 seconds (the API service may make two downstream service calls serially with 600ms timeout each, accounting for one call sequence and the upstream response construction time; a 1.5-second total timeout provides margin for non-serial paths), end-to-end budget of 2 seconds (the frontend timeout) covering the API service timeout plus network and client-side rendering overhead. The timeout values in this hierarchy are derived from each other — a change to the database timeout propagates upward through the chain, constraining what is possible at each layer. A resilience decision record that documents the derivation chain means that when a database is upgraded and the P99 query time changes from 15ms to 5ms, the team can recalculate the downstream service and API timeouts to tighten the budget and reduce the concurrency saturation threshold, rather than leaving the original conservative values in place indefinitely. This connects to the microservices vs. monolith decision record: a monolith makes database calls directly from request handlers, and the timeout between the handler and the database is the only timeout in the chain; a microservice fleet has multiple timeout layers, and the product of the layers' timeouts determines the worst-case recovery time for the fleet. Teams that move from a monolith to microservices without redesigning the timeout hierarchy typically inherit the monolith's generous timeouts at each new service boundary, producing a cascade amplification factor that is multiplicatively worse than the original monolith.

Context propagation — passing a deadline or timeout from the original request through every service call in the chain — reduces the cascade amplification factor by ensuring that downstream calls inherit the remaining time budget of the original request rather than starting fresh timeouts at each boundary. A request with a 2-second end-to-end deadline that spends 500ms in the API service before calling a downstream service should pass a 1.5-second deadline to the downstream service, not start a new 2-second timeout. If the downstream service then spends 800ms processing and calls the database, it should pass a 700ms deadline to the database call, not start a new 200ms timeout. Context propagation ensures that when the end-to-end deadline is exceeded, all layers cancel their in-progress work simultaneously rather than each layer completing its full local timeout independently — the worst case where each of three layers waits for its full timeout before the request is considered failed. gRPC's deadline propagation is the canonical implementation; HTTP-based services typically implement context propagation via a request header (the standard is to use the HTTP context or a custom header like X-Request-Deadline) that the downstream service reads and uses to bound its own processing. The decision to implement context propagation is a fleet-level decision — it requires every service to both read the incoming deadline header and pass it through to all outbound calls — and must be documented in the resilience decision record as a requirement for new service implementations. Without the decision record, context propagation is implemented only in services where the implementing engineer happened to know about the pattern, producing a fleet where some services propagate deadlines and others start fresh timeouts, and the aggregate worst-case recovery time is unknowable without auditing each service's implementation.

Five sections the circuit breaker and resilience pattern decision record should address

1. Service dependency graph and failure mode inventory

The resilience decision record begins with a map of the service dependency graph — which services call which other services, what operations are performed on each dependency, and what the business impact is if each dependency becomes unavailable. The dependency graph serves as the basis for all subsequent decisions: which dependencies require circuit breakers, what the timeout hierarchy is, which operations require idempotency guarantees, and where fallback responses should be provided. Without the dependency graph, circuit breakers and timeout configurations are applied locally to individual services without a global view of the failure propagation paths. A dependency that is called by one service may appear low-risk (if that one service fails, only that one service is affected), but the same dependency called by five services may be a single point of failure for the entire platform if none of those services have circuit breakers configured on the dependency.

For each dependency in the graph, document the failure mode inventory: the set of failure scenarios and their expected impact. A dependency that can fail in three modes — slow responses (latency exceeds threshold but requests eventually complete), error responses (requests fail immediately with an error status), and complete unavailability (the network connection is refused or times out at the TCP level) — requires different circuit breaker configurations for each mode. Slow responses are the most dangerous mode for cascading failures because they hold connections and threads open at the caller side; the circuit breaker's failure detection threshold must include latency-based failure detection (marking a call as failed if it exceeds a response-time threshold) in addition to error-rate-based detection. Error responses are easier to detect and circuit-break on, but the error-rate threshold must be calibrated against the dependency's normal error rate to avoid false trips. Complete unavailability is the simplest case — connection refused or TCP timeout — but the TCP timeout value at the OS level may be much longer (90 seconds for some configurations) than the application-level timeout, requiring the application-level timeout to be set explicitly rather than relying on the OS TCP connection timeout. The failure mode inventory should include the expected duration of each failure type based on historical incident data or the dependency's SLA documentation — this is the input to the circuit breaker reset timeout calibration. This connects to the event-driven architecture decision record: services that communicate asynchronously via a message queue have a different failure mode than services that communicate synchronously via HTTP — a queue that is unavailable causes producers to accumulate messages locally or to block on publish calls, not to cascade failures to consumers; the resilience pattern for queue-based dependencies focuses on publish-side backpressure and consumer-side dead-letter handling rather than circuit breakers at the call site.

The business impact classification for each dependency determines the fallback strategy: a dependency whose unavailability means the user's request cannot be completed at all (the payment processor for a checkout flow) requires a different treatment than a dependency whose unavailability means the user's request is fulfilled with degraded quality (the recommendation engine for a product page). For business-critical dependencies with no fallback, the circuit breaker's failure detection should be aggressive (low threshold, short window) to fail fast and surface the outage to monitoring and to the user quickly, rather than hanging on slow responses for 30 seconds before returning a 504. For enhancement dependencies with fallback, the circuit breaker can be more conservative (higher threshold, longer window) to distinguish transient errors from sustained degradation before switching to the fallback, reducing the frequency of unnecessary fallback activations that degrade the user experience for the majority of requests that would have succeeded.

2. Circuit breaker configuration and threshold rationale

For each dependency identified in the failure mode inventory, document the circuit breaker configuration: the failure threshold percentage, the monitoring window duration, the minimum number of requests in the window before the threshold applies (the volume threshold, which prevents a single failed request out of two total requests from tripping a 50%-failure-rate circuit), the reset timeout duration, the half-open probe count and success threshold, and the failure detection mode (error-rate-based, latency-based, or both). Document the rationale for each parameter value, including the data source — the dependency's SLA, historical incident data, or an empirical measurement of the dependency's normal error rate and response time distribution.

The library selection for circuit breaker implementation should also be documented: the resilience library used (Hystrix, Resilience4j, Polly, go-circuit-breaker, or a custom implementation), the version, and the configuration API. Library selection affects the available configuration parameters — some libraries support latency-based failure detection (marking calls as failed if response time exceeds a threshold), half-open state probe configuration, and bulkhead patterns (limiting the number of concurrent calls to a dependency regardless of circuit state); others support only error-rate-based detection and binary open/closed state. The library selection is a fleet-level decision because inconsistent library choices produce inconsistent behavior across services — a fleet where some services use Resilience4j with latency-based failure detection and others use a simple error-count circuit breaker will have different protection against slow-response degradation depending on which service is the entry point for a given request path. The resilience decision record should specify the canonical library for each programming language used in the fleet and explain why alternatives were rejected. This connects to the service mesh decision record: a service mesh (Istio, Linkerd) can implement circuit breaker behavior at the proxy level, outside the application code, with consistent configuration across all services regardless of programming language. If a service mesh is in use, the resilience decision record should specify which resilience behaviors are implemented at the mesh level (circuit breaking, retry, timeout) and which are implemented at the application level (idempotency key generation, fallback response construction, business-logic-level retry decisions), to prevent double-implementation — a circuit breaker at both the application level and the mesh level can produce unexpected behavior when both trip independently on the same downstream degradation.

Include the circuit breaker disabled list: services or call sites where circuit breakers are explicitly not applied, with the rationale for each exception. The disabled list prevents ambiguity — an engineer reading a service that does not have a circuit breaker cannot tell whether the absence was intentional (the dependency is internal and the team decided the overhead of a circuit breaker is not justified) or accidental (the engineer who built the service did not know the policy required circuit breakers). An explicit disabled list documents the intentional absences and makes accidental absences visible by difference: any dependency not covered by a circuit breaker and not on the disabled list is a gap in the failure detection surface that should be addressed. The disabled list should be reviewed when a new dependency is added to any service on the list, because a dependency that was safe to leave unprotected when it was called by one service may no longer be safe if it is called by five services.

3. Retry policy and backoff strategy

Document the retry policy for each dependency type: the maximum number of retry attempts, the backoff algorithm (fixed, exponential, or exponential with jitter), the initial delay, the maximum delay, the jitter range, and the total retry budget (the maximum elapsed time across all attempts including delays). The total retry budget must be shorter than the upstream timeout that bounds the retry sequence — document the upstream timeout constraint explicitly as the reason for the retry budget limit, so that if the upstream timeout changes, the reviewer knows to reconsider the retry budget as well. Include the list of retriable error conditions: the specific HTTP status codes, gRPC status codes, database error codes, or exception types that the retry predicate considers retriable. Errors not on the retriable list should not be retried — the default should be to not retry and to propagate the error, with retries requiring explicit justification per error type.

Document the idempotency requirements per operation type. The retry policy and the idempotency surface interact: a retry policy that retries write operations without idempotency keys is a source of duplicate side effects. For each downstream dependency, enumerate the write operations and specify whether they require idempotency keys, and if so, the idempotency key format and scope — how the key is generated (UUID per logical operation, hash of the request body, combination of user ID and request timestamp), how long the server retains keys for deduplication (the minimum retention must exceed the maximum retry window), and how the server signals that a request was a duplicate (typically a 200 OK with the original response body, rather than a 409 Conflict that might be misinterpreted as a retriable error). The idempotency key format and retention policy should be documented in the API contract of the downstream service — the resilience decision record references the API contract and specifies how callers are required to use idempotency keys. A service that consumes a downstream API should not implement its own idempotency workarounds (such as querying for the existence of a resource before creating it) when the downstream API provides idempotency key support — the query-then-create pattern introduces a time-of-check to time-of-use race condition and may not work correctly under concurrent load. This connects to the queue and messaging decision record: message queues typically provide at-least-once delivery guarantees, meaning consumers must treat all message processing as idempotent — a message may be delivered more than once, and the consumer must produce the same outcome for all duplicate deliveries. The idempotency requirement for queue consumers is the same as the idempotency requirement for retried HTTP calls, and the idempotency key is the message's unique identifier rather than a caller-generated UUID.

The retry policy should also specify the backpressure behavior — what the service does when all retry attempts are exhausted and the operation has failed. For synchronous request-handling, the exhausted retry path returns an error to the upstream caller. For asynchronous processing (message queue consumers, background job workers), the exhausted retry path must decide between writing the operation to a dead-letter queue (for later manual or automated retry after the dependency recovers), dropping the operation (acceptable for non-durable data like analytics events, not acceptable for user-initiated writes), or blocking the worker (waiting for the dependency to recover before processing the next message, which may cause the queue to accumulate but prevents data loss). The backpressure behavior should be documented per operation type, matching the durability requirement of the operation — a user-facing payment operation that fails all retries should go to a dead-letter queue for manual review; an analytics event that fails all retries may be dropped without business impact. Without documenting the backpressure behavior, engineers implement it independently, producing a fleet where some services drop failed operations silently and others block indefinitely, and the behavior is only discovered during an incident when the impact of the chosen behavior becomes visible at scale.

4. Timeout hierarchy and deadline propagation

Document the timeout value for each service-to-dependency call in the dependency graph, with the derivation chain: the downstream dependency's P99 response time in normal operation, the number of retry attempts the timeout must accommodate, the headroom factor for normal variability, and the upstream timeout that constrains the value. The documentation should be structured as a table or diagram that shows the full timeout chain from the end-to-end SLA down to the innermost dependency timeout, so that the relationship between the values is visible and changes at any layer can be evaluated against the full chain. A timeout value change that appears safe in isolation (tightening the database timeout from 200ms to 100ms) may violate the downstream service's P99 distribution — if the database's P99 is 85ms in normal operation, a 100ms timeout will trip on 1% of normal queries, triggering unnecessary retries. The P99 and P999 response time distributions of each dependency should be included in the timeout derivation, with the timeout set at a multiple of the P99 that accommodates normal variability without impacting normal operations.

For context propagation, document the implementation pattern: the header name or gRPC metadata key used to carry the deadline, the format of the deadline value (absolute UTC timestamp or relative remaining milliseconds), and how each service reads the incoming deadline, applies a processing-time deduction, and passes the remaining deadline to its outbound calls. The context propagation pattern must be implemented consistently across all services — a service that does not propagate the incoming deadline starts a fresh local timeout for all outbound calls, breaking the end-to-end deadline chain. Include the failure mode for expired deadlines: a service that receives a request with an expired deadline (the remaining time is zero or negative) should return a fast error immediately without processing the request, since any work done on the request will be discarded by the upstream caller that has already timed out. Documenting the expired-deadline behavior prevents a class of wasted work: a service that receives and processes a request after the upstream has already returned a timeout error to the user will produce a state change (a database write, a queue publish) that the user's client will never see, but that may conflict with a retry that the client or upstream service initiates. This connects to the CI/CD pipeline decision record: the timeout hierarchy should be validated in integration tests — a test that verifies that a slow downstream dependency produces a fast error at the service level within the configured timeout, and that the service returns the error before the test's own timeout expires, catches timeout misconfiguration before it reaches production. Without the integration test, the timeout configuration is validated only during production incidents.

Include the alert thresholds for timeout proximity: the P99 response time of each downstream dependency relative to the configured timeout for that dependency. An alert that fires when the P99 approaches 70% of the timeout value (for example, P99 is 140ms and the timeout is 200ms) gives the team advance warning before the P99 exceeds the timeout and retry amplification begins. The threshold for the alert should be derived from the expected variability of the dependency's response time — a dependency with a P99 of 140ms and a P999 of 180ms has little margin for the tail to exceed a 200ms timeout under elevated load; a dependency with a P99 of 50ms and a P999 of 90ms has more margin. The alert threshold calibration belongs in the resilience decision record alongside the timeout values, because the alert is the monitoring mechanism that catches timeout-configuration drift: when the dependency's P99 rises gradually over weeks (due to data growth, query plan degradation, or capacity reduction), the alert fires before the P99 exceeds the timeout, giving the team time to adjust the timeout, the dependency's capacity, or both before the retry amplification begins at scale.

5. Observability instrumentation for resilience patterns

The resilience patterns in the decision record are only effective if their behavior is visible in monitoring. Document the required metrics and logs for each resilience pattern. Circuit breakers should emit metrics for the number of calls in each state per circuit (closed calls, open circuit rejections, half-open probe calls), the circuit state transition events (closed-to-open, open-to-half-open, half-open-to-closed, half-open-to-open), and the failure rate at the time of the state transition. These metrics provide the signal for validating circuit breaker calibration — if circuits are tripping frequently on transient errors that resolve before the reset timeout expires, the failure threshold or monitoring window needs adjustment. If circuits are not tripping during dependency degradations that cause user-visible errors, the threshold is too high or the latency-based detection is missing. The circuit breaker metrics are also the primary diagnostic tool during an incident: the circuit-open metric identifies which dependency is degraded before the on-call engineer has traced the error logs manually, reducing the mean time to identify the root cause.

Retry metrics should record the number of retry attempts per operation type (labeled by service, dependency, and operation), the retry success rate (the fraction of retried operations that eventually succeed versus exhaust all retries), and the distribution of retry counts before success (how many operations required 1 retry versus 2 retries versus exhausting all retries). The retry success rate distinguishes retriable transient errors (which succeed on retry at a high rate) from sustained errors (which fail all retries at a high rate). A high retry count with a low retry success rate indicates that the downstream service is degraded, the retry policy is not helping, and the circuit breaker should be tripping to stop the retry amplification. A low retry count with a high retry success rate indicates that the retry policy is handling transient errors correctly. The distribution of retry counts identifies dependencies where the second or third retry is rarely needed — reducing the maximum retry count on those dependencies reduces the retry amplification factor without impacting the operation success rate. The retry metrics connect to the observability platform decision record: the retry metrics generate a volume of data proportional to the request rate multiplied by the average retry count per request, which can be significant under load. The observability platform must be able to handle the metric volume during a retry storm — the period when the retry metrics are most critical is also the period when they are being generated at the highest rate.

Timeout proximity alerts should be configured for the P99 response time of each dependency relative to the configured timeout, as described in the timeout hierarchy section. Additionally, alert on the circuit breaker open rate per dependency — the fraction of the monitoring window during which the circuit is in the open state — as a measure of dependency reliability over time. A dependency whose circuit is open for more than 5% of the monitoring window is a dependency with a reliability gap that may require negotiation with the provider or architectural mitigations (caching, fallback, removal from the critical path). The alert thresholds for circuit open rate should be calibrated per dependency based on the dependency's SLA and the business impact of its unavailability. A payment processor circuit that is open for 1% of the monitoring window represents a meaningful failure rate for a critical business operation; a recommendation engine circuit that is open for 5% of the monitoring window may be within acceptable operational bounds given the availability of fallback responses. Document the alert thresholds and their business-impact rationale in the resilience decision record, so that when thresholds are triggered, the on-call engineer knows whether the alert represents a critical incident requiring immediate response or a known degradation within the acceptable range.

What to do with the circuit breaker and resilience pattern decision record

The resilience decision record's primary audience is the engineer who adds a new downstream dependency to an existing service, the engineer who builds a new service that will call existing dependencies, and the on-call engineer investigating a cascading failure incident. The engineer adding a new dependency needs to know whether the dependency requires a circuit breaker (consult the failure mode inventory and the disabled list), what library to use, what threshold values to start with, and whether the operation type requires idempotency key support before retry logic is added. The engineer building a new service needs the timeout hierarchy derivation chain to set the correct timeout values from the start, rather than inheriting defaults that are inconsistent with the fleet's timeout budget. The on-call engineer investigating a cascading failure needs the dependency graph to trace the propagation path, the circuit breaker metrics to identify which circuit opened first and when, and the retry metrics to quantify whether retry amplification is contributing to the ongoing load. All three audiences are served by a single document that covers the dependency graph, the circuit breaker configuration, the retry policy, the timeout hierarchy, and the observability instrumentation.

If the resilience decision record was not written when the service fleet was small enough to reason about holistically, the most useful reconstruction starts with the dependency graph. Map every outbound call in every service — who calls whom, with what operation — and identify the failure propagation paths. Then audit the circuit breaker coverage: which dependencies have circuit breakers, which do not, and which are on the disabled list for documented reasons. The gaps are the unprotected paths that a future failure will propagate through. Prioritize filling the gaps for the highest-traffic and business-critical dependencies first, since those are the paths through which a degradation will amplify most aggressively. A circuit breaker added to the checkout service's calls to the payment processing service — the gap identified in the e-commerce cascade incident — would have contained that incident to the payment tier. Document the chosen threshold values as decisions with rationale, not as constants in a configuration file — the rationale is what allows the values to be revisited when the dependency's behavior changes.

The ADR template provides the starting structure for the decision record. The WhyChose extractor can surface resilience-related decisions from ChatGPT or Claude conversations during the service design phase — the discussion where circuit breaker library options were compared, where the timeout value for a database call was chosen, or where the team debated whether a non-critical service call should block or degrade gracefully may contain reasoning that was never written into the architecture documentation and is now recoverable from the chat export. The decisions never written down essay covers why resilience configuration is among the decisions whose absence is least visible until a production incident — circuit breakers that are present work silently, and circuit breakers that are absent also work silently until the exact dependency combination and load level that reveals the gap occurs during a production incident. The microservices vs. monolith decision record covers the architectural context in which the resilience decision record is most critical: a monolith's failure surface is bounded by the process boundary, while a microservice fleet's failure surface is the full dependency graph, and the resilience decision record is the document that makes the fleet's failure surface legible before it is discovered during an incident.