The saga pattern decision record: why the coordination model you chose determines your compensating transaction idempotency surface and your partial completion recovery exposure

A 35-person e-commerce company decomposed its monolithic checkout flow into a set of microservices after a scaling review identified the checkout handler as the bottleneck preventing horizontal scaling during traffic spikes. The new checkout architecture comprised four services: an order service that created and tracked order records, a payment service that charged the customer's card, an inventory service that decremented available stock, and a shipping service that initiated carrier fulfillment. The team chose choreography-based saga coordination: rather than a central orchestrator that invoked each service in sequence, each service would listen on a Kafka topic for events from the preceding service, perform its step, and emit an event for the next service. The payment service listened for OrderCreated events, charged the card, and emitted PaymentProcessed. The inventory service listened for PaymentProcessed events, decremented stock, and emitted InventoryReserved. The shipping service listened for InventoryReserved events and initiated fulfillment.

The architecture had been designed and reviewed in two sessions focused on service decomposition, event schema design, and Kafka topic configuration. Neither session addressed what would happen if a consumer service was offline when its trigger event was produced. The Kafka consumer group configuration for the inventory service's listener used the default auto.offset.reset=latest setting — the default Kafka behavior for a consumer group encountering a topic it has no committed offset for, or reconnecting after a gap. The engineers who configured the consumer group were aware that latest was the default; they had not explicitly set it to earliest, and the founding sessions that designed the choreography did not specify what offset reset behavior was required for each consumer in the saga chain.

During the company's largest holiday sale — a 36-hour promotional window with 12x normal order volume — the inventory service's Kafka consumer process was killed by an OOM error after processing 2,100 consecutive PaymentProcessed events loaded a high-cardinality inventory cache beyond its memory limit. The consumer was offline for 8 minutes while the Kubernetes pod restarted and the service initialized its cache. During those 8 minutes, the payment service continued processing orders and emitting PaymentProcessed events to the Kafka topic. The topic accumulated 340 events corresponding to 340 successfully charged orders. When the inventory service's consumer reconnected, its consumer group had no committed offset for the most recent 8-minute window — it had last committed an offset before the OOM kill. With auto.offset.reset=latest, the consumer group resumed from the current head of the partition, skipping the 340 accumulated events entirely. The inventory service decremented stock for all orders placed after its restart. It never decremented stock for the 340 orders placed during the outage. Those 340 orders had been charged, confirmed to the customer, and assigned order numbers. Inventory records showed none of them as decremented.

The discrepancy was not discovered by monitoring — the inventory service's metrics showed normal processing rates after restart, because it was processing the post-restart events correctly. It was discovered three days later during a weekly physical stock audit in which warehouse staff found that 47 product SKUs had negative available stock according to the system. The reconciliation took four days of engineering time to match order records against inventory decrement logs and identify the 340 orders that had not triggered decrements. The company applied manual inventory decrements, voided 23 orders that had been placed for items that were genuinely out of stock, issued $8,200 in refunds, and sent apology emails to affected customers. The founding sessions that designed the choreography saga had documented "Kafka-based event routing, one topic per service transition." They had not specified the required delivery guarantee for each saga step event — at-least-once for any step with a side effect, which for the inventory step meant auto.offset.reset=earliest and explicit offset management to prevent skipping events on consumer restart.

A 25-person B2B SaaS company built an orchestration-based saga for its customer onboarding flow. The saga coordinated four steps executed in sequence by a central orchestrator process: provision the account workspace in the company's infrastructure, charge the customer's card for the first billing period, send a welcome email with login credentials, and configure a default project and dataset. The orchestrator stored its progress in a saga_log table in PostgreSQL, recording each step's status, start time, and completion time. If any step failed after the charge step had completed, the orchestrator ran compensating transactions in reverse order: send a failure notification email, issue a refund for the charge, and deprovision the workspace. The design was reviewed in a session focused on ensuring that failed onboarding did not leave customers in a partially provisioned state with an outstanding charge.

The refund compensation was implemented as a function that called the payment gateway's refund API, passing the charge ID from the original charge step. The function did not generate or pass an idempotency key. The payment gateway's documentation stated that idempotency keys were optional but recommended for refund operations, and that without an idempotency key, each API call was treated as a distinct refund request regardless of whether an identical request had been issued recently. The founding session that designed the compensating transactions had not discussed idempotency. The session's goal was to ensure that every failed onboarding resulted in a refund; the team considered the compensation flow correct as long as the refund API returned a success response.

In production, a specific failure pattern emerged for a subset of customers: the welcome email step timed out. The email provider had a documented P99 response time of 8 seconds; the saga orchestrator had a 30-second step timeout; under normal conditions the step completed in 2 to 4 seconds. During a period of elevated load on the email provider's API, response times degraded to 35 to 50 seconds for a subset of requests. For 180 customers across two weeks, the welcome email step timed out at 30 seconds, the orchestrator marked the step as failed, and the compensating flow began. The compensation issued a refund for the charge step by calling the payment gateway's refund API — which succeeded and returned HTTP 200. The orchestrator then attempted to write the compensation result to the saga_log table to mark the refund step as complete. That write failed with a deadlock error: a concurrent database connection from the orchestrator's health-check process was holding a lock on the same saga log row. The deadlock was detected and the write was rolled back. The orchestrator's retry logic, which was designed to handle transient database errors, retried the entire compensation step — which meant calling the payment gateway's refund API again, without a new idempotency key (because no idempotency key had been stored from the first call to reuse on retry). The payment gateway processed the second refund call as a new, distinct refund request and issued a second refund. For 180 customers, two refunds were issued: one from the first compensation call that succeeded before the saga log write failed, and one from the retry. The total double-refund amount was $23,400.

The engineering team identified the failure six days after the first occurrence, when a reconciliation between payment gateway records and the saga log showed 360 refund API calls for 180 saga compensation records. The remediation required issuing 180 chargeback-correction requests to the payment gateway, each of which required manual review by the payment operations team and took 2 to 5 business days. The founding session that designed the compensating transactions had documented "issue refund via payment gateway refund API on onboarding failure." It had not documented that the refund call must be idempotent, that an idempotency key must be generated before the call and stored in the saga log before the call is issued, or that compensation steps are subject to the same retry semantics as forward steps and must therefore meet exactly the same idempotency requirements as forward steps with external side effects.

A 20-person logistics SaaS company built an orchestration-based saga for its shipment fulfillment flow. The saga coordinated five steps: validate the delivery address against a geolocation API, reserve a carrier slot with the preferred carrier's booking API, charge the customer's account, generate a shipping label via the carrier's label API, and send a shipment confirmation notification to the customer. The orchestrator stored each step's status in a saga log and was designed to retry each step up to three times before marking it as failed and triggering manual review. The company added a monitoring job that queried the saga log every 5 minutes and sent an alert to the on-call engineer when any saga had been in a non-terminal state for more than 45 minutes.

The alert was added in a session focused on operational visibility — the engineering team had noticed that some sagas appeared to stall without triggering an explicit failure, and wanted to ensure that stuck fulfillments were caught before the customer contacted support. The session documented "add a 45-minute timeout alert for in-progress sagas." It did not document what the engineer receiving the alert should do when they investigated and found a saga in a partially completed state. The assumption in the session was that the engineer would use their judgment based on what they found in the saga log — which step had failed, what the error was, and whether retrying it made sense. The session did not produce a recovery procedure table mapping each possible partial state to a set of safe and unsafe recovery actions.

Three months after the monitoring was deployed, the alert fired for a shipment saga that had been in progress for 52 minutes. An on-call engineer queried the saga log and found the following state: validate_address = completed, reserve_carrier = completed, charge_account = completed, generate_label = failed (the carrier's label API had returned HTTP 500 after 43 seconds), notify_customer = not_started. The engineer's investigation took 22 minutes, during which they reviewed the carrier API's status page (showing no incidents), the orchestrator logs (showing a single HTTP 500 response from the carrier API followed by three retry attempts that all timed out), and the carrier API's documentation (which stated that label generation requests were idempotent if the same booking reference was used). The engineer interpreted "idempotent if the same booking reference is used" as confirmation that retrying was safe, and manually triggered a retry of the generate_label step by calling the carrier's label API with the booking reference from the carrier slot reservation step.

The retry succeeded and returned a label. The engineer marked the saga as complete and closed the alert. Three days later, 67 customers whose shipments had been processed during the same 2-hour window contacted support reporting that they had received two complete shipments — two boxes, both with valid carrier labels, both containing the correct items. The root cause was that the original generate_label HTTP 500 response had been returned by the carrier's API after the label had been generated and the carrier charge had been applied. The 500 was an error in the carrier API's response serialization layer: the label data had been written to their database, the carrier charge had been debited, and the label image had been rendered, but the HTTP response writer had encountered an error marshaling the response body and returned a 500 instead of the 200 with the label payload. The carrier API had correctly applied the action; only the response delivery had failed. When the engineer retried the call with the same booking reference, the carrier API checked its database, found an existing label for the booking reference, and returned that label — along with creating a new shipment record and generating a second carrier charge and label, because the carrier's idempotency implementation was scoped to prevent duplicate label generation within a 60-second window, not across all time. The retry occurred 74 minutes after the original call, outside the 60-second idempotency window. Sixty-seven shipments were fulfilled twice, each with a valid label and a carrier charge billed to the company. The total cost of the duplicate shipments — return shipping, repackaging, second carrier charges, and customer support time — was $18,400.

The founding session that added saga timeout monitoring had documented "alert when a saga is stuck for 45 minutes." It had not documented the set of possible partial states the saga could be in when the alert fired, the recovery procedure for each state, the idempotency window and scope of the carrier label API, or the distinction between a step that failed before the external service applied the action and a step that failed after the external service applied the action but before the response was delivered — which is the indeterminate state that makes retrying without a verified-safe idempotency contract dangerous.

Structural properties set by the saga pattern decision

Three structural properties are determined when a team decides how to implement distributed transaction coordination across services. None appear explicitly in the session that picks orchestration or choreography, the session that adds compensating transactions, or the session that adds operational monitoring — they are the operational consequences of design choices made under the assumption that coordination, compensation, and visibility together constitute a complete saga implementation.

Property 1: The compensating transaction and the idempotency requirement. Every compensating action in a saga must be idempotent, because the saga orchestrator will retry compensating transactions on failure just as it retries forward steps on failure. The retryability of sagas — their primary advantage over two-phase commit — is inseparable from the idempotency requirement: a saga can survive orchestrator failures, network partitions, and transient errors only if every step and every compensation can be retried safely. For external service calls, idempotency requires generating a stable key before the call is issued and storing that key in the saga log before the call is made. The correct sequence is: generate the key, write it to the saga log, issue the call with the key, update the saga log on success. Reversing any two steps in this sequence breaks the idempotency guarantee: generating the key after the call loses the key if the call succeeds and the key generation fails; writing the key to the saga log after the call loses the key if the call succeeds and the database write fails; the retry that follows has no stored key and issues a new call that the external service treats as a new request.

The idempotency contract must also specify the scope and duration of the external service's deduplication window. A service that deduplicates idempotency keys for 60 seconds provides no protection for retries that occur after 60 seconds — which is the common case when the retry follows a manual investigation or an extended timeout. The webhook delivery decision record documents the same requirement from the receiving side: at-least-once delivery with idempotent processing is the only delivery contract that survives retries; the deduplication key must be scoped to the event identity (not just a time window), and the key must be stored before the action is applied, not after. The same invariants apply to saga compensating transactions, which are structurally identical to webhook event processing — an external trigger that must be processed exactly once regardless of how many times it is delivered.

Property 2: The choreography event delivery guarantee and the service unavailability surface. A choreography-based saga routes each step's trigger as an event on a message broker. The set of failure modes available to the saga is bounded by the delivery guarantees of the broker and the consumer configuration that the saga design specifies. For any saga step that has a side effect — decrementing inventory, provisioning infrastructure, charging a payment method — the required delivery guarantee is at-least-once: the event must be delivered to the consumer and processed at least once, even if the consumer was offline when the event was first produced. At-least-once delivery requires three things from the broker configuration: durable topic storage (the broker must retain events until they are consumed, not until they are produced), acknowledged consumer offsets (the consumer must commit its offset only after processing the event and applying the side effect, not immediately on receipt), and explicit offset reset behavior on consumer restart — auto.offset.reset=earliest for consumer groups that must process events accumulated during an outage, rather than the default latest which skips them.

At-least-once delivery for a step with a side effect implies that the consumer processing the event must itself be idempotent — the same event may be delivered more than once (if the consumer processes the event and applies the side effect but crashes before committing its offset, the event will be redelivered on restart). The saga design must specify both the delivery guarantee for each saga step event and the idempotency requirement for the consumer processing each event. The message broker decision record documents the durability configuration that enforces at-least-once delivery: replication factor, acknowledgment mode (all replicas vs. leader only), and retention policy; the saga design must verify that the broker configuration matches the delivery guarantee it requires before assuming the guarantee is in place. The event-driven architecture decision record documents the broader delivery guarantee model: at-least-once with idempotent consumers is the standard pattern for event-driven workflows where message loss is more dangerous than message duplication, because duplication can be deduplicated while loss requires reconstructing a state that may not be recoverable.

Property 3: The saga state and the recovery procedure for partial completion. A saga in a partially completed state — some forward steps completed, some did not — requires a documented recovery path for each possible partial state, because the set of safe recovery actions depends entirely on which steps completed and what those steps did on the external services they called. The number of distinct partial states grows as the number of steps increases: a five-step saga has 32 possible state combinations (each step either completed or not), though in practice the valid partial states are constrained by the sequential execution order — only states in which all steps up to a certain point completed are reachable under normal operation without concurrent modification. Even with this constraint, a five-step saga has six meaningful partial states (zero through five steps completed), and for each state the recovery procedure depends on the reversibility of the completed steps.

The critical distinction in partial state recovery is between the failed state and the indeterminate state for each step that calls an external service. A step is in the failed state if the external service definitively did not apply the action — the request was rejected before processing, or a 4xx response was received indicating an invalid request. A step is in the indeterminate state if the external service may have applied the action before the error occurred — a connection timeout after the request was delivered, a read timeout after the request was sent, or a 5xx response that may have been produced after the action was committed. Retrying a step in the failed state is safe if the retry uses a new or the same idempotency key. Retrying a step in the indeterminate state is safe only if the external service supports idempotent retry with the same idempotency key across an unbounded time window — not just within a 60-second deduplication window. The error handling strategy decision record documents the retry safety classification for each error type: connection errors before request delivery are definitively failed; timeout errors after request delivery are indeterminate; 4xx errors are definitively failed; 5xx errors are indeterminate. The saga recovery procedure must document, for each step and each error class, whether the step is safe to retry, whether manual verification of the external service's state is required before retry, and what the consequence of an unsafe retry is if it is executed in error.

What the founding session records and what it omits

The founding saga session typically records the coordination model selected (orchestration or choreography), the services and steps in the saga flow, the event topics or orchestrator step names, the retry count and timeout for each step, and the high-level description of the compensating flow ("refund the charge if any step after the charge fails"). It may record the rationale for choosing orchestration over choreography or vice versa — the team's preference for centralized visibility, or the desire to avoid coupling services directly to each other through a shared orchestrator. What it does not record is the idempotency specification for each step that calls an external service — whether the call is idempotent, what key is used, how the key is generated and stored, and what the deduplication window and scope of the external service's idempotency guarantee is. It does not record the delivery guarantee for each saga event and the consumer configuration required to enforce it. It does not record the partial state map — the set of reachable partial states, the recovery procedure for each, and the distinction between steps in the failed state and steps in the indeterminate state.

The idempotency omission produces a failure class that is visible only under retry conditions, which means it is invisible during testing (which does not test failure-followed-by-retry sequences for external calls) and invisible during normal operation (which does not trigger compensation flows). The failure surfaces during the combination of a transient compensation failure followed by an orchestrator retry — a combination that happens regularly in production but almost never in test environments. The double refund is the canonical form: the compensation call succeeds, the orchestrator's record-keeping fails, the orchestrator retries the call, and the external service processes it as a new request because no idempotency key was stored to reuse. The distributed locking decision record documents an analogous pattern in the context of distributed state management: a write that succeeds on the server but fails to acknowledge to the client is retried, and without a stored lock token or idempotency key, the retry creates a second write that the server treats as new. The remedy in both cases is identical: generate the key before the call, store it before the call, use it on every attempt.

The delivery guarantee omission produces a failure that is specific to choreography-based sagas and is triggered by consumer restarts under load — an event that happens regularly in production Kubernetes environments due to OOM kills, rolling deployments, and liveness probe failures. The failure is invisible in testing because test environments do not typically reproduce the consumer offset reset behavior of production consumer groups, and the auto.offset.reset configuration is not part of the typical code review for a Kafka consumer. The failure surfaces during the combination of a consumer restart and a burst of events produced during the outage window — which happens most severely during traffic spikes, which are also the conditions under which OOM kills are most likely. The circuit breaker resilience decision record documents the interaction between service restarts under load and event processing: a circuit breaker on the consumer's downstream dependencies can prevent the OOM kill that causes the restart, but only if the consumer's resource usage is bounded by a circuit limit rather than allowed to grow unbounded with event backlog processing. Preventing the OOM kill is a mitigation; the correct fix is specifying the offset reset behavior that ensures event delivery survives restarts regardless of cause.

The partial state recovery omission produces a failure that requires both a monitoring alert (to detect the stuck saga) and a human decision (to determine the recovery action) before it manifests. The failure is the human decision made without a documented recovery procedure — the engineer who retries a step that is in the indeterminate state, because the engineer does not know that the step is indeterminate rather than definitively failed, and does not know that the external service's idempotency window has expired. The observability strategy decision record documents the distributed trace correlation requirement for saga state visibility: each saga step must carry the saga ID as a trace attribute, so that the full execution path of any saga — including the external API calls made by each step and their outcomes — is reconstructable from the distributed trace without requiring the engineer to join across multiple log sources under time pressure during an incident. Trace correlation does not prevent the partial completion; it makes the partial state visible enough that the engineer can determine whether the step is definitively failed or indeterminate before taking a recovery action. The microservices vs. monolith decision record documents the operational cost of distributed transaction coordination that the saga pattern is designed to manage: the saga pattern is the correct approach for coordinating multi-service transactions, but it transfers the atomicity guarantee from the database (which provides it for free via ACID transactions) to the application (which must implement it through idempotency, delivery guarantees, and recovery procedures — none of which the database provides). The saga design must account for this transfer of responsibility explicitly.

The WhyChose decision extractor finds the founding saga sessions in your ChatGPT and Claude export — the "how should we handle the checkout flow across services?" architecture conversation, the "we need compensating transactions for failed onboarding" design session, the "let's add monitoring for stuck sagas" observability discussion. It extracts the coordination model selected and the options considered, not the surrounding architectural debate that buries the idempotency specification question in forty messages about Kafka topic naming conventions and consumer group configurations. The transaction isolation decision record documents the local transaction that each saga step must use to ensure its own state is consistent: each step's application of its side effect and its update to the saga log must happen in the same local database transaction, so that a crash between the two leaves the saga log in a state that accurately reflects whether the side effect was applied. Without the local transaction wrapping both writes, a crash between them produces a saga log state that does not reflect reality, and the recovery procedure for that state is indeterminate for the same reason that the carrier API call was indeterminate — the action may or may not have been applied, and there is no record to consult.

The five ADR sections for a saga pattern decision

Section 1: Coordination model selection. Specify whether the saga uses orchestration (a central coordinator that explicitly invokes each step and drives compensating transactions) or choreography (each service emits events consumed by the next service in the chain, with no central coordinator), and document the rationale. Orchestration produces a centralized saga log that is the single source of truth for saga state, making partial state visibility and recovery procedure execution straightforward — but adds a single component whose availability determines whether any saga can progress. Choreography avoids the centralized component but distributes the saga state across the event log of each participating service, making the saga state reconstructable only by joining across multiple event streams, and puts the delivery guarantee burden on the broker and consumer configuration rather than the orchestrator's retry logic. Document for the chosen model: the saga log location and schema (for orchestration), or the event topics and consumer group assignments (for choreography); the retry count and timeout for each step; and the conditions under which a saga transitions to a terminal failed state requiring manual intervention rather than continued automated retry. The microservices vs. monolith decision record documents the broader trade-off: saga complexity is a direct cost of microservices decomposition, and the decision to decompose should account for the operational investment in saga design that decomposition requires.

Section 2: Compensating transaction design and idempotency specification. For each step in the saga that has a side effect, specify the compensating transaction that reverses it and the idempotency contract for both the forward and compensating calls. Document: the idempotency key generation method (deterministic from saga ID, step name, and attempt number), the storage location for the key (the saga log row for the step, written before the call), the external service's idempotency support (whether the service accepts an idempotency key header, the deduplication scope, and the deduplication window), and the failure behavior if the external service does not support idempotent calls (manual deduplication layer, or compensation flow that prevents double invocation by design). For steps where the compensation is not possible — a step that sends an irreversible notification or triggers a physical process that cannot be recalled — document the compensation as "notification only" (an alert is sent to the operations team) and document the partial state in which this step has completed and a subsequent step has failed as one requiring manual handling rather than automated compensation. The idempotency specification for each step is the single most important section of the saga decision record, because every other failure mode in saga operation — duplicate charges, double shipments, duplicate refunds — is a consequence of a missing or incomplete idempotency specification.

Section 3: Event delivery guarantee and consumer configuration (for choreography). For each event in the saga event chain, specify the required delivery guarantee and the broker and consumer configuration that enforces it. For events triggering steps with side effects, the required guarantee is at-least-once: the broker must retain the event until it is acknowledged by the consumer, the consumer must commit its offset only after applying the side effect and updating its own state, and the consumer's auto.offset.reset must be set to earliest to process events accumulated during an outage rather than skipping them. Document the consumer group ID for each consumer, the offset management strategy (auto-commit disabled, manual offset commit after processing), the maximum message retention period for each topic (which must exceed the maximum expected consumer outage duration), and the dead-letter topic configuration for events that fail processing after the maximum retry count. For each consumer, document the idempotency requirement: because at-least-once delivery may deliver the same event more than once, every consumer processing a saga step event must be idempotent — applying the same event twice must produce the same state as applying it once. The message broker decision record documents the broker durability configuration that is the prerequisite for at-least-once delivery.

Section 4: Saga log and state persistence. Specify the saga log schema, the local transaction boundary for each step, and the concurrency control for concurrent saga log writes. The saga log must record, for each step: the step name, the current status (pending, in_progress, completed, failed, compensation_in_progress, compensation_complete), the idempotency key generated for the step's external call (stored before the call is made), the idempotency key generated for the step's compensating call (stored before the compensation call is made), the start time, the completion time, and the error details for failed steps. The local transaction boundary for each step must wrap both the step's domain write (the side effect applied to the step's own service's database) and the saga log status update in a single ACID transaction. This is the "transactional outbox" pattern applied to saga state: the saga log row and the domain write are committed atomically, so a crash between them is impossible. The transaction isolation decision record documents the isolation level required for the saga log transaction: read committed is sufficient for preventing dirty reads of saga state; serializable is required only if concurrent saga operations on the same saga ID are possible and must be prevented from interleaving. Specify the behavior when the saga log transaction fails (the step is retried from the beginning, using the idempotency key already stored in the log before the transaction began) and the behavior when two saga orchestrator instances attempt to process the same saga simultaneously (the second instance must detect the in-progress lock via an optimistic lock or a select-for-update on the saga log row and back off).

Section 5: Partial completion and indeterminate state recovery procedures. Enumerate each reachable partial state for the saga — the set of states in which some steps have completed and some have not — and document the recovery procedure for each. For each partial state, document: which steps are completed (and confirmed as applied on the external service), which steps are failed, whether any failed step is in the definitively-failed state (the external service did not apply the action) or the indeterminate state (the external service may have applied the action before the error was returned), the safe recovery action for definitively-failed steps (retry with the stored idempotency key), the required verification procedure for indeterminate steps before any recovery action (query the external service's status endpoint using the transaction reference from the saga log to verify whether the action was applied), and the escalation path if the external service's status endpoint is unavailable or returns an ambiguous response. The indeterminate state procedure must document explicitly that retrying a step in the indeterminate state without first verifying the external service's state is unsafe, and that the consequence of an unsafe retry is a duplicate application of the step's side effect — a second charge, a second shipment, a second label generation — with real financial and operational consequences. The observability strategy decision record documents the distributed trace requirements that make partial state investigation tractable: each saga step's external API calls must carry the saga ID as a trace attribute, so that the call outcome — including whether the external service applied the action before returning the error — is reconstructable from the distributed trace without requiring log correlation under time pressure.