The API idempotency decision record: why the idempotency key model you chose determines your duplicate charge surface and your retry safety contract

A 12-person payments startup built a checkout API that accepted card payments from mobile clients. The API delegated charge processing to Stripe: it received the payment parameters from the client, called Stripe's charges endpoint, and returned the charge result to the mobile app. The mobile client's network handling used a 5-second timeout — if no response arrived within 5 seconds, the client retried the request once. The retry was implemented straightforwardly: if the first attempt timed out, the client submitted the same payment request to the same endpoint with the same parameters.

A customer on a degraded mobile connection attempted to purchase an annual subscription for $299. The request reached the startup's API server. The API server called Stripe, Stripe accepted the charge and returned a successful response, and the API server began composing its response to the mobile client. The connection dropped during the response transmission — the TCP ACK for the HTTP response never arrived at the server. From the mobile client's perspective, the request timed out. The client retried. The retry arrived at the API server as a fresh HTTP POST with the same payment parameters. The server had no mechanism to recognize that this was a retry of the operation it had already completed. It called Stripe again. Stripe charged the card a second time. The customer was billed $299 twice.

The customer noticed the duplicate charge within an hour and called support. The startup issued a refund for the second charge. But the refund took three business days to appear on the customer's card. The customer filed a dispute with their card issuer before the refund appeared, triggering Stripe's dispute process and a $15 dispute fee. The customer cancelled their subscription that evening.

The team's post-incident review identified the fix: add Stripe's idempotency key mechanism to the API. Stripe accepts an optional Idempotency-Key header; if two requests arrive with the same key, Stripe processes the first and returns the cached response to the second without charging the card again. The team added idempotency key support to their payment endpoint: the mobile client generated a UUID for each payment intent and sent it as the X-Idempotency-Key header; the server forwarded this key to Stripe. Future retries of the same payment intent would be safely deduplicated by Stripe.

The fix resolved the double-charge problem but introduced a new one that was not discovered until three months later. The mobile client generated idempotency keys deterministically from the payment intent ID — a stable identifier that persisted across the user's session and across app restarts. When a customer's payment failed (the card was declined), the mobile app showed an error and allowed the customer to retry with a different card. But the retry was submitted with the same idempotency key, because the payment intent ID had not changed. Stripe received the retry with the original idempotency key and returned the cached failure response from the first attempt — the new card was never charged. Customer support calls began accumulating from users who reported that they could not pay even after entering a valid card. The issue was traced to idempotency key reuse across what were semantically different operations — the original failed charge and the new-card retry were correctly treated by the client as retries of the same intent, but they required Stripe to attempt a new charge rather than replay the cached failure.

The founding session that added Stripe's idempotency key mechanism had not documented what idempotency keys mean for the client's key generation strategy, when a new key is required versus when the existing key is correct, or how the system should behave when a customer legitimately wants to retry a failed operation with different parameters. These were discoverable only when the customer-support pattern became a data point three months after the fix was merged.

A 20-person SaaS platform built a REST API for enterprise customers who integrated it with their internal automation systems. One of the API's core features was a batch create endpoint — customers could POST an array of up to 500 records in a single request. Enterprise automation systems called this endpoint on schedule, typically nightly, to synchronize records from their internal systems into the platform.

One customer's automation had its own retry logic, common in enterprise integration middleware: if an API call returned a network error or a 5xx response, the middleware retried up to three times with exponential backoff starting at 10 seconds. The retry behavior was configured at the middleware layer, independent of the specific API endpoint being called — the middleware treated all 5xx responses as transient and all endpoints as safe to retry.

During a period when the platform's database was experiencing elevated write latency due to a long-running migration, the customer's nightly sync submitted a batch create request for 500 records at 11:45 PM. The migration had increased write latency to 800ms per record. The platform's API server began processing the batch sequentially, writing each record to the database. After 36 seconds — 45 records short of completion — the middleware's 30-second timeout fired and declared the request failed. The middleware retried immediately.

The platform's API server received the retry as a fresh request. It began processing all 500 records from the beginning. The database migration had completed by this point, restoring normal write latency, so the retry completed in 4 seconds and returned HTTP 200. The customer's automation marked the sync as successful. But the original request had not been abandoned on the server — it had been processing independently and completed its remaining 45 records approximately 8 seconds after the retry started. By the time the retry completed, 455 records had been written twice.

The platform's batch create endpoint had no idempotency mechanism — there was no way for the server to recognize that the retry was a re-submission of the same batch operation. The duplicate 455 records were propagated to the customer's downstream systems via webhooks, creating 455 duplicate entries in the customer's own database. The cleanup required a 4-hour coordinated operation involving both the platform's engineering team and the customer's data team to identify and remove the duplicates using the created_at timestamps and record content hashes.

After the incident, the platform added idempotency keys to the batch create endpoint. The team chose Redis as the deduplication store because the database was already under load and adding synchronous idempotency record writes to the same database would worsen the latency problem that caused the incident. The deduplication window was set to 1 hour — a compromise between the engineer who argued for 10 minutes (covering the middleware's 30-second timeout with plenty of margin) and the engineer who argued for 24 hours (covering the scenario where an operator manually resubmits a batch that appeared to fail).

Three weeks after the fix was deployed, the same customer's automation submitted a batch at 11:45 PM and received a timeout at 12:15 AM — 30 minutes into processing. The middleware retried at 12:50 AM, 65 minutes after the original submission. The idempotency record for the original request had been purged from Redis at the 1-hour mark. The retry was processed as a new operation. The original request had completed its full batch of 500 records. The retry created 500 more. The customer had 1,000 records in the platform, 500 of which were duplicates. A second cleanup operation was required.

The founding session that added idempotency keys had not documented the relationship between the deduplication window and the client's maximum retry interval. The 1-hour TTL was a reasonable-sounding compromise that was shorter than the actual retry interval in the scenario it was meant to prevent. The session also had not documented the behavior under Redis unavailability — which became relevant two weeks after the second duplicate incident, when a Redis maintenance window caused the deduplication store to be unreachable for 12 minutes. The team had not decided whether to reject all requests during that window (safe but unavailable) or process them without deduplication (available but potentially duplicating). The on-call engineer made an ad-hoc decision to process without deduplication. No records were duplicated during the 12-minute window, but the behavior was undocumented and might not have been the correct decision had the window coincided with a batch sync.

Structural properties set by the API idempotency decision

Three structural properties are determined when a team decides how to handle idempotency in their API. None appear explicitly in the founding session that added retry logic to the mobile payment client or configured the enterprise integration middleware to retry on 5xx — they are the operational consequences of choices made under the pressure of making the API reliable for clients who experience network failures.

Property 1: The idempotency key model and the duplicate operation surface. An idempotency key is a client-supplied token that the server uses to deduplicate requests. When the server receives a request with an idempotency key it has seen before, it returns the cached result of the original request without re-executing the operation. The idempotency key model determines which operations are covered by this guarantee, how keys are generated and scoped, and what happens when a key is reused in a context the server cannot reconcile with the original operation.

The key generation strategy determines who is responsible for ensuring uniqueness and what the key represents. Client-generated UUIDs (Version 4, random) are the most common approach: each key is unique with overwhelming probability, the client generates it at the point of initiating the operation, and the server accepts any opaque string as a valid key. The risk of UUID-based keys is that the client must manage key lifecycle explicitly — when to generate a new key (a new operation that has never been attempted) and when to reuse an existing key (a retry of an operation that timed out or returned an ambiguous error). A client that always generates a new UUID for each HTTP request will never trigger the deduplication path, making idempotency keys useless for retry protection. A client that reuses the same UUID across semantically different operations (as the payments startup's mobile client did when retrying a declined charge with a new card) will trigger deduplication when it should not.

Deterministic key generation — computing the key as a hash of the operation's parameters — solves the new-UUID-per-retry problem but introduces parameter scope questions. For a payment operation, should the key be a hash of the amount, the customer ID, the payment method, and the timestamp? Including the timestamp means each new attempt is a new key (defeating deduplication). Excluding the payment method means a retry with a different card reuses the same key (producing the erroneous cache hit the payments startup encountered). The key must hash the parameters that define the semantic identity of the operation — "charge customer X exactly once for this purchase intent" — not the parameters that vary between legitimate retries.

The scope of idempotency key uniqueness determines whether two different clients can accidentally use the same key. Per-client scoping (the server namespaces keys by API key or session token) means that client A and client B can use the same UUID without collision. Global scoping means any two clients using the same UUID are treated as the same operation — typically not the intent. The API schema design decision record documents the API authentication model; the idempotency key scope must be documented in relation to the authentication model, because per-client scoping requires the server to associate each idempotency record with the authenticated client identity.

The response for a key collision with different parameters — where the client reuses an idempotency key for what appears to be a different operation — must be HTTP 409 Conflict, not a silent replay of the cached result. A silent replay for a mismatched request would apply the first operation's result to the second operation's context (returning a successful payment result when the client intended to charge a different amount, or returning a batch-create result when the client intended to delete records). HTTP 409 with a response body that explains the mismatch — "this idempotency key was previously used for a payment of $299; your request specifies $149" — allows the client to either generate a new key for a genuinely new operation or confirm that the prior result is the correct outcome and adopt it. The error handling strategy decision record documents the error response format; the idempotency ADR must document the specific error response structure for each idempotency violation (key reuse with parameter mismatch, key reuse during processing, key that references a completed operation).

Property 2: The deduplication store and the window contract. The deduplication store retains idempotency records for the duration of the deduplication window. Each record contains the idempotency key, the original request fingerprint (typically a hash of the request body), the response body, the response status code, and the processing result. When a request arrives with a key that matches an existing record, the server returns the cached response without executing the operation. The store's durability, eviction policy, and availability determine whether the deduplication guarantee holds across the full range of retry scenarios.

An in-memory store (a hash map in the API server process) is the simplest implementation and the most dangerous. It cannot survive process restarts: when the server restarts during the window between a client's original request and its retry, the idempotency record is lost, and the retry is treated as a new operation. In-memory stores also cannot be shared across server instances in a horizontally scaled deployment — if the client's original request was processed by instance A and the retry reaches instance B, the key is not found and the operation is processed again. In-memory idempotency is appropriate only for single-instance deployments where the deduplication window is short enough that a restart is unlikely to occur within it — a scenario that should be explicitly documented as a constraint, not assumed.

A Redis store survives process restarts and is accessible across all server instances in a horizontally scaled deployment. The risk is TTL-based eviction: Redis expires keys based on the configured TTL, and if the TTL is shorter than the client's maximum retry interval, the deduplication guarantee lapses before the client stops retrying. Redis also evicts keys under memory pressure when the maxmemory-policy is set to an eviction policy (allkeys-lru, volatile-lru) rather than noeviction — a Redis instance that is evicting keys under load may evict idempotency records that are within their TTL, silently weakening the deduplication guarantee. The idempotency ADR must document the Redis maxmemory-policy requirement (noeviction for the idempotency key namespace, or a dedicated Redis instance for idempotency records) and the behavior when Redis is unavailable. The caching strategy decision record documents the caching infrastructure; the idempotency deduplication store is a specialized cache with durability requirements that may differ from the general application cache.

A database store (a dedicated idempotency_records table in the application database) is the most durable option: records survive restarts, are accessible across instances, and are retained until explicitly expired by a background cleanup job. The cost is a synchronous write on the critical path of every idempotent request — the server must write the idempotency record before processing the operation (to prevent two concurrent requests with the same key from both processing), and must update the record after processing (to store the cached response). This adds two database round-trips to every idempotent operation. For payment processing at low volume, this overhead is acceptable. For high-throughput batch APIs, the additional database writes may be the binding constraint on request throughput. The database vendor decision record documents the primary database and its write throughput characteristics; the idempotency ADR must document whether the idempotency record table uses the primary database, a read replica, or a separate store, and how write contention for the idempotency record is handled when two concurrent requests with the same key arrive simultaneously (a distributed lock or a unique constraint on the idempotency key column with conflict detection).

The deduplication window must be longer than the maximum retry interval in the client's retry policy, measured from the original request timestamp, not from the most recent retry. A client that configures exponential backoff with a 10-second initial delay, a 2x multiplier, and a maximum of 5 retries has a maximum retry interval of approximately 10 + 20 + 40 + 80 + 160 = 310 seconds — about 5 minutes from the original request. A 10-minute deduplication window covers this client's retry policy. A client that configures a manual retry option for human operators has an unbounded retry interval — the support engineer who retries a failed payment 18 hours after the original attempt is still a legitimate retry, not a new operation. Payment APIs typically use a 24-hour deduplication window to cover human-initiated retries within the same business day. The circuit breaker and resilience decision record documents the retry policies for service clients; the idempotency ADR must document the required deduplication window as a function of the client's configured retry policy, not as an independently chosen value.

Property 3: The retry safety classification and the client contract. A mutation endpoint is retry-safe if submitting the same operation twice produces the same final state as submitting it once — the operation is idempotent either by design or by idempotency key enforcement. The retry safety classification determines which endpoints clients and load balancers may retry automatically on error, and which endpoints require the client to first determine whether the original operation completed before deciding to retry.

Naturally idempotent operations do not require keys because the operation's semantics guarantee safety. PUT that sets a resource to an explicit state (PUT /users/123 with a full user object) produces the same user state whether called once or ten times — the hundredth call overwrites the ninety-ninth with the same values. DELETE of a specific resource by ID is idempotent when the server returns HTTP 404 for a resource that no longer exists rather than HTTP 500 — deleting a resource twice produces the same state (the resource does not exist) as deleting it once, provided the server handles the not-found case as a success. Endpoints classified as naturally idempotent must be documented with the specific conditions under which the natural idempotency holds, because the guarantee is not unconditional. DELETE /users/123 is idempotent only if the second call returns 204 or 404, not if it returns 500 because a foreign key constraint prevents deletion of a user with associated records — in that case, the second attempt fails for a different reason than the first, and the failure is not safe to retry without checking the cause.

Non-idempotent operations require explicit idempotency key enforcement and must be documented with the key generation contract. POST that creates a new resource assigns a new server-generated ID on every call — POSTing the same record twice creates two records with different IDs. Financial operations (charge, transfer, refund) move money on every call — charging the same amount twice charges twice. Batch operations (bulk create, bulk update, bulk delete) apply the operation to every item in the batch on every call. For each of these endpoint types, the idempotency ADR must document the key generation contract: who generates the key, what parameters the key is derived from, when a new key must be generated versus when the existing key should be reused, and what the server returns when the key is first used versus when it is a recognized duplicate.

The ambiguous category — operations that are partially idempotent depending on how they are called — requires the most explicit documentation. PATCH is idempotent when it sets a field to an explicit value (PATCH /orders/123 with {status: 'cancelled'} produces the same state regardless of how many times it is called) and non-idempotent when it applies a delta (PATCH /accounts/123 with {balance_delta: -50} debits $50 on every call). The idempotency classification must be documented per-operation, not per-HTTP-method. The retry safety classification is consumed by three different systems: the client SDK (which must send idempotency keys on non-idempotent operations and must not retry non-idempotent operations on 5xx without first checking the operation's status), the load balancer retry policy (which must not retry non-idempotent operations at the infrastructure level), and the API gateway (which must propagate idempotency keys from the client through to the backend service, not discard them). The API gateway decision record documents the gateway configuration; the idempotency ADR must document whether the gateway is responsible for deduplication (gateway-level idempotency, where the gateway intercepts duplicate requests before they reach the backend) or whether deduplication is handled by the backend service (requiring the gateway to pass idempotency keys through transparently).

What the founding session records and what it omits

The API idempotency decision is almost always made in two phases: an initial phase when the first retry-sensitive endpoint is built (usually a payment or resource creation endpoint) and a second phase when the first duplicate-operation incident occurs. The initial phase produces an idempotency key implementation that addresses the specific scenario that motivated it. The second phase patches the gaps that the first phase didn't anticipate. Neither phase produces a documented idempotency contract that specifies the key model, the deduplication window, the store durability requirements, and the retry safety classification across all mutation endpoints.

Three types of AI chat sessions generate these gaps:

The "how do we prevent duplicate charges when the network drops?" session. The engineer is implementing a payment endpoint and asks how to handle the case where the client retries a timed-out request. The session explains idempotency keys, recommends storing a key-to-response mapping in Redis with a reasonable TTL, and produces a code example of checking the Redis store before processing the payment and writing to it after. The session does not ask: what TTL should the Redis key use, and how does that TTL relate to the client's maximum retry interval? What is the key generation strategy — client-generated UUID, or a deterministic hash of the payment parameters, and what parameters should be included or excluded? What happens when the Redis store is unavailable — should the server reject the request to prevent duplicates, or process it and accept the risk? What should the server return when it receives a request with an idempotency key that matches a prior request but with different parameters? The session produces a working implementation for the specific scenario of network-dropped payment retries. It does not produce a documented idempotency contract that covers the full range of client retry behaviors, operational scenarios, and edge cases. The payment processor decision record documents the payment gateway integration; the idempotency ADR must document how the payment gateway's own idempotency mechanism (Stripe's Idempotency-Key header, PayPal's PayPal-Request-Id header) is used in conjunction with the API's own deduplication, and whether the API's deduplication store is a caching layer in front of the gateway's own idempotency or an independent deduplication layer.

The "how do we make our API safe to retry?" session. The engineer is adding retry logic to a client SDK or configuring a load balancer to retry on 5xx and asks which requests are safe to retry automatically. The session explains the difference between idempotent and non-idempotent HTTP methods (GET and PUT are safe to retry, POST is not), recommends retrying only on network errors and specific 5xx status codes (502, 503, 504), and suggests adding idempotency keys to POST requests. The session does not ask: which specific endpoints in this API are retry-safe, and under what conditions? What is the difference between a 500 that means the operation failed before making any changes and a 500 that means the operation completed but the server crashed before sending the response? If the server returns 500 for a payment that Stripe accepted before the server's database write failed, is that 500 a safe retry (the payment did not complete from the API's perspective) or an unsafe retry (the payment completed from Stripe's perspective and retrying would charge the card again)? The session produces general guidance on HTTP idempotency that does not account for the specific failure modes of this API's operations. An engineer who implements the general guidance — retry all 5xx responses for idempotent HTTP methods, send idempotency keys for POST — will retry a payment POST with the same idempotency key after a 500, which is correct if the 500 means the payment was rejected, and catastrophically incorrect if the 500 means the payment was accepted by Stripe but not recorded locally. The distributed locking decision record documents the concurrency model; idempotency key handling requires a lock or a unique constraint to prevent two concurrent requests with the same key from both proceeding to process the operation simultaneously — the lock acquisition strategy and its failure behavior must be documented as part of the idempotency model, not assumed to be handled by the database constraint alone.

The "what HTTP status code should we return for a duplicate idempotency key?" session. The engineer is implementing the idempotency key deduplication logic and asks what to return when a duplicate key is detected. The session explains that returning the cached response from the original request is the correct behavior for a genuine retry (the client receives the same 200 as the first call, making the retry transparent), and that returning HTTP 409 is appropriate when the duplicate key is accompanied by different parameters. The session does not ask: how should the response distinguish between three distinct states — the original request is still processing, the original request completed successfully, and the original request failed? If the original request is still processing when the duplicate arrives, should the server wait for it to complete and then return the result, return HTTP 202 Accepted immediately, or return HTTP 409 with a "processing" status? If the original request failed (the payment was declined), should the server return the cached failure response to the retry (causing the client to show the same decline error without attempting to charge) or return HTTP 409 with an explanation that the key maps to a failed operation and a new key is required for a new attempt? The session produces a deduplication response strategy for the happy path — genuine retries of successful operations — without documenting the response strategy for failed operations, in-progress operations, or parameter-mismatched retries. These gaps produce incorrect client behavior: a client that retries a payment after a network timeout, receives a cached decline response, and presents the decline error to the user when the original payment actually succeeded and the decline response was from a different prior attempt with the same key. The message broker decision record documents the event streaming infrastructure; in architectures where payment events are published to a message broker after the API records the payment, the idempotency guarantee must extend to the event publication — a duplicate API request that is correctly deduplicated at the HTTP layer must not publish a duplicate payment event to downstream consumers.

The WhyChose extractor surfaces the founding payment retry session, the first duplicate-charge incident session, and the batch-create timeout session from AI chat history. The API idempotency ADR converts the implicit choices in those sessions — Redis with an hour TTL, UUID keys, client-side retry on 5xx — into a documented deduplication store with an explicit durability requirement, a deduplication window derived from the client's maximum retry interval, and a retry safety classification that specifies per-endpoint which responses are safe to retry and which require a status check before retrying.

The five sections of an API idempotency ADR

Section 1: Idempotency key model and operation classification. Document the idempotency coverage model: which operations in the API require idempotency key enforcement, which are naturally idempotent, and which are neither (read-only operations that require no idempotency guarantee because they have no side effects). The classification must be documented per endpoint, not per HTTP method, because the HTTP method is an imprecise proxy for idempotency — POST endpoints that create resources are non-idempotent, POST endpoints that serve as action triggers (POST /payments/123/capture that captures an already-authorized payment exactly once per authorization) can be made idempotent with keys, and PATCH endpoints vary depending on whether they set absolute values or apply deltas.

Document the key format specification: client-generated UUID v4 (random), UUID v5 (namespace + name hash for deterministic generation), or a custom hash of the operation parameters. For UUID v4, document the key lifecycle — the key must be generated once per operation intent (not per HTTP request, since a retry of the same intent reuses the same key), stored by the client until the operation is confirmed to have completed or failed definitively, and discarded after the operation result is known and accepted. For deterministic keys, document the specific parameters that are hashed and the parameters that are explicitly excluded, with a rationale for each exclusion: "the payment method ID is excluded from the key hash because legitimate retries with a different payment method (after a decline) must be treated as new operations, not as duplicates."

Document the key collision response policy for three distinct cases. For a genuine retry (same key, same parameters, operation completed): return the cached response verbatim, including the original status code and body, with a Idempotent-Replayed: true response header so the client can distinguish a cached response from a freshly processed one. For a key reuse with parameter mismatch (same key, different body): return HTTP 409 Conflict with a body that includes the original request parameters, the current request parameters, and an explicit message that the idempotency key has been used with different parameters — this is always a client bug and must not be silently handled. For a key received while the original request is still processing (concurrent duplicate): return HTTP 409 with a body that indicates the operation is in progress, the original request ID if available, and a retry-after interval after which the client can check the operation's status. The API schema design decision record documents the response envelope format; the idempotency key collision responses must conform to the standard error response schema while carrying the additional idempotency-specific context fields.

Section 2: Deduplication store and window specification. Document the deduplication store selection with the durability rationale: in-memory (acceptable only for single-instance deployments with TTLs short enough that a restart is negligible), Redis (acceptable when the eviction policy is noeviction for the idempotency key namespace, or when a dedicated Redis instance is used for idempotency records, and when the behavior under Redis unavailability is explicitly defined), or database (required when idempotency records must survive service restarts and the throughput overhead of additional database writes is acceptable at the expected request rate). Document the per-store failure behavior: when the deduplication store is unreachable, the server must either reject all requests that include an idempotency key (safe — prevents duplicates at the cost of availability) or process requests without deduplication (available — accepts the risk of duplicates during the outage). This decision cannot be made ad-hoc by the on-call engineer during a Redis maintenance window; it must be documented in the ADR and implemented as the default behavior.

Document the deduplication window per endpoint category, derived from the client's documented retry policy. Payment endpoints: minimum 24 hours to cover human-initiated retries within the same business day. Batch create endpoints: minimum 2 hours to cover automation middleware that may retry with up to 3 attempts at exponential backoff starting at 10 seconds. Resource creation endpoints (invoices, subscriptions, orders): minimum 30 minutes to cover mobile client retries with 5-second timeouts and 3 retry attempts. The window must be stated in the client SDK documentation as an explicit contract: "idempotency keys are honored for 24 hours from the original request timestamp; after 24 hours, a request with the same key is processed as a new operation." Clients must treat requests submitted more than 24 hours after the original as semantically new operations and generate new idempotency keys for them.

Document the idempotency record schema: the key fields retained in the deduplication store and their purpose. Required fields: idempotency_key (the client-supplied token, indexed), client_id (the authenticated client identity, for per-client scoping), endpoint (the operation type, for per-endpoint scoping), request_fingerprint (a hash of the request body, for mismatch detection), response_status (the HTTP status code of the original response), response_body (the full response body to replay), created_at (the original request timestamp, for TTL calculation), and processing_state (pending, processing, completed, failed — for concurrent duplicate handling). The processing_state field is critical for concurrent requests: when the first request sets the state to 'processing', a concurrent duplicate arriving before the first request completes finds the 'processing' state and returns the in-progress conflict response rather than proceeding to execute the operation simultaneously. The database indexing strategy decision record documents the index design for the primary database; if the deduplication store is the primary database, the idempotency_records table requires a unique composite index on (client_id, endpoint, idempotency_key) and a TTL-based cleanup job that runs frequently enough to prevent the table from growing unboundedly.

Section 3: Request fingerprinting and response caching policy. Document the request fingerprinting model: whether the deduplication check validates only the idempotency key (key-only matching) or also compares the current request body against the stored fingerprint of the original request (key-plus-fingerprint matching). Key-only matching is simpler — a request with a known key always receives the cached response regardless of what the current request body contains. The risk is that a client bug that reuses a key with different parameters (a payment amount that changed after a pricing calculation error) is silently applied the wrong result — the cached response from the original parameters — without any indication that the mismatch occurred. Key-plus-fingerprint matching detects the mismatch and returns HTTP 409, making the client bug visible. The overhead is a fingerprint comparison on every idempotent request — typically a constant-time hash comparison, which is negligible. Key-plus-fingerprint matching is strongly recommended for financial operations; key-only matching may be acceptable for low-stakes resource creation where parameter mismatch is rare and the cached response is acceptable regardless.

Document the response caching policy: what is stored and what is replayed. The response body must be stored verbatim as returned by the original request — not re-serialized, not transformed, not updated. A replayed response must be byte-for-byte identical to the original, including timestamps and resource IDs that were generated at the time of the original request. A client that receives a cached response must be able to process it identically to a fresh response. Document the response headers that are modified for replayed responses: the Idempotent-Replayed: true header indicates that the response is from the deduplication cache, allowing clients and monitoring systems to distinguish replays from fresh executions. The Date header in the replayed response should reflect the current timestamp, not the original request timestamp, to avoid client-side cache conflicts. Document the response storage size limit: very large response bodies (binary file uploads, large JSON arrays) may be too expensive to store in the deduplication cache; for these endpoints, the idempotency policy may store only the response status code and a minimal response body, with the full response reconstructed from the stored operation result rather than from a verbatim cache of the original response bytes.

Section 4: Client retry protocol and idempotency key lifecycle. Document the client-side idempotency contract as a specification that the client SDK must implement. The contract has three components. First, the key generation policy: when to create a new key (initiating a new operation that has never been attempted), when to reuse the existing key (retrying an operation that returned a network error, a 5xx response, or a timeout before a response was received), and when to create a new key for an operation that previously failed definitively (a payment that was declined requires a new key for a retry with the same or different parameters, because the server must not treat the declined-card retry as a replay of the cached decline — the client wants a new charge attempt, not a cached decline response). The distinction between "ambiguous failure" (no response received) and "definitive failure" (decline response received) must be explicit in the SDK documentation.

Second, the operation status check: when a client receives an ambiguous response (timeout, network error, 500 without a body that clarifies whether the operation was attempted) and has an existing idempotency key, the client must be able to determine the operation's actual status before deciding whether to retry. Document the status check endpoint: a GET endpoint that accepts the idempotency key as a parameter and returns the current state of the operation (pending, processing, completed with the original response, failed with the failure reason). This endpoint allows a client to ask "did my payment go through?" without submitting a new payment request. Without a status check endpoint, a client that receives a 500 after a payment has no safe path: retrying with the same key may replay a successful payment and cause the client to show a success it didn't cause; retrying with a new key may charge the customer twice if the original payment succeeded; not retrying may leave the customer in a broken state. The status check endpoint converts this trilemma into a deterministic protocol. Document the status check endpoint path, the accepted query parameter name for the idempotency key, and the response schema that distinguishes the four states: not found (the key is not in the deduplication store — either the window expired or the key was never used), pending (the original request has not yet started processing), processing (the original request is in progress), completed (the original request completed — response body included), failed (the original request failed — failure reason included, and whether a retry with the same key or a new key is appropriate).

Third, the key storage policy: the client must persist idempotency keys in durable storage (local database, persistent storage) rather than in memory, so that keys survive client restarts. An application that stores the current payment's idempotency key in memory and restarts before the payment is confirmed will generate a new key on restart and retry as a new operation, potentially charging the user twice. The client SDK must document its storage mechanism explicitly. The observability strategy decision record documents the client-side instrumentation; idempotency key lifecycle events (key generated, key reused, status check performed, key expired) should be emitted as client-side metrics to enable detection of client bugs (high rate of key reuse with parameter mismatch indicates a key generation logic error) and to verify that the deduplication mechanism is being used correctly (zero key reuse in a payment client that should be retrying frequently indicates that the client is generating new keys on every retry rather than reusing existing ones).

Section 5: Idempotency observability and duplicate detection. Document the idempotency-specific monitoring surfaces that are not captured by standard API error metrics. The duplicate detection rate — the ratio of replayed responses to total responses on idempotent endpoints — is the primary health metric for the idempotency system. A duplicate detection rate that is consistent with the expected client retry rate (typically 1–5% during normal operations) indicates the system is working as designed. A rate of zero indicates that clients are not sending idempotency keys (a client SDK misconfiguration) or that the deduplication store is not being checked (a server-side implementation bug). A rate above 10% indicates that clients are retrying more aggressively than expected — either due to elevated error rates that are driving retries, or due to a client bug that generates the same key for multiple distinct operations.

The key collision rate — the ratio of HTTP 409 responses to duplicate idempotency key requests — distinguishes legitimate retries (which receive 200 with the cached response) from client bugs (which receive 409 because the key is being reused with different parameters). A non-zero key collision rate with parameter mismatch indicates a client-side key generation bug that is actively producing incorrect behavior. Document the alert threshold: any non-zero key collision rate with parameter mismatch should trigger an immediate investigation, because each collision is a client that received the wrong response — either a cached success for an intended new operation, or a 409 error for what the client believed was a straightforward retry.

The deduplication store latency percentiles — P50, P95, and P99 — measure the overhead of the idempotency check on the critical path of every idempotent request. Because the idempotency check must complete before the server begins processing the operation, the P99 deduplication store latency adds directly to the P99 API response time for all idempotent endpoints. A Redis deduplication store with P99 latency above 5ms should be investigated: elevated Redis latency may indicate memory pressure causing evictions, network congestion between the API server and Redis, or a hot key (a single idempotency key being checked at very high frequency, causing cache locking contention). Document the alerting threshold: P99 deduplication store latency above 10ms should alert the on-call team, because at that latency the idempotency mechanism is adding measurable overhead to payment processing times that users experience as checkout latency.

The window expiry rate — the rate at which clients submit requests with idempotency keys that are not found in the deduplication store because the deduplication window has expired — identifies the case where the deduplication window is shorter than the client's actual retry interval. A non-zero window expiry rate for a client that should be retrying within the window indicates either that the deduplication window is too short, that the client's retry interval exceeds the documented maximum, or that the deduplication store is evicting keys before the window expires (under Redis memory pressure). Each window expiry is a potential duplicate — the client submitted what it believed was a retry and received a fresh operation. For financial operations, every window expiry must be investigated to determine whether the client's original operation succeeded (in which case the fresh operation is a duplicate that must be reversed) or failed (in which case the fresh operation is correct and the client should update its key lifecycle handling). The data pipeline orchestration decision record documents the batch processing infrastructure; for batch APIs where idempotency key expiry could cause large-scale duplicates, the window expiry monitoring must integrate with the batch job orchestration to alert when a batch job's retry arrives after the window has expired, before the retry is executed, so that the team can decide whether to allow the retry or hold it pending manual verification of the original job's status.

None of these monitoring surfaces are configured in the founding session that added idempotency keys to prevent double charges or duplicate batch records. The founding session produces an implementation that handles the specific scenario that motivated it. The API idempotency ADR produces the operational model that covers the full range of client retry behaviors, deduplication store failure modes, window expiry scenarios, and key lifecycle edge cases — so the next engineer who asks "why did a customer get charged twice after we added idempotency keys?" has the deduplication window contract and the Redis eviction policy documented, not the answer to "why didn't we add idempotency keys?" that would have been the question before the founding session.

FAQs