The webhook delivery decision record: why the delivery guarantee model you chose determines your data consistency surface and your retry storm ceiling

Published 2026-07-15 · WhyChose

Webhook delivery decisions are made in the integration session, the reliability hardening session, or the API productization session — whenever the team first needs to push event notifications to an external consumer or expose event-driven integrations as a product feature. The AI session that implements webhook delivery is practically grounded and gets to a working state quickly: it implements an HTTP POST from the producer to the consumer endpoint, includes a retry mechanism for failed deliveries, selects a shared secret scheme for request authentication, and verifies that the consumer receives and acknowledges the event payload. The session ships a webhook sender that works correctly in the test environment. At founding scale — a small number of known consumers, a low event volume, a simple payload shape, and test conditions where the consumer endpoint is always available — the webhook system does exactly what it was built to do.

What the AI session does not produce is the webhook system's second half. The session answers "which events, which endpoint, and how do we authenticate the request?" and delivers a working sender. It does not ask: what is the delivery guarantee tier — is each event delivered at most once (the consumer misses it permanently if its endpoint is unavailable when the event fires) or at least once (the event is retried until the consumer acknowledges receipt), and if at-least-once, what is the idempotency model that prevents consumer state corruption when the same event is delivered twice because the consumer processed it but returned a 5xx before the sender received the acknowledgment? What is the retry policy — how many retries are attempted, on what backoff schedule, and does the backoff include jitter, because without jitter a consumer outage that causes thousands of events to accumulate at the same retry count will produce a thundering herd where every queued event fires simultaneously when the consumer recovers, re-collapsing an endpoint that was just restored? What is the endpoint verification model — does the consumer validate the webhook signature on every request, what is the shared secret rotation policy, and does the verification include a timestamp check that rejects replayed requests older than five minutes, because without replay prevention an attacker who captures a valid webhook request can re-deliver it to the consumer endpoint any number of times? What is the payload schema versioning contract — are consumers expected to handle additive schema changes without a version bump, what is the notification and migration timeline for non-additive changes, and what is the version negotiation mechanism for consumers that cannot migrate by the deprecation date? What is the dead letter queue policy — after the maximum retry count is exhausted, where does the event go, who is responsible for monitoring it, and what is the consumer's recovery path for events that land in the dead letter queue during an extended outage? Each of these questions has an answer that is not derivable from "we POST to the consumer endpoint and retry three times on failure" as the team frames the implementation. Each answer determines whether the webhook system is a reliable integration primitive that keeps consumer data consistent during producer events, or becomes the mechanism by which a database failover during a high-volume payment period leaves hundreds of orders in a permanently-pending state, a consumer outage produces a synchronized retry burst that re-collapses the consumer the moment it recovers, and an attacker who monitors network traffic to the consumer endpoint can replay valid signed requests months after they were originally delivered. The answers exist in the AI sessions. They are the operational commitments behind the delivery mechanism. They are almost never written down.

Two ways webhook delivery decisions produce the wrong outcome in production

The at-most-once payment event and the permanently-pending orders

A payments platform builds its first webhook integration in the second year of operation. Until this point, merchant integrations have used polling: each merchant's backend calls the platform's REST API every thirty seconds to check whether the status of a payment has changed. Polling works at small scale but produces a significant API load increase as the merchant base grows, and several merchants have complained that the thirty-second polling interval introduces a noticeable delay in their order confirmation flow. The decision is made to offer webhook push notifications for payment events. The founding session implements the webhook sender: when a payment transitions to the succeeded, failed, or refunded state, the platform sends an HTTP POST to the merchant's registered callback URL with a JSON payload containing the payment ID, the new status, the amount, and the currency. The session adds a basic retry: if the first POST returns a non-2xx status or times out, the sender retries once after thirty seconds, and if that also fails, it logs the failure and continues. The session selects an HMAC-SHA256 signature scheme: the payload is signed with a per-merchant shared secret and the signature is included in the X-Webhook-Signature header. The webhook system ships and the first merchants migrate from polling to push within two weeks.

Eleven months after launch, one of the platform's largest merchants — an e-commerce company processing approximately 4,000 orders per hour during peak periods — undergoes a database failover. The failover takes six minutes and forty seconds from the primary failure to full replica promotion, during which the merchant's order management system cannot write to its database. During this window, the payments platform fires approximately 440 payment.succeeded webhooks for orders that completed payment while the merchant's endpoint was unavailable. Each webhook POST returns a 500 during the failover window. The single retry fires thirty seconds later — still within the failover window for the events that fired early in the window, and in the recovery phase for those that fired late. Of the 440 events, 312 still receive a 500 or a connection timeout on the retry. After the second failure, the platform's webhook sender logs the error and marks the delivery as failed. No third attempt is made.

The merchant's order management system recovers from the database failover and resumes normal operation. The 312 orders whose payment.succeeded webhook was not successfully delivered remain in the "payment processing" state — the initial state set when payment was initiated — because the order management system transitions orders to "confirmed" only in the webhook handler, not by polling. The merchant does not discover the issue for three hours, when the customer service team begins receiving escalations from customers whose payments were charged but who never received an order confirmation. An audit of the payment platform's logs reveals the 312 failed webhook deliveries. The merchant's engineering team implements a manual replay: they query the payment platform's REST API for all payments with succeeded status in the past 24 hours, compare against their confirmed orders, and manually trigger the order confirmation flow for the 312 orders where a payment.succeeded event was never processed.

The manual replay takes four hours of engineering time and generates 312 duplicate confirmation emails to customers — the merchant's confirmation flow is not idempotent, so manually triggering it for orders that had already sent a "payment received" email (sent at payment initiation) produces a second confirmation email that confuses customers and generates a second wave of customer service contacts. The founding session that chose at-most-once delivery with a single retry did not address: what is the delivery guarantee required for payment state-transition events whose loss causes permanent consumer data inconsistency, and does at-most-once delivery meet that requirement? What is the retry count and retry window required to survive a database failover of the expected duration, and is one retry in thirty seconds sufficient for a failover that takes up to ten minutes? What is the consumer's recovery path when events are permanently lost — is there a dead letter queue with replay functionality, or does recovery require a separate polling-based reconciliation that the consumer must implement and maintain? Three questions that determined whether 312 orders could be recovered via webhook replay or required four hours of manual engineering work and 312 duplicate customer emails.

The synchronized retry burst and the consumer that re-collapsed on recovery

A SaaS developer tools platform builds a public webhook API in its third year, allowing enterprise customers to receive real-time notifications when builds complete, tests pass or fail, and deployments are promoted or rolled back. The founding session implements at-least-once delivery with exponential backoff retries: on the first failure, the retry fires after one minute; on the second, after two minutes; on the third, after four minutes; up to a maximum of eight retries with a maximum interval of sixty minutes. The session adds retry exhaustion handling: after eight failures, the event is moved to a dead letter queue and an email notification is sent to the customer's registered webhook contact. The session does not add jitter to the retry intervals — the backoff schedule is deterministic: a failure at time T is retried at exactly T+1m, T+3m, T+7m, T+15m, T+31m, T+63m, T+123m, and T+183m.

Eight months after launch, one of the platform's largest enterprise customers — a financial services company with a large CI/CD pipeline that generates approximately 600 build completion events per hour during the business day — has a 90-minute webhook consumer outage caused by a Kubernetes pod scheduling failure. The outage begins at 11:15 AM and is resolved at 12:45 PM. During the 90-minute outage window, the platform accumulates approximately 900 undelivered webhook events for this customer's endpoint. Each of the 900 events has failed delivery at time T (distributed across the 90-minute window), and each is currently on its second or third retry attempt. Because the retry intervals are deterministic and the outage is resolved at 12:45 PM, the retry schedule for the 900 events produces the following delivery pattern as the consumer comes back online: events that failed on their first retry between 11:44 AM and 12:14 PM are scheduled for their second retry in a 30-minute window starting at 12:45 PM — approximately 200 events. Events that failed on their second retry between 11:37 AM and 12:07 PM are scheduled for their third retry starting at 12:45 PM — approximately 160 events. Events on their first retry are still coming in from the tail end of the outage window. At 12:45 PM, when the consumer endpoint comes back up, it receives 360 webhook requests in the first 30 seconds — 12 times the normal delivery rate of 30 requests per second — because all accumulated retries fire at their scheduled times without any rate limiting or jitter spreading them out.

The consumer's webhook handler is backed by a worker pool sized for the expected throughput of 30 requests per second. At 360 requests per second, the worker pool is exhausted within eight seconds. New requests queue behind the worker pool's bounded queue, which fills in approximately thirty seconds. After the queue fills, the consumer's HTTP server begins returning 503 Service Unavailable responses — which the platform's webhook sender treats as a delivery failure, incrementing the retry count and scheduling the next exponential backoff retry. The customer's webhook consumer is now in the same failed state it was in during the outage, but the cause is no longer a Kubernetes scheduling failure — it is the retry burst from the platform's own retry system. The consumer oscillates between brief recovery windows (when the burst rate momentarily drops) and overload-induced 503 waves for the next forty minutes before the platform's retry backoff intervals spread out enough that the burst rate drops below the consumer's capacity. Over 200 of the 900 events have their retry count exhausted and move to the dead letter queue during this oscillation window, even though the consumer was operationally healthy — the dead letter queue entries are a consequence of the retry storm, not of a consumer fault.

The founding session that chose deterministic exponential backoff without jitter did not address: what is the thundering herd interaction between the retry backoff model and a consumer outage that accumulates a large backlog — do deterministic retry intervals cause all accumulated events to fire simultaneously when the consumer recovers, and what is the burst magnitude for the expected backlog size? What is the maximum delivery rate per consumer endpoint, and is there a rate limiter that prevents the retry system from overwhelming a recovering endpoint regardless of how many events are queued? What is the jitter model — is the retry interval randomized within a window to prevent synchronized retry waves, and is the jitter window sized proportionally to the backoff interval so that longer backoff intervals produce wider jitter windows that spread more events? Three questions that determined whether a 90-minute consumer outage would be absorbed by the retry system and resolved when the consumer recovered, or would produce a retry storm that extended the effective outage by forty minutes and moved 200 events to the dead letter queue unnecessarily.

Three structural properties that webhook delivery decisions determine

The delivery guarantee tier and the idempotency model

The delivery guarantee tier determines what the webhook system promises the consumer about event delivery — and therefore what the consumer can assume about the completeness of the event stream it receives. The guarantee tier is a choice with two valid options: at-most-once or at-least-once. There is no webhook-native exactly-once delivery; achieving exactly-once semantics requires at-least-once delivery combined with consumer-side idempotency, which together produce an effective guarantee of exactly-once processing even though the delivery mechanism itself may deliver the same event more than once.

At-most-once delivery makes the weakest guarantee: each event is attempted once (or a small fixed number of times with no persistence of the delivery attempt), and if the consumer's endpoint is unavailable, the event is permanently lost from the webhook stream. At-most-once delivery is correct when the webhook event is an informational notification where loss is acceptable — a low-priority status update, a notification where the consumer can recover by querying the producer's API for current state, or an event in a high-frequency stream where missing individual events is expected and handled by the consumer's reconciliation logic. It is incorrect for any event that represents a state transition the consumer must apply to remain consistent with the producer's authoritative state: payment state transitions, subscription lifecycle events, access permission changes, or any event where a missed delivery leaves the consumer's data permanently inconsistent with the producer's unless the consumer has a separate and explicitly-designed reconciliation mechanism. The decision record must document whether the event types conveyed by the webhook system are informational notifications or state-transition events, and whether the consumer has a reconciliation mechanism that recovers from missed events — because these two properties determine whether at-most-once delivery is correct or silently incorrect for the event types the system conveys.

At-least-once delivery requires that every delivery attempt is persisted — recorded in a delivery log before the first attempt is made, not after — so that the retry system can enumerate failed attempts and schedule retries without relying on the in-memory state of the sender process. A delivery log entry that is written after the first attempt is made, rather than before, has a race condition: if the sender process crashes after making the attempt but before writing the log entry, the delivery attempt is lost and the event is never retried, producing the same at-most-once behavior that the retry system was intended to prevent. The delivery log must be written in the same database transaction as the event state transition that triggers the webhook — if the payment.succeeded state is written to the payments table and the webhook delivery log entry is written in a separate subsequent operation, a crash between the two writes leaves the payment in the succeeded state with no pending delivery record, and the consumer never receives the event. The transactional atomicity of the event state write and the delivery log entry write is the foundational requirement of at-least-once delivery, and it is the implementation detail most likely to be missing from a founding session that adds "retry on failure" without considering the persistence model for delivery attempts. The event-driven architecture decision record is the upstream document: the transactional outbox pattern — writing the event payload to an outbox table in the same database transaction as the state change, and having a separate process poll the outbox and deliver events — is the standard mechanism for achieving this atomicity without requiring the webhook sender to participate in the producer's database transaction directly.

At-least-once delivery introduces the duplicate delivery scenario: a consumer's endpoint processes the event and updates its state, but before returning a 2xx response to the sender, the consumer's process crashes or its network connection to the sender is interrupted. The sender receives a failure (a connection reset or a timeout) and schedules a retry. The consumer's next retry receives the same event a second time. If the consumer's event handler is not idempotent — if processing the same event twice produces a different outcome than processing it once — the consumer's state will be corrupted by the duplicate delivery. The idempotency model must specify the idempotency key: the field in the event payload that uniquely identifies this delivery attempt and can be used by the consumer to detect and skip duplicate events. The standard idiom is an event ID included in the payload (a UUID or a comparable unique identifier generated by the producer when the event is created, not when it is delivered — the same event ID must be included in every retry of the same event, not a new ID per delivery attempt). The consumer stores processed event IDs in a deduplication table and checks the incoming event ID against the table before processing: if the ID is already present, the consumer returns a 2xx without processing (acknowledging receipt to the sender so the retry stops) and discards the duplicate. The deduplication table has a retention window matching the webhook system's maximum retry duration plus a safety margin — if retries are attempted for up to 72 hours, the deduplication table retains event IDs for at least 96 hours. The idempotency key must be documented in the webhook API specification as a required consumer implementation detail, not as an optional optimization.

The retry policy and the thundering herd surface

The retry policy determines how the webhook system behaves when a consumer endpoint is unavailable, and its design determines two independent properties: the maximum data loss surface (how long a consumer endpoint can be unavailable before events are permanently lost to the dead letter queue) and the thundering herd surface (how large a synchronized retry burst the system can produce when a consumer recovers from an outage).

The retry count and the maximum retry duration are the first parameters to document. The maximum retry count (8, 24, 72 — typically expressed as attempts, not hours) combined with the backoff schedule determines the maximum window during which a consumer can recover and still receive all queued events without data loss. A retry policy with 8 attempts using exponential backoff up to a 1-hour maximum interval retries events for approximately 6 hours (1m + 2m + 4m + 8m + 16m + 32m + 60m + 60m = 183 minutes ≈ 3 hours from the first failure to the last retry, but the actual window is longer because each retry fires after the interval from the previous failure, not from the original event). A retry policy with 72 attempts at hourly maximum interval retries events for approximately 3 days. The maximum retry duration is the SLA implicit in the retry policy: the producer is promising the consumer that a consumer outage shorter than the maximum retry duration will not result in permanent event loss. This implicit SLA must be documented explicitly, because consumer teams make architectural decisions based on it — a consumer that knows events are retained for 72 hours can absorb a weekend outage without designing a reconciliation mechanism; a consumer that does not know the retention duration cannot make that assumption safely.

The backoff algorithm and the jitter model are the second properties to document. Fixed-interval retries — retry every N minutes regardless of the failure count — have no burst spreading: all events that failed at time T are scheduled for T+N, T+2N, T+3N, and so on, and a burst of events that all failed within a short window will retry in synchronized waves. Exponential backoff — retry at 1m, 2m, 4m, 8m — spreads events that failed at slightly different times into different retry waves, because the backoff multiplier amplifies small differences in failure timing into larger differences in retry timing. But pure exponential backoff without jitter does not prevent the synchronized retry burst: events that failed at exactly the same time (during a consumer outage where every attempt returns the same 503 within a few milliseconds) have identical retry schedules and will fire simultaneously. Jitter adds a random component to the retry interval — typically a multiplicative jitter where the interval is scaled by a random value between 0.5 and 1.5, or an additive jitter where a random value between 0 and a cap (such as 30% of the base interval) is added — so that events whose original failure timestamps are within a few seconds of each other will be scheduled for slightly different retry times, distributing the burst across a time window rather than concentrating it at a single moment. The jitter range must scale with the backoff interval: for a 1-minute base interval, a ±30-second jitter spreads events across a 1-minute window; for a 60-minute maximum interval, a ±18-minute jitter spreads events across a 36-minute window, absorbing much larger backlogs without synchronized bursts. The decision record must document the jitter model as a specific implementation detail — "we use a multiplicative jitter of random(0.75, 1.25) applied to each retry interval" — not as a vague statement that retries are "randomized," because the thundering herd behavior is sensitive to the specific jitter range and distribution.

The per-endpoint delivery rate limit is the third component of the retry policy. Even with well-designed jitter, a large consumer outage accumulating tens of thousands of queued events will produce a delivery rate on recovery that is proportional to the backlog size, not the consumer's normal throughput. A per-endpoint rate limit — enforced by the webhook sender regardless of the retry queue depth — caps the delivery rate to a consumer endpoint at a configurable maximum (50 events per second, 100 events per second, or the consumer's declared capacity), absorbing the retry burst gradually rather than at full queue throughput. The per-endpoint rate limit interacts with the retry policy: events that are rate-limited (held back because the endpoint's rate limit is currently saturated) should not have their retry count incremented — they are not failed deliveries, they are queued-but-not-yet-attempted deliveries. Incrementing the retry count for rate-limited events would cause high-backlog consumers to exhaust their retry budget faster than intended, landing events in the dead letter queue because the consumer was absorbing the backlog at the rate-limited pace, not because the consumer was failing. The rate limit must be applied as a flow control mechanism separate from the retry failure accounting. The API rate limiting decision record is the companion document: the per-endpoint delivery rate limit in the webhook system is a producer-side rate limiter that protects the consumer; the design principles — token bucket vs sliding window, per-endpoint vs per-account limits, the burst allowance model — apply directly to webhook delivery rate limiting with the additional requirement that the rate limiter must distinguish between rate-limited-and-queued events and genuinely-failed events when counting against the retry budget.

The dead letter queue policy is the terminal behavior when the retry budget is exhausted. The dead letter queue must be a durable, queryable store — not simply a log entry — so that events can be replayed after the consumer resolves the underlying issue. The replay mechanism must be documented as part of the dead letter queue design: can the consumer trigger a replay via an API call, or does replay require contacting the producer's support team? Is replay idempotent — will replaying a dead letter queue event for a consumer that has already reconciled via an alternative path result in a duplicate processing? Does the dead letter queue preserve the original event payload and metadata (event ID, event type, original timestamp, failure reason, delivery attempt log) so that the consumer can use the information to diagnose the outage and identify which events need manual reconciliation versus automated replay? The dead letter queue retention window must be documented separately from the delivery retry window: the retry policy retains events in the active delivery queue for 72 hours; the dead letter queue should retain events for at least 30 days so that consumers who discover a data inconsistency days or weeks after an incident can replay the missing events rather than performing manual state reconstruction.

The endpoint verification model and the consumer coupling surface

The endpoint verification model determines how the consumer validates that an incoming webhook request was sent by the legitimate producer and was not forged, replayed, or tampered with in transit. Webhook verification is a security control that is frequently omitted from founding sessions because the test environment delivers webhooks from a known sender to a known receiver with no adversarial actors in between, and adding signature verification appears to be optional infrastructure complexity rather than a required security control. The absence of endpoint verification is an injection vulnerability: any party who knows the consumer's webhook endpoint URL can send a forged request claiming to be the producer, causing the consumer to process a fabricated event. For a payment.succeeded event, a forged request would cause the consumer to fulfill an order for which no payment was actually made. For an access.granted event, a forged request would cause the consumer to provision access that was not authorized by the producer.

HMAC-SHA256 signature verification is the standard webhook authentication mechanism. The producer computes a signature over the request payload (the raw byte representation of the JSON body, before any deserialization) using a per-consumer shared secret and the HMAC-SHA256 algorithm. The signature is included in a request header — typically X-Webhook-Signature or Webhook-Signature — formatted as a hex-encoded or base64-encoded string. The consumer recomputes the HMAC-SHA256 over the received raw body bytes using the shared secret, and compares the result against the header value. The consumer must compare the signatures using a constant-time comparison function — not a naive string equality check — to prevent timing attacks that would allow an attacker to determine the correct signature byte-by-byte by measuring how long the comparison takes before finding a mismatch. The comparison must be performed over the raw body bytes, not the deserialized JSON object, because JSON serialization is not guaranteed to be canonical: two equivalent JSON objects may serialize to different byte sequences depending on key ordering and whitespace, and comparing signatures over deserialized objects will produce false negatives when the serialization at the consumer differs from the serialization at the producer.

The shared secret lifecycle must be documented as part of the verification model. The shared secret must be generated with a cryptographically secure random number generator and must have sufficient entropy — at least 256 bits (32 bytes) of randomness — to resist brute-force attacks. The secret must be stored at the consumer as a secret, not as an application configuration value: it must not appear in source code, environment variable definitions committed to version control, or application logs. The secret rotation policy must specify how often the secret is rotated, the rotation mechanism (the producer exposes a rotate-secret API that generates a new secret and returns it to the consumer; the producer supports a grace period during which both the old and new secrets are valid, so that the consumer can update its secret verification logic without a delivery gap), and the revocation procedure if the secret is believed to be compromised. The secrets management decision record governs how the shared webhook secret is stored, rotated, and revoked — the webhook verification model must be consistent with the secrets management model for the consumer's application, not implemented as a separate ad-hoc secret handling mechanism.

Timestamp validation prevents replay attacks: an attacker who captures a valid signed webhook request can store it and replay it to the consumer endpoint at any future time, because the signature over a static payload with a static secret is always the same. Replay prevention requires that the request payload includes a delivery timestamp (the Unix timestamp at which the producer sent the request) and that the consumer rejects requests whose timestamp is older than a configurable threshold (typically 5 minutes). The timestamp must be included in the signed payload — either as a field in the JSON body or as a component of the signature input alongside the body — so that an attacker cannot modify the timestamp without invalidating the signature. If the timestamp is included in a separate header that is not part of the signature input, an attacker can replay the original request with a fresh timestamp header, bypassing the timestamp validation while still presenting a valid signature. The 5-minute replay prevention window requires that the producer and consumer have synchronized clocks within a few seconds — both must use NTP-synchronized clocks, and the consumer must reject requests where the clock skew between the delivery timestamp and the local clock exceeds a configurable tolerance (typically 60 seconds) to accommodate legitimate network delivery delays without opening the replay window. The decision record must document whether timestamp validation is implemented, the replay window duration, the clock synchronization requirement, and whether the timestamp is included in the signature input or in a separate unsigned header.

Payload schema versioning is the third component of the consumer coupling surface. The webhook payload schema changes over time as the producer adds new event types, adds fields to existing event payloads, and occasionally needs to remove or rename fields in ways that break backward compatibility. The schema versioning model determines how much of this change the consumer is required to absorb without code changes, and how the producer communicates non-additive changes that require consumer migration. The standard model is a versioned envelope: each payload includes a version identifier field at the top level — the date-based ISO 8601 string (e.g. "version": "2025-01-01") or a monotonic integer (e.g. "version": 2) — that the consumer uses to route the payload to the appropriate deserialization and processing handler for that version. Additive-only changes — new optional fields added to an existing payload version — are backward-compatible and do not require a version bump, provided the consumer's deserialization logic ignores unknown fields rather than treating them as an error. The additive-only constraint is a formal rule that applies to all schema changes within a version: once a version is published, no fields may be removed, renamed, or have their type changed without a new version number. Non-additive changes require a new version, a consumer migration timeline with a specific deprecation date for the old version, and a mechanism for consumers who have not migrated by the deprecation date to continue receiving the old version during a grace period rather than receiving the new version without warning. The API versioning decision record is the companion document for the schema versioning contract: the same principles — additive-only compatibility within a version, explicit version negotiation, deprecation timeline with communication — apply to webhook payload schemas as they do to REST API response schemas, with the additional constraint that webhook consumers receive payloads passively rather than requesting them, so they cannot negotiate the version per-request the way an API client specifies an Accept-Version header.

Three AI session types that embed webhook delivery decisions without documenting them

The integration session is where webhooks first appear in a system — typically when the team needs to consume events from a third-party service (a payment processor, an identity provider, or a communication platform) or when the team needs to push events to a first external consumer (a customer's internal system, a partner's API, or a monitoring platform). When consuming third-party webhooks, the founding session implements the consumer endpoint, verifies the incoming signature using the third-party's documented HMAC scheme, and writes a handler that processes the received event. What the session does not address is the consumer's own reliability model: what happens when the handler fails — does the third-party retry, and if so, is the consumer's handler idempotent? The consumer's endpoint is written as a straightforward handler — receive event, update database, return 200 — without considering that the third-party may retry a failed delivery, and that a handler that receives the same payment.succeeded event twice may double-credit the customer's account or double-fulfill the order. The idempotency model is not designed into the handler because in the test environment, the third-party's webhook test tool sends each event exactly once, and the duplicate delivery scenario does not occur in testing. When the consumer experiences a database connection timeout in production, returns a 5xx, and receives a retry from the third-party's robust at-least-once delivery system, the handler processes the retried event without checking whether it has already been processed, and the duplicate outcome surfaces as a data inconsistency that is difficult to trace because the duplicate-processing log and the original-processing log appear to be two separate successful operations. The open-source extractor surfaces these founding integration sessions from AI chat history, recovering the handler logic, the database update patterns, and the success/failure branching that determine whether the consumer is idempotent without the team having to audit the handler code against every possible retry scenario at the time the incident occurs.

The reliability hardening session is where at-least-once delivery is added to an existing at-most-once webhook sender. The session is motivated by a production incident — typically an event loss incident like the one described above, where a consumer outage or a network partition caused events to be missed permanently — or by a customer escalation from a consumer whose data is inconsistent due to missed events. The session adds retries to the webhook sender: a retry loop that re-attempts delivery on failure, an exponential backoff schedule, and a retry count limit with failure logging. What the session does not address is the thundering herd interaction: the backoff schedule is chosen based on intuition about appropriate retry intervals (1 minute, 2 minutes, 4 minutes), not based on an analysis of the expected backlog accumulation rate during a consumer outage and the burst delivery rate that the accumulated backlog would produce when the consumer recovers. The jitter design, if jitter is added at all, is typically a small additive random value (plus or minus a few seconds) applied to each retry interval — not a multiplicative jitter proportional to the interval that would effectively spread the synchronized burst for longer backoff intervals. The per-endpoint delivery rate limit is not added because the session's focus is on ensuring that events are retried, not on protecting the consumer from excessive retry throughput after an outage. The session tests the reliability improvement by simulating a short consumer outage (returning 500s for 30 seconds) and verifying that events are successfully delivered after the consumer recovers. The thundering herd scenario — a 90-minute outage accumulating thousands of queued events — is not part of the test scenario, and the synchronized retry burst does not manifest in the test environment. The queue messaging decision record is the companion document: the webhook delivery retry system is fundamentally a persistent message queue with delivery semantics and a consumer health feedback loop, and the queue design decisions — persistence model, retry policy, backpressure, dead letter queue — apply directly to the webhook delivery backing queue even when the team does not recognize the retry system as a queue.

The API productization session is where the team publishes webhooks as a public-facing integration feature for their customers, rather than using them only for internal system-to-system communication. The session is motivated by a customer request ("can we get notified when our users' subscription status changes instead of polling every hour?") or by a competitive requirement (competing products offer webhooks as a standard integration feature). The session implements the webhook management UI (endpoint registration, secret generation, event type selection), the webhook delivery infrastructure (event fan-out from the source event to all registered consumer endpoints), and the developer documentation for the webhook API. What the session does not address is the schema versioning contract — the commitment the platform is making to its customers about how the webhook payload schema will evolve over time, because the session is focused on shipping the feature, not on its long-term maintenance implications. The first version of the payload schema is whatever the team finds natural to include in the event — often mirroring the internal event model directly without considering whether the internal model's structure is appropriate as a public API contract. When the internal event model changes eight months later due to a database schema refactor or a feature addition, the team discovers that the webhook payload schema is tightly coupled to the internal model, and that changing the payload requires either a non-additive schema change that breaks all existing consumers or maintaining a mapping layer between the internal model and the versioned payload schema that was never designed into the original implementation. The new CTO onboarding problem in a webhook-heavy system is a versioning problem: an incoming technical leader finds a webhook payload schema that directly exposes internal database column names, three different event types that serialize timestamps in three different formats (ISO 8601, Unix seconds, Unix milliseconds), no version field in the payload, and no documented schema versioning policy — a technical debt surface that requires careful migration planning to address without breaking the consumer integrations of every enterprise customer who built against the original payload schema.

The five sections of a webhook delivery decision record

The first section documents the delivery guarantee tier and the idempotency model. The guarantee tier must be stated explicitly — "this webhook system provides at-least-once delivery for all event types in the payment.* namespace, and at-most-once delivery for informational notification events in the notification.* namespace" — rather than inherited implicitly from the implementation. The at-least-once delivery guarantee must document the persistence model for delivery attempts: are delivery attempts written to a delivery log in the same database transaction as the state change that triggers the event, or are they written asynchronously after the state change? The atomicity gap — the window between the state change and the delivery log write — is the failure mode that converts an intended at-least-once delivery into de facto at-most-once delivery, and its existence or absence must be documented explicitly. The idempotency key must be named: "the event ID field in the payload uniquely identifies this delivery attempt; the same event ID is included in every retry of the same event; consumers must implement an idempotency check using this field before processing the event body." The consumer's responsibility for implementing idempotent event handlers must be stated as a requirement in the webhook API documentation, not as an optional recommendation, because at-least-once delivery produces incorrect results whenever a consumer processes a duplicate delivery of a non-idempotent handler. The distinction between the event ID (the identifier for the logical event, shared across all retries) and the delivery attempt ID (the identifier for a specific delivery attempt, unique per retry) must be documented: the event ID is the idempotency key; the delivery attempt ID is the identifier the consumer can use to report a specific delivery failure to the producer's support team. The decisions never written down in a webhook delivery system are the delivery guarantee tier, the atomicity model for the delivery log, and the idempotency key — three properties that together determine whether at-least-once delivery prevents event loss or merely adds complexity without guaranteeing the property it appears to provide.

The second section documents the retry policy. The retry count, the backoff algorithm, the jitter model, the maximum retry interval, and the dead letter queue trigger must all be documented as specific values, not as "we retry with exponential backoff." The retry count documentation must include the implicit SLA: "events are retried for up to 72 hours; a consumer endpoint that is unavailable for more than 72 hours will have its undelivered events moved to the dead letter queue." The backoff algorithm must be specified with the formula — "retry interval = base_interval × 2^(attempt_number - 1) × jitter_factor, where jitter_factor is a random value drawn uniformly from [0.75, 1.25], with a maximum retry interval of 3600 seconds" — because "exponential backoff with jitter" is not a specific enough description to evaluate the thundering herd behavior for a given backlog size and outage duration. The per-endpoint delivery rate limit must document the rate cap and the rate control mechanism — "webhook deliveries to a single consumer endpoint are rate-limited to 100 requests per second using a sliding window counter; events that are rate-limited are held in the delivery queue and are not counted as failed deliveries for retry count purposes." The dead letter queue policy must document the dead letter trigger (maximum retry count exhausted), the retention window (30 days), the replay mechanism (consumer-initiated replay via the management API), and the notification mechanism (email to the registered webhook contact when events are moved to the dead letter queue, with the event count and event type distribution in the notification body). The dead letter queue replay must be documented as an idempotent operation — replaying a dead letter queue event for an endpoint that has already processed the event via a reconciliation path must not produce a duplicate side effect — which requires that the event ID idempotency check is implemented consistently across the primary delivery path and the dead letter queue replay path.

The third section documents the endpoint verification model. The signature scheme must be named with specificity: "HMAC-SHA256 over the raw request body bytes, using a per-consumer 32-byte random secret, with the signature base64-encoded and included in the X-Webhook-Signature header." The signed input must be documented — "the signature is computed over the raw body bytes concatenated with the delivery timestamp; the delivery timestamp is the Unix timestamp in seconds at which the producer initiates delivery, included in the X-Webhook-Timestamp header and required to be part of the signature input to prevent timestamp substitution attacks." The timestamp validation requirement must specify the maximum age threshold (5 minutes), the clock synchronization requirement (both producer and consumer must use NTP-synchronized system clocks with a maximum 60-second skew), and the handling of requests that fail the timestamp check (return 401 with a response body that distinguishes a signature mismatch from a timestamp rejection, so that consumers can distinguish misconfigured secrets from clock skew issues in their error logs). The constant-time comparison requirement must be stated explicitly — "consumers must use a constant-time string comparison for the HMAC signature to prevent timing attacks; language-standard string equality (== in Python, equals() in Java, === in JavaScript) is not constant-time and must not be used for HMAC verification" — because the timing attack failure mode is non-obvious and is absent from most webhook integration examples that consumers follow when implementing their endpoint. The secret lifecycle must document the rotation mechanism: "the producer provides a rotate-secret endpoint that generates a new secret; during a 10-minute grace period after rotation, the producer accepts both the old and new secrets as valid, allowing consumers to update their secret configuration without a delivery gap; after the grace period, only the new secret is valid." The secrets management decision record must reference the webhook shared secret as a managed secret with its own rotation schedule and revocation procedure, separate from other application secrets.

The fourth section documents the payload schema versioning model. The version field must be defined with its location in the payload structure — "every webhook payload includes a top-level 'version' field containing an ISO 8601 date string representing the schema version (e.g. '2025-01-01'); the version field is a string, not a date type, to avoid deserialization ambiguity across consumer languages" — and its update policy: "the version is incremented only for non-additive schema changes; additive changes (new optional fields) are deployed to the current version without a version bump; non-additive changes require a new version string and a migration timeline." The additive-only constraint must be stated as a formal rule: "within a published version, fields may be added but not removed, renamed, or have their type changed; the addition of new optional fields is backward-compatible and does not require consumer code changes provided the consumer's deserialization ignores unknown fields." The non-additive change migration process must document the notification timeline (minimum 90 days' notice before the old version is deprecated), the grace period (old version delivered in parallel with new version for 90 days after the new version is published), the consumer opt-in mechanism (consumers register their supported versions via the management API; if a consumer has not registered support for the new version, the old version is delivered until the grace period expires), and the post-deprecation behavior (after the grace period, all consumers receive the new version regardless of registration status; consumers who have not migrated will receive the new version with the schema changes they have not prepared for). The event type inventory — the full list of event types conveyed by the webhook system, with the payload schema for each type and the version history — must be documented as a living document that is updated each time a new event type is added or an existing schema is versioned. The API versioning decision record provides the framework for the schema versioning contract: the same consumer impact analysis — who is affected by this change, what is the migration path, what is the deprecation timeline — applies to webhook schema changes as it does to REST API version changes, with the additional consideration that webhook consumers receive payloads passively and cannot pin to a version per-request.

The fifth section documents the observability model and the consumer health feedback loop. The webhook delivery system's observability must provide four views: the delivery success rate view (the fraction of delivery attempts that receive a 2xx response on the first attempt, grouped by event type and consumer endpoint, to identify consumers with elevated failure rates before their retry queues grow large); the retry queue depth view (the number of events queued for retry per consumer endpoint, broken down by retry count, so that the operations team can identify consumers approaching their retry budget before events begin moving to the dead letter queue); the dead letter queue age view (the age of the oldest event in the dead letter queue per consumer endpoint, to identify consumers who have not replayed dead letter events and whose data may be inconsistent with the producer's authoritative state); and the consumer error distribution view (the HTTP status code distribution of failed delivery attempts per consumer endpoint — 503 Service Unavailable indicates a consumer capacity issue that benefits from rate limiting intervention; 401 Unauthorized indicates a secret mismatch that benefits from a secret rotation event alert; 422 Unprocessable Entity indicates a consumer schema deserialization error that may indicate a version mismatch requiring migration support). The consumer health feedback loop — the mechanism by which the producer alerts consumers to delivery failures before they lose events — must be documented as a notification policy: "email notification to the registered webhook contact when: (1) a consumer endpoint has a delivery failure rate exceeding 50% over a 15-minute window; (2) a consumer endpoint's retry queue depth exceeds 1,000 events; (3) events are moved to the dead letter queue for any consumer endpoint; (4) a consumer endpoint returns 401 Unauthorized on three consecutive delivery attempts, indicating a possible secret compromise." The consumer dashboard — a management UI showing the consumer's own delivery success rate, retry queue depth, and dead letter queue contents — must be part of the webhook product design so that consumers can self-diagnose delivery issues without contacting the producer's support team. A consumer who cannot see their own delivery failure rate cannot distinguish between "my endpoint is healthy and receiving events correctly" and "my endpoint is failing and the producer has been retrying silently for two hours" without external monitoring of their own webhook endpoint's response rates.

The integration session that first consumed a third-party webhook, the reliability hardening session that added retries to an at-most-once sender, and the API productization session that published webhooks as a public integration feature each produced webhook delivery decisions whose long-term operational cost — 312 orders permanently stuck in a pending state because at-most-once delivery with a single retry could not survive a six-minute database failover, because the founding session chose a retry policy without documenting the delivery guarantee required for state-transition events or providing a dead letter queue with replay functionality; or a retry storm that extended a 90-minute consumer outage by forty additional minutes because deterministic exponential backoff without jitter produced a synchronized burst of 360 requests per second against an endpoint sized for 30, because the founding session added retries without analyzing the thundering herd interaction; or a payload schema locked to internal database column names with three timestamp serialization formats and no version field, requiring a major migration coordination effort with every enterprise customer who built integrations against the original schema — exceeds what a delivery guarantee tier document, a retry policy with jitter specification, and a schema versioning contract would have cost at the time the founding decisions were made. The decisions are in the AI sessions: the HTTP POST that "retries if it fails," the HMAC secret stored in an environment variable the team will "move to a secrets manager later," the payload that "just serializes the internal model for now." WhyChose's open-source extractor surfaces these founding webhook sessions as structured decision records before the next database failover produces permanently-pending orders, the next consumer outage produces a retry storm, or the next schema change requires a months-long migration coordination sprint with enterprise customers who integrated against a payload schema that was never meant to be a stable public API contract. The decisions never written down in a webhook delivery deployment are the delivery guarantee tier, the thundering herd interaction with the retry policy, and the schema versioning contract — the three operational commitments that determine whether the webhook system is a reliable integration primitive that keeps consumer data consistent during producer events and infrastructure incidents, or becomes the mechanism by which silent event loss, retry storms, and schema breaking changes create operational complexity disproportionate to the simplicity of the original HTTP POST that started it all.

Further reading