What is the Redlock algorithm and why doesn't it provide the same guarantees as consensus-based distributed locking?

The Redlock algorithm, designed by Redis's creator, attempts to provide mutual exclusion across N independent Redis nodes (typically 5) by requiring that a lock acquisition succeed on a majority (N/2 + 1) of nodes within a validity window. The validity window is derived from the lock TTL minus the time elapsed during acquisition attempts, providing a conservative estimate of how long the lock holder can safely assume exclusivity. Redlock improves on single-node Redis locking by eliminating the single-node failure mode: if one Redis node crashes, the remaining majority still refuses to grant the lock to a new holder until the crashed node's TTL expires. The algorithm's documented limitations come from two sources. First, Redlock relies on bounded clock drift between nodes — the algorithm's proof of correctness assumes that clocks on all N nodes advance at roughly similar rates, so that the TTL expiry calculations on each node remain synchronized within a small epsilon. Cloud VMs running NTP correction can experience clock jumps of several seconds, particularly during VM live migration or hypervisor maintenance, which can cause one node's TTL to expire significantly earlier than the others — creating a window where the node considers the lock expired and grants it to a new holder while the remaining majority still considers it held. Second, Redlock's safety proof assumes that the majority of nodes' state is current — specifically, that nodes coming back online after a crash return with the lock state they had before crashing. Redis's persistence model (RDB snapshots or AOF logs) may not have persisted the most recent lock acquisition before the crash, meaning a node that crashes after granting a lock may restart without that lock in its state and grant the same lock key to a new holder, producing a majority (from the new holder's perspective) even while the original holder still considers itself the lock owner. Consensus-based backends (etcd, ZooKeeper) avoid both limitations by using a consensus protocol (Raft, Zab) that provides linearizable writes — the lock is only granted when a majority of nodes have durably recorded the acquisition, and the consistency guarantee is maintained even under network partitions between individual nodes, as long as a quorum remains reachable. The cost of consensus-based locking is higher latency (multiple round trips for consensus) and operational complexity (a properly replicated etcd or ZooKeeper cluster rather than independent Redis nodes). The choice between Redlock and consensus-based locking is a choice between probabilistic mutual exclusion with documented failure windows under specific infrastructure conditions, and deterministic mutual exclusion with higher latency and infrastructure requirements.

What is the difference between a session-scoped and transaction-scoped Postgres advisory lock, and which should I use for background job serialization?

Postgres advisory locks come in two scopes: session-scoped and transaction-scoped. A session-scoped advisory lock (acquired with pg_advisory_lock or pg_try_advisory_lock) is held until the session explicitly releases it or the session terminates. If the application crashes while holding a session-scoped advisory lock, the lock is automatically released when Postgres detects the session termination — typically within a few seconds via TCP keepalive or connection timeout. A transaction-scoped advisory lock (acquired with pg_advisory_xact_lock or pg_try_advisory_xact_lock) is held for the duration of the current transaction and released automatically on COMMIT or ROLLBACK, regardless of whether the application explicitly releases it. For background job serialization, the choice depends on whether the job's critical section aligns with a single database transaction. If the entire job runs inside a single transaction, transaction-scoped advisory locks provide the cleanest semantics: the lock is automatically released when the transaction ends, whether through success (COMMIT) or failure (ROLLBACK or crash). If the job's critical section spans multiple transactions or includes operations outside the database (external API calls, file writes, message publishes), session-scoped locks are required because there is no single transaction boundary at which to anchor the lock release. Session-scoped locks require explicit release on both the success and failure paths — a crash releases them automatically, but an exception that is caught and handled without releasing the lock will hold the lock for the life of the session. The practical failure mode for session-scoped locks in background job frameworks that use connection pooling is that the job acquires a session-scoped lock, completes (or fails), and returns the connection to the pool — but the pool does not automatically reset the session state (including advisory lock releases) between uses. The next job that happens to use the same pooled connection may find advisory locks held from the previous job's incomplete cleanup. Explicitly calling pg_advisory_unlock_all() at job completion — both on the success path and in the exception handler — is the safest pattern for session-scoped advisory locks in connection-pooled environments.

How should I set the lock TTL for a background job, and what is the double-execution risk if I set it too short?

Lock TTL for background jobs should be derived from the worst-case runtime of the job under load, not the expected runtime under normal conditions. The critical distinction is that TTL serves two separate purposes that engineers commonly conflate: (1) as a crash recovery timeout — if the lock holder crashes, the lock will expire and be available for a new holder after TTL seconds, preventing permanent deadlock; (2) implicitly, as a maximum exclusivity window — after TTL seconds, the lock backend will grant the lock to a new requester even if the original holder is still running. The double-execution failure mode activates when actual job runtime exceeds the TTL. The risk is not that the TTL is set 'too conservatively short' — it is that the estimate of worst-case runtime under load is systematically lower than the actual worst-case runtime. Jobs that take 90 seconds in testing will run longer under database load (slower queries), network congestion (slower external API calls), or GC pressure (paused threads). The failure mode is asymmetric: jobs that take longer than expected do so precisely during the conditions that are most likely to produce high load — month-end billing runs, peak traffic periods, database maintenance windows. Setting TTL to three times the expected runtime is not sufficient if expected runtime was measured under low-load conditions and worst-case runtime under high-load conditions can be ten times the expected runtime. The correct derivation is: measure job runtime at maximum realistic load (or the load at which SLA commitments are required), identify the P99.9 runtime rather than the P50, multiply by a safety factor (2-3×), and document that this derivation must be reviewed whenever the job's workload characteristics change — because a billing job that processes 100 customers in 90 seconds will process 1000 customers (after 10× user growth) in 900 seconds if the job is not redesigned to process in batches. The alternative to TTL-based locking for long-running jobs is heartbeat-based lock renewal: the job holds a short TTL lock but renews it every N seconds (N < TTL) as long as it is still running, and the lock expires naturally if the heartbeat stops (indicating a crash or hang). Heartbeat renewal prevents the double-execution failure mode from TTL expiry during extended job runtime, at the cost of requiring the job to actively maintain the lock and the monitoring infrastructure to detect missed heartbeats.

What should a distributed locking decision record include?

A distributed locking ADR needs five sections. First, lock backend selection and failure model: which backend is used (Redis single-node, Redis Redlock, Postgres advisory locks, etcd, ZooKeeper), the specific failure scenarios the backend cannot prevent (single-node Redis cannot prevent double execution during a Redis restart with stale state; Redlock cannot prevent it under bounded clock drift violations or AOF persistence gaps; Postgres advisory locks cannot prevent it if the connection pool reuses connections without session reset), the scenarios the backend does prevent (crash recovery, explicit release on session termination), and the rationale for why the chosen backend's failure modes are acceptable for the specific use case. Second, lock TTL and heartbeat model: the TTL derivation methodology (measured P99.9 runtime at maximum load times a documented safety factor), the review trigger (workload change, processing volume change), and whether heartbeat-based renewal is implemented for long-running jobs and under what renewal interval. Third, lock acquisition and release protocol: what happens when lock acquisition fails (skip this run, queue for next window, alert and skip), what the retry model is for acquisition failures, how explicit release is handled on both success and failure paths, and whether the lock token (the unique value set during acquisition) is verified before release to prevent releasing a lock held by a different worker. Fourth, idempotency requirement and the lock's role: whether the protected operation is idempotent or non-idempotent, and if non-idempotent, what additional correctness mechanism exists beyond the distributed lock (idempotency keys, database unique constraints, database conditional updates using optimistic concurrency) — the lock is a serialization aid, not a correctness proof for non-idempotent operations. Fifth, lock observability and monitoring: what metrics are collected (lock hold duration, time from acquisition to release as percentage of TTL, lock acquisition failure rate, lock renewal success rate for heartbeat-based locks), what alert thresholds signal risk (hold duration exceeding 80% of TTL warrants investigation before the expiry-based double-execution failure mode is reached), and how stale locks are detected and cleaned up.

2026-07-23 · ~26 min read · Decision records

The distributed locking decision record: why the lock acquisition model you chose determines your double-execution surface and your correctness ceiling under partial failure

Distributed locking decisions are made in three AI sessions: the initial background job session that asks ChatGPT how to prevent concurrent execution and gets Redis SETNX with a TTL set to "comfortably longer than the expected runtime," the incident response session that migrates to Redis Redlock after a single-node Redis failure, and the compliance session that adds an audit log to the billing job without reviewing whether the lock provides the mutual exclusion guarantees the audit assumes. What none of these sessions produce is the lock backend failure model, the TTL derivation methodology, or the idempotency requirement. Those absences determine whether distributed locking provides a correctness guarantee or a false correctness confidence that fails precisely under the conditions where correctness matters most.

The billing run that expired mid-execution

A developer tools startup builds monthly subscription billing. The billing background job iterates over all active subscriptions, charges each customer's stored payment method, and updates subscription status. An engineer asks Claude: "How do I make sure this job only runs once at a time? We have multiple workers and I don't want to double-charge anyone." Claude returns a Redis SETNX implementation: acquire a lock with a unique value, set a TTL, run the job, release the lock by deleting the key only if its value matches the acquired value. The TTL is set to 300 seconds — five minutes — because billing "normally finishes in about ninety seconds in testing, so five minutes is very conservative."

The implementation is correct. In testing, under development database load with 200 customer records, billing completes in 112 seconds. The lock expires after 300 seconds, well after completion. The pattern holds for eight months as the customer base grows to 1,800 accounts. Billing runs in 190 to 280 seconds depending on payment processor response times. The lock's 300-second TTL remains above the actual job runtime. The billing system works.

In month nine, the billing run coincides with two unrelated database events: a DBA has scheduled an index rebuild on the subscriptions table during the early morning billing window (the previous month had slow queries during billing; the DBA is fixing the root cause), and the payment processor is experiencing elevated API latency (a status page incident, acknowledged but classified as "degraded performance" rather than "outage"). The index rebuild acquires table-level read locks during certain phases, causing billing queries to queue. The payment processor's P99 response time rises from its normal 340ms to 4.2 seconds. Billing runs for 423 seconds — seven minutes and three seconds — before completing.

At second 300, the Redis lock's TTL expires. The billing job has charged 1,340 of 1,800 customers and is mid-iteration. At second 301, a second worker scheduled for a retry check (part of the billing system's retry logic for workers that appear to have stalled) acquires the lock successfully — Redis granted it because the TTL has expired and the key is gone. The second worker begins billing from the beginning of the customer list.

The first worker continues billing the remaining 460 customers, completing at second 423. The second worker charges all 1,800 customers from the start, completing at second 731. When both workers finish, 1,340 customers have been charged twice. The double charges are for amounts ranging from $9 to $299. The payment processor's charge log confirms both runs. Forty-seven customers notice immediately — the charges are large enough to trigger bank notifications.

The post-mortem identifies the TTL as the root cause. The engineering team revisits the original implementation. The five-minute TTL was set because billing "should never take more than three minutes, so five minutes is conservative." The estimate was made against a 200-customer test dataset under development conditions, not against a 1,800-customer production dataset under concurrent database load. The TTL was a crash recovery timeout set to the expected worst-case runtime — but the actual worst-case runtime exceeded the estimate by 2.4× under conditions the engineer did not anticipate when making the estimate.

The decision to set the TTL to 300 seconds was made in a single AI chat session that produced working code. It was not in any ADR. The code was reviewed for correctness (the SETNX pattern is correct), not for the appropriateness of the 300-second value or the methodology that produced it. The double-execution failure mode became possible the moment the TTL was set to a value derived from expected runtime rather than worst-case runtime under load, and it actualized eight months later when the actual runtime first exceeded that estimate.

The Redlock migration that narrowed the window without closing it

A B2B SaaS team runs a schema migration orchestrator: a background job that applies pending database migrations one at a time, waits for each to complete, verifies the schema state, and advances a migration pointer. The job is protected by a Redis single-node lock to prevent two migration workers from applying migrations concurrently. The system works reliably for fourteen months.

In month fifteen, the Redis primary experiences a hardware failure during a migration run. Redis goes down while the migration lock is held. The migration job crashes. The lock is gone — it existed only in Redis, and Redis crashed. A retry worker starts immediately, acquires the lock on the recovered Redis node (which comes back online with empty state — the crash happened before the RDB snapshot cycle), and begins running the next migration. The prior migration was mid-application when the crash occurred: it had applied ALTER TABLE but had not yet committed. The retry worker sees the schema in an intermediate state and applies the next migration on top of it. The resulting schema is corrupted.

The incident triggers a migration to Redis Redlock. The team deploys five independent Redis nodes, configures the Redlock algorithm to require majority acquisition (three of five), and migrates the migration orchestrator to use a Redlock client library. The improvement is real: a single Redis node failure no longer grants the lock to a new holder immediately, because the remaining four nodes still refuse to grant it. The majority condition prevents the crash-and-immediate-reacquisition failure mode that caused the original incident.

Seventeen months later, the team upgrades its Redis nodes from one major version to the next using a rolling upgrade procedure. The upgrade procedure takes one node offline, upgrades it, brings it back online, verifies it joins the cluster, then proceeds to the next node. During the upgrade of the third node, the following sequence occurs: the third node goes offline for upgrade, taking the migration lock's third majority vote with it; the second node processes a Redis operation during the offline window; the third node comes back online and begins AOF replay to recover its state from before the upgrade — but the AOF log on the third node was persisted 47 seconds before the node went offline, meaning its replayed state does not include the most recent operations, including the current migration lock's SETNX entry.

The migration lock is now in this state: nodes 1, 2, 4, and 5 know the lock is held by Worker A. Node 3, with its replayed pre-upgrade state, does not have the lock entry — it appears to node 3 as if no lock is held. Worker B, a retry-check process that was waiting for what it believed was a stalled migration (the network monitoring dashboard had shown elevated latency on the node-3 upgrade traffic and the retry process interpreted this as the worker being unresponsive), sends lock acquisition requests to all five nodes. It receives grant responses from nodes 3 (which doesn't have the key) and — because the timing coincided with node 4 processing the request during a brief window when its own clock jumped forward 2.1 seconds due to NTP correction, causing its TTL expiry calculation to evaluate the key as already expired — node 4. Worker B has acquired two of five nodes: not a majority, it does not consider the lock acquired, and it does not proceed.

Fourteen seconds later, node 5 experiences its own brief clock correction of 1.8 seconds. Combined with the accumulated time since lock acquisition, node 5's TTL calculation evaluates the lock as expired. Worker B retries. It now receives grant responses from nodes 3, 4, and 5 — a majority. Worker B acquires the Redlock lock. Worker B begins applying the next migration.

Worker A, which had legitimately held the lock and was in the middle of a long-running migration involving a full-table column type change, is still running. Both Worker A and Worker B hold the Redlock lock simultaneously for a 23-second window — the time between Worker B's acquisition and the TTL expiry on nodes 1 and 2, which finally releases Worker A's lock references. During the 23-second overlap, Worker B applies the next migration while Worker A is still mid-ALTER on the current one. The resulting schema state is undefined.

The Redlock algorithm's documented limitations include exactly this failure mode: clock drift across nodes combined with AOF persistence gaps can produce a window where two holders simultaneously believe they hold the lock. The library's README cited the academic critique of Redlock (Martin Kleppmann's 2016 analysis) and the original author's response to that critique. Neither document was read during the migration decision session. The team knew Redlock was "more reliable than single-node Redis." They did not know the specific failure modes that Redlock improved and the specific failure modes it did not eliminate.

Three structural properties the distributed locking decision determines

1. The lock backend and its correctness guarantee

Every distributed locking backend makes a specific correctness claim and has a specific set of failure scenarios it cannot prevent. Choosing a lock backend without understanding both the claim and the uncovered failure scenarios is choosing a correctness ceiling without knowing where the ceiling is.

Single-node Redis SETNX provides mutual exclusion under the assumption that the Redis node is available and its state is current. It correctly prevents concurrent execution when both conditions hold. It fails to provide mutual exclusion under two scenarios that are not exotic: (1) the Redis node crashes while the lock is held, and restarts with an empty or stale state (RDB-based recovery may miss the most recent lock acquisitions); (2) the Redis node experiences a network partition that isolates the lock holder from the node while other workers remain reachable. Under scenario 1, the restarted node grants the lock to a new holder immediately, before the original holder knows the node has restarted. Under scenario 2, the partitioned worker may continue running (believing it still holds the lock) while a new worker acquires the lock from the worker's perspective of Redis.

The practical risk of scenario 1 depends on how the Redis node is deployed. A Redis node with appendonly yes (AOF persistence) and appendfsync everysec loses at most one second of data on crash — the lock acquisition will be in the AOF log if it occurred more than one second before the crash. A Redis node with RDB-only persistence may lose several minutes of data on crash, losing any lock acquisitions in that window. The TTL provides a secondary protection: even if the lock is lost on crash, the lock will become available again after the TTL expires, and the original holder should detect the Redis unavailability through a failed heartbeat or renewal attempt.

Redis Redlock improves on the crash scenario by requiring majority acquisition across N nodes — a single node crash does not remove the lock from the majority's knowledge, preventing immediate reacquisition. It does not improve on clock drift scenarios (where a node's TTL calculation becomes incorrect relative to the other nodes' calculations) or on AOF persistence gaps on individual nodes coming back online after being removed from the majority. The Redlock algorithm's failure probability under these conditions is not zero — it is bounded by the probability of the specific infrastructure events (clock correction magnitude, AOF persistence lag) occurring during the lock validity window. For most applications, this probability is acceptably low. For applications where double execution has severe consequences (charging customers, applying irreversible schema changes), the question is whether "acceptably low probability" is an acceptable correctness claim for a non-idempotent operation.

Postgres advisory locks provide correctness guarantees that depend on the Postgres session model. A session-scoped advisory lock is held until the session explicitly releases it or the Postgres connection is terminated. When a worker crashes, Postgres detects the terminated connection and releases the session-scoped lock within the TCP keepalive timeout (typically 30-120 seconds, configurable via tcp_keepalives_idle). This is a deterministic release mechanism — the lock will always be released when the session terminates, without requiring TTL estimation. The failure mode for Postgres advisory locks is not probabilistic like Redlock — it is structural: if the connection pool reuses connections without resetting session state, a session-scoped advisory lock held by a previous job may appear held on a connection that the pool considers fresh. The determinism of session-based release creates a different operational requirement: the connection pool must either reset session state on return or the advisory lock must use a transaction-scoped variant that is released on transaction end regardless of pool behavior.

Consensus-based backends (etcd, ZooKeeper) provide the strongest correctness guarantee by using distributed consensus protocols (Raft, Zab) that ensure linearizable writes — a lock is granted only when a majority of nodes have durably recorded the acquisition, and the consistency guarantee is maintained under network partitions between individual nodes as long as a quorum remains reachable. The TTL for consensus-based locks is implemented as a lease: the lock holder must renew the lease before it expires, and the lease expiry is enforced by the consensus cluster (not by a single node's clock). The cost is higher latency (multiple round trips for consensus), operational complexity (a properly deployed etcd or ZooKeeper cluster is more operationally demanding than independent Redis nodes), and a hard dependency on the consensus cluster's availability — if the consensus cluster loses quorum, no locks can be acquired or renewed, which is a different failure mode than Redis Redlock (where Redlock degrades gracefully as individual nodes fail, as long as the majority threshold is maintained).

The lock backend decision should be framed as: what failure scenarios does this job's double execution require protection against, and which backend's correctness guarantee covers those scenarios? A job where double execution has minor consequences (generating a report that will be deduplicated at the consumer) can tolerate single-node Redis's failure modes. A job where double execution has severe consequences (charging a customer, applying a non-reversible schema change, publishing a financial transaction to a ledger) requires either a backend with stronger correctness guarantees than single-node Redis, or an idempotency mechanism that makes double execution safe even when the lock fails.

2. The lock TTL model and the double-execution surface

The lock TTL serves two separate purposes, and the systematic confusion between them is the root cause of the billing story's double-execution failure.

As a crash recovery timeout, the TTL specifies how long after a holder crashes before the lock becomes available for a new holder. This use of TTL is about liveness — preventing permanent deadlock when the lock holder crashes and never explicitly releases the lock. For crash recovery, the TTL should be long enough that a recovering system can detect the crash, start a replacement worker, and begin processing without the lock expiring during recovery — typically measured in minutes rather than seconds.

As a maximum exclusivity window, the TTL implicitly specifies the maximum duration for which the lock backend will refuse to grant the lock to a new requester. This use of TTL is about correctness — preventing concurrent execution. For correctness, the TTL must be longer than the actual job runtime under all realistic conditions, because the lock backend will grant the lock to a new holder when the TTL expires, regardless of whether the original holder is still running.

These two uses require different TTL values derived through different methodologies, but they are implemented as the same TTL parameter. When an engineer sets the TTL to "longer than expected runtime," they are usually setting a crash recovery timeout that they believe will also serve as a correctness window — because "expected runtime" sounds conservative, and a "conservative" estimate of expected runtime sounds like it would cover the worst case. It does not, because expected runtime and worst-case runtime under load diverge under the conditions that are most likely to cause extended job duration: database slow queries, external API latency spikes, GC pressure, resource contention.

The correct TTL derivation methodology for correctness requires measuring worst-case job runtime, not expected runtime. The measurement should be made at the maximum realistic processing volume (accounting for future growth), under realistic load conditions (not a quiet test environment), and should use a high percentile (P99 or P99.9) rather than average or median. The derived worst-case runtime should be multiplied by a documented safety factor (2-3×) to account for conditions not represented in the measurement, and the TTL should be re-derived whenever the job's processing volume or workload characteristics change.

For jobs where worst-case runtime is genuinely unpredictable — because job duration depends on data characteristics that can vary by orders of magnitude (a migration job that processes a table of 10,000 rows and the same job processing a table that grew to 10,000,000 rows) — the correct solution is heartbeat-based lock renewal. The job acquires a short-TTL lock (30-60 seconds) and renews it every N seconds (N must be less than TTL, with renewal frequency typically at TTL/3 to TTL/2 to provide headroom for renewal failures). If the job is running, it continuously renews the lock. If the job crashes, the heartbeat stops, and the lock expires after the short TTL — achieving fast crash recovery. The double-execution failure mode from TTL expiry during extended runtime is eliminated because the lock never expires while the job is alive and renewing it.

Heartbeat renewal introduces its own requirements: the renewal must be atomic (the renewal should extend the TTL only if the lock's token matches the acquirer's token, preventing renewal of a lock that was re-acquired by a different holder); the renewal failure mode must be handled (if renewal fails because Redis is unavailable, the job must decide whether to continue running, risk double execution when another holder acquires the lock, or abort and release resources); and renewal intervals must be monitored to detect jobs that are renewing successfully but making no progress (indicating a hung job that is holding the lock but not completing).

The distinction between TTL as crash recovery and TTL as correctness window also determines the appropriate response to lock acquisition failure. If the previous lock holder crashed (and the TTL has not yet expired), waiting for the TTL to expire and then retrying is the correct behavior — the previous holder is gone and the new holder needs to wait for the crash recovery window. If the previous lock holder is still running (and simply holding the lock longer than expected), waiting for the TTL to expire and then acquiring the lock produces double execution — both the running job and the new job will execute concurrently after the TTL expires.

3. The idempotency requirement and the correctness ceiling

Distributed locks reduce the probability of concurrent execution under partial failure scenarios — they do not eliminate it. Every distributed locking mechanism has documented failure scenarios under which two holders can simultaneously hold the lock, as described above. This means the system's correctness ceiling under lock failure is determined by whether the protected operation is idempotent.

An idempotent operation is one where executing it multiple times produces the same result as executing it once. Generating a read-only report is idempotent: two concurrent report generation runs produce two identical reports, and the consumer deduplicates them (or the second one overwrites the first without harm). Sending a notification email with a nonce in the subject line is idempotent if the consumer deduplicates based on the nonce. For idempotent operations, distributed locking is a performance optimization — it prevents redundant work, not correctness failures. If the lock fails and two workers execute concurrently, the result is two redundant executions, not a correctness violation.

A non-idempotent operation is one where executing it multiple times produces a different result than executing it once. Charging a customer is non-idempotent: two concurrent billing runs produce two charges. Applying a database ALTER TABLE is non-idempotent: applying the same ALTER TABLE twice produces an error (if the column already exists) or a different schema state than intended (if the ALTER was cumulative). Publishing a financial transaction to a ledger is non-idempotent: two publishes create two ledger entries.

For non-idempotent operations, distributed locking is not sufficient as the sole correctness mechanism. The lock reduces the probability of concurrent execution — but the residual probability under documented failure scenarios means that the correctness ceiling is determined by the idempotency of the operation, not by the lock. An additional correctness mechanism is required.

The most robust additional mechanism is an idempotency key paired with a database unique constraint. Before executing a non-idempotent operation, the worker inserts a record into an idempotency table with a unique key representing this specific execution (for billing: the billing period + customer ID + billing attempt number). The insert is atomic and the unique constraint causes the second concurrent insert to fail with a unique constraint violation. The worker that receives the constraint violation knows that another worker is already executing this operation and aborts. The first worker to successfully insert the idempotency record is the only worker that proceeds with the non-idempotent operation. This mechanism provides correctness even if the distributed lock fails and both workers proceed past the lock check — the idempotency table provides a second serialization layer backed by the database's ACID guarantees.

An alternative is optimistic concurrency control: each record subject to the non-idempotent operation carries a version number, and the update is conditioned on the version number matching the expected value. If a second worker reads a record with version N and the first worker has already updated it to version N+1, the second worker's UPDATE WHERE version = N matches zero rows — the second worker detects the conflict and either retries or aborts. This is the mechanism that event sourcing systems use for concurrent write serialization without distributed locks. Optimistic concurrency adds a conflict detection step (read-modify-write with version check) that pessimistic locking (acquire lock, execute, release) does not need — but it provides correctness guarantees backed by the database's MVCC guarantees rather than the distributed lock backend's consistency model.

The message queue is an architectural alternative that avoids the distributed locking problem entirely for certain use cases. If the non-idempotent operation can be expressed as a message consumer processing from a partitioned queue with exactly-once or at-least-once delivery guarantees, the queue's delivery semantics can provide serialization without a separate distributed lock. Billing a customer can be expressed as: produce one "bill customer X for period Y" message, configure the consumer with exactly-once delivery (if supported), process the message, commit the offset. The queue's offset-based delivery ensures the message is processed once per consumer group. This shifts the correctness responsibility from a distributed lock to the queue's delivery semantics — which may be a better abstraction, or may trade one set of partial-failure failure modes for another set, depending on the queue infrastructure.

The new technical leader inheriting a billing system with Redis-based distributed locking cannot determine from the code alone: what the TTL derivation methodology was, whether the protected operation is idempotent or non-idempotent (the code may be correct in isolation but incorrect when executed twice, and the correctness issue only manifests during lock failure), or whether there is an additional correctness mechanism beyond the lock. These are decisions made in the initial implementation session that are invisible in the code if not documented.

Five ADR sections the distributed locking decision record needs

1. Lock backend selection and failure model

Document the chosen lock backend and the specific failure scenarios it prevents and cannot prevent. The cannot-prevent list is as important as the can-prevent list — it names the residual risk that must be addressed through other means (idempotency keys, operational procedures, alternative architecture).

For Redis single-node: specify the persistence configuration (RDB-only, AOF with fsync mode) and the resulting maximum data loss window on crash. Document whether the lock provides correctness under Redis primary failure with replica promotion — a promotion that restores from an AOF that does not include the most recent lock acquisition is a correctness gap if the promoted replica grants the lock to a new holder while the original holder continues running. Specify whether a circuit breaker on the Redis connection is in place to handle Redis unavailability — when Redis is unavailable, lock acquisition will fail, and the response (skip execution, queue for retry, execute without lock) is a correctness decision that must be made explicitly.

For Redis Redlock: document the number of nodes, the majority threshold, and the persistence configuration of each node. Note the documented limitations (clock drift, AOF persistence gaps on rejoining nodes) and the risk assessment for whether the application's non-idempotent operations require protection against these failure modes. Document the library version and the specific implementation's handling of the clock drift mitigations (validity window calculation, retry intervals). Note whether the Redlock implementation verifies the lock token before the critical section — some client libraries acquire the lock and begin the critical section without verifying that the lock's validity window has not expired between acquisition and execution start.

For Postgres advisory locks: document whether session-scoped or transaction-scoped locks are used, the rationale for the choice, the connection pool configuration (whether it resets session state on connection return), and the explicit release pattern in both the success path and all exception handlers. Note the database dependency this creates — the lock backend is the same infrastructure as the application database, which means database slowdowns or connection pool exhaustion affect both the protected operation and the lock acquisition.

2. Lock TTL and heartbeat model

Document the TTL value, the measurement methodology that produced it, and the conditions under which the TTL must be recalculated. Specify whether the TTL represents a crash recovery timeout (and therefore must cover the time between crash detection and replacement worker startup) or a maximum exclusivity window (and therefore must cover the worst-case job runtime under production load), or both — and if both, how the two requirements are reconciled if the crash recovery timeout requirement produces a shorter value than the worst-case runtime requirement.

Document the worst-case runtime measurement: what dataset size was used, what load conditions were present during measurement, what percentile was used (P99 or P99.9 rather than average), and what safety factor was applied to the measured value. Record the measured value and the applied factor separately so that future engineers can update each component independently — if the dataset grows 5×, the measured value can be re-derived without re-deriving the safety factor.

If heartbeat-based renewal is used, document the renewal interval, the renewal protocol (token verification on renewal), the response to renewal failure (abort and release, continue with a reduced confidence window, alert and wait), and the monitoring for heartbeat health (renewal success rate, time since last successful renewal for running jobs). Document the maximum number of consecutive renewal failures before the job self-terminates — a job that continues running indefinitely after renewal failures is holding resources without holding a valid lock, and may execute concurrently with a new holder that acquired the lock after the renewal failures caused the TTL to expire.

3. Lock acquisition and release protocol

Document what happens when lock acquisition fails. The three options have different correctness and liveness implications. Skipping the current execution window (the job does not run if the lock is unavailable) is safe from a double-execution perspective but reduces liveness — if the lock is stuck (because the previous holder crashed after the TTL was set too short and has not yet released the lock), the job will not run until the TTL expires. Queuing for retry (the job waits and retries acquisition after a backoff interval) maintains liveness but introduces a race condition: if the previous holder is still running when the retry succeeds, the new holder will execute concurrently. Executing without the lock (the job proceeds if acquisition fails after N retries, on the assumption that the lock backend is unavailable and the operation should not be blocked by a lock backend failure) maintains liveness but eliminates the lock's correctness guarantee for non-idempotent operations.

Document the lock token generation and verification protocol. A correctly implemented distributed lock generates a unique token at acquisition time (typically a UUID or a cryptographic random value), stores the token as the lock key's value, and verifies that the lock key's value matches the stored token before executing the critical section and before releasing the lock. The token check before the critical section detects the case where the lock was acquired, time passed, the TTL expired, a second holder acquired the lock, and then the first holder proceeds to execute — the token mismatch tells the first holder that it no longer holds the lock. The token check before release prevents the first holder from releasing the second holder's lock after the first holder's TTL expires.

Specify the lock release protocol on both the success path (explicit release with token verification) and the failure path (exception handlers must release the lock on the failure path; unhandled exceptions that bypass the release code produce stuck locks that hold until TTL expiry). For Postgres advisory locks, document the use of pg_advisory_unlock_all() as a session cleanup call in the connection return hook, as a defense against missed releases on non-standard failure paths.

4. Idempotency requirement and the lock's role

Document explicitly whether the protected operation is idempotent or non-idempotent. This is the most consequential section of the distributed locking decision record because it determines whether the lock is the correctness mechanism or a performance optimization.

For non-idempotent operations, document the additional correctness mechanism. Options in order of increasing implementation cost and correctness strength: (1) idempotency table with a unique constraint on the execution key — the simplest and most broadly applicable mechanism; (2) optimistic concurrency control on the records being modified — appropriate when the modified records carry a natural version column; (3) conditional writes using database-supported atomic check-and-set operations — appropriate when the operation is a single conditional update; (4) exactly-once message delivery via a partitioned queue — appropriate when the operation can be expressed as a consumer-side idempotent application of a message.

Specify the scope of idempotency: which operations within the job are non-idempotent, and which can be retried safely. A billing job has three distinct operation types: reading subscription state (idempotent), charging the payment method (non-idempotent), and updating subscription status after a successful charge (idempotent if conditioned on the charge having been recorded). The idempotency mechanism applies specifically to the charge operation — the charge API should receive an idempotency key (most payment processors support this natively) that prevents the second charge from being processed by the payment processor even if the billing job submits the same charge twice.

For idempotent operations, document that the lock is a performance optimization and specify what happens when the lock fails and concurrent execution occurs: are there resource constraints that multiple concurrent executions would violate (memory, CPU, network bandwidth, downstream rate limits), and if so, is the lock failure response to accept resource contention or to implement a more robust serialization mechanism? A report generation job that is idempotent in terms of correctness but requires significant memory allocation may still need reliable serialization to prevent resource exhaustion under concurrent execution.

5. Lock observability and monitoring

Document the metrics collected for each lock. The minimum set for detecting the failure modes described in this record:

Lock hold duration as a percentage of TTL: a job that routinely completes at 70-80% of its TTL is approaching the threshold at which a slower run will cross it. This metric should alert at 80% of TTL to provide remediation time before the double-execution failure mode activates. The alert should trigger a review of the TTL derivation — either the TTL should be extended, or the job should be redesigned to run faster (batch size reduction, query optimization, parallel processing that reduces wall-clock time while increasing database load).

Lock acquisition failure rate: the rate at which workers attempt to acquire the lock and find it already held. A baseline acquisition failure rate reflects the job's normal overlap window (the overlap between the job's scheduled interval and its actual runtime). An elevated acquisition failure rate indicates either that job runtime has increased (the same worker is now taking longer, causing more overlap with the next scheduled invocation) or that multiple workers are being scheduled simultaneously (a misconfiguration in the job scheduler). Elevated acquisition failure rate is an early warning for the conditions that produce TTL expiry failures.

For heartbeat-based renewal: renewal success rate and time since last successful renewal for each running job. A renewal failure is an early warning of Redis connectivity issues that will eventually expire the lock. The time since last renewal should be compared to the TTL to estimate the remaining exclusivity window — a job that has not renewed for TTL × 0.7 seconds is at risk of lock expiry within the next TTL × 0.3 seconds.

For Postgres advisory locks: the count of advisory locks held per session, compared to the expected count (one per active job). A session holding more advisory locks than it should indicates missed releases on previous executions using that connection. This metric is queryable from pg_locks filtered to locktype = 'advisory', and monitoring it catches the connection pool session-reuse failure mode before it produces double execution.

Stale lock detection: a lock held for longer than N × TTL (where N is the expected maximum number of TTL-renewal cycles for a single job execution, or simply the worst-case job runtime divided by the TTL for non-heartbeat locks) should trigger an alert and an investigation. A stale lock indicates either a job that has been running unusually long (approaching or exceeding the TTL for non-heartbeat locks), a heartbeat renewal failure that has not yet caused TTL expiry (for heartbeat locks), or a lock that was not released after job completion and is being held by a pooled connection. The response to a stale lock alert should be investigation first — the lock may be held by a legitimately long-running job — and forced release only after confirming the holder is no longer active.