The distributed systems consensus decision record: why the consensus model you chose determines your consistency guarantee ceiling and your partition tolerance surface

A 14-person fintech building a payment reconciliation platform had a staff architect who had spent the previous four years scaling a high-throughput analytics pipeline at a previous company. The central challenge at that company had been write throughput: millions of events per second flowing into a logging system that had to ingest without bottlenecks. The architect's mental model for storage design was shaped by that experience: choose a system that writes fast, reads eventually, and scales horizontally without coordination overhead. When she joined the fintech and began designing the data layer, Cassandra was the obvious choice. Write-anywhere, multi-datacenter replication, linear scaling with node count, eventual consistency that handled the high-volume transaction log ingestion the platform required. The decision was documented in the team's internal wiki as "Cassandra for the data layer — scales for our transaction volume, handles multi-region replication without a write bottleneck."

The platform launched with three primary data types: a transaction log table recording every payment event with timestamp and event type, a payment status table recording the current state of each payment (pending, processing, completed, settled, failed), and a reconciliation summary table aggregating daily settlement figures per merchant. The transaction log was a perfect fit for eventual consistency: each row was an immutable append, reads were historical, and a brief replication lag between two Cassandra nodes produced no incorrect behavior. The reconciliation summary was similarly fine: merchant settlement figures were computed from the transaction log in a nightly job, and a brief divergence between two nodes during the computation window had no correctness consequence. The payment status table was different. Payment status records were read by four downstream systems: the merchant dashboard (display only), the webhook dispatcher (which sent payment confirmation webhooks to merchant endpoints), the refund processor (which checked payment status before authorizing a refund), and the fraud detection system (which flagged status transitions that violated expected patterns). Each of these consumers treated the payment status table as a state machine: they read the current status, made a decision based on that status, and expected the status to reflect the most recent write.

Six months after launch, on a Saturday at 2:14 AM, a network partition between two Cassandra nodes lasted 4.1 seconds before automatic repair. During those 4.1 seconds, node A received a status update marking payment ID 449127 as "completed." Node B, partitioned from node A, still showed payment ID 449127 as "pending." The webhook dispatcher read from node B (the closest replica in its availability zone), saw "pending," and re-queued the payment for processing. The webhook dispatcher read "pending" three times across the 4.1-second window — the polling interval was 1.5 seconds — and fired the payment confirmation webhook to the merchant endpoint three times. The merchant's endpoint was idempotent for the first retry but not for the third: the idempotency window was 3 seconds, and the third webhook arrived 4.5 seconds after the first, outside the window. The merchant's order management system processed the confirmation three times, generating three order fulfillments. Fourteen similar payments were affected in the 4.1-second window before the partition healed and Cassandra's last-write-wins reconciliation resolved the divergence to the correct "completed" status across all nodes. The duplicate fulfillments cost the fintech $4,200 in refunds and customer credits, plus four days of engineering time to identify the cause and patch the webhook dispatcher to read from a consistent quorum rather than from the nearest replica.

The architectural fix was not to replace Cassandra. It was to classify the payment status table as a strong-consistency requirement and route status reads through Cassandra's QUORUM read consistency level rather than LOCAL_ONE — a configuration change requiring every consumer to specify consistency level on each read, enforced through a shared data access library rather than left to individual service implementations. The transaction log and reconciliation summary tables remained at LOCAL_ONE reads, where the original design had been correct. The original decision had treated consistency as a database-level property — "Cassandra for the data layer" — rather than as a per-data-type requirement that varies within the same database depending on the semantics of each table. The documentation of "Cassandra for the data layer" contained no data type classification, no consistency level specification per table, and no partition tolerance analysis for the tables where eventual consistency would produce correctness failures. The constraint reasoning existed in the architect's head as "transaction logs are eventually consistent, that's fine" without ever being written down as "payment status is a state machine read by systems that make irreversible external effects, and eventual consistency for payment status produces duplicate webhook delivery during a replication lag." Connect this to the caching strategy decision record: the same classification problem recurs when downstream caches are introduced — a cache in front of a payment status reader inherits the consistency model of the underlying store, and a cache with a 30-second TTL compounds the replication lag into a 34-second window of stale payment status reads if the underlying store's consistency model is not explicitly specified as a constraint on any upstream cache configuration.

A 37-person B2B SaaS building a cloud infrastructure orchestration platform had an engineering team of eighteen and a distributed systems engineer who had worked with ZooKeeper-based coordination services for three years before joining. When the team needed a reliable source of truth for their distributed configuration — the set of feature flags, resource limits, and service endpoint addresses consumed by every microservice in their platform — he recommended etcd: a battle-tested Raft-based key-value store with linearizable reads and writes, strong consistency guarantees appropriate for configuration data where a stale read could cause a microservice to use a deprecated endpoint or an incorrect resource limit. The recommendation was correct. The decision was documented as "etcd for distributed configuration — linearizable reads and writes, Raft consensus, battle-tested by Kubernetes." The document included the cluster sizing (three nodes for quorum, one in each availability zone), the key schema, and the client library. It did not include the partition tolerance strategy: what would happen to reads and writes during a node failure, a network partition, or a cluster upgrade, and how the 400 microservices consuming configuration from etcd would behave during an etcd write unavailability window.

Fourteen months after launch, the distributed systems engineer began an etcd cluster upgrade from version 3.4.18 to 3.5.4 to address three CVEs. The upgrade procedure was a rolling restart: take down one node, upgrade it, bring it back up, repeat for the remaining two nodes. In the staging environment, with zero client traffic, the upgrade took eight minutes and produced no errors. In production, at 14:07 UTC on a Tuesday, the engineer took down the first etcd node (etcd-node-2) for upgrade. With three nodes and one down, the cluster had a two-node quorum and continued serving reads and writes normally. The first node upgrade completed in four minutes. At 14:11 UTC, the engineer took down the second node (etcd-node-3) for upgrade. With two nodes and two down, the cluster had a one-node quorum — insufficient for Raft consensus, which requires a majority of three, meaning at least two nodes. The single remaining node (etcd-node-1) was unable to reach quorum and began a leader election. The default Raft election timeout in etcd is 1000ms per tick with 5 ticks per election timeout — approximately 5 seconds for the first election attempt, up to 10 seconds if the first attempt fails. etcd-node-1 attempted leader election with itself as the only node, which Raft correctly rejected because a single node cannot form a quorum of three. Writes to etcd blocked for 7.3 seconds.

During those 7.3 seconds, the 400 microservices consuming configuration from etcd were healthy — they used cached configuration values and did not need to read from etcd for most operations. But 180 of the 400 microservices had a health check implementation that called etcd to verify connectivity as part of their liveness probe, under the assumption that etcd connectivity was a proxy for the service's ability to serve traffic. The liveness probes began failing at second 3 of the etcd write block (the TCP connection to etcd established but the read timed out at 3 seconds per the client library's default). The load balancer pulled 180 microservices from its upstream pool — 60% of the platform's service instances — within 4 seconds of the liveness probe failure. The platform served traffic from the remaining 40% of instances at degraded throughput for 12 seconds until the upgrade completed, etcd-node-3 rejoined, quorum was restored, and the liveness probe health checks returned to passing. No data was lost, no configuration was corrupted, and the upgrade completed successfully. But the 12-second 60%-availability degradation affected three enterprise customers during a peak usage window, generating two support escalations and one SLA breach notification.

The root cause was not the rolling upgrade procedure, which was correct. It was the combination of three undocumented assumptions: the election timeout behavior during a two-node-down scenario (undocumented in the ADR), the liveness probe design that treated etcd connectivity as a health signal for services that could serve traffic without etcd (undocumented in any service's operational runbook), and the absence of a partition tolerance strategy specifying what the acceptable write unavailability window was for an etcd cluster with three nodes (undocumented — the ADR said "linearizable reads and writes" without specifying what happened to reads and writes when the cluster could not reach quorum). The fix required changes to three separate systems: the upgrade procedure (pre-upgrade health check verifying all three nodes are healthy before beginning, graceful leader stepdown on the active leader before taking it down), the election timeout configuration (adjusted to 500ms per tick with 3 ticks for faster recovery), and the liveness probe implementations (removed etcd connectivity check from liveness, kept it in readiness where a failed etcd connectivity check removes the pod from the load balancer without restarting it). Each fix addressed a different aspect of the same underlying gap: the partition tolerance surface of the consensus algorithm was not part of the original design documentation, and the operational procedures for every system that depended on etcd did not account for the etcd write unavailability window that the consensus algorithm produced during a cluster state change. Connect this to the observability strategy decision record: the 12-second degradation was detected only through customer support escalations and a post-incident SLA breach review — the platform had no alerting on etcd election timeout events, no dashboard tracking etcd write latency against a threshold that would have triggered an on-call notification before the availability degradation reached customers, and no synthetic transaction test that would have detected the liveness probe failure cascade before it produced a visible availability drop.

A 52-person enterprise SaaS building an e-commerce operations platform had an engineering team of twenty-nine organized into three product teams: a Catalog team owning product listings and pricing, an Inventory team owning warehouse stock and reservation management, and a Billing team owning order payment and invoicing. The platform's core operation — placing an order — required atomic coordination across all three services: reserve inventory in the Inventory service, create the order record in the Catalog service's order database, and initiate payment processing in the Billing service. When inventory reservation, order creation, and billing initiation were implemented as independent service calls without coordination, the platform experienced three categories of correctness failures: orders created and billed without successful inventory reservation (overselling), orders created and inventory reserved without successful billing (inventory locked for unpaid orders), and billing initiated twice for the same order due to a retry in the order creation service that did not check whether billing had already been initiated. The CTO convened a working session with the three team leads to design a transaction model. The solution they landed on was a two-phase commit coordinator: a dedicated OrderCoordinator service that managed the prepare-and-commit protocol across all three services, ensuring that either all three operations committed successfully or all three rolled back.

In the six months after the 2PC coordinator was deployed, the platform's correctness failures dropped to near zero. The prepare-commit-rollback protocol handled all three failure categories correctly, and the coordinator's write-ahead log ensured that coordinator crashes could be recovered by replaying the log to the commit or rollback state. In development and staging, commit latencies averaged 45ms end-to-end — fast enough to be imperceptible to users. The engineering team treated the coordinator as solved infrastructure and moved on to building the Catalog and Inventory features that the quarterly roadmap required. The partition tolerance surface of the 2PC coordinator — what happened to in-flight transactions when any of the three services experienced latency spikes or network issues — was not analyzed because the development and staging environments had never produced it.

The first Black Friday after the 2PC coordinator was deployed, the Billing service began experiencing Stripe API latency spikes at 11:43 AM EST — typical behavior for Stripe during peak commerce periods. The prepare phase of the 2PC protocol required the Billing service to call Stripe's payment intent creation API to verify payment method validity before responding "prepared" to the OrderCoordinator. During normal operation, Stripe's prepare call took 15-50ms. During the Black Friday latency spike, it took 800ms to 2.4 seconds. The 2PC coordinator held distributed locks on the Inventory service and the Catalog service during the prepare phase — the Inventory service reserved the items and held the reservation under a write lock, and the Catalog service created a draft order record under a row lock, both locks held until the coordinator received the "prepared" response from the Billing service and proceeded to the commit phase. With the Billing prepare call taking 2 seconds, those locks were held for 2 seconds per order. At the platform's normal order rate of 200 orders per second, a 2-second lock hold time produced a lock contention backlog that exceeded the Inventory service's connection pool within 90 seconds. Order throughput dropped from 200 orders per second to 12 orders per second as the vast majority of order creation attempts blocked waiting for connection pool availability. Three coordinator instances crashed under the load, leaving their in-flight transactions with locks held indefinitely until the engineering team manually intervened to resolve the locks. The Inventory service was effectively unavailable for 8 minutes until the engineering team disabled new order creation entirely, waited for in-flight transactions to resolve, and brought the system back online at reduced capacity.

The post-incident analysis identified the architectural mismatch: two-phase commit provides atomicity guarantees that are correct for the platform's consistency requirements, but 2PC's lock-holding model makes write availability during a participant's latency spike a function of the slowest participant's response time, multiplied by the concurrent transaction rate, multiplied by the lock duration per transaction. At 200 transactions per second and 2-second lock durations, 2PC required 400 concurrent locks to be held at any instant — a number that exceeded the Inventory service's connection pool and produced cascading availability failures. The saga pattern — where each service commits its own local transaction and publishes an event, with compensating transactions handling rollback — would have provided the same atomicity guarantee across the eventual execution of all three steps while each service committed independently, eliminating the distributed lock contention entirely. The saga pattern accepted a brief window of partial consistency: between the Inventory service committing its reservation and the Billing service committing its payment initiation, the order would show as "processing" rather than as "confirmed." That window was approximately 100ms under normal conditions and would have been 2 seconds under Stripe's Black Friday latency — visible to users as a slightly longer order confirmation spinner, not as an overselling failure or an availability incident. The platform spent the following six months migrating from 2PC to a saga pattern using event sourcing, with idempotency keys on all Stripe API calls and compensating transactions for each step. Connect this to the API versioning decision record: the saga pattern's compensating transactions required each service to expose a stable, versioned rollback API that the saga orchestrator could call idempotently — an API contract that must be versioned and maintained with the same rigor as any external-facing API, and whose deprecation must be coordinated across all saga participants before the rollback endpoint can be removed from any service's API surface.

Structural properties set by the distributed systems consensus decision

Two structural properties are determined by the founding decisions about which consistency model is selected for each data type, which consensus algorithm produces those guarantees, and which transaction model manages cross-service correctness invariants. Neither property is visible during normal operation — they surface during the partition events, leader elections, and peak load scenarios that expose the gap between the system's design assumptions and its actual operating environment.

Property 1: Consistency guarantee ceiling. The consistency guarantee ceiling is the strongest consistency level the system can provide for a given data type, determined at system design time by the consensus model embedded in the data layer. Four levels form a hierarchy from strongest to weakest. Linearizability (single-copy semantics): every operation appears to execute atomically between its invocation and response, and operations are totally ordered across all clients — a read that begins after a write completes will always return that write's value or a later value, regardless of which replica handles the read; Raft and Multi-Paxos with quorum reads produce linearizability; Cassandra at QUORUM read level produces linearizability for a single key. Sequential consistency: all clients see operations in the same total order, but that order need not correspond to real-time wall-clock ordering — a read may see a write from the past if the write was not yet visible in the total order; most CP systems in practice provide linearizability rather than sequential consistency because linearizability is simpler to reason about. Causal consistency: operations that are causally related are seen in causal order by all clients, but concurrent operations may be seen in different orders by different clients; causally consistent systems (like CockroachDB's follower reads with bounded staleness) provide more throughput than linearizable systems at the cost of visibility ordering guarantees for concurrent writers. Eventual consistency: all replicas converge to the same value in the absence of new writes, with no ordering guarantee and no bound on the convergence time; Cassandra at LOCAL_ONE, DynamoDB in its default mode, and CouchDB provide eventual consistency. The practical implication: the consistency ceiling for a given data type cannot be raised after the data model is built without changing the read and write paths for that data type — and changing the read and write paths for data that has been in production requires a migration that touches every producer and consumer of that data type. Misclassifying a data type — applying eventual consistency to a state machine that requires linearizability, as the fintech's payment status table illustrates — produces correctness failures that manifest at the first partition or replication lag event rather than at system design time, which is why the classification must happen during the design phase when the cost of changing it is low. Connect this to the capacity planning decision record: the consistency ceiling has a direct throughput implication — a linearizable system requires quorum acknowledgment on every write, which adds two to three network round trips compared to a LOCAL_ONE write; at 10,000 writes per second, the additional latency from quorum acknowledgment is a capacity cost that must be planned for in the system's throughput budget, because the consistency guarantee and the write throughput budget are set simultaneously by the consensus model selection rather than independently.

Property 2: Partition tolerance surface. The partition tolerance surface is the operational behavior of the system's reads and writes during a network partition, a node failure, or a leader election — the set of conditions under which reads degrade, writes block, or both, and the duration and character of that degradation. In a CP system using Raft or Multi-Paxos, the partition tolerance surface has three components: writes block until quorum is restored (duration = detection latency + election timeout + leader stabilization time, which ranges from milliseconds under normal conditions to seconds under configuration with conservative election timeout settings); reads from the leader remain consistent during a partition that does not affect the leader; reads from followers may return stale values if the cluster is configured to allow follower reads. The 37-person SaaS example's 7.3-second write block was the partition tolerance surface of a Raft cluster with default election timeout settings, exposed by a rolling upgrade that temporarily reduced the cluster to a single node. In an AP system using eventual consistency, the partition tolerance surface has a different character: writes continue to all reachable replicas during a partition, diverging in value until the partition heals and the reconciliation model (last-write-wins, vector clocks, CRDTs) resolves the divergence; the surface is not write unavailability but write divergence, and the duration is the partition healing time plus the reconciliation propagation time. The 14-person fintech's 4.1-second window of payment status divergence was the partition tolerance surface of an AP system, exposing the consistency ceiling gap for a data type that required linearizability. The partition tolerance surface must be specified explicitly in the consensus model ADR rather than derived from the CAP theorem at a high level of abstraction: "linearizable reads and writes" as the sole description of an etcd cluster's consistency model contains no information about the partition tolerance surface that production operations require — the election timeout, the acceptable write unavailability window, the behavior of client connections during leader election, and the detection and alerting mechanisms that make the partition tolerance surface visible before it causes an availability incident.

The distributed systems consensus ADR: five sections

Section 1: Data consistency classification. The consistency classification enumerates each data type or table owned by the system and specifies the consistency level required for that data type, with explicit rationale. The classification should address three questions per data type. First, what is the semantic model for reads of this data: is the data used to make decisions with irreversible external effects (payment webhooks, inventory reservations, access control grants), or is it displayed to a user where a brief staleness window has no consequence, or is it consumed by an analytical system where eventual convergence is sufficient? Second, what is the semantic model for writes: does this data type have concurrent writers, and if so, can write conflicts be resolved by last-write-wins, vector clocks, or application-level merge logic, or does write serialization require strong consistency? Third, what is the consequence of a consistency failure — a moment where two readers see different values for the same key — and is that consequence a data integrity failure (duplicate payment), a display artifact (stale dashboard count), or a compliance violation (audit log with missing entries)? The classification output is a per-table or per-data-type table with columns: data type name, required consistency level (linearizable / sequential / causal / eventual), rationale (one sentence), and consequence of consistency failure (one sentence). This table is the canonical input to every subsequent section of the ADR and to every future decision about read and write path configuration for the system. A data type not in the table is a gap in the classification — and a gap in the classification is a latent correctness failure waiting for the first partition event to expose it.

Section 2: Consensus algorithm and replication model selection. The consensus algorithm selection specifies which algorithm or replication model will be used for each consistency tier identified in the classification. The selection should address four criteria explicitly. Consistency ceiling produced: which consistency level does the algorithm provide, and does it match the requirements identified in the classification (Raft and Multi-Paxos produce linearizability; Cassandra at QUORUM produces per-key linearizability for single-partition reads; CRDTs produce strong eventual consistency without coordination)? Throughput and latency profile: what is the write latency under the algorithm at the expected write rate, and what is the throughput ceiling — Raft's leader-serialized writes produce lower peak throughput than AP stores' multi-leader writes, and that difference is a capacity budget implication that must be quantified against the expected write volume? Operational maturity and tooling: what is the operational complexity of running the algorithm at the required cluster size, what monitoring and alerting does the algorithm's implementation expose, and what recovery procedures are documented for the failure scenarios the algorithm produces? Migration path: if the consistency requirements of a data type change — because the system grows into a use case requiring stronger or weaker consistency — what is the migration path from the current algorithm to the alternative, and what is the operational cost of that migration? The selection should explicitly reject alternatives with rationale: "we chose Cassandra at QUORUM for payment status rather than PostgreSQL with read replicas because our write throughput exceeds PostgreSQL's single-leader write ceiling at our projected scale, and QUORUM reads provide the per-key linearizability our payment state machine requires without requiring a separate strongly-consistent store for state machine data alongside an eventually-consistent store for transaction logs" contains the constraint reasoning that a future migration decision requires, because the constraint (write throughput ceiling) and the consequence (data type classification) are both explicit.

Section 3: Partition tolerance strategy. The partition tolerance strategy specifies what happens to reads and writes for each consistency tier during a network partition, node failure, or leader election, and what the acceptable degradation profile is. The strategy should address four components. Write availability during partition: for CP systems, what is the acceptable write unavailability window (specifying the election timeout configuration that produces that window), and what is the behavior of write clients during that window (block, retry with backoff, fail fast and surface the error to the application layer)? Read availability during partition: for CP systems with follower reads enabled, are stale follower reads acceptable during a leader election window, or must reads block until quorum is restored? For AP systems, are reads from a partitioned replica acceptable knowing the replica may be behind by up to the partition healing time? Graceful degradation path: what is the degraded-mode behavior for each consumer of data stored in the consensus system during a partition event — can the consuming service continue serving requests from a local cache for the duration of the acceptable partition window, or does any partition cause the consuming service to degrade its own availability? Recovery procedure: what is the operational procedure for recovering from a partition event — how are split-brain scenarios detected, which node's state is authoritative after a partition heals, and what monitoring surface detects partition events before their consumer impact is reported through support channels? The 37-person SaaS's incident would have been prevented by a partition tolerance strategy that specified: "Raft election timeout configured for 300ms per tick with 3 ticks (900ms maximum election time); rolling upgrade procedure requires active leader stepdown before any node is taken offline; etcd-connectivity health checks are readiness signals, not liveness signals, and consuming services must cache their last-known-good configuration values for up to 30 seconds to tolerate a partition event of that duration without degrading their own availability."

Section 4: Cross-service transaction model. The cross-service transaction model specifies how operations that span multiple service boundaries maintain their correctness invariants, and the explicit tradeoff between consistency strength and availability under the expected latency and failure profiles of each participant. The model should address three decision points. Atomicity model: what is the atomicity guarantee required — all-or-nothing (2PC) versus eventual atomicity (saga with compensating transactions) versus best-effort with idempotency (each operation is idempotent and retried independently until it succeeds)? The choice depends on the consequence of partial execution: if a partially-executed cross-service operation produces a correctness failure that cannot be resolved by a compensating transaction after the fact (irreversible external effects with no rollback path), 2PC may be required; if partial execution produces a state that compensating transactions can resolve (inventory reservation without billing initiation can be compensated by releasing the inventory reservation), sagas are appropriate; if all operations are idempotent and retryable, best-effort with idempotency keys requires no coordination. Latency sensitivity: 2PC's prepare phase holds distributed locks on all participants for the duration of the slowest participant's prepare response — under normal conditions and under the worst-case latency profile each participant may encounter (external API calls, database lock contention, garbage collection pauses); if the worst-case prepare latency for any participant multiplied by the concurrent transaction rate would exceed the connection pool or lock table capacity of any participant, 2PC produces the availability failure the 52-person enterprise SaaS experienced; sagas do not hold distributed locks and scale to concurrent transaction rates independently of participant latency. Rollback semantics: for saga patterns, what is the compensating transaction for each forward operation, and is the compensating transaction guaranteed to succeed under all failure scenarios (a compensation that can itself fail requires a compensation for the compensation, and so on recursively until the compensation is guaranteed idempotent or the partial execution is accepted as a permanent inconsistency); compensating transactions that can fail permanently must be handled explicitly — either through a dead letter queue and manual intervention path, or through designing the forward operations to be reversible at the data model level.

Section 5: Consistency violation detection and monitoring model. The consistency violation detection model specifies how the system detects when its consistency guarantees have been violated in production and what the alerting and response procedure is. The model should address three detection mechanisms. Invariant checking: for each data type with a strong consistency requirement, what application-level invariants can be checked periodically as a consistency probe — a payment status that transitions backward (completed to pending), two payments for the same order in non-terminal states simultaneously, an inventory reservation record without a corresponding order record within the expected saga completion window; these invariant checks should run as background processes on a schedule (every minute for correctness-critical invariants) and produce alerts when violations are detected, before the violation manifests as a customer-visible incident. Read path monitoring: what metrics on the read path indicate a consistency degradation before an invariant is violated — replication lag above a threshold, quorum read failure rate above a threshold, leader election events, follower read staleness above the consistency classification's acceptable window; these metrics should be in the system's observability dashboards and produce alerts at thresholds that allow the on-call engineer to investigate before a consistency ceiling breach produces a customer impact. Synthetic transaction testing: for cross-service transaction models, does the monitoring suite include a synthetic transaction that executes the full saga or 2PC protocol end-to-end against production infrastructure on a scheduled basis (every 5 minutes for critical paths), measuring end-to-end commit latency, detecting partial execution failures, and alerting when the synthetic transaction's behavior deviates from the expected model? The 14-person fintech's duplicate charge incident was detected through customer support channels after 14 affected payments — a detection latency that a read-path replication lag alert with a 2-second threshold would have reduced to under 10 seconds from the start of the partition event. Connect this to the observability strategy decision record: consistency violation detection requires instrumenting the application layer with business-semantic metrics (payment state transitions, saga step completion rates, replication lag per data type) rather than relying solely on infrastructure-layer metrics (CPU, memory, connection count) — the infrastructure metrics for the 14-person fintech's Cassandra cluster showed nothing anomalous during the 4.1-second partition event because the partition was within the cluster's normal self-healing behavior; only the application-layer metric (payment state machine divergence per minute) would have exposed the consistency gap between the infrastructure's correct self-healing behavior and the application's incorrect assumption that eventual consistency was appropriate for payment state machine data. Running the AI chat sessions in which the team worked through the consistency model selection through the open-source extractor before designing the consistency violation detection suite produces the data type classification that identifies which invariants to check — a consistency violation detection suite that does not know which data types are supposed to be linearizable cannot check for linearizability violations; the detection suite is only as complete as the consistency classification that drives it.

FAQ

Further reading

• Caching strategy decision record — caching decisions interact with the consensus model because a cache in front of a data store inherits the consistency properties of the underlying store, compounded by the cache TTL: a payment status reader with a 30-second cache TTL backed by an eventually-consistent store produces a correctness window that is the sum of the replication lag and the cache TTL rather than just the replication lag alone; the caching strategy ADR should include an explicit section on consistency implications for each data type cached, and any cache TTL for a data type requiring strong consistency should be treated as a consistency ceiling constraint — a cache TTL greater than zero on a linearizability-required data type means the system's effective consistency guarantee for that data type is "consistent within the TTL window," not linearizable, and that reclassification must be explicit and deliberate.
• Observability strategy decision record — consistency violations in production are detectable only through application-layer metrics that reflect the business semantics of the data type, not through infrastructure metrics that reflect the health of the consensus algorithm itself; a Cassandra cluster that is self-healing correctly through a partition event shows no anomalous infrastructure metrics while producing stale reads in the application layer; the observability strategy for a system with strong consistency requirements must include application-layer invariant checks (payment state machine transitions, saga step completion rates, replication lag per data type mapped against the consistency classification) as first-class monitoring targets with alerting thresholds calibrated to the acceptable consistency failure window specified in the consensus ADR.
• Capacity planning decision record — the consistency model selection has a direct write throughput implication that must be accounted for in the system's capacity plan: a linearizable system using Raft requires quorum acknowledgment on every write, adding two to three network round trips compared to a single-replica local write; at write volumes above a few thousand writes per second, the quorum acknowledgment overhead is a meaningful fraction of the write latency budget, and the capacity model that ignores it will produce throughput estimates that are consistently optimistic; the consensus ADR and the capacity plan should cross-reference each other, with the capacity plan explicitly modeling the consensus overhead for each consistency tier as a per-write latency cost rather than treating it as an implementation detail of the underlying database.
• API versioning decision record — saga patterns replace distributed locks with event-driven compensation, but the compensating transaction APIs exposed by each saga participant must be versioned and maintained with the same rigor as any external-facing API; a compensating transaction endpoint that is removed or incompatibly changed breaks the saga orchestrator's ability to roll back in-flight transactions, producing the same partial-execution correctness failures the saga pattern was designed to prevent; the API versioning model for each service that participates in a saga must include the compensating transaction endpoints as first-class versioned APIs with explicit deprecation policies, and the saga orchestrator's compatibility matrix should be part of the API versioning ADR's dependency section.
• Open-source extractor — consensus decisions are made in extended AI chat sessions where engineering teams work through CAP theorem analysis for specific data models, evaluate partition tolerance strategies against specific production scenarios, and reason through 2PC versus saga tradeoffs against specific transaction semantics and latency profiles; those sessions contain the constraint reasoning — the specific data type classification that drove the consistency ceiling selection, the partition scenario that drove the election timeout configuration, the throughput estimate that drove the rejection of 2PC — that is rarely captured in the final architectural document; running those sessions through the extractor before a consensus model review or a migration to a new data store produces the decision record the review needs: what was decided, why those choices were made over the alternatives considered, and what production signals were designated as triggers for revisiting the model.