What is replication lag and why does it matter for read replicas?

Replication lag is the delay between when a write is committed to the primary database and when that write is visible on a read replica. In PostgreSQL streaming replication (the default for AWS RDS read replicas and most self-managed PostgreSQL replicas), the primary writes changes to its Write-Ahead Log (WAL) and streams those WAL segments to the replica. The replica's apply worker reads and applies the WAL changes, making them visible to queries. The lag is the gap between the LSN (Log Sequence Number) of the most recent commit on the primary and the LSN of the most recent applied change on the replica. Lag accumulates when the primary's write rate temporarily exceeds the replica's apply rate — during large batch operations, schema migrations that generate significant WAL volume, or traffic spikes. A replica that is normally 1–3 seconds behind can fall 30–120 seconds behind during a large data backfill. Replication lag matters for read replicas because every query served from the replica reflects the database state as of the replica's current applied LSN, not the primary's current LSN. A query that reads account balance from a replica during a 30-second lag window may return the pre-transaction balance — the balance the account had 30 seconds ago — even if a debit completed 2 seconds before the read. The severity depends entirely on which queries are routed to the replica and what consistency requirements those queries have. Browse queries (product listings, search results, content pages) typically tolerate seconds of lag without user impact. Queries that immediately follow writes (reading your own write: confirming an order, checking inventory after a reservation, viewing an account balance after a transaction) have strong consistency requirements that replication lag violates. The routing policy — which queries go to the replica and which go to the primary — determines the actual impact of replication lag on application correctness.

What is the difference between Multi-AZ standby and a read replica in AWS RDS?

AWS RDS Multi-AZ and read replicas both involve a replica database instance, but they use different replication modes and serve different purposes. Multi-AZ standby uses synchronous replication: every write to the primary is simultaneously written to the standby before the primary confirms the write to the client. The standby is always exactly in sync with the primary. When the primary fails, RDS automatically promotes the standby to primary in approximately 60–120 seconds with zero data loss (RPO = 0). The standby cannot serve read traffic during normal operation — it is exclusively a failover target. Multi-AZ is a high-availability feature, not a read-scaling feature. Read replicas use asynchronous replication: the primary confirms the write immediately without waiting for the replica. WAL segments are streamed to the replica, which applies them independently. Read replicas can serve SELECT queries, distributing read load away from the primary. The tradeoff is replication lag: the replica applies changes after the primary has confirmed them, so the replica may be seconds or minutes behind the primary depending on write load. Read replicas can be promoted to primary manually if the primary fails, but unlike Multi-AZ, the replica is not in sync at the moment of failure — there is a data loss window equal to the replication lag at failure time. In AWS RDS PostgreSQL, a Multi-AZ deployment typically has one synchronous standby for HA plus optionally one or more asynchronous read replicas for read scaling. In Amazon Aurora, the architecture is different: Aurora replicas share the same underlying distributed storage as the primary, so Aurora replica lag is typically measured in milliseconds (not seconds) and replica promotion is fast with no data loss — the promoted replica was already reading from the same shared storage volume. Understanding which type of replica you have is essential for understanding the actual RPO and RTO of your failover strategy.

What is read-after-write consistency and how is it implemented with read replicas?

Read-after-write consistency is the guarantee that a client that writes data to the database and then immediately reads it back receives the updated value, not a stale pre-write value. With read replicas, this guarantee is violated when the read is served from a replica that has not yet applied the write. The violation produces a specific failure mode: a user submits a form, the write goes to the primary, the confirmation page loads and reads from the replica, the replica hasn't replicated the write yet, and the confirmation page shows the pre-write state (the item is not in the cart, the account shows the old balance, the comment is not visible). Three implementation patterns exist for maintaining read-after-write consistency with read replicas. Session-level primary routing: after any write operation in a session, route all subsequent reads in that session to the primary for a fixed time window (typically equal to the observed 95th-percentile replication lag, e.g., 5 seconds). Simple to implement; wastes primary capacity on reads that could have been served from the replica. LSN-based routing: the write operation returns the LSN (log sequence number) of the commit on the primary. The client passes this LSN as a consistency token in subsequent read requests. The database proxy or application layer routes the read to the replica only if the replica's applied LSN is greater than or equal to the token LSN; otherwise it routes to the primary. Precise and efficient, but requires the proxy layer to support LSN-aware routing (most off-the-shelf proxies like PgBouncer and RDS Proxy do not support this natively — it requires custom implementation or application-level routing logic). Application-level write routing: the application tracks which operations involve writes that require read-after-write consistency and always routes those operations and their immediate follow-up reads to the primary, routing all other reads to the replica. This requires the application to be explicitly aware of the routing decision at each query call site — it is the most precise approach but requires the most application-level discipline to maintain as the codebase evolves. The choice of read-after-write strategy must be documented in the decision record because the choice affects the proxy technology selection, the application code architecture, and the behavior when the implementation drifts (e.g., a new engineer adds a read after a write without following the routing convention).

What should a database failover procedure include and how is it tested?

A database failover procedure is the documented sequence of steps for recovering from a primary database failure, covering both automated failover (Multi-AZ, Aurora) and manual failover (read replica promotion when no synchronous standby is available). A complete failover procedure must cover: (1) failure detection — how the team learns the primary is unavailable (monitoring alert, AWS health notification, application error spike); (2) RPO assessment — checking the replica's current replication lag to determine the data loss window before deciding to promote; (3) promotion trigger — the person authorized to trigger promotion, the command or console action to execute, and the expected behavior during promotion; (4) connection string update — how the application is redirected to the promoted replica (DNS failover using Route 53 weighted routing, RDS endpoint update, application config change, connection pool restart); (5) post-promotion verification — confirming the promoted instance is accepting writes, checking application error rates, verifying that the replication lag at promotion time matches the data loss observed in the application; (6) downstream notification — notifying users of an outage window, communicating with customer success about affected accounts if data was lost, informing compliance if the data loss affects audit records. A failover procedure that exists only in a document but has never been executed is not a tested procedure. Testing must cover: failover drill (promoting a non-production replica to verify the procedure steps are accurate), connection string update timing (how long does it take for all application instances to reconnect to the promoted replica?), and lag-at-promotion verification (measuring replication lag in the seconds before promotion to validate the RPO assumption). RDS provides a Reboot with Failover option for Multi-AZ instances that triggers the failover mechanism without an actual primary failure — this tests the Multi-AZ failover path. Testing read replica promotion requires actually promoting a replica, which terminates the replication relationship permanently — the test must use a disposable replica, not a production replica that the team plans to continue using after the test.

The database read replica decision record: why the replication lag policy you chose determines your read consistency surface and your failover time ceiling

July 10, 2026 · WhyChose

A read replica is added to a database during a sprint framed as a performance improvement or a cost-reduction initiative. The primary database is under load — read queries are competing with write queries for CPU and I/O, latency is climbing during peak hours, and the on-call runbook has an entry about restarting the primary when query queues back up. The solution is well-known: add a read replica, split the read traffic off the primary, and the primary handles only writes plus a subset of reads. The sprint ships the replica, the ORM gets a read-write splitting configuration, query latency drops, the team closes the ticket. The configuration is documented as an infrastructure change, not as an architectural decision. No one writes down which queries can safely read stale data, what the replication lag ceiling is, or what the failover procedure requires when the primary fails.

The read replica configuration is not an infrastructure change. It is a set of architectural decisions that determine the read consistency surface of the application — which operations return data that is current and which return data that may be seconds or minutes old — the data loss window the team has implicitly accepted for their disaster recovery scenario, and the actual steps required to restore write capability if the primary fails. These decisions are made by default at the moment the replica is provisioned: the replication mode (synchronous or asynchronous), the routing policy (which queries go to the replica), the consistency guarantee (what read-after-write consistency means for the application), and the failover model (whether promotion of the replica is automatic or manual, how long it takes, and how much data may be lost). All four are consequences of the replica configuration, not separately documented decisions. The lag ceiling — the maximum acceptable delay between what the primary has committed and what the replica has applied — is the most consequential: it sets the floor for read staleness on every query routed to the replica and the ceiling for data loss in every disaster recovery scenario where the replica is promoted.

The missing decision record is invisible while the replica works. The read-write splitting middleware routes reads to the replica and reads come back slightly stale but not noticeably so under normal load. The application behaves correctly on the happy path because most reads do not have strict consistency requirements and the replication lag is typically low. The undocumented assumptions surface during peak load events (when the replica falls behind and queries return visibly stale data), during write-heavy operational events like database migrations (when the replica lag spikes while applying the migration's WAL output), and during failure scenarios (when the primary goes down and the team discovers that the replica has a 4-minute lag that the disaster recovery plan said was zero). By then, the decision is two or three years old, the engineer who configured the replica may not be available, and the team is reconstructing the intended behavior under production pressure.

Two things that happen when the decision is not written down

The stale-read incident: overselling 47 products during a Black Friday sale because no routing policy was written down

A 35-person e-commerce company ran PostgreSQL 14 on AWS RDS. They had been operating for three years with a single RDS instance — reads and writes on the same primary. In the month before their second Black Friday, a load test revealed that query queue depth would exceed the primary's capacity at projected peak traffic: the test simulated 8,000 concurrent users and produced a peak of 14,000 read queries per second against an instance that was provisioned for approximately 9,000. The solution was a read replica. An engineer added one RDS read replica (same instance class as the primary) and configured SQLAlchemy's read-write splitting via a custom middleware that overrode the session's bind based on the statement type: all SELECT statements were routed to the replica connection, all INSERT, UPDATE, DELETE, and BEGIN statements were routed to the primary. The middleware was implemented correctly — it worked exactly as specified. The specification was wrong.

The e-commerce application had three categories of reads. The first category was browse reads: product listings, search results, category pages, product detail pages (excluding the inventory count badge), and editorial content. These were served from the replica at all times. Lag of 10–30 seconds was acceptable — a product description that is 30 seconds stale is not incorrect. The second category was session-state reads: cart contents, user account preferences, order history. These were also routed to the replica. Lag on cart reads was problematic but not immediately obvious because most cart reads happened more than 30 seconds after the most recent cart write — a user who adds an item to their cart and then browses for 2 minutes before refreshing the cart page would not notice 5 seconds of replication lag. The third category was inventory reads: the quantity available for each SKU, checked during the "Add to Cart" action to determine whether the item could be added, and checked again during checkout to confirm the item was still available before processing payment. These were also routed to the replica. The routing policy was "all SELECTs go to replica" — there was no distinction between browse reads (acceptable for eventual consistency) and inventory reads (requiring current primary state).

During Black Friday peak traffic, the replica fell behind. The write rate on the primary spiked as orders were placed, inventory counts were decremented, and cart sessions were created — all generating WAL at a rate higher than the replica's normal apply throughput. At 11:47 AM, the replica was 8 seconds behind the primary. By 12:15 PM, the lag had grown to 34 seconds as the write volume sustained and the replica's apply worker could not keep pace. At 12:31 PM, lag peaked at 42 seconds before the write rate moderated and the replica began catching up.

During the 34–42 second lag window, the inventory system was returning stale data. The inventory check in the "Add to Cart" flow queried the replica for SELECT quantity_available FROM inventory WHERE sku_id = $1. If the primary had processed 15 orders for a SKU in the last 40 seconds, reducing quantity_available from 22 to 7, the replica was still returning 22. Users who added the SKU to their cart and received a confirmation ("Item added — 22 available") were doing so against a phantom inventory count. The checkout flow also read from the replica for the inventory confirmation step — the final payment processing step read from the primary, but the checkout flow's pre-payment confirmation page (which displayed "Your order is confirmed, we're processing your payment") read inventory from the replica and showed the stale available count without detecting the oversell condition. The payment processor step, which did read from the primary, did not re-check inventory against the primary before processing payment — it relied on the checkout flow's confirmation result. The actual oversell detection happened downstream, in the warehouse fulfillment system, which had its own inventory database that was synced from the primary via a separate ETL job. Warehouse fulfillment marked 47 orders as unfulfillable approximately 90 minutes after the peak. Customer service handled 47 refund requests, 31 complaint calls, and sent 47 apology emails with a 15% discount code. Direct cost: $8,400 in refunds and customer service time. The underlying cause was a routing policy that had never been specified: the middleware's "all SELECTs to replica" rule was the only documented behavior, and no one had written down which reads required current primary data.

The fix — routing inventory reads to the primary — required modifying the SQLAlchemy session middleware to check the query's table targets and redirect specific tables to the primary connection regardless of query type. This took three days to implement correctly because the middleware's routing logic had to be updated in 47 call sites spread across four services. The fix would have taken 90 minutes to implement at replica configuration time if the routing policy had been written down as part of the initial decision: "Reads from inventory, cart_items, and account_balance tables route to primary; all other reads route to replica." Three words and a table list would have prevented $8,400 in losses.

The failover incident: a 247-second data loss window in a disaster recovery plan that documented RPO as zero

A 55-person B2B SaaS company operated PostgreSQL 15 on AWS RDS with a Multi-AZ deployment in us-east-1 (primary in us-east-1a, synchronous standby in us-east-1b) and a single read replica in eu-west-1 for reporting queries from their European customer base. The Multi-AZ configuration provided synchronous replication to the standby, ensuring zero data loss and automatic failover within 60–120 seconds in the event of a primary or AZ failure. The eu-west-1 read replica used asynchronous streaming replication to serve EU-timezone reports without sending cross-Atlantic read traffic to the primary. Both the Multi-AZ standby and the eu-west-1 replica were described as "database replicas" in the company's infrastructure documentation. The disaster recovery plan document, written during a SOC 2 audit preparation process, stated: "In the event of a us-east-1 complete outage (both AZs affected), the eu-west-1 read replica can be manually promoted to serve as the primary database. RTO: 30 minutes. RPO: zero." The "RPO: zero" statement referred to the Multi-AZ synchronous replication property — which was accurate for the Multi-AZ scenario. The disaster recovery plan applied it to the eu-west-1 replica scenario, where it was not accurate.

The replication lag on the eu-west-1 replica was not a monitored metric. The engineering team's CloudWatch dashboard for the database included: CPU utilization, free storage space, database connections, read latency, write latency, and replication slot lag (in bytes — for the Multi-AZ standby's logical replication slot). The eu-west-1 replica's replication lag in seconds was not on the dashboard. RDS published it as the ReplicaLag metric in CloudWatch, but no alarm had been configured. During a routine audit of the infrastructure, an engineer had noted that the metric existed and that it typically showed values of 15–45 seconds during the European business day (when reporting queries were running and the cross-Atlantic streaming replication was under load) and dropped to 3–8 seconds overnight. This observation was mentioned in a Slack thread and never recorded in any document.

Eight months after the DR plan was written, a misconfigured database migration caused an unplanned load event. A senior engineer ran a migration that added a composite index on a 890-million-row table in the analytics schema — a table that had grown far larger than its original capacity estimate, but the capacity planning review had been deferred. The index creation used PostgreSQL's CREATE INDEX CONCURRENTLY to avoid locking reads, but CONCURRENTLY index creation generates substantial WAL volume over its entire duration because every page read during the scan is written to the WAL for consistency. The migration ran for 6 hours and 40 minutes on the primary, generating a sustained WAL output of approximately 180 MB per minute during the scan phase. The eu-west-1 replica's apply worker received the WAL stream but could not apply it as fast as it was generated — the cross-Atlantic transfer latency added 85ms to each WAL segment's delivery time, and the replica's apply worker was single-threaded (the default in PostgreSQL streaming replication). The replica fell progressively behind during the migration window. At the migration's completion, the replica had accumulated a lag of 4 minutes and 23 seconds. It continued to apply the backlogged WAL segments over the next 3 hours until it caught up with the primary.

Eighteen hours after the migration completed, at 03:47 UTC, an AWS maintenance event affected both us-east-1a and us-east-1b simultaneously — both the primary RDS instance and the Multi-AZ standby were impacted by a hypervisor-level maintenance procedure that AWS had not announced in advance. Both instances became temporarily unavailable for approximately 8 minutes while AWS performed the maintenance. The Multi-AZ failover, which should have automatically promoted the us-east-1b standby, did not complete because both instances were affected simultaneously — the automatic failover mechanism detected the primary failure and attempted to promote the standby, but the standby was also unreachable during the maintenance window. The RDS API returned the us-east-1 endpoint as "in failover" state for 11 minutes before the maintenance completed and both instances became available again. During those 11 minutes, the application's database connections timed out and the application's health checks began failing. The on-call engineer, following the incident response runbook, reached the step "If us-east-1 primary is unavailable and failover is not progressing: initiate DR failover to eu-west-1 read replica per DR-003 procedure." The engineer checked the eu-west-1 replica status in the RDS console: Connected, Replication status: Replicating. No lag figure was visible on the replica's summary page — the lag metric required navigating to the CloudWatch metrics for the replica, which the engineer had not done before under time pressure.

The engineer initiated the replica promotion by clicking "Promote" in the RDS console at 03:58 UTC, 11 minutes after the incident began. The promotion completed at 04:01 UTC, making the eu-west-1 instance writable. At the moment of promotion, the replica had a replication lag of 247 seconds relative to the last WAL LSN applied from the primary before the promotion disconnected the replication stream. The 247-second lag was a residual from the previous night's migration: the replica had caught up from its 4-minute-23-second post-migration lag to approximately 3 minutes 15 seconds of residual lag by the time the incident occurred (it was still catching up slowly). The promotion absorbed that lag as data loss: 247 seconds of committed transactions on the primary that the replica had received in the WAL stream but had not yet applied before the promotion command disconnected the replication.

The data loss surfaced progressively over 72 hours. 23 user account registrations that had completed between 03:43 UTC and 03:47 UTC (when the primary went offline) were absent from the promoted replica. 156 API keys generated for those accounts were also absent. 341 subscription event records (billing cycles, plan changes, and feature usage events) from the same window were missing. The customer success team began receiving reports of users who "couldn't log in" because their accounts didn't exist in the promoted primary. Because the original primary had been unreachable for 11 minutes and then came back online as a secondary (not the current primary), the team had no mechanism to query its transaction log for the specific commits that were lost — it was demoted to a standby, and its WAL history was inaccessible via the RDS console. The 247 seconds of lost data had to be reconstructed by cross-referencing email confirmation logs (which showed which emails were sent to registered users), payment processor webhook logs (which showed which subscription events had been processed), and customer support tickets. The reconstruction took 3 engineers 2 days and was not complete — 4 accounts could not be reconstructed because the only trace of their existence was in the lost database records. The company's SOC 2 auditor, informed of the incident, required a formal RCA and an updated DR plan. The updated DR plan included: eu-west-1 replica lag monitoring with a configured CloudWatch alarm, a documented RPO of "up to 10 minutes for the eu-west-1 failover scenario," and a prohibition on considering the eu-west-1 failover an equivalent substitution for Multi-AZ failover in any compliance document. The "RPO: zero" statement in the original DR plan was replaced with "RPO: 0 seconds for Multi-AZ us-east-1 failover; RPO: variable (target < 10 minutes) for eu-west-1 disaster recovery failover."

Three structural properties that are set at read replica configuration time

The replication model and the lag ceiling

The replication lag ceiling — the maximum lag the replica can accumulate relative to the primary under the expected write workload — is the most consequential property set at replica configuration time, because it determines both the read staleness floor for every query routed to the replica and the data loss window for every disaster recovery scenario that involves replica promotion. The ceiling is not a configuration parameter that can be set directly; it is an emergent property of the replication mode, the network path between the primary and replica, the replica's apply throughput capacity, and the primary's write workload.

Asynchronous streaming replication — the default mode for PostgreSQL read replicas, MySQL replicas, and AWS RDS read replicas — means the primary confirms a write to the client immediately after writing to its own WAL, without waiting for the replica to receive or apply the change. The replica receives WAL segments from the primary continuously; the apply worker on the replica reads those segments and applies each WAL record to the replica's data files. The lag accumulates when the primary's WAL generation rate exceeds the replica's effective apply rate. The apply rate is bounded by three independent limits: network throughput (how fast WAL bytes can transit from the primary to the replica — typically 125–250 MB/s for intra-region transfer, 15–40 MB/s for cross-region transfer over public or AWS backbone peering), replica CPU and I/O throughput (the apply worker is single-threaded in physical replication by default; a busy replica serving many concurrent read queries competes with the apply worker for I/O), and WAL record complexity (a bulk INSERT generates simple WAL records that apply quickly; a large UPDATE that rewrites many index pages generates complex WAL records that take longer per byte to apply). The lag ceiling is the point at which the primary's write rate consistently exceeds the replica's effective apply rate — at that point, lag grows unboundedly until the write rate drops. A replica that reaches the lag ceiling during a write spike does not immediately return to low lag when the write rate drops; it must first apply all the accumulated backlog, which can take minutes to hours depending on the size of the backlog.

Synchronous replication changes the semantics fundamentally. PostgreSQL's synchronous_commit = remote_write setting requires the primary to wait for the replica to acknowledge receipt of the WAL record (not necessarily its application) before confirming the write to the client. synchronous_commit = remote_apply requires waiting for the replica to both receive and apply the WAL record. Either setting eliminates replication lag for the confirmed writes — the replica is, by definition, up-to-date with every write that the primary has confirmed — at the cost of adding the replica's acknowledgment latency to every write's response time. For a replica in the same AZ, the round-trip latency overhead is typically 1–3ms per write. For a cross-region replica, the round-trip latency overhead is 30–120ms per write, depending on the network path — this is why synchronous replication is only used for intra-region standby replicas in HA configurations, not for cross-region read scaling replicas. Amazon Aurora provides a middle path: Aurora replicas share the same underlying distributed storage volume as the primary. When the primary commits a write, the storage layer applies it to the shared volume and notifies both the primary and any Aurora replicas; the replicas do not replay WAL but instead refresh their buffer pool from the shared volume. Aurora replica lag is typically 10–100 milliseconds for data that has been committed to the storage layer — effectively eliminating the lag ceiling that async streaming replication creates, because the replica does not need to receive, buffer, and apply a WAL stream. The Aurora model connects to the database vendor decision record: the decision between PostgreSQL-compatible Aurora and standard RDS PostgreSQL affects the replica lag model, the read consistency surface, and the failover behavior — the two products have the same SQL interface but fundamentally different replication architectures with materially different consistency properties.

The decision record must specify the replication mode in use, the expected lag under normal write load (as a 95th-percentile measurement, not an average), the expected lag during known write-heavy events (large migrations, batch data loads, peak transactional periods), and the monitoring and alerting configuration for lag. A lag target of "less than 5 seconds at the 95th percentile under normal load" is a specifiable, measurable, and alertable constraint. "Low lag" is not. The difference is whether the next engineer who operates the replica knows whether a 30-second lag reading is normal, concerning, or critical — without a documented target, that judgment requires context that may not be available. This connects to the observability platform decision record: replication lag, measured in seconds and in bytes of WAL not yet applied, is the primary health signal for a read replica — it must be instrumented as a time-series metric, displayed on the primary database dashboard, and alarmed at a threshold that gives the operations team time to respond before the lag exceeds the application's acceptable staleness window.

The read routing policy and the consistency requirements per query

The routing policy is the set of rules that determines which database queries are sent to the read replica and which are sent to the primary. The routing policy is either explicitly documented and consistently enforced, or it is implicitly defined by the ORM or database proxy's default behavior — which is typically "all SELECTs to the replica, all writes to the primary." The default is wrong for most applications because it treats all read queries as having equivalent consistency requirements, which they do not.

Read queries have a spectrum of consistency requirements. At one end are eventually consistent reads — queries where receiving data that is 5–60 seconds stale does not affect application correctness or user experience. Product catalog pages (descriptions, images, pricing that is updated infrequently), search results (where index freshness is already a design consideration), content pages, and analytics dashboards (where the user understands data is not real-time) are examples. These queries can safely route to the replica at any lag level the application tolerates. At the other end are strong consistency reads — queries where receiving stale data produces incorrect application behavior. Inventory availability checks before purchase, account balance reads before a financial transaction, permission checks that must reflect the most recent role assignment, and any read that is the basis for a decision that will be written back to the database require current primary state. These queries must route to the primary regardless of the replica's current lag. In the middle are read-after-write consistency reads — queries that must reflect the result of a write that the same session or user just performed, but that can use replica state once the replica has applied that specific write. Cart contents after an add-to-cart, order details after order placement, profile after a settings update are examples. These queries require either routing to the primary for a bounded window after the write, or an LSN-based routing mechanism that confirms the replica has applied the specific write before serving the read.

Implementing the routing policy requires a mechanism at the database connection layer. Three mechanisms are common. The first is an ORM-level read-write splitting configuration with custom routing hooks. SQLAlchemy with a get_bind override, Django's multiple database routing via a custom Router class, and Sequelize with a read-write pool configuration all support custom logic for routing individual queries. The custom logic receives information about the query (table names, query type, current session state) and returns the connection to use. This approach keeps routing logic in the application layer, where it can access session state (which tables the current request has written, what LSN the last write returned), but it requires each ORM call site to be aware of the routing intent or requires the router to infer it from query characteristics. The second mechanism is a database proxy with routing rules. ProxySQL (for MySQL), pgpool-II (for PostgreSQL), and HaProxy with custom configuration can route queries to primary or replica based on query text patterns, connection attributes, or custom extensions. RDS Proxy with reader and writer endpoints provides a managed version. Proxy-based routing is transparent to the application — the application connects to a single proxy endpoint, and the proxy routes each query. The limitation is that proxy-based routing cannot access application-level session state (which operations the current user has performed), making read-after-write consistency harder to implement than with application-level routing. The third mechanism is explicit dual-connection patterns: the application maintains a separate primary connection and replica connection, and each query explicitly chooses which connection to use. This is the most explicit approach — there is no ambiguity about which database a query reads from — but it requires every query call site to make an explicit routing choice, and it makes it easy for a new engineer to default to the primary connection for everything, defeating the read-scaling benefit.

The routing policy decision record must enumerate the query categories that require strong consistency, the categories that can tolerate eventual consistency, and the categories that require read-after-write consistency. It must specify the routing mechanism (ORM hooks, proxy, explicit connections), the read-after-write consistency strategy (primary routing for N seconds after write, LSN-based routing, or primary routing for identified read-after-write query pairs), and the process for reviewing new queries to determine which routing category they belong to. Without the enumeration, each new feature adds queries whose routing behavior is determined by the default — which is eventually "all reads go to replica" because that requires no explicit decision. This connects to the database migration strategy decision record: a schema migration that adds a new table typically also adds application code that reads and writes that table; the read routing policy for the new table must be determined as part of the migration planning, not after the code ships. If the migration adds a table that holds financial data, and the routing policy default routes SELECTs to the replica, the new financial reads will route to the replica by default without anyone having decided that this is acceptable.

The failover model and the RPO commitment

The Recovery Point Objective (RPO) is the maximum acceptable data loss measured in time — how many seconds or minutes of committed transactions the team is willing to lose if the primary database fails and cannot be recovered. The RPO is not a policy that is set in a configuration file; it is the consequence of the replication mode and the replication lag at the moment of failure. The team may document an RPO of "zero" in their disaster recovery plan. The actual RPO is the replication lag at the moment the primary becomes unavailable — a number that changes continuously as the write workload varies and as write-heavy events cause lag to spike.

Three failover scenarios have different RPO properties. The first is Multi-AZ automatic failover (AWS RDS, Azure SQL with geo-redundancy in synchronous mode, GCP Cloud SQL with high-availability instances). The standby uses synchronous replication — every committed write on the primary has been acknowledged by the standby before the primary confirmed it to the client. When the primary fails, the standby has received every committed transaction. RPO = 0. The failover is automatic, typically completing in 60–120 seconds for RDS Multi-AZ. The standby's endpoint becomes the primary endpoint; the application reconnects (after the connection pool clears timed-out connections and establishes new connections to the now-primary endpoint) within 1–3 minutes depending on connection pool configuration. The key limitation: the Multi-AZ standby is in the same region as the primary. A region-level failure (affecting both primary and standby AZs) removes the Multi-AZ HA guarantee entirely and makes the cross-region replica the only available database. This connects to the disaster recovery decision record: the Multi-AZ HA configuration and the disaster recovery cross-region failover configuration are different layers of availability that must be documented separately — their RPO properties, RTO properties, and failure scenarios are distinct, and conflating them in a single "RPO: zero" statement (as the company in the second story did) produces a compliance document that accurately describes one scenario while misrepresenting another.

The second scenario is manual replica promotion after primary failure with no available synchronous standby (the common cross-region disaster recovery case). The cross-region read replica uses asynchronous replication; at the moment of primary failure, the replica has received some but not all WAL segments that the primary had committed. The RPO equals the replication lag at failure time — which is the number of seconds' worth of committed transactions that the replica had received in the WAL stream but had not yet applied (or had not yet received at all, if the network buffer had not flushed the most recent WAL segments before the primary went down). The promotion procedure involves: checking the replica's current lag before proceeding, deciding whether the lag is acceptable given the business impact of waiting for the replica to catch up (it cannot catch up once the primary is unreachable — the choice is between promoting now with the current lag, or waiting if the primary might recover), executing the promotion command, updating the application's database endpoint to point to the promoted instance, restarting connection pools, and verifying application function after the connection updates. Each step takes time (the RTO) and has failure modes that must be documented. If the team has not practiced the procedure, the RTO and the promotion outcome under pressure are unknown.

The third scenario is Aurora's read replica promotion. Because Aurora replicas read from the same shared storage volume as the primary, Aurora replica lag is typically less than 100 milliseconds — effectively zero for practical purposes, because the committed data is on the shared storage volume before the primary confirms the write. Promoting an Aurora replica to primary involves making the replica's endpoint the write endpoint and stopping the replication of LSN metadata from the primary. The promotion typically completes in under 30 seconds, and because the replica was reading from the same storage volume, no committed data is lost — the promoted replica has all committed transactions. This property — Aurora's shared-storage model producing a cross-AZ failover with near-zero data loss — is architecturally different from asynchronous streaming replication's variable-lag data loss window, but it is described using the same "read replica" terminology. Teams that migrate from RDS PostgreSQL with cross-region read replicas to Aurora PostgreSQL may assume that cross-region Aurora replicas have the same near-zero RPO as same-region Aurora replicas — they do not. Cross-region Aurora Global Database uses asynchronous replication between regions with a typical lag of 1–2 seconds, not the same shared-storage mechanism used for same-region Aurora replicas. The RPO for a cross-region Aurora Global Database failover is the replication lag at failure time, which is bounded differently from streaming replication lag but is not zero. This connects to the database sharding decision record: the sharding architecture determines which database instance is authoritative for which data; a read replica failover strategy must account for which shard's primary has failed and which replica covers that shard's data — a cross-shard failover where the shards use different replica configurations produces different RPO properties per shard, which complicates both the DR procedure and the RPO documentation.

The failover model decision record must specify: the replication topology and the role of each replica in the failover hierarchy (which replica is the first promotion target, which is the DR fallback); the expected RPO range for each failover scenario (not a point estimate — a range that reflects observed lag variation); the promotion procedure as a tested, step-by-step runbook; and the connection string update mechanism (how the application is redirected to the promoted instance, and how long that redirection takes). A failover procedure that has never been tested under realistic conditions is a hypothesis, not a plan. This connects to the database connection pooling decision record: the connection pool's behavior during a primary failure and subsequent promotion — whether it reconnects automatically, how long it holds failed connections before attempting reconnection, and whether it requires an explicit pool drain and restart — determines a significant portion of the RTO. A connection pool that retries with exponential backoff for up to 10 minutes before failing a connection will delay the application's recovery from a failover by up to 10 minutes after the promoted instance is writable, regardless of how fast the promotion itself completes.

Five sections the database read replica decision record should address

1. Replication topology selection and lag target

Document the full replication topology: the primary instance (instance class, storage type, AZ, region), each replica (type, instance class, AZ or region, replication mode — synchronous or asynchronous), and the role of each replica (read scaling, DR failover tier-1, DR failover tier-2). For AWS RDS, distinguish clearly between Multi-AZ standby instances (not read replicas; synchronous replication; cannot serve reads; automatic failover) and read replicas (asynchronous replication; serve reads; manual promotion). For Amazon Aurora, distinguish between same-region Aurora replicas (shared-storage model, < 100ms lag, automatic failover support) and cross-region Aurora Global Database replicas (asynchronous, 1–2 second lag, requires manual promotion). For self-managed PostgreSQL, document the replication configuration: whether synchronous_standby_names is set (and for which replicas), whether synchronous_commit is on, remote_write, remote_apply, or local, and whether cascading replication is in use (a replica of a replica, which compounds lag and adds additional failure modes).

Document the replication lag target as a measurable constraint, not a general objective. A useful lag target specification reads: "eu-west-1 read replica lag target: < 8 seconds at P95 during normal business hours write load; < 60 seconds during known high-write events (database migrations, monthly data backfills). Alert threshold: 30 seconds sustained for > 5 minutes. Escalation threshold: 120 seconds sustained for > 2 minutes (page on-call, consider routing EU reporting queries to primary)." This specification is specific enough to configure a CloudWatch alarm, identify when the target is being violated, and determine when an operational response is required. It also documents the expected lag range during high-write events, which prevents false alarms during known conditions and ensures that genuinely anomalous lag is distinguishable from expected lag. The lag target must be calibrated against observed behavior — the specification above is a template; the actual numbers must be measured from the replica's observed behavior under the production write workload, not estimated from benchmarks or vendor documentation. This connects to the capacity planning decision record: the replica's instance class must be sized to maintain the apply throughput required to stay within the lag target under the expected peak write workload, plus headroom for write spikes. A replica that is undersized relative to the primary's peak write throughput will accumulate lag during every write spike and never recover during high-load periods.

2. Read routing policy and consistency requirements

Document the routing policy as an explicit classification of query types, not as a description of the routing mechanism. The routing mechanism (ORM hooks, proxy configuration, explicit connection selection) is secondary to the policy it enforces. A useful routing policy specification enumerates: (a) queries that always route to the primary — list them by table, operation type, and the consistency requirement that mandates primary routing; (b) queries that always route to the replica — list the characteristics that make them safe for eventual consistency; (c) queries that require read-after-write consistency — specify the consistency strategy (primary routing for N seconds post-write, LSN token-based routing, or explicit connection selection at the call site) and the write operations that trigger the requirement.

The routing policy must be enforceable by a future engineer without requiring them to understand the full history of why each routing decision was made. A specification that says "reads from the inventory table route to primary because inventory is checked before purchase and must reflect the most recent deductions" is enforceable — a new engineer adding an inventory read can check the policy, see that the table requires primary routing, and implement accordingly. A specification that says "sensitive reads go to primary" is not enforceable — the definition of "sensitive" is in the original author's head. Document the routing implementation: which ORM methods or configuration settings enforce the policy, where the routing logic lives in the codebase (a specific middleware file, a database router class, a proxy configuration file), and what the review process is for new queries that don't obviously fit an existing classification. This connects to the data pipeline decision record: batch data pipelines that read from the database to process or transform records frequently route all reads to the replica to reduce primary load — a reasonable default for analytics pipelines, but a problematic default for operational pipelines that read records immediately after writing them (e.g., a pipeline that writes processed event records and immediately reads them back to trigger downstream actions). The routing policy for data pipelines is a separate classification from the routing policy for application services and must be explicitly addressed.

3. Read-after-write consistency strategy

Document the read-after-write consistency strategy explicitly, covering the implementation mechanism, the query pairs it applies to (which write operations trigger a consistency requirement, which reads must observe those writes), and the fallback behavior when the mechanism fails or is unavailable. The three implementation mechanisms (session-level primary routing window, LSN token-based routing, explicit dual-connection) have different implementation complexity, different accuracy, and different performance implications.

Session-level primary routing window routes all reads to the primary for a fixed duration (typically 2–10× the P95 replication lag) after any write in the session. This is the simplest mechanism to implement and the most conservative — it errs toward primary reads when in doubt. The cost is wasted read capacity: during the routing window, reads that could safely go to the replica are instead going to the primary. In applications where writes are frequent (active sessions that write on every page interaction), the routing window may cover most or all reads in the session, eliminating the read-scaling benefit of the replica for active users. The window duration must be calibrated against the observed replication lag: if the window is set to 5 seconds and the replica falls behind by 30 seconds during a write spike, reads in the 5-second window after the write may still be routed to the replica before the replica has applied the write, and the consistency guarantee fails. The window should be set against the P99 lag, not the P50 or P95, to provide a robustness margin. LSN token-based routing is more precise — it routes to the primary only if the replica hasn't caught up to the specific write, and routes to the replica as soon as it has. This eliminates unnecessary primary routing for writes where the replica has already applied the change, but requires the application to track LSN tokens per write and the routing layer to compare them against the replica's current applied LSN. This mechanism is architecturally cleaner but requires custom implementation because no standard ORM or proxy supports it natively. If the replica's current LSN is not queryable via the connection pool (some poolers do not expose it), the mechanism requires a separate LSN-check query that adds latency to every routed request. This connects to the CI/CD pipeline decision record: the read-after-write consistency mechanism must be tested in CI with a test that deliberately creates a replication lag condition and verifies that the application correctly serves recent writes rather than stale replica data — an integration test that only runs against a replica without simulated lag will not detect consistency failures that only manifest under lag conditions.

4. Failover procedure and RPO/RTO documentation

Document the failover procedure as an operational runbook, not as a high-level description. The runbook must include: the monitoring signals that indicate the primary is unavailable (specific CloudWatch alarm names or monitoring tool alert names), the decision tree for which failover path to execute (Multi-AZ automatic failover, read replica manual promotion, cross-region DR), the command or console sequence for each path, the expected duration for each step, the verification steps after each action (confirming the promoted instance is accepting writes, confirming the application's connection pool has reconnected, confirming the application is serving reads and writes correctly), and the communication protocol (who to notify, what to communicate, when to declare the incident resolved).

Document the RPO range for each failover scenario separately. The Multi-AZ failover RPO is 0 — no data loss, synchronous replication. The cross-region read replica failover RPO is a range bounded by the observed replication lag distribution: "eu-west-1 replica failover RPO: typically 8–45 seconds under normal write load; up to 300 seconds following a high-write event (migration, monthly backfill). Actual RPO at failure time equals the replica's replication lag at the moment of promotion, which varies and cannot be determined precisely until promotion is initiated." Document the procedure for assessing the RPO before executing promotion: "Before promoting the eu-west-1 replica, retrieve the ReplicaLag CloudWatch metric for the past 30 minutes. If lag is < 30 seconds, proceed with promotion and note the lag value as the expected data loss. If lag is > 30 seconds, evaluate whether to wait for the primary to recover (if outage is expected to be brief) or accept the higher data loss and promote immediately. Document the decision and the lag value at the time of promotion in the incident record." Documenting the RPO range and the pre-promotion assessment step addresses the most common RPO misunderstanding: teams that have never measured their actual lag during a high-write event assume their RPO is approximately the average lag they see on the monitoring dashboard, not the peak lag that occurs when write throughput is elevated.

Document the connection string update mechanism and its expected duration. If the application uses a hardcoded primary endpoint hostname, updating to the promoted replica's endpoint requires redeploying the application configuration — which may take 5–15 minutes if a CI/CD pipeline is involved. If the application uses a Route 53 CNAME that points to the primary's endpoint, updating the CNAME takes effect after the TTL expires (typically 60 seconds for database DNS records) and requires an additional reconnection step when existing connections detect the DNS change. If the application uses RDS Proxy or a similarly abstracted endpoint, the proxy's endpoint may remain constant and the routing update happens at the proxy layer — but the proxy must be reconfigured to point to the promoted replica. Each mechanism has a different update latency, and the RTO is dominated by the slowest step among promotion, connection string update, and connection pool reconnection. All three must be measured in a drill before they determine the RTO in production.

5. Observability instrumentation for replication health

Document the required instrumentation for each read replica in the topology. The minimum instrumentation set includes: replication lag in seconds (the primary operational metric — how far the replica lags the primary in time), replication lag in bytes (the WAL volume not yet applied — a leading indicator of lag growth before it appears in the seconds metric), replica connection status (connected or disconnected — binary but essential; a disconnected replica is not replicating at all), apply rate in bytes per second (the replica's WAL apply throughput — a drop in apply rate under constant primary write load indicates a replica-side bottleneck), and LSN delta between primary and replica (the specific offset in the WAL that quantifies how much data the replica has not yet applied). These metrics must be measured and stored in the observability platform as time-series data with at least 14 days of retention — the incident investigation in the second story was unable to reconstruct the replication lag at the moment of promotion because the CloudWatch metric for ReplicaLag was not stored with sufficient retention and the team had no historical visibility into the lag spike that occurred during the migration 18 hours before the incident.

Document the alerting thresholds and the escalation path. The alert threshold must be set relative to the lag target documented in section 1 — if the lag target is < 8 seconds at P95, the alert threshold should be approximately 25–30 seconds (a lag that has been exceeded long enough to be anomalous, with enough headroom to investigate before reaching a level that causes visible consistency failures). The escalation threshold — the lag level at which the on-call engineer must take action, such as redirecting traffic from the replica to the primary to prevent consistency failures — should be based on the application's acceptable staleness window for its most time-sensitive routed queries. If the inventory check uses a 10-second acceptable staleness threshold and the replica is currently at 45 seconds of lag, the inventory reads must be temporarily rerouted to the primary. Documenting this as an explicit operational procedure ("if eu-west-1 replica lag exceeds 40 seconds, redirect all reads to the primary via REPLICA_ROUTING_ENABLED=false in the feature flag system and alert the data team") prevents the situation where the on-call engineer can see that the lag is high but does not know whether to act, what action to take, or what the impact of waiting is.

Document the replication health checks that should be run after any database maintenance event — a primary restart, a schema migration, a version upgrade, a compute resize — to verify that replication resumed correctly after the event. A schema migration that runs on the primary generates a large WAL burst and may cause the replica to fall significantly behind; the post-migration health check should include verifying that the replica is connected, the lag is decreasing (the replica is catching up), and the apply rate has returned to the pre-migration baseline. A primary restart that completes a failover should be followed by a verification that the new primary's replication stream is being received by all replicas and that each replica's lag has not grown during the restart. These health checks are the operational equivalent of the integration tests that verify code correctness — they catch replication-layer failures early, before they surface as read consistency failures or a disaster recovery scenario where the replica is unexpectedly far behind the primary. This connects to the database connection pooling decision record: the connection pool's health check configuration must account for the time it takes for a replica to reconnect to the primary after a primary restart — a pool that health-checks replica connections too aggressively during a failover event will generate a spike of connection errors that may be mistaken for application errors rather than replication reconnection transients.

What to do with the database read replica decision record

The read replica decision record's primary audience is the database administrator or infrastructure engineer who configured the replica, the application engineer who implements read-write splitting in the ORM or database layer, and the on-call engineer who will respond to a primary database failure. The infrastructure engineer needs the replication topology and lag target to size the replica correctly, configure the monitoring, and know which metrics indicate a problem requiring escalation versus a transient condition expected to resolve. The application engineer needs the routing policy and read-after-write consistency strategy to implement query routing correctly and to know the classification of any new query they add — without the classification, every new engineer defaults to whatever the ORM default is, which is typically "all reads to replica" regardless of consistency requirements. The on-call engineer needs the failover procedure and the RPO documentation to respond correctly under time pressure without having to reconstruct what the replica's expected lag is, what the promotion procedure involves, or whether the data loss from promoting now is acceptable given the current lag reading.

If the read replica decision record was not written when the replica was added, the most useful reconstruction starts with a lag audit. Retrieve the replication lag metric time series for the past 30–90 days from the monitoring system. Plot it against the primary's write throughput during the same period. The correlation will show whether the lag is driven by write spikes (high-write events cause temporary lag accumulation that recovers after the event) or by sustained write throughput that consistently exceeds the replica's apply capacity (indicating the replica is undersized for the current write workload). From the lag time series, calculate the P95 lag under normal load and the maximum observed lag over the audit period — these are the actual performance bounds that should be documented as the lag target, not an aspirational number. The routing policy reconstruction requires auditing the ORM or proxy configuration to enumerate which queries are currently routed to the replica and checking each against the consistency requirement it actually has. Inventory reads that route to the replica and are used for pre-purchase availability checks are a specific pattern to look for — they are a common source of oversell incidents in e-commerce systems with async replication. The ADR template provides the structure for documenting the current state as a decision — the topology as it exists today, the lag target derived from the audit, the routing policy as implemented, the consistency strategy (or the absence of one), and the consequences of the current configuration that have been observed (including any incidents where stale reads or unexpected RPO at failover time surfaced as production issues).

The WhyChose extractor can recover database read replica decisions from ChatGPT or Claude conversations during the scaling sprint that added the replica — the conversation where the replica type was selected, the thread where the routing strategy was first discussed, or the message where the DR plan was drafted and someone asked "what's our RPO if the primary goes down?" These conversations contain the reasoning that was never formalized into documentation. The decisions never written down essay covers why database configuration decisions are particularly likely to be missing from decision records: they are made during operational sprints framed as infrastructure improvements rather than architectural decisions, by engineers whose primary goal is to resolve the immediate performance problem, not to document the consistency implications. The result is that the properties that govern the application's data consistency surface — which reads return current data, which return stale data, and how stale they can get — exist only in the default behavior of the ORM configuration and surface as constraints when a write spike or a primary failure reveals the undocumented assumptions.