The change data capture decision record: why the CDC mechanism you chose determines your schema migration coupling surface and your downstream consumer delivery guarantee failure mode

A 36-person developer tooling company built a SaaS platform for infrastructure teams managing configuration deployments across multi-cloud environments. The platform's primary data flow was event-driven: every configuration change, deployment execution, audit event, and approval decision written to the primary Postgres database needed to appear in a downstream event stream consumed by five services — a real-time dashboard, an audit trail indexer, a notification fanout service, a compliance reporting service, and a data warehouse export pipeline. The engineering team evaluated three approaches and chose Debezium running as a Kafka Connect connector against the Postgres WAL, with a Confluent Schema Registry for Avro-encoded event schemas.

The choice was made with clear reasoning: log-based CDC produced zero write latency overhead on the source database, Debezium's Postgres connector was well-documented and production-proven, and Kafka Connect's distributed mode provided connector failure isolation. The five downstream consumers were built and tested over three months. The system worked reliably in production for eight months. The Debezium connector ran on a dedicated Kafka Connect worker with a single-node configuration and a connector schema registry that tracked the current schema version of each monitored Postgres table.

In month nine, the database team ran a schema migration on the deployments table — the highest-volume table in the schema, receiving approximately 1,400 writes per minute during business hours. The migration added a config_hash column (a SHA-256 digest of the deployment configuration payload) to support a deduplication feature that the product team had requested. The migration followed the standard runbook: a CREATE INDEX CONCURRENTLY to pre-build the index before the column was added, then an ALTER TABLE deployments ADD COLUMN config_hash TEXT. The migration completed in eleven seconds on a low-traffic Saturday morning. The migrating engineer verified that the column existed in the production database and closed the migration ticket.

The Debezium connector was not paused before the migration. The connector schema registry was not updated with the new column. The migration runbook had no CDC coordination step — the runbook had been written before Debezium was adopted and had never been updated. When the ALTER TABLE DDL statement appeared in the Postgres WAL, the connector read the DDL event and attempted to update its internal schema representation for the deployments table. The connector's schema update logic in the version deployed (1.9.x) compared the WAL DDL event against the Confluent Schema Registry's registered schema for the deployments topic. The registered schema did not include the config_hash column. The connector threw a DataException: Schema does not match expected schema error and entered a FAILED state, stopping all event production across all five monitored tables — not just the deployments table — because the connector instance was shared across the full schema.

The five downstream consumers stopped receiving events. The real-time dashboard showed configuration deployments as pending indefinitely. The notification fanout service queued notifications that never fired. The compliance reporting service's consumer group lag counter began incrementing at 1,400 events per minute — the rate at which production writes continued into the now-silent event stream. The on-call engineer received a PagerDuty alert for the consumer group lag threshold at 3:40 AM, forty-two minutes after the connector failed. The alert message said "deployments consumer group lag exceeds 10,000 events" — it did not say "Debezium connector in FAILED state." The on-call engineer spent thirty minutes investigating the deployment pipeline before checking the Kafka Connect REST API and finding the connector status: FAILED with the schema exception message.

The fix required three steps that took two hours to execute safely: first, update the Confluent Schema Registry to register the new Avro schema for the deployments topic with the config_hash field as an optional string with a null default; second, restart the Debezium connector; third, verify that the connector resumed processing from its last committed WAL offset without replaying the entire event history. The connector resumed successfully. Consumer group lag cleared within eighteen minutes at the connector's production replay throughput. Total event gap: four hours and three minutes, during which 336,000 deployment events were produced to the WAL but not delivered to the five downstream consumers. The consumers replayed the backlog correctly once the connector resumed — no data was lost, because the WAL retained the events — but the four-hour gap had produced stale dashboard states for 87 active customers during business hours in US timezones.

The retrospective produced two changes: the database migration runbook was updated with a mandatory CDC coordination section (pause connector → run DDL → update schema registry → resume connector → verify), and the Kafka Connect worker was configured to send a Slack alert on connector state transition to FAILED. Neither change addressed the root problem: the decision to use Debezium had been documented in a design document as "log-based CDC via Debezium," but the schema migration coupling — the requirement that every DDL change to any monitored table requires a corresponding connector schema registry update as part of the migration execution — had never been documented as a property of that decision. Database migrations and connector management were operated by different engineers who shared a migration runbook that predated Debezium's adoption.

A 29-person B2B SaaS company built a contract management platform for professional services firms — statement of work creation, approval workflows, e-signature collection, billing milestone tracking, and client portal access. The platform's backend was a Node.js API layer over a Postgres database, and the product roadmap called for an event-driven integration layer that would allow clients to receive real-time webhooks on contract status changes and push events to their CRM and ERP systems.

The engineering team evaluated Debezium and application-level dual writes for the integration layer. The team lead summarized the trade-off: Debezium required a dedicated Kafka Connect deployment, WAL access configuration on the Postgres instance, a schema registry, and operational expertise in Kafka Connect that the five-person engineering team didn't have. Application-level dual writes required adding a Kafka producer call after each database write — a pattern the team's senior engineer had used at a previous company and could implement in an afternoon. The team chose dual writes. The decision was recorded in a Notion page titled "Integration Layer Architecture" as: "We'll use application-level event emission after database writes. Simpler than CDC infrastructure."

The implementation added a publishEvent(eventType, payload) call after each db.query() call in the contract state machine's transition functions. The pattern was applied consistently across eleven state transitions: draft creation, draft submission, approval routing, approval decision, revision request, final approval, signature request, signature completion, billing milestone creation, milestone approval, and contract closure. In development and staging environments, the publisher was mocked; in production it called a Kafka producer configured with acks=1 (the broker leader acknowledges the write without waiting for follower replication).

Eighteen months after the integration layer launched, a compliance audit required the company to produce a complete record of all contract state transitions for all contracts processed in the prior fiscal year. The audit firm's methodology was to compare the event stream record against the database audit log (a separate contract_audit_log table populated by Postgres triggers — the company had triggers on the contracts table that had been added before the event integration layer). The comparison revealed that the event stream was missing 3,400 events that existed in the trigger-populated audit log.

The missing events were concentrated in two patterns. The first pattern accounted for 2,100 of the 3,400 gaps: these were events from a six-week period when the API service had been running on a Kubernetes cluster during an infrastructure migration that produced frequent pod restarts due to a misconfigured readiness probe. Pod restarts under the misconfigured probe averaged eleven per day across three API replicas. The sequence of events during a pod restart was: the application wrote the contract state change to Postgres (which committed), the Kubernetes liveness probe killed the pod before the Kafka producer's send() call completed, the in-flight Kafka write was abandoned, and the event was permanently lost. The Kafka producer was not configured with a retry policy for producer-side failures — the application relied on the acks=1 acknowledgment that never arrived.

The second pattern accounted for 1,300 gaps: these were events from a period when the database and the Kafka broker were both healthy but the Kafka broker's leader election for the events topic had produced a 340 ms leadership pause. The Node.js Kafka producer's default socket timeout was 300 ms. Producer calls that hit the leadership pause timed out, threw a KafkaJSRequestTimeoutError, and were swallowed by a try/catch block in publishEvent() that logged the error and continued — a defensive error handling pattern that prioritized not failing the API request over guaranteeing event delivery.

The three-week audit remediation produced a reconciliation job that cross-referenced the trigger audit log against the event stream for the prior 18 months and replayed the 3,400 missing events into the Kafka topic with a replayed: true field to prevent downstream deduplication issues. The remediation also added the outbox pattern to the contract state machine: the dual write was replaced with a write to a contract_events_outbox table within the same Postgres transaction as the contract state change, with a Debezium connector reading the outbox table via WAL CDC and publishing to Kafka. The infrastructure investment the team had avoided 18 months earlier — dedicated Kafka Connect deployment, WAL access configuration, schema registry — was implemented during the remediation sprint at a cost of 14 engineer-days, compared to the original estimate of 1 engineer-day for dual writes. The "simpler" option had been simpler to write and harder to operate reliably.

A 44-person e-commerce enablement company built a platform for direct-to-consumer brands — order management, inventory tracking, fulfillment routing, and merchant analytics — as a white-label infrastructure layer that mid-market brands could deploy behind their storefronts. The platform handled order state transitions, inventory reservation and release, and fulfillment provider routing for approximately 30 merchant accounts in its first year of operation.

The engineering team needed to propagate changes from the primary Postgres database to three downstream systems: a read replica for the merchant analytics dashboard (which couldn't run queries against the primary during high-traffic periods), an Elasticsearch index for order search, and an event stream consumed by the fulfillment routing service. The team chose PostgreSQL row-level triggers as the CDC mechanism for all three integration targets. The choice was made in year one when the company had twelve merchant accounts and peak order volume of approximately 800 write operations per minute across all merchants combined. At that volume, trigger overhead of 2–4 ms per write was invisible in P99 latency metrics that hovered around 18 ms at peak. The trigger implementation was straightforward: an AFTER INSERT OR UPDATE OR DELETE trigger on the orders table populated a cdc_events staging table, and a polling relay process read from the staging table and routed events to the three destinations.

The trigger approach worked reliably for fourteen months. In month fifteen, the company onboarded six new merchants in a single quarter — three of them high-volume brands with order rates that individually exceeded the company's entire prior peak. Total platform write volume grew from 800 to 6,600 operations per minute over eight weeks. The engineering team treated this as a capacity success and did not audit the write path performance during the onboarding sprint.

The performance degradation surfaced through merchant escalations rather than monitoring. In week three after the high-volume onboarding, two merchants reported that order confirmation pages were loading slowly — the confirmation page waited for a synchronous API response that included the order record fetched after the write. Support tickets described confirmation page load times of 6–9 seconds in the evening peak window, compared to under 2 seconds previously. The platform's synthetic monitoring had no order write latency alert; the existing alerts covered API response time at the load balancer level, which was averaging the slow evening peak writes with normal daytime writes and staying below the threshold.

The database team ran an EXPLAIN (ANALYZE, BUFFERS) on the order write path during an evening peak window and found a P99 write latency of 94 ms on the orders table — up from 12 ms in the pre-onboarding baseline. The breakdown showed that the orders table write itself (the INSERT into the primary table) took 8 ms at P99. The trigger execution — the AFTER trigger's INSERT into cdc_events with all 47 columns of the orders table captured — took 86 ms at P99. The trigger had been benchmarked at 2–4 ms in the year-one environment against a 12-column version of the orders table; the table had grown to 47 columns over fourteen months of feature development, and the trigger captured all columns because the downstream consumers needed full row state. At 6,600 writes per minute, the cdc_events staging table was receiving more than 6,600 rows per minute, and contention on the staging table's index under concurrent trigger writes was compounding the per-row overhead into 86 ms P99 values.

The immediate mitigation was reducing the captured columns to the thirty-one fields that the downstream consumers actually used, which brought trigger P99 overhead down to 41 ms — still a 3× increase over the year-one baseline. The engineering team's database lead confirmed that at the projected write volume for the next two years (the sales pipeline included three more high-volume merchants), trigger-based CDC was not viable: even optimally tuned, trigger execution within the write transaction would continue to add latency that scaled with write volume.

The migration to Debezium log-based CDC took eleven weeks: two weeks for the Kafka Connect infrastructure deployment (the team had no prior Kafka Connect experience), two weeks to configure the Debezium Postgres connector with the correct WAL access permissions and schema registry, three weeks to rebuild the three downstream integrations to consume from Kafka topics rather than polling the cdc_events staging table, two weeks of parallel-run verification comparing trigger output to Debezium output for consistency, and two weeks of staged merchant migration to the new event stream with rollback readiness maintained. After migration, P99 write latency on the orders table returned to the 8–12 ms range at full production volume. The trigger approach had been the right choice at year-one volume and the wrong choice at year-three volume. The decision had never been documented with a volume threshold — there was no written record that said "this approach is valid up to N writes per minute, and above that threshold we migrate to log-based CDC." The migration was reactive rather than planned.

Structural properties set by the CDC mechanism decision

Three structural properties are determined when a team decides — or fails to explicitly decide — how to implement change data capture from a relational database: what the CDC mechanism selection determines about the schema migration coordination requirement as the source database schema evolves and the CDC infrastructure's schema representation must evolve in lockstep, what the consumer delivery semantics determine about the idempotency requirement as at-least-once delivery produces duplicate events that consumers built on exactly-once assumptions will mishandle, and what the dual write model determines about the crash recovery gap as application writes to both the database and the event stream without transactional coordination between them. None of these properties are labeled as decisions in the conversations that produce them. The CDC mechanism selection emerges from a "we already use Kafka" or "simpler to implement" infrastructure preference without specifying how schema changes are coordinated across the migration runbook and the connector configuration. The consumer delivery semantics emerge from the connector's default configuration, which is at-least-once, without a corresponding consumer design specification that mandates idempotent writes. The dual write model emerges from a "publish after write" pattern that appears to be a straightforward extension of the database write without recognizing that the two writes are not atomically coordinated.

Property 1: The CDC mechanism selection and the schema migration coupling surface. Log-based CDC (Debezium, AWS DMS, Maxwell, PeerDB) reads the database's write-ahead log or binary log directly — the same log that the database uses internally to support replication and crash recovery. Because the WAL contains both DDL events (schema changes) and DML events (data changes), the CDC connector must interpret the schema context of each DML event correctly; if the connector's cached schema representation does not match the schema under which the DML event was written, the connector cannot deserialize the event and fails. This coupling exists at the level of the migration execution process: the database schema change and the connector schema registry update must happen as a coordinated pair, not as independent operations in separate runbooks owned by separate teams. The schema migration coupling surface is the set of DDL changes that can cause a connector failure if applied without connector coordination — it includes adding a NOT NULL column, removing a column that the connector schema includes, renaming a column, and changing a column's data type. The coupling surface is zero for trigger-based CDC (triggers read the row after the DML operation and are unaffected by schema changes, because the AFTER trigger sees the row in its new schema) and for application-level dual writes (the application controls the event payload shape independently of the database schema). The CDC mechanism decision should specify: which mechanism is selected, the schema migration coordination requirement for that mechanism (none for triggers, connector pause + registry update for log-based CDC), and the runbook section that implements the requirement. Connect this property to the database schema migration decision record: the zero-downtime schema migration procedure — the expand/contract pattern, the backward-compatible column addition, the multi-step migration sequence — determines the order in which DDL changes appear in the WAL; the CDC migration coordination step must be inserted at the correct point in the expand/contract sequence, typically after the additive DDL step and before the application deployment that begins writing the new column's values. A migration runbook that does not include CDC coordination as a mandatory step alongside the standard DDL sequence produces connector failures on every additive schema change, at a rate proportional to the schema's change velocity.

Property 2: The consumer delivery semantics and the idempotency requirement. CDC connectors operating over Kafka provide at-least-once delivery by default: the connector commits its WAL offset to Kafka after processing a batch of events; if the connector fails between processing events and committing the offset, it will replay from the last committed offset on restart, delivering some events more than once. The idempotency requirement is the property that downstream consumers must satisfy to process replayed events correctly: a consumer is idempotent with respect to a CDC event if applying the event more than once produces the same result as applying it once. For consumers that write records to a database based on CDC events, idempotency is implemented as an upsert keyed on the source record's primary key and the WAL transaction sequence number (LSN). For consumers that trigger side effects — sending a notification, charging a payment, calling an external API — idempotency requires a processed-events ledger: record the event's unique identifier (source table name + primary key + LSN) before triggering the side effect, and check the ledger before triggering; if the identifier is already present, skip the side effect. The idempotency requirement is a consumer design specification that must be written before the first consumer is built, not retrofitted after the first duplicate processing incident. The failure mode is specific: a consumer built with exactly-once delivery assumed uses blind INSERT rather than upsert; when the connector replays a batch of events after a restart, the consumer's blind INSERT fails with a unique constraint violation or — worse — silently succeeds on a table without unique constraints and produces duplicate records. Connect this property to the event sourcing decision record: event-sourced systems built on top of CDC share the same at-least-once delivery characteristic as CDC consumers — the projection builder must be idempotent with respect to replayed events from the event store; the idempotency key for the projection builder is the event's position in the event log (the LSN for a Postgres-backed event store); a projection builder that uses the event's LSN as a watermark and skips events whose LSN is less than or equal to the stored watermark handles connector replays without producing duplicate projection writes. The API idempotency decision record covers the parallel requirement at the API layer: the same LSN-keyed idempotency pattern that protects CDC consumers from duplicate processing also applies to API endpoints that receive retried requests; the idempotency key model should be consistent across the event processing layer and the API layer so that end-to-end duplicate processing is handled by a single, uniform pattern rather than by ad hoc deduplication logic at each consumer.

Property 3: The dual write model and the crash recovery gap. Application-level dual writes — writing to the database and to the event stream in sequence within the same application request handler — do not provide transactional consistency unless the database and the event stream support distributed two-phase commit. In practice, Postgres and Kafka do not support distributed two-phase commit with standard client libraries. The crash recovery gap is the set of events that are permanently lost when the application process terminates between the successful database commit and the successful event stream write. The gap is bounded by the application's write throughput and the crash frequency — at 100 writes per minute and one crash per week, the expected gap is approximately the number of writes in-flight at the moment of the crash. At 1,400 writes per minute and eleven crashes per day (as in the contract management SaaS during the Kubernetes probe misconfiguration period), the gap accumulates rapidly. The outbox pattern closes the gap by making the event write part of the same ACID transaction as the domain state change: the application writes an event record to an outbox table within the database transaction, and a CDC connector (typically log-based, for zero write latency overhead) reads the outbox table via WAL and publishes to the event stream. The outbox entry exists in the database if and only if the domain state change committed. The relay publishes the event if and only if the outbox entry exists. The crash recovery gap is zero: if the application crashes after writing to the outbox table and before the relay publishes, the relay will publish on its next polling cycle. The dual write decision should be evaluated against the crash recovery gap explicitly: if a missing event has a business consequence — a missed billing event, a compliance audit gap, a cross-service state synchronization failure — the dual write model is not appropriate, and the outbox pattern with a CDC relay is the correct implementation. Connect this property to the saga pattern decision record: distributed sagas that coordinate state changes across multiple services rely on event delivery guarantees between saga steps; a saga implemented on top of dual writes with a crash recovery gap will produce saga instances that advance past a compensatable state without emitting the event that triggers the next step or the compensating transaction; the outbox pattern provides the delivery guarantee that saga steps require, and the saga pattern decision should specify the event delivery model — outbox-backed CDC or a transactional event store — as a prerequisite for the saga's correctness guarantee. The WhyChose extractor finds the CDC mechanism decisions buried in your AI chat history — the architecture session where the team compared Debezium and dual writes and chose dual writes for simplicity without documenting the crash recovery gap, and the incident review where the team first encountered the connector schema mismatch and discovered that the migration runbook had no CDC coordination step.

The CDC decision ADR: five sections

Section 1: CDC mechanism selection and write path impact specification. Specify the CDC mechanism with a written evaluation of its write path impact at the expected production write volume in three years, not at the current write volume. The three primary options and their write path impacts: (1) Log-based CDC (Debezium, AWS DMS, Maxwell) reads the database's WAL or binary log outside the write transaction path — zero write latency overhead at any volume. Operational overhead: dedicated connector process, WAL access configuration (Postgres logical replication slot, MySQL binlog row format), schema registry, connector monitoring. Schema migration coupling: mandatory coordinator step in every DDL migration runbook. (2) Trigger-based CDC executes within the write transaction — 2–6 ms per write at low column count and low concurrent write volume; overhead grows with column count and write concurrency. Operational overhead: trigger maintenance (triggers must be updated when the source table schema changes to avoid capturing deprecated columns); no external process required. Schema migration coupling: none — triggers read the row in its current schema after each DML. (3) Application-level dual writes execute in the application request handler — zero database write overhead; overhead is the event stream producer call latency (typically 1–10 ms at p50). Crash recovery gap: events are permanently lost when the application crashes between the database commit and the event stream write unless the outbox pattern is used. Document the write volume threshold at which trigger-based CDC becomes unacceptable given the source table's column count and the P99 write latency SLO. For a 40-column table and a 20 ms P99 write latency SLO, trigger-based CDC is viable up to approximately 2,000 writes per minute under standard Postgres trigger execution overhead — above that threshold, log-based CDC is required. Connect to the database connection pooling decision record: log-based CDC connectors require a dedicated Postgres replication connection (a logical replication slot) that counts against the max_connections limit; the connection pooling model must reserve capacity for CDC replication connections alongside application pool connections; a PgBouncer configuration that pools only transaction-mode connections will not work for CDC replication because replication connections require session-mode persistence.

Section 2: Schema migration coordination procedure and runbook integration. For log-based CDC, specify the schema migration coordination procedure as a mandatory section in the database migration runbook. The procedure must be documented as an ordered sequence of steps, not as a general principle: (1) Before the DDL migration: identify all CDC connectors that monitor the affected table by querying the Kafka Connect REST API (GET /connectors/{name}/status); verify connector status is RUNNING and lag is below the acceptable threshold; pause each connector (PATCH /connectors/{name}/config with connector.class set to a paused state, or use the Kafka Connect PAUSE endpoint). (2) During the DDL migration: run the DDL change using the standard expand/contract procedure. (3) After the DDL migration and before connector resume: register the new schema version in the schema registry for each topic produced by the affected connector; verify the registered schema is backward compatible with the previous version using the schema registry compatibility API. (4) Resume the connector and verify: resume each connector; monitor the connector error rate metric and the consumer group lag for the affected topics for 10 minutes; confirm lag is clearing at the expected throughput. This procedure should be verified against a non-production Postgres replica before it is executed in production — schema registry interactions vary between Debezium versions and Confluent Schema Registry versions, and an untested procedure will produce unexpected behavior during a production migration window. For trigger-based CDC, specify the trigger update procedure: triggers that capture all columns must be updated when columns are added or removed to avoid capturing deprecated columns and to include new columns that downstream consumers need. Document which downstream consumers require which columns, to make the trigger column list an explicit contract rather than a "capture everything" default. Connect to the CI/CD pipeline decision record: the CDC schema migration coordination step should be encoded as a pre-migration check in the deployment pipeline — a CI job that queries the schema registry for the current schema version and the Kafka Connect REST API for connector status should run before the DDL migration step, and a post-migration check should verify that the connector resumed successfully and lag is clearing; encoding these checks in the pipeline converts the manual runbook procedure into an automated verification step that cannot be skipped.

Section 3: Consumer delivery semantics and idempotency contract. Specify the expected delivery semantics for all CDC consumers before the first consumer is built: at-least-once (the default for Kafka-backed CDC connectors) or exactly-once (available with Kafka's transactional producer and idempotent consumer configuration, at a throughput cost of approximately 20–30% versus at-least-once). For at-least-once delivery, specify the idempotency requirement for every consumer category: database write consumers must use upserts keyed on the source record's primary key and the WAL LSN; side-effect consumers (notifications, payments, external API calls) must use a processed-events ledger keyed on the event identifier; aggregate consumers (counters, sums, derived state) must use a LSN watermark to skip already-processed events. Document the idempotency key schema for each consumer as part of the CDC decision record — the primary key field names, the LSN field name in the Debezium event envelope, and the watermark table schema. A consumer that is built without an explicit idempotency specification will default to the developer's intuition about event delivery, which in a healthy system that has never experienced a connector restart will be exactly-once. The idempotency failure mode surfaces only during a connector restart, a rebalance, or a network partition — all of which are infrequent enough in development environments that the missing idempotency is not discovered until the first production incident. Include an idempotency test in the consumer's integration test suite that replays the same event twice and asserts that the consumer produces the correct result rather than a duplicate. Connect to the event-driven architecture decision record: the idempotency requirement for CDC consumers applies equally to all event-driven consumers regardless of the event source; a unified idempotency pattern across CDC consumers, event sourcing projection builders, and API webhooks reduces the surface area of duplication handling to a single library with a consistent interface, rather than requiring each consumer to implement its own deduplication logic independently.

Section 4: Outbox pattern adoption criteria and dual write evaluation. Specify explicitly whether the application uses the outbox pattern or direct dual writes for each event emission path, with a written evaluation of the crash recovery gap for dual writes. The evaluation should cover: the application's process restart rate (under normal Kubernetes scheduling, a pod restarts on average once every 2–7 days in a production cluster with rolling deployments); the average number of in-flight writes at the moment of a restart (write rate × average write latency in seconds); the expected event loss rate per month (in-flight writes per restart × restart rate per month); and the business consequence of a missing event for each event type. If the business consequence of a missing event is acceptable — the event type is informational, the gap can be reconciled after the fact, and the loss rate is below the acceptable threshold — direct dual writes with a resilient Kafka producer (configured with acks=all, a retry policy with an exponential backoff, and error logging that triggers an alert rather than silently swallowing failures) is a reasonable choice. If the business consequence is not acceptable — the event type is a compliance audit record, a billing event, or a saga coordination message — the outbox pattern is required. Document this evaluation per event type, not as a blanket policy, because different event types within the same application will have different acceptable loss thresholds. The outbox table schema should include: the event type, the aggregate type, the aggregate ID, the event payload as JSON, the creation timestamp, and a published flag or a published-at timestamp used by the relay to track which events have been delivered. Connect to the background job infrastructure decision record: the outbox relay process — the component that reads unpublished outbox records and publishes them to the event stream — is a background job with specific delivery requirements: it must run continuously, process events in insertion order per aggregate, handle Kafka producer failures with retries, and mark events as published atomically (the mark must happen within the same Postgres transaction as the relay's acknowledgment of the Kafka write, or use a separate idempotency check). The background job infrastructure decision specifies the execution model (Kubernetes CronJob, a dedicated worker process, or a sidecar container) and the failure handling policy that the outbox relay must satisfy.

Section 5: CDC monitoring, lag alerting, and consumer health model. Specify the monitoring model for the CDC pipeline as a complete observability specification: the connector health metrics (connector status as a binary RUNNING/FAILED metric exported to the alerting system, not only to a Kafka Connect UI dashboard), the consumer group lag metrics (expressed in seconds, not only in event count, derived from dividing the lag in events by the historical events-per-second rate for each topic), and the business-impact metrics that are derived from CDC consumer health (order confirmation write latency for an order management CDC consumer, compliance report freshness for an audit trail CDC consumer). The lag alert threshold should be specified in seconds rather than event count, because the same event count gap has different business impact at different throughput rates: a 10,000-event lag at 100 events per second is 100 seconds of staleness; at 10,000 events per second it is 1 second. The connector FAILED state should trigger an immediate P2 alert — a failed connector produces no events at all, which is distinguishable from high lag only if the connector status metric is monitored separately from the consumer group lag metric. Without a connector status alert, the failure mode from story one in this post recurs: the lag alert fires 30–40 minutes after the connector fails, and the on-call engineer investigates the consumer rather than the connector. The consumer health model should define: the maximum acceptable lag in seconds per topic, the escalation path when lag exceeds the threshold, the connector restart runbook (including the schema registry verification step), and the consumer replay procedure for periods when the connector was failed and the consumer group lag represents a real event backlog rather than a false alarm. Connect to the alerting threshold decision record: the CDC lag threshold — the number of seconds of consumer lag that constitutes an alertable condition — is a threshold that must be calibrated against the business impact of that lag for each consumer type; a compliance reporting consumer that produces daily reports has a different acceptable lag from an order notification consumer that sends emails within 30 seconds of order confirmation; the alerting threshold decision should document the lag tolerance per consumer type and the escalation policy for each, rather than using a single platform-wide lag threshold.

FAQ

Further reading

• Database schema migration decision record — the expand/contract procedure, the backward-compatible migration sequence, and the zero-downtime constraint that apply to every DDL change on a table monitored by a CDC connector; the CDC connector coordination step must be inserted at the correct point in the expand/contract sequence, and the migration runbook must encode both the DDL procedure and the connector pause/resume procedure as a single ordered checklist.
• Event sourcing decision record — the event store selection, the projection design, and the event schema evolution protocol that share the same at-least-once delivery and idempotency requirements as CDC consumers; event-sourced projection builders built on CDC share the same LSN-watermark idempotency pattern that CDC consumers require, and the schema evolution protocol for event-sourced aggregates follows the same backward-compatibility and schema registry requirements as the CDC connector schema coordination procedure.
• Saga pattern decision record — the distributed saga coordination model, the compensating transaction design, and the step-ordering guarantee that depend on reliable event delivery between saga steps; a saga built on top of dual writes with a crash recovery gap will advance past a compensatable state without emitting the coordination event that triggers the next step; the outbox pattern is the prerequisite delivery guarantee for saga correctness.
• Event-driven architecture decision record — the event delivery model, the consumer design contract, and the failure isolation strategy that apply across all event-driven consumers regardless of source; the idempotency pattern for CDC consumers (upsert keyed on primary key + LSN) is the same pattern required for all event-driven consumers, and a unified idempotency library reduces the deduplication surface area across CDC consumers, event sourcing projection builders, and API webhooks.
• Alerting threshold decision record — the lag threshold calibration methodology and the connector health metric specification; CDC consumer lag expressed in seconds rather than event count, the connector FAILED state as a P2 alert distinct from the consumer lag alert, and the per-consumer-type threshold calibration against the business impact of staleness for each consumer.
• Open-source extractor — find the CDC mechanism decisions buried in your AI chat history: the architecture session where the team compared Debezium and dual writes and chose dual writes for simplicity without documenting the crash recovery gap, and the incident review where the team discovered that the migration runbook had no connector coordination step after the first schema-mismatch connector failure.