The data pipeline reliability decision record: why the reliability model you chose determines your downstream data correctness surface and your incident blast radius ceiling

A 30-person SaaS company building a customer analytics platform had a data pipeline that ingested event streams from three third-party data sources, applied a set of transformation and aggregation steps, and wrote the outputs to the tables that powered its customers' dashboards. The pipeline's monitoring setup was thorough by the standards of the engineering team's prior experience: Datadog was configured to alert on Kafka consumer lag above 60 seconds, processing error rate above 1%, and worker memory utilization above 80%. The on-call runbook covered the three failure modes the team had encountered during the pipeline's first year: consumer lag spikes from batch size mismatches, worker OOMKills from large record payloads, and periodic delivery gaps when one of the upstream sources had maintenance windows. The infrastructure had been reliable for eighteen months. The engineering team had not received a serious pipeline incident page in six months.

One of the three upstream sources changed the format of a timestamp field in their event payload. The field had previously been delivered as a Unix epoch timestamp in integer seconds; the source updated it to an ISO 8601 string format as part of a data model migration they communicated in a release note on their developer portal. The pipeline's ingestion stage parsed the field as an integer and passed the parsed value downstream to the transformation step, which used it to compute event timestamps for bucketing and aggregation. When the source began delivering ISO 8601 strings, the integer parser received a value like "2026-09-15T14:23:11Z" and produced NaN, which the downstream aggregation step coerced to zero, which the timestamp bucketing step interpreted as January 1, 1970. The pipeline processed every record successfully. The error rate remained at zero. Consumer lag remained within bounds. The infrastructure metrics showed nothing wrong.

For six weeks, a fraction of every customer's dashboard showed date-bucketed data that spiked anomalously in the 1970 bucket and showed a corresponding depression in the current-period bucket proportional to the fraction of events delivered by the affected upstream source. The product team's internal monitoring flagged the "1970 data anomaly" as a display issue and created a ticket for the frontend team to investigate whether a date formatting bug had been introduced in a recent release. Three enterprise customers flagged the anomaly to their account managers during quarterly business reviews; one ran a reconciliation comparison against their own internal event data and documented a 23% discrepancy in the affected time period. In the following quarter, three enterprise customers churned. All three cited data accuracy in their exit interviews: "We can't build our team's reporting on data we can't trust" and "We audited the last quarter's numbers and found discrepancies we couldn't explain" and "When we compared your numbers to our own source data, they didn't match." The pipeline's infrastructure metrics had been green for the entire six-week contamination window. Connect this to the vendor dependency governance decision record: the upstream schema change was not a breach of the source's API contract — the source had published a release note describing the change three weeks before the migration, but the pipeline team's vendor governance model had no mechanism for surfacing API release notes from the source's developer portal to the team responsible for the consuming integration; the governance failure that enabled the schema change to corrupt six weeks of customer data was not the source's decision to change the format but the pipeline team's absence of a process for tracking and responding to upstream format changes, and a vendor dependency governance model that includes a change notification protocol for upstream data sources would have surfaced the release note before the migration date rather than six weeks after the damage was done.

A 52-person analytics platform building a B2B SaaS for marketing attribution had invested in infrastructure monitoring depth: Datadog dashboards tracked CPU utilization, memory pressure, Kafka consumer lag, processing latency, and event throughput across twelve pipeline stages. The engineering team had documented SLAs for each infrastructure metric and had a clear on-call runbook for each alert type. The pipeline processed attribution events from customer-provided ad platforms and converted them into revenue attribution records that customers used for budget allocation decisions. Data accuracy was a contractual requirement for the platform's enterprise customers; the agreement stated that attribution accuracy would exceed 98% on a rolling 30-day basis measured against customer-provided revenue data.

A routine dependency upgrade to the platform's internal JSON parsing library introduced a silent type coercion change. The previous library version had preserved numeric strings — values like "9007199254741023" that exceeded the precision of a 64-bit float — as strings through the parsing step, leaving their handling to the downstream code that explicitly declared them as either string or integer depending on their semantic purpose. The upgraded library applied JavaScript's default numeric coercion rules: values that could be represented as numbers were converted to numbers, and values exceeding float64 precision were rounded to the nearest representable float. Attribution event records contained an ad platform's internal campaign ID field that used a 64-bit integer identifier; values above 2^53 — approximately 12% of the platform's customer traffic based on the distribution of campaign IDs in active accounts — were being silently rounded to the nearest representable float64, producing two distinct campaign IDs that resolved to the same float value and were treated as the same campaign in the attribution model. The infrastructure metrics showed nothing wrong: no increase in error rate, no queue buildup, no latency change, no memory pressure. The pipeline was processing every record successfully. It was just processing 12% of them incorrectly.

The bug went undetected for four months. A customer's annual compliance audit, which required reconciling the platform's attribution output against the customer's own transaction ID records, discovered that 8% of the attributed revenue in the prior quarter had been assigned to incorrect campaigns — revenue attributed to Campaign A that the customer's transaction records showed had been driven by Campaign B — because the two campaigns shared the same float64-rounded identifier and the attribution model had been merging their results. The platform's post-incident analysis identified that the dependency upgrade had been logged in an internal CHANGELOG entry, reviewed by the infrastructure team as a non-breaking upgrade, and deployed without a regression test that specifically checked the behavior of high-precision numeric IDs. The pipeline had no data quality assertion on output campaign ID uniqueness — a uniqueness check on the IDs in each processed batch would have detected the collision pattern in the first batch after the upgrade was deployed. The monitoring had no assertion on the distribution of campaign ID values — a check on the maximum observed ID value in each batch would have shown a sudden decrease in the observed maximum after the upgrade, which would have been an anomaly signal even without knowledge of the float64 precision boundary. Connect this to the incident severity classification decision record: when the attribution error was discovered, the on-call team's initial severity classification was uncertain — the pipeline was operating normally from an infrastructure perspective and the error was not a service outage; the team's severity framework had no tier for silent data quality corruption that was discovered by a customer audit rather than by the platform's own monitoring, and the absence of a severity definition for this class of failure produced a three-hour escalation delay while the team debated whether the incident required immediate executive notification; a severity framework that includes a tier for customer-impacting data quality failures discovered after the fact, with the same escalation path as a service outage, would have produced an immediate executive notification and a customer communication within the first hour rather than after the escalation debate resolved.

A 75-person fintech building a payment analytics platform had, by its third year, built what its engineering leadership considered a mature data reliability practice. Data quality assertions ran at every transformation stage: freshness assertions alerting if a batch had not arrived within 15 minutes of the expected delivery time, completeness assertions flagging batches whose record count fell more than 20% below the rolling 30-batch average for that source, schema conformance assertions rejecting records with null values in required fields, and value distribution assertions comparing each batch's key field distributions against the rolling 30-batch mean with a two-standard-deviation alert threshold. The on-call runbook documented a specific response procedure for each assertion type. The engineering team had responded to and resolved fourteen data quality incidents in the prior year without any incident producing downstream consumer contamination — the assertion layer had caught every failure before the affected data crossed to the next pipeline stage.

An upstream payment data provider sent a batch containing 847 records with null values in the transaction_currency field, which the pipeline's schema conformance assertion flagged as a quality failure. The assertion triggered the pipeline's rejection logic, which logged the failed batch, sent an alert to the on-call engineer, and queued the batch for reprocessing after the upstream source corrected the malformed records. The pipeline's reprocessing logic was designed for throughput: it re-sent the entire batch to the transformation stage once the upstream source acknowledged the correction. The transformation stage had no idempotency guard on its downstream write path — it had been designed with the assumption that each batch would be processed exactly once, and the write path used insert semantics rather than upsert semantics. The 847 records with null currency values were in the last 5% of the batch; the first 95% — 16,087 records — had been successfully written to the downstream consumer before the conformance assertion rejected the batch at the 16,087th record.

When the reprocessing logic re-sent the entire batch, the downstream consumer received 16,087 records it had already written, applied insert semantics to each of them, and doubled the counts for every transaction in those records. The downstream consumer was an incremental aggregation pipeline — it did not store individual records, only running aggregates — so it had no mechanism to detect that it was receiving duplicate records. The aggregated totals for the affected time period were now 2x the correct value. The value distribution assertion on the output layer detected the anomaly as a sudden doubling of aggregate values and triggered a quality failure alert. The reprocessing logic interpreted the quality failure alert as a signal that the batch had not been fully processed and re-sent it a third time. Five retry cycles completed before the on-call engineer was paged, by which time the downstream consumer had aggregated each of the 16,087 records five times and was reporting values that were 5x the correct amount. The on-call engineer halted the reprocessing job and assessed the damage: 16,087 payment records for a 4-hour window were contaminated with 5x-multiplied aggregates; 6 downstream reports used the affected aggregates as inputs; a full reprocess from source was required to restore correct state; estimated recovery time was 8 hours. The reprocessing system had been designed to process 10,000 records per second — it was fast, and it had retried efficiently; the blast radius of the single malformed upstream batch was proportional to the efficiency of the reprocessing logic's retry throughput. Connect this to the change data capture decision record: the downstream consumer's CDC mechanism had been designed to propagate incremental updates from the analytics pipeline to the platform's customer-facing reporting layer — it was optimized for the incremental update case and had no support for the full-reprocess correction pattern that the idempotency failure required; the recovery required constructing a manual reconciliation workflow outside the CDC path that had not been designed or tested, and the 8-hour recovery time was in part a function of the gap between the pipeline's reprocessing capability and the CDC model's correction capability — a CDC model that includes a reconciliation mode designed to handle correctness corrections, not just incremental updates, would have reduced the recovery time from 8 hours to the 2 hours required for the pure reprocessing work without the manual correction scaffolding.

Structural properties set by the data pipeline reliability decision

Two structural properties are determined by the founding decisions about reliability model scope, data quality contract completeness, reprocessing path design, and incident classification. Neither is visible in the infrastructure metrics dashboard until a downstream consumer receives contaminated data, a customer audit discovers a discrepancy, or a retry cascade turns a contained quality failure into a multi-hour reconciliation incident — at which point the recovery cost is proportional to the correctness surface and blast radius ceiling that the reliability model's design either bounded or allowed to grow unchecked.

Property 1: Downstream data correctness surface. The downstream data correctness surface is the set of downstream consumers whose output data can be contaminated by an upstream quality failure without triggering a detection signal before the contaminated data is consumed. It is set by the completeness of data quality contracts at each pipeline stage, the detection latency for quality failures, and the pipeline's ability to prevent incorrect data from crossing to a new consumer stage before it has been validated by the producing stage's assertions. A pipeline with no data quality assertions has a correctness surface equal to all downstream consumers — every failure that produces incorrect output propagates to every consumer without constraint. A pipeline with data quality assertions only on the infrastructure-observable failure modes — null pointer exceptions, schema parse errors, queue timeouts — has a correctness surface equal to all downstream consumers of the semantically incorrect data that passes the infrastructure layer: the silent type coercion that produces the wrong numeric value, the field format change that converts a correct integer to NaN, the precision truncation that collapses two distinct identifiers into the same value — these produce outputs within the expected type signature while containing values that are semantically wrong, and the infrastructure layer has no observation point for them. The correctness surface has two dimensions that multiply to produce the total contamination scope: breadth, the number of downstream consumers receiving data from a failing stage before the failure is detected, and depth, how far into each consumer's derived state the contamination propagates before it can be localized and corrected. The surface can be reduced by shortening detection latency — the time between when the quality failure begins producing incorrect data and when an assertion detects it — and by adding isolation boundaries that prevent data from crossing to a new consumer stage before the producing stage's quality assertions have passed. A pipeline stage with no quality assertions and no isolation boundary has an indefinite detection latency for semantically incorrect output and propagates that output to all downstream consumers immediately; a pipeline stage with a value distribution assertion and an isolation boundary that holds data at the boundary until the assertion passes has a detection latency bounded by the assertion's evaluation frequency and a correctness surface bounded by the isolation boundary. The surface reduction from adding a value distribution assertion is not marginal — it converts an unbounded correctness surface, where the failure propagates to all downstream consumers for as long as the pipeline runs, into a bounded correctness surface, where the failure is detected within the assertion's evaluation window and contained at the boundary before propagation. Connect this to the SLA design decision record: the data correctness SLA — the percentage of downstream consumers receiving correct data over a measurement period — is jointly determined by the correctness surface and by the detection latency; a correctness surface that spans all downstream consumers combined with a detection latency of six weeks produces a data correctness SLA failure that is retrospective and un-remediable for the contaminated period; a correctness surface bounded by isolation boundaries and a detection latency bounded by assertion frequency produces a correctness SLA that is prospective — a failure is detected and contained before it propagates far enough to affect customer-visible outputs — and the SLA commitment to customers can be calibrated against the actual detection and containment capability of the monitoring model rather than against an optimistic assumption that the infrastructure metrics would have caught any quality problem.

Property 2: Incident blast radius ceiling. The incident blast radius ceiling is the maximum downstream scope of a pipeline reliability failure, set by the idempotency guarantees in the reprocessing path, the isolation model between pipeline stages, and the reversibility of downstream writes when a quality failure is detected and the pipeline must reprocess from a known-good state. A pipeline stage that writes to a downstream consumer without idempotency has a blast radius that is a function of retry count rather than of failure scope: the first retry doubles the contamination for the records processed before the failure, the second retry triples it, and the pipeline designed for throughput will complete five retry cycles before the on-call engineer is paged, producing a 5x contamination of the originally-affected records that requires full state reconciliation rather than targeted correction. The ceiling grows with two design choices that are independently made but compound together: the absence of idempotency on the write path, and the presence of a retry logic that is designed for throughput without awareness of whether the data being retried has already been written downstream. The blast radius ceiling is also a recovery time ceiling because the time required to return the downstream consumer to a correct state is proportional to the amount of downstream state that was contaminated. A contamination of 5x-multiplied aggregates over a 4-hour window requires reprocessing the source data for that window, re-deriving all downstream aggregates from scratch, propagating the corrections through any consumers of those aggregates, and verifying that each stage of the reprocess produced correct output — a process that takes hours for a window that would have taken minutes to reprocess if the idempotency guarantee had limited the contamination to a single-pass error rather than a five-pass cascade. The ceiling is an architectural property that is set at design time and is expensive to change after deployment because idempotency requires a deduplication key in the write path, and adding a deduplication key to a write path that was not designed to include one typically requires a schema migration on the downstream consumer's storage — the migration that should have been part of the original design must now be executed against a live consumer with production data and an on-call incident as the forcing function. The ceiling is most accurately understood as a design parameter to be specified before building, not a risk to be managed after deployment: what is the maximum acceptable contamination scope of a single batch-level quality failure? The answer to that question determines the idempotency requirements for the reprocessing path, the isolation model required between stages, and the reconciliation capability required of the downstream consumers' CDC mechanism. Connect this to the vendor dependency governance decision record: the blast radius ceiling is influenced by the governance model for upstream data sources because the class of failures most likely to trigger a large blast radius — a malformed batch from an upstream source that passes partial validation before rejection — requires a reprocessing workflow that is designed specifically for the case where the upstream source is correcting a previously-delivered error; a vendor dependency governance model that includes a defined protocol for upstream source corrections, specifying what the source must do before a corrected batch is delivered and what the pipeline must do before reprocessing begins, reduces the blast radius ceiling by ensuring that the reprocessing workflow knows the boundaries of the failure and is not triggered with the assumption that the entire batch is being reprocessed cleanly when only the erroneous records are actually being corrected.

The data pipeline reliability ADR: five sections

Section 1: Reliability model selection and SLA definition. The reliability model specifies what constitutes a reliability failure in the data pipeline as distinguished from a reliability failure in the infrastructure that runs the pipeline, and how those failures are translated into SLA commitments that are measurable, auditable, and actionable for the teams responsible for the pipeline's operation. The first decision is whether the reliability model covers infrastructure reliability only or infrastructure reliability plus data quality reliability. An infrastructure-only model monitors the pipeline's computational components and defines SLA in terms of uptime, processing latency, and error rate; this model is appropriate only if the pipeline's downstream consumers have no quality requirements — if correct output and incorrect output that passes the infrastructure layer are equally acceptable — which is the condition that no serious data pipeline consumer actually meets. An infrastructure-plus-quality model monitors both the computational components and the data flowing through them; this model is appropriate for any pipeline whose downstream consumers make decisions based on the pipeline's output, which is every pipeline worth building. The SLA distinction between infrastructure and quality matters because the two failure modes have different detection mechanisms, different recovery procedures, and different impact severities, and conflating them produces an SLA commitment that cannot be measured — an overall uptime SLA of 99.9% that a pipeline achieves while producing six weeks of incorrect data is not a useful reliability commitment. The per-consumer reliability contract specifies what each downstream consumer is entitled to receive: a freshness commitment (data representing a time period will be available within a specified window of that period ending), a completeness commitment (the pipeline will deliver at least a specified fraction of the expected records for each batch), a correctness commitment (the values in each output field will be semantically correct according to the field's declared meaning), and a correction commitment (when a quality failure is discovered that has produced incorrect data that the consumer has already consumed, the pipeline will provide a reprocessing mechanism within a specified time window that allows the consumer to reconcile its state). The correction commitment is the SLA component most often omitted and the one with the highest impact when it is missing: a pipeline that can detect and contain quality failures before propagation but has no correction mechanism for failures that were detected after propagation has a correctness surface that includes the historical period back to the last known-good state, and the recovery cost for each historical incident is bounded only by the consumer's storage and the pipeline's reprocessing throughput rather than by a defined commitment about what correction the pipeline will provide. Connect this to the SLA design decision record: the data quality SLA faces the same measurement challenge as the availability SLA — the commitment is only meaningful if the measurement methodology is defined alongside the commitment; a correctness SLA commitment of "98% of output records are semantically correct over a rolling 30-day window" requires a methodology for sampling and verifying record correctness, and a methodology that relies on customers reporting discrepancies has a measurement latency that is six weeks in the worst case; the SLA is not fulfilled just by setting a target — it is fulfilled by building the measurement infrastructure alongside the commitment so that the pipeline can proactively verify its own correctness before the customer does.

Section 2: Data quality contract and monitoring design. The data quality contract specifies what correct output looks like for each pipeline stage, and the monitoring design specifies how compliance with that contract is measured and what happens when a measurement falls outside the contract's acceptable range. Four assertion classes form the minimum viable data quality contract. Freshness: data representing a time period should arrive and be available to downstream consumers within a specified window after that period ends; the window is calibrated against the source's historical delivery latency plus a buffer that accounts for occasional delays without producing false alerts; the alert triggers when the expected batch has not been received within the window, not when the batch processing has not completed — distinguishing a delivery failure from a processing failure changes the on-call response from "check the pipeline's worker health" to "check the upstream source's delivery health." Completeness: the record count for each batch should fall within an expected range relative to the upstream source's count or the prior batch's count; a drop below 80% of expected count triggers a quality failure alert and holds the batch from propagating to downstream consumers; the range is calibrated from the source's historical volume with a tolerance for seasonal and day-of-week variation, which means the completeness check requires a volume baseline model rather than a static threshold; a batch whose volume is 30% below normal is a different signal than a batch whose volume is 30% below an absolute count that was set when the pipeline was first deployed and has not been updated as the source's volume grew. Schema conformance: every required field is non-null, every typed field is parseable as its declared type without loss or coercion, and every enumerated field contains only values in the declared enumeration; schema conformance assertions run at ingestion before any transformation logic, so that a schema change by the upstream source is detected at the boundary between external data and internal processing rather than discovered downstream as an unexpected transformation output; the assertion should include a precision check for numeric fields whose values are known to include high-precision integers — asserting that no numeric ID field loses precision when converted from the source format to the internal representation catches the float64 truncation class of failures at ingestion rather than four months later in a customer audit. Value distribution: the statistical distribution of values in key output fields should not shift abruptly between batches; the assertion computes the current batch's distribution of each key field and compares it against the rolling 30-batch mean and standard deviation; values beyond two standard deviations from the mean trigger an anomaly alert that routes to the on-call engineer for manual review before the batch is propagated; the assertion does not need to know what the correct value is — it only needs to know that abrupt distribution shifts are anomalous and that anomalous batches should be held for review rather than silently written to downstream consumers. The quality assertion failure action is as important as the assertion itself: an assertion that detects a quality failure but allows the batch to propagate to downstream consumers produces a detection event without a containment event, and the correctness surface grows during the time between detection and remediation; the default action for a quality assertion failure should be to hold the failing batch at the current stage's output boundary until the failure is investigated and either the batch is corrected or the assertion is overridden with explicit documentation of why the failure is acceptable. Connect this to the change data capture decision record: the data quality contract for a pipeline that uses CDC to propagate changes to downstream consumers must include a contract for how quality failures that were detected after propagation are corrected through the CDC mechanism; the CDC model is optimized for the case where the source of truth is correct and the downstream consumer needs to catch up; the correction case — where the source of truth was incorrect and the downstream consumer needs to undo a change that was propagated — is structurally different and requires explicit support in the CDC model design, typically through a tombstone or compensating event pattern that the downstream consumer's idempotency logic can apply to produce a correct state from an incorrect one without requiring a full reprocess from source.

Section 3: Failure propagation containment model. The failure propagation containment model specifies how pipeline stages are isolated from each other so that a quality failure in one stage does not propagate to downstream stages before the failure is detected and resolved, and how the reprocessing path is designed to bound the blast radius of a failure that was not detected before propagation. Stage isolation is implemented through quality assertion boundaries: each stage's output is validated by the producing stage's assertions before being made available to the consuming stage; data that fails an assertion is held at the boundary in a quarantine state rather than written to the consuming stage's input queue; the consuming stage proceeds only with data that has passed the producing stage's assertions. The quarantine design requires four components: a quarantine storage location where failed batches are held with their failure metadata (which assertion failed, what the observed value was, what the expected value or range was), an alert routing that notifies the on-call engineer of the quarantine event with enough context to determine whether the failure requires manual intervention or automatic retry, a retry protocol that specifies when an automatic retry is appropriate and what conditions must be met before the retry is triggered, and a manual override protocol that allows the on-call engineer to release a quarantined batch with documented justification for cases where the assertion failure is a known anomaly rather than a data quality failure. Idempotency in the reprocessing path: every write operation in the pipeline's transformation stages should implement upsert semantics using a deduplication key that uniquely identifies each record across retry attempts; the deduplication key is typically a composite of the source-provided record identifier and the batch identifier that delivered it; the write operation checks whether the deduplication key already exists in the downstream consumer's state and applies an update rather than an insert if it does; this design ensures that a record processed twice produces the same downstream state as a record processed once, which converts the retry blast radius from a multiplying function of retry count into a constant bounded by the scope of the single failure. The isolation model for partially-processed batches — batches where the failure occurred after some records were written successfully — requires special handling: the partial write should be tracked in the quarantine metadata, and the retry protocol should resume from the failed record rather than reprocessing from the beginning of the batch; resumption-from-failure is significantly more complex than full-batch retry and requires that the pipeline track per-record processing state rather than per-batch state, but it is the design that bounds the blast radius to the actually-failed records rather than to the entire batch. For teams where per-record state tracking is not feasible, idempotency on the downstream write path is the fallback that bounds blast radius even for full-batch retries: the correctly-processed records in the batch are written again by the retry, the deduplication key causes them to produce an update-in-place rather than a new insert, and the downstream consumer's state is identical to what it was after the first partial write. Connect this to the incident severity classification decision record: the failure propagation containment model determines what class of incident a pipeline quality failure produces; a quality failure that is caught at the stage boundary by an assertion produces an operational incident — a quarantined batch requiring on-call attention and reprocessing — while a quality failure that propagates to downstream consumers before detection produces a customer-impacting data integrity incident; the two classes require different escalation paths, different customer communication protocols, and different root cause analysis scopes; a severity framework that treats both classes as "pipeline incidents" will apply the same escalation procedure to a quarantined batch as to a six-week customer data corruption, and the customer-impacting class will receive the operational response designed for the infrastructure failure class rather than the customer communication and data integrity response it requires.

Section 4: Recovery and backfill strategy. The recovery and backfill strategy specifies what events trigger a backfill, what the backfill scope is relative to the failure's detection latency and correctness surface, how backfill operations are sequenced to restore downstream consumers to a correct state in a predictable order, and how downstream consumers and customers are communicated with during the recovery window. Backfill trigger events: the strategy should define which quality failure types trigger an automatic backfill, which require manual authorization, and which do not require a backfill at all because the failure was contained before propagation. An assertion failure that holds a batch in quarantine and is resolved by reprocessing the corrected batch through the normal pipeline path does not require a backfill — the downstream consumer never received incorrect data, so there is nothing to correct. An assertion failure that was not detected before propagation because the assertion did not cover the failure mode — the silent type coercion, the format change — requires a backfill whose scope is the full detection latency window: from the time the failure is estimated to have begun producing incorrect data to the time the pipeline was restored to correct operation. Backfill scope: the scope is bounded by two factors. The forward bound is the restoration time — the time at which the pipeline was confirmed to be producing correct output, typically the time at which the quality failure was identified and the root cause was corrected. The backward bound is the onset time — the time at which the pipeline began producing incorrect output; for failures caused by a specific event (a dependency upgrade, an upstream schema change, a deployment), the onset time can be identified precisely; for failures caused by gradual drift (a value distribution that shifted slowly over many batches), the onset time is an estimate based on the point at which the distribution began deviating from the baseline. The backfill writes should use the same idempotency guarantees required in the normal processing path — the backfill is a reprocessing operation that will write records that were previously written with incorrect values, and the correction write should overwrite the incorrect state rather than append alongside it. Downstream consumer sequencing: the backfill should be applied to downstream consumers in the order of their dependency on the affected pipeline stage — the directly-consuming stage first, the consumers of that stage next, continuing through the dependency graph until all consumers have been corrected; this sequencing ensures that each consumer is backfilled from a correct upstream state rather than from a state that itself requires correction, which would produce a cascade of re-corrections if the sequencing were reversed. Customer communication: for failures that produced customer-visible incorrect data, the communication should specify the time window affected, the nature of the error (what was wrong with the data and why), the correction timeline (when the backfill will complete and correct data will be available), and the customer's recommended action (re-run any reports or exports generated during the affected window after the correction is confirmed complete). The communication should not wait for the backfill to complete before notifying customers — a customer who discovers the discrepancy independently before the company has communicated about it experiences both a data quality failure and a trust failure; a customer who receives a proactive communication before they have independently discovered the discrepancy experiences the same data quality failure but with a company that identified the problem and is fixing it, which is the difference between churn and retention for the customers whose trust is on the margin. Connect this to the vendor dependency governance decision record: when the source of a data quality failure is an upstream vendor's schema change or incorrect data delivery, the backfill strategy must include a protocol for working with the vendor to obtain the corrected data; a backfill that reprocesses data from the pipeline's own storage of the original incorrect records will produce the same incorrect output regardless of how many times the backfill is run; the corrected output requires the corrected source data, which means the vendor governance model must include a defined process for requesting and receiving source corrections that the pipeline can use as the input to the backfill operation rather than the original incorrect delivery.

Section 5: Pipeline reliability review cadence and incident retrospective model. The pipeline reliability review cadence specifies how often the reliability model is evaluated against the actual failure patterns the pipeline has encountered, what metrics are tracked to surface trends in reliability model effectiveness, and how the retrospective findings are translated into changes to the assertion library, the containment model, and the backfill strategy. Three metrics form the core of the reliability tracking model. Data quality assertion failure rate: the number of assertion failures per period, broken down by assertion type and pipeline stage, with a trend toward zero failures interpreted not as "the pipeline is getting more reliable" but as "the assertion library may be insufficiently sensitive" — a mature pipeline in a production environment encounters quality failures regularly; a pipeline whose assertion failure rate drops to near zero over several periods is a pipeline whose assertions have become calibrated to pass the data that is actually being produced, including data that is semantically incorrect but within the assertion's tolerance bounds; the drift from a useful assertion to a tolerance-expanded assertion is the primary mechanism of assertion value decay over time. Correctness surface exposure time per incident: for quality failures that were not contained by assertions before propagation, the time between the failure onset and the detection of the failure, measured in hours; this metric reveals the pipeline's blind spots — the failure modes that the current assertion library does not detect within an acceptable latency — and drives the prioritization of new assertions; an exposure time of six weeks is a signal that the assertion library has no detection capability for the failure mode that produced the six-week window, not that the engineering team failed to monitor adequately within the existing model's coverage. Blast radius per incident: for each incident, the number of downstream consumers whose state required correction and the total reprocessing time required to restore correct state, measured in consumer-hours; this metric drives the prioritization of idempotency improvements and isolation boundary additions — a recurring high blast-radius incident pattern in a specific pipeline stage identifies that stage as the highest-priority candidate for idempotency investment; a pattern where blast radius grows with retry count identifies the absence of idempotency on the write path as the cause regardless of which stage is affected. The quarterly retrospective uses the three metrics to answer three questions: which failure modes did the current assertion library detect within an acceptable window, which failure modes were detected after propagation or after customer discovery, and which failure modes produced a blast radius that required more than the acceptable recovery time? The failures detected within an acceptable window confirm that the assertion coverage for that failure class is working; the failures detected after propagation or customer discovery identify the assertion library's current detection gaps and drive the addition of new assertions targeting those failure modes; the failures with unacceptable blast radius identify the reprocessing path's idempotency gaps and drive the addition of deduplication keys or isolation boundaries at the affected stages. AI sessions where the pipeline's architecture was originally designed or where a specific stage's transformation logic was worked through — sessions where an engineering lead debugged a complex field mapping, discussed the tradeoffs between full-batch and incremental processing, or designed the retry protocol — should be run through the open-source extractor before the quarterly reliability retrospective to recover the original design reasoning that explains why specific assertion types were included, what failure modes each isolation boundary was designed to contain, and what alternative reprocessing strategies were explicitly rejected; this context is the reference for deciding whether a proposed change to the assertion library or containment model is an improvement targeting a newly-discovered failure mode or a regression toward a failure mode the original design explicitly addressed, and without the extracted reasoning the retrospective treats the current design as arbitrary rather than as a deliberate response to the specific failures and tradeoffs the pipeline team was weighing when the design was made.

FAQ

Further reading

• Change data capture decision record — the CDC mechanism that propagates changes from the analytics pipeline to downstream reporting consumers is designed for the incremental update case — the source of truth is correct and the consumer needs to catch up; the correction case, where the source of truth was incorrect and the consumer needs to undo a propagated change, is structurally different and requires explicit support in the CDC model design, typically through a tombstone or compensating event pattern; a CDC model that does not include a reconciliation mode will require manual correction scaffolding for every data quality incident that propagated through the CDC path before detection, converting a 2-hour reprocessing operation into an 8-hour recovery effort that involves constructing and testing a one-off correction workflow outside the normal pipeline path; the CDC model should specify before deployment whether the reconciliation case is supported and what the supported recovery pattern is, because adding reconciliation support to a CDC model after a live incident is significantly more expensive than designing for it at the outset.
• Incident severity classification decision record — the severity framework's categories for data pipeline incidents should distinguish between quality failures that were contained by the assertion layer before propagation and quality failures that were detected after producing customer-visible incorrect data; the first class is an operational incident — a quarantined batch, a reprocessing task, no customer impact — that routes through the normal on-call escalation; the second class is a customer-impacting data integrity incident that requires customer communication, executive notification, and a root cause analysis scoped to why the assertion library did not detect the failure before propagation; a severity framework that treats both classes as operational incidents will apply the response designed for a quarantined batch to a six-week customer data corruption, producing the wrong escalation path, the wrong communication protocol, and the wrong root cause analysis scope; the severity distinction between "contained before propagation" and "detected after propagation" is the most important classification boundary in a data pipeline severity framework.
• Vendor dependency governance decision record — the upstream data sources a pipeline ingests are vendors in the governance framework's sense: they have release cadences, schema change policies, and notification practices that affect the pipeline's reliability regardless of the pipeline team's engineering decisions; a vendor governance model that does not include a change notification protocol for upstream data sources will receive schema changes, format updates, and field deprecations through the pipeline's quality assertion failures rather than through a proactive communication before the migration date; the protocol should specify how the pipeline team subscribes to change notifications from each upstream source, what lead time is required before a breaking change to the source's output format is made, and what the source provides as a corrected delivery when the pipeline's assertions reject a batch that the source acknowledges as incorrect; the governance model's value in the reliability context is not preventing upstream sources from making changes but ensuring that those changes arrive as planned maintenance rather than as undetected data corruption.
• SLA design decision record — the data correctness SLA faces the same measurement challenge as the availability SLA: the commitment is only meaningful if the measurement methodology is defined alongside the commitment; a correctness SLA of "98% of output records are semantically correct over a rolling 30-day window" requires a methodology for sampling and verifying record correctness, and a methodology that relies on customers reporting discrepancies has a measurement latency that is six weeks in the worst case — the pipeline can meet its contractual correctness SLA by the letter while producing customer-visible data quality failures that drive churn, because the SLA measurement methodology did not include the monitoring infrastructure required to detect failures within an acceptable window; the SLA design and the data quality monitoring design must be developed together so that the pipeline's monitoring capability is calibrated to detect the failure modes that would violate the correctness commitment within a window short enough to correct them before the commitment period ends, rather than designed independently and discovered to be misaligned when the first correction-cycle SLA violation is measured retroactively.
• Open-source extractor — the data pipeline architecture decisions that are hardest to recover when a new infrastructure engineer joins or when a pipeline component requires a significant redesign are the decisions about what failure modes each assertion was designed to detect, why the isolation boundaries were placed at the specific stages they were placed at rather than at adjacent stages, what alternative idempotency approaches were considered and why the chosen deduplication key structure was preferred, and what specific production incidents drove the addition of the value distribution assertion that was added six months after the initial deployment; running the ChatGPT or Claude sessions from the original pipeline design conversations and from post-incident architecture reviews through the extractor before a pipeline reliability retrospective or a component redesign recovers the design reasoning that explains the current model's structure — the specific failure modes each decision was designed to prevent and the tradeoffs that were accepted when the design was chosen — which is the context that determines whether a proposed change to the assertion library or isolation model is an improvement targeting a newly-discovered failure mode or a regression toward a failure mode the original design explicitly addressed and whose avoidance has since been forgotten.