What is the difference between schema validation and data quality monitoring?

Schema validation is a subset of data quality monitoring that checks structural conformance: does the data have the expected fields, are the field types correct, are required fields present, do values fall within defined ranges or enum sets. Schema validation catches structural errors — a field that should be an integer contains a string, a required field is missing, a date field contains a value that cannot be parsed as a date. It does not catch semantic errors — a field that contains a syntactically valid integer but represents a logically incorrect value (a customer_id of -1, a revenue figure of 0.00 for a transaction that should have a non-zero amount, or a field that is present and correctly typed but whose value is now populated from a different upstream source than it was last week because the upstream API was restructured). Data quality monitoring is the broader practice that includes schema validation but extends to completeness checks (did all expected records arrive, are there gaps in a time series that should be continuous), accuracy checks (do values match expected distributions or known reference values, do cross-field relationships hold), consistency checks (does the same entity appear with the same attributes across different tables or different time periods, are aggregates consistent with their source rows), timeliness checks (did the data arrive within the SLA window, is the freshness timestamp within acceptable bounds), and uniqueness checks (are there duplicate records for entities that should appear once). The distinction matters for data quality decision records because the quality model determines which types of errors the monitoring system catches, and a model that covers only schema validation will miss the accuracy and consistency failures that produce incorrect downstream outputs without any structural warning. A column rename in an upstream API that maintains the same data type passes schema validation but breaks every JOIN that relied on the old column name — a semantic error invisible to structural checking.

What should the anomaly detection threshold calibration policy include?

The anomaly detection threshold calibration policy must specify five things. First, the baseline window and its exclusions: the historical window used to compute the expected distribution (rolling 28-day average, trailing 90-day percentiles, same-week-prior-year comparison) and which periods are explicitly excluded from the baseline because they represent known anomalous conditions — prior holiday campaigns, outage windows, data backfills, promotional events. Including a Black Friday spike in the rolling average raises the baseline, making future non-holiday volumes appear low and future holiday volumes appear less anomalous; excluding it keeps the baseline calibrated to steady-state behavior and requires a separate mechanism for handling planned spikes. Second, the anomaly classification and response matrix: not all anomalies warrant the same response. A volume spike of 1.5x baseline may warrant an alert and a log entry while continuing to process. A spike of 5x baseline may warrant a halt and an immediate escalation. A volume drop of 50% below baseline may warrant a halt for under-delivery investigation. The response matrix must document the thresholds for each response level and the escalation path for each response. Third, the planned event integration mechanism: a campaign calendar, maintenance window registry, or similar system that allows the anomaly detector to receive advance notice of expected volume changes and either adjust thresholds for the notified window or suppress halt behavior in favor of alert-and-continue. Without this mechanism, every planned traffic spike requires a manual threshold adjustment before the event, which is error-prone and frequently missed. Fourth, the threshold recalibration policy: how often thresholds are reviewed and updated to reflect new seasonal patterns, product growth, or structural changes in data volume. A threshold calibrated when the product had 10,000 DAU may be wildly miscalibrated at 500,000 DAU — not because the data is worse, but because the baseline has shifted. The policy should specify the recalibration frequency (quarterly, or triggered by sustained anomaly alert rates above a threshold), who owns the recalibration, and what the approval process is for threshold changes. Fifth, the false-positive and false-negative tracking: the monitoring system's anomaly detection accuracy must be measured. Every alert that is manually dismissed as a false positive is evidence that the threshold is miscalibrated. Every data quality incident that the anomaly detector missed is evidence that the threshold is too permissive. Tracking false positives and false negatives over time provides the feedback loop for improving the calibration model.

Where should schema enforcement be placed in a data pipeline?

Schema enforcement placement is a decision with a latency-blast-radius trade-off at each boundary. Enforcement at the source API boundary (rejecting API responses that violate the expected schema before ingestion begins) has the shortest feedback loop — the pipeline fails immediately on the first record that violates the contract, and the blast radius is limited to the current batch — but requires the data platform team to have sufficient control over the API client to enforce rejection rather than passive logging, and it requires coordination with the source team to understand when schema changes are intentional versus accidental. This placement is appropriate when the source API is internal and the data platform team can implement strict contract enforcement through shared tooling. Enforcement at the ingestion boundary (validating records as they are written to the raw layer of the warehouse) catches schema violations before they propagate to transformation and consumption layers, but records that fail validation must be quarantined rather than dropped — dropping them silently creates completeness gaps that are harder to detect than schema failures. The quarantine strategy (a dead-letter table, a rejected-records log, a reprocessing queue) must be specified in the quality model because it determines what happens to the data that fails validation. Enforcement at the transformation boundary (validating that transformation outputs conform to the expected output schema before publishing downstream) catches errors introduced by the transformation itself in addition to upstream schema changes, but failures at this boundary have already consumed the transformation compute cost and may leave downstream consumers waiting for data that is not coming. Enforcement at the consumption boundary (validating data before it is used in a report, model, or API response) provides the latest possible detection but also the largest blast radius — errors that reach the consumption boundary may have already propagated through multiple transformation steps, making the root cause harder to trace. The correct placement depends on the pipeline architecture, the source API's contract stability, and the cost of downstream impact. Most mature data platforms enforce at multiple boundaries with different response behaviors: hard rejection at the source boundary for structural violations, quarantine at the ingestion boundary for semantic violations, alert-and-continue at the transformation boundary for output schema drift, and alerting at the consumption boundary for freshness and completeness SLA breaches.

What is a data freshness SLA and how should it be defined?

A data freshness SLA is the maximum acceptable age of data in a dataset at the time it is consumed by a downstream user or system. It has two components: the pipeline SLA (the maximum time from when source data is generated to when it is available in the warehouse or serving layer) and the consumption SLA (the maximum age of data that a downstream consumer will tolerate before the data is considered stale for its intended use). These two SLAs are related but not identical. A pipeline that runs hourly has a pipeline SLA of up to 60 minutes of lag. A dashboard that displays revenue data used for real-time pricing decisions may require a consumption SLA of under 10 minutes — which means an hourly pipeline cannot meet the consumption SLA for that use case, even if it runs perfectly. A dashboard used for weekly reporting may have a consumption SLA of 24 hours, which the hourly pipeline satisfies with margin. The freshness SLA must be defined per-dataset and per-consumer, not globally. A single warehouse with ten downstream consumers may have ten different consumption SLAs, and the pipeline must be designed to meet the most demanding one for each dataset — or the consumers must be explicitly documented as receiving data with a lag that may not meet their ideal freshness requirement. The freshness SLA definition must specify: the measurement point (freshness is measured from the source event timestamp, not from the pipeline run time — a pipeline that runs at midnight and successfully loads data from the previous day has a pipeline SLA of 24 hours, even if the pipeline itself ran in 5 minutes), the monitoring query (the specific query or metric that measures current data age against the SLA threshold), the breach notification path (who is alerted when the SLA is breached, and what the expected response is), and the SLA suspension policy for planned maintenance windows or pipeline outages (what downstream consumers should do when the pipeline SLA cannot be met and how they are notified in advance).

2026-07-25 · ~23 min read · Decision records

The data quality monitoring decision record: why the quality model you chose determines your stale data detection latency and your downstream correctness surface

Data quality decisions are made during three sessions that never document the model — the pipeline build session that adds row-count validation without defining what correct data looks like, the anomaly detection session that sets thresholds on historical averages without calibrating for planned traffic spikes, and the schema enforcement session that validates structure without validating semantics. What none of these sessions produce is the quality dimension inventory (which of the six quality dimensions are monitored, at what granularity, with what enforcement action), the anomaly threshold calibration policy (how thresholds reflect seasonal patterns and planned campaigns, what distinguishes a genuine data error from a genuine traffic spike), the schema enforcement placement (at which pipeline boundary schema and semantic contracts are enforced, with what response on violation), or the downstream protection model (how downstream consumers are shielded from quality failures, and what they receive instead of bad data) — the four gaps that turn a silently renamed JOIN key into 23 days of NULL revenue attribution and a holiday campaign spike into a 4-hour pipeline halt at peak traffic.

The pipeline that validated every row and produced 23 days of broken attribution

A logistics SaaS company runs a daily ETL pipeline that extracts order data from their e-commerce platform's Orders API, transforms it against a fixed mapping schema, and loads it into their analytics warehouse. The pipeline has two quality gates: row count validation (today's order count must be within 15% of the prior 7-day average) and pipeline completion time monitoring (the pipeline must finish within the 4-hour nightly maintenance window). For eight months, both gates pass every night. The data team considers the pipeline reliable.

In month nine, the e-commerce platform ships a breaking change to the Orders API response schema. The field that previously contained the customer identifier — named customer_id in the API response — is renamed to account_id to align with the platform's new unified identity system. The platform's changelog marks this as "non-breaking with deprecation period: customer_id will be populated alongside account_id for 90 days." The ETL pipeline reads both fields and maps account_id to the warehouse column named customer_id. Row counts are unaffected — every order still arrives. Pipeline completion time is unaffected. Both quality gates pass.

The problem is in the mapping: the ETL's transformation step maps account_id (the new primary field) to the warehouse column customer_id, which is correct. But the pipeline also still reads the deprecated customer_id field from the API — which the e-commerce platform is now populating with a different identifier format as part of their identity system migration. The deprecated customer_id field now contains a UUID-format string where the warehouse schema expects an integer. The ETL's schema validation checks that customer_id is present in the API response and non-null — both remain true. The type check was added to the ETL six months ago as a soft warning (log but don't fail), so the UUID-format values load silently with a warning that nobody monitors.

The revenue attribution pipeline, which runs separately, joins the orders data to the customer account table on customer_id. For the 23 days after the e-commerce platform's API migration, every order's customer_id in the warehouse is a UUID-format string that does not match any row in the integer-keyed customer account table. The JOIN succeeds but returns zero matching rows for every new order. Revenue attribution for every customer acquired in the past 23 days is NULL. Customer lifetime value calculations, cohort analysis, and subscription renewal attribution all contain 23 days of NULL customer linkage. The data analysts interpret the NULLs as "new customers with no prior history" — a plausible explanation for recently acquired customers — and do not escalate.

The issue surfaces 23 days after the API migration when a senior data analyst runs a cohort conversion report and finds that the "day-1 activation rate" for a recent acquisition campaign is 0.0% — every customer in the campaign cohort has zero attribution. The investigation traces the NULL attributions to the JOIN failure, which traces to the UUID-format customer_id values, which traces to the API migration. The fix takes 4 hours: correct the ETL mapping, backfill 23 days of affected attribution data, and reprocess the downstream reports. The 23-day window represents $1.4M in orders with incorrect attribution that must be manually reconciled with the finance team.

The decision that was never written down was not just the schema mapping — it was the quality model: which dimensions of data quality the pipeline was responsible for monitoring, what "correct" data meant for each field in the context of its downstream use, and what the enforcement action was when a quality check detected a problem. The ETL's row-count and completion-time gates validated delivery and timeliness but not accuracy. The accuracy check — does the customer_id field in the warehouse contain a value that will produce valid JOIN results in the revenue attribution pipeline — was never defined as a quality dimension, never assigned an owner, and never had a monitoring rule written for it.

The anomaly detector that halted the pipeline at the campaign's peak 4 hours

An e-commerce company's data platform team implements anomaly detection on their order ingestion pipeline in Q1. The detector uses a rolling 28-day average of daily order volume. If any day's order count exceeds 2.5 standard deviations above the 28-day mean, the pipeline halts and pages the on-call data engineer. The team chose 2.5 standard deviations after reviewing six months of historical data: genuine data errors (duplicate batch submissions, malformed API responses that created phantom records) had produced volume spikes of 3x to 5x the daily average, while organic traffic variation had stayed within 2 standard deviations. A 2.5σ threshold was considered conservative and accurate.

The company is ten months old when the anomaly detector is deployed. It has never experienced a major holiday campaign season. The rolling 28-day baseline is built entirely from steady-state traffic — no Black Friday, no Cyber Monday, no holiday promotional campaigns appear in the calibration window. The team's historical spike analysis reflects the error cases they had observed (duplicate loads and phantom records), not the traffic cases they had not yet experienced.

In November, the company launches its first major holiday campaign with email, social, and influencer channels coordinated. The campaign goes live at midnight on a Thursday. By 2:47 AM, hourly order volume has reached 3.9x the rolling 28-day average. The anomaly detector fires: volume anomaly detected, pipeline halted, on-call paged. The on-call data engineer — who works on the data platform team and was not briefed on the campaign launch — receives a page for what appears, from the alert, to be a 4x volume spike that is consistent with the historical signature of a duplicate-load error (the most common prior incident). Standard procedure for a duplicate-load suspicion is to halt processing and investigate before loading potentially duplicate records into the warehouse, because duplicate records in the orders table propagate through all downstream aggregations and require expensive deduplication backfills to fix.

The investigation takes 4 hours. The on-call engineer checks the API source logs, the ingestion pipeline logs, and the row-level sample to determine whether the volume spike represents real orders or duplicate submissions. At 6:52 AM, the engineer contacts the marketing team to ask whether anything unusual was planned for the previous night. The marketing team confirms the campaign launch. The engineer manually approves the pipeline to resume processing, suppresses the anomaly alert for the next 72 hours, and begins the reprocessing backlog for the 4-hour halt window.

The 4-hour halt during peak campaign hours means the inventory management system does not receive the order signals it uses for real-time stock allocation, the personalization engine serves recommendations based on pre-campaign behavioral data for the first six hours of the campaign's peak window, and the morning executive dashboard — reviewed at 8 AM by the leadership team who launched the campaign — shows order volume that ends at 2:47 AM and resumes at 6:52 AM with no indication that a pipeline halt had occurred. The company processes $2.1M in orders during the campaign's peak window, all of which arrive correctly. None of them cause a data quality problem. The anomaly detector halts the pipeline not because the data is wrong but because the quality model had no mechanism to distinguish a genuine traffic spike from a genuine data error.

The data pipeline architecture decision determines what mechanisms are available for the anomaly detector to receive context — whether a campaign calendar integration, a manual override API, or a threshold adjustment workflow exists in the pipeline infrastructure. The quality model decision determines whether the pipeline responds to anomalies by halting (protecting the warehouse from bad data) or by alerting and continuing (protecting downstream consumers from unavailability). Both responses are correct for different anomaly types — a duplicate-load spike warrants a halt to prevent data corruption; a genuine traffic spike warrants alert-and-continue to prevent downstream unavailability. A quality model that uses a single response for all anomaly types will optimize for one failure mode and create the opposite failure mode for the other.

Three structural properties the data quality monitoring decision determines

1. The quality model and the validation scope

Data quality has six dimensions, and the quality model decision is which dimensions to validate, at what granularity, and with what enforcement action when a validation fails. Completeness: did all expected records arrive? Accuracy: do values correctly represent the real-world entities they describe? Consistency: do values agree across sources and across time periods? Timeliness: did data arrive within the SLA window? Validity: do values conform to schema constraints and business rules? Uniqueness: are there duplicate records for entities that should be unique?

Most data pipelines validate completeness (row counts) and timeliness (pipeline completion time) and nothing else. These two dimensions catch the delivery failures — the pipeline didn't run, the source API returned empty results, the batch transfer was truncated. They do not catch the correctness failures — the data arrived but means something different than it used to. The attribution pipeline failure in the first story was an accuracy failure: the values in the customer_id column were syntactically valid (non-null, present) but semantically incorrect (UUID-format values in a field used as an integer JOIN key). Row-count and completion-time validation cannot detect accuracy failures because accuracy is defined relative to the downstream use, not relative to the source structure.

Accuracy validation requires knowing what "correct" means for each field in the context of its downstream use. For a customer_id field used as a JOIN key, "correct" means "present in the customer account table." The accuracy check is a referential integrity check: join the orders warehouse table to the customer account table on customer_id and verify that the match rate is above a threshold (95% of orders must match a valid customer account, accounting for a reasonable fraction of genuinely new customers without prior account history). This check must be defined by someone who understands the downstream use — the data analyst who builds the revenue attribution model knows that the JOIN must succeed for attribution to work. The data platform engineer who builds the ETL may not know what the downstream JOIN dependency is unless it is explicitly documented in the quality model.

Consistency validation catches the class of errors where values are individually valid but contradict each other across time or across sources. An order total that is $0.00 on the first day an order appears in the warehouse and $89.50 on the second day (after a late-arriving update batch is loaded without deduplication) is consistent with no individual validity constraint but is inconsistent across time — the same order cannot have two different totals. A customer record that appears in the CRM export with plan=pro and in the billing system export with plan=free on the same date is consistent with no individual validity constraint but is inconsistent across sources. Consistency checks require multi-source or multi-period queries that are more expensive than single-row validation, which is why they are commonly omitted — but they catch the failures that produce the most expensive downstream corrections.

The quality model must document, for each dataset and each dimension: whether the dimension is monitored (yes, monitored / no, explicitly out of scope), the monitoring rule (the specific query or metric that operationalizes the dimension), the threshold for a quality failure (row count within 15% of 7-day average; referential integrity match rate above 95%; freshness timestamp within 60 minutes of SLA deadline), and the enforcement action on threshold breach (halt the pipeline, quarantine the failing records, alert and continue, or log only). The enforcement action is the most consequential parameter of the quality model — it determines whether quality failures produce data unavailability (halt) or data incorrectness (continue). Both outcomes are harmful; the choice of which harm to accept depends on the downstream consumers' relative tolerance for missing data versus wrong data, which varies by use case and must be explicitly decided.

The observability platform decision determines whether the infrastructure exists to monitor data quality metrics alongside application metrics — whether the same alerting system that monitors API error rates can also monitor pipeline row-count deviations and freshness SLA breaches, or whether the data quality monitoring system is separate and requires separate alerting configuration and runbook maintenance.

2. The anomaly detection model and the threshold calibration surface

Statistical anomaly detection for data pipelines works by comparing current observations to a baseline derived from historical data. The baseline represents "normal," and the threshold determines how far from normal an observation must be before it is flagged. The fundamental limitation of historical baselines is that they can only represent patterns that have occurred in the historical window. A baseline calibrated on 10 months of steady-state traffic cannot represent the holiday campaign pattern because it has never occurred. A threshold set at 2.5 standard deviations above a steady-state baseline will flag the holiday campaign spike as anomalous even though it is entirely correct data.

The calibration policy must address seasonal decomposition: separating the time series into a trend component (the gradual growth of the product), a seasonal component (the recurring weekly pattern — orders are typically lower on Monday and higher on Friday), and a residual component (the unexplained variation that represents genuine anomalies and genuine errors). Anomaly detection should operate on the residual after removing the trend and seasonal components, not on the raw time series. A pipeline that processes 10,000 orders on Friday and 6,000 orders on Monday should not flag Monday as anomalous relative to Friday — the weekly seasonal pattern is expected variation, not a quality signal. Removing it from the baseline before computing the anomaly threshold reduces false positives from expected variation and improves the signal-to-noise ratio for genuine errors.

Planned event integration is the mechanism that the calibration system uses to handle known-in-advance volume changes that fall outside the historical pattern. A campaign calendar that the anomaly detection system reads before each pipeline run allows the detector to suppress halt behavior in favor of alert-and-continue during a campaign window, or to dynamically adjust the threshold based on the campaign's projected volume (if the campaign is expected to produce 3x baseline volume, temporarily raise the halt threshold to 5x and set a lower alert threshold at 2x to catch genuine errors within the expected elevated range). Without planned event integration, every campaign launch requires a manual threshold adjustment by an engineer who has been specifically notified about the campaign — a process that fails whenever the notification doesn't happen, which is reliably the case at midnight during the first major campaign of the company's history.

The threshold calibration policy must also specify the response matrix: what action the pipeline takes at each anomaly magnitude. A volume spike of 1.5x to 2x baseline in the absence of a known planned event warrants a warning log and an alert to the data engineering channel — someone should look, but processing should continue, because the blast radius of loading a few percent of potentially bad records is smaller than the blast radius of halting the pipeline. A spike of 3x to 5x baseline warrants halting new loads into the production warehouse while routing to a quarantine table, alerting the on-call data engineer, and automatically notifying the campaign and marketing channels asking for context. A spike of 10x or above warrants a full halt and immediate escalation, because at that magnitude the spike is consistent with a catastrophic error (a pipeline loop that is processing the same batch repeatedly) rather than an organic traffic event. The response matrix makes the anomaly detection system's behavior explicit and reduces the cognitive load on the on-call engineer who must decide what to do at 2:47 AM with an alert that says "volume anomaly detected."

The pipeline orchestration decision determines whether the infrastructure supports partial halts (quarantine failing records while continuing to process valid ones), conditional routing (send anomalous records to a dead-letter table while loading normal records to production), and campaign calendar integration (read a schedule from a shared calendar API before each run). An orchestration system that supports only binary halt-or-run cannot implement a nuanced response matrix regardless of how well the quality model defines one.

3. The schema enforcement placement and the contract ownership model

Schema enforcement placement determines at which boundary in the data pipeline schema violations are detected and what happens when they are. The four canonical placement options — source boundary, ingestion boundary, transformation boundary, and consumption boundary — differ in detection latency, blast radius, and who owns the enforcement rule.

Source boundary enforcement (validate the API response schema before beginning ingestion) provides the earliest detection and the smallest blast radius: the current batch fails before any records are written to the warehouse. The cost is that the enforcement rule is owned by the data platform team but depends on the source API's schema definition, which is owned by a different team. When the source team makes a schema change — intentional or accidental — the source boundary enforcement fails immediately and noisily, which is the correct behavior. The risk is false positives from intentional schema changes: if the source team adds a new optional field to the response schema without notifying the data team, source boundary enforcement may flag the addition as a violation if the enforcement rule was written to reject any field not in the expected schema. The source boundary enforcement rule must be written to allow additive changes (new optional fields are acceptable) while rejecting destructive changes (removed required fields, renamed fields, type changes). This distinction requires the enforcement rule to encode the data contract's additive-change policy, not just the current schema snapshot.

Ingestion boundary enforcement (validate records as they are written to the raw layer) catches violations that pass the source boundary (because the API response schema was structurally valid but semantically incorrect) and violations in non-API sources (file uploads, database exports, manually constructed datasets). The ingestion boundary enforcement must handle the disposition of failing records: hard reject (the record is not written to the warehouse at all, producing a completeness gap), soft reject with quarantine (the record is written to a quarantine table instead of the production table, preserving it for inspection and reprocessing without contaminating production data), or soft reject with logging (the record is written to production with a quality flag column set to indicate the violation, allowing downstream consumers to filter by quality flag). The quarantine approach is most common for ingestion boundary enforcement because it preserves all data while preventing known-bad records from propagating — but it requires a quarantine table, a reprocessing workflow, and a completeness reconciliation step to ensure that quarantined records are eventually either corrected and loaded or explicitly dropped from the dataset.

The contract ownership model addresses who writes and maintains the enforcement rules. The data platform team can write and maintain enforcement rules, but they may not have enough business context to define accuracy and consistency rules — they know the schema, not the business meaning. The data consumers (the analysts and data scientists who build reports and models on the data) have the business context, but they may not have the technical access or tooling to write enforcement rules against the pipeline. The most robust ownership model is collaborative: the data platform team owns the structural validity rules (schema conformance, type checking, null checks) and the data consumers define the accuracy and consistency rules (referential integrity checks, value distribution expectations, cross-source agreement rules) with the data platform team implementing them in the monitoring infrastructure. This split requires a formal process for consumers to register quality rules — a shared quality rule registry, a PR process against a rules-as-code repository, or a data catalog with quality rule annotations — and a process for notifying consumers when their rules begin failing.

The API contract testing decision addresses the related problem of validating API response schemas in automated test environments. Source boundary enforcement in the production pipeline and API contract tests in the test pipeline solve the same underlying problem — detecting schema changes before they propagate to downstream consumers — but at different points in the change lifecycle. Contract tests catch schema changes before they are deployed; source boundary enforcement catches schema changes that were deployed without test coverage. The two mechanisms are complementary: contract tests for internal APIs where the data platform team can contribute tests to the source service's test suite, source boundary enforcement for external APIs where the data platform team cannot influence the upstream testing process.

Five ADR sections the data quality monitoring decision record needs

1. Quality dimension inventory and validation scope definition

For each dataset owned by the team, document which of the six quality dimensions are monitored (completeness, accuracy, consistency, timeliness, validity, uniqueness), the specific monitoring rule for each monitored dimension (the query or metric that operationalizes the check), the threshold for a quality failure, and the enforcement action on threshold breach.

The quality dimension inventory must be anchored to downstream use, not to the source schema. The question to answer for each field is: "what would go wrong downstream if this field contained an incorrect value?" For a customer_id field used as a JOIN key, the wrong value produces JOIN misses and NULL attributions. The monitoring rule for this field is therefore a referential integrity check against the join target, not just a null-check against the source schema. For a revenue_amount field used in financial reporting, the wrong value produces incorrect revenue figures in executive dashboards. The monitoring rule is a value distribution check — revenue amounts should not spike to 10x the daily average unless a high-value event is known to have occurred, and zero-value revenue records should not represent more than 5% of order volume — plus a cross-source consistency check against the billing system's revenue figures for the same period.

The enforcement action matrix must distinguish between dimensions and magnitudes. Completeness failures (missing records) warrant different enforcement than accuracy failures (wrong values in records that arrived). A 5% completeness gap may be acceptable as a delayed-records situation that self-resolves when the source backfills; a 0.1% accuracy rate on the customer_id JOIN key means 0.1% of revenue is unattributed, which may be acceptable for operational dashboards but unacceptable for financial reporting. The quality model must document these tolerances explicitly rather than treating all failures as equivalent emergencies, because treating a 5% completeness gap with the same response as a 50% completeness gap will exhaust the on-call rotation's alert attention on minor deviations and reduce the urgency response to genuine failures.

The database vendor decision affects which quality dimension checks can be enforced natively at the storage layer (database constraints for validity and uniqueness) versus which must be enforced through external monitoring (application-level checks for accuracy, consistency, and timeliness). Postgres's CHECK constraints enforce value validity rules at write time with zero additional infrastructure; referential integrity enforcement through FOREIGN KEY constraints enforces uniqueness and JOIN validity at write time. External monitoring must cover the dimensions the database cannot enforce natively — primarily accuracy (semantic correctness relative to downstream use), consistency across sources, and timeliness (freshness relative to SLA).

2. Anomaly detection model and threshold calibration policy

Document the baseline model (rolling average, percentile distribution, seasonal decomposition, or a combination), the baseline window (how many prior periods are included), the planned event integration mechanism (how the detector receives advance notice of expected volume changes and what it does with that notice), the response matrix (which response is triggered at which anomaly magnitude for which anomaly type), and the recalibration policy (how often thresholds are reviewed and updated, who owns the recalibration, and what the approval process is for threshold changes).

The baseline model choice determines the false-positive rate for organic traffic variation. A simple rolling average without seasonal decomposition will flag every Monday's order volume as a partial anomaly relative to Friday's peak — the weekly seasonal pattern will appear as weekly quasi-anomalies in the residual. A seasonal decomposition model that removes the weekly pattern before computing the threshold will have a lower false-positive rate for day-of-week variation but requires more historical data to estimate the seasonal component reliably (at minimum one full seasonal cycle — 28 days for weekly patterns, 52 weeks for annual patterns). A model deployed in month one of a product's existence cannot have a well-calibrated seasonal component; it must use a simpler baseline with wider tolerances until sufficient history accumulates, with a documented plan for upgrading the model as history grows.

The response matrix must be specified as a matrix, not a single threshold-and-action pair. A useful matrix has three axes: anomaly type (volume spike, volume drop, schema violation, completeness gap, freshness breach), anomaly magnitude (minor, moderate, severe — defined as specific thresholds for each type), and context (planned event window versus no planned event). The response options are: log only (record the anomaly in the quality log, no alert, continue processing), alert and continue (page the on-call channel, continue processing, expect acknowledgment within 30 minutes), quarantine and continue (route failing records to the dead-letter table, continue loading valid records, alert the on-call), or halt and escalate (stop all loading, alert on-call, escalate to data engineering lead if not acknowledged within 15 minutes). A volume spike of 2x in a planned campaign window might be "log only" while a volume spike of 2x outside a planned campaign window might be "alert and continue." A volume drop of 50% in any context is "halt and escalate" because under-delivery means downstream consumers are receiving incomplete data that will produce incorrect aggregates.

The background job infrastructure decision determines the technical mechanism for implementing the anomaly detection checks — whether they run as pipeline steps (embedded in the pipeline DAG and blocking pipeline progress when they fail), as parallel monitoring jobs (running concurrently with the pipeline and sending alerts without blocking pipeline progress), or as post-load validation jobs (running after the pipeline completes and alerting on the loaded data rather than the in-flight data). Each implementation has different latency characteristics: pipeline-step enforcement catches violations before loading (preventing bad data from reaching the warehouse); post-load validation catches violations after loading (ensuring the warehouse's current state is correct but requiring remediation rather than prevention).

3. Schema enforcement placement and contract ownership

Document which pipeline boundaries enforce schema and semantic contracts, what enforcement action is taken at each boundary when a violation is detected, who owns the enforcement rules at each boundary, the process for registering new quality rules (who can add them, how they are reviewed, how they are tested against historical data before activation), and the process for handling rule violations (who is notified, what the first response is, and how the violation is resolved — correction and reprocessing, or acceptance with documentation).

The contract ownership assignment is the most operationally consequential part of the enforcement placement decision. For each quality rule, the owner is accountable for: defining the rule (translating a business requirement into a specific monitoring query and threshold), maintaining the rule as the source schema and downstream requirements evolve (updating the rule when the source API changes intentionally), and responding when the rule fails (investigating whether the failure is a genuine quality problem or a false positive from the rule definition). Without explicit ownership, quality rules drift: the rule that was accurate when written becomes increasingly inaccurate as the upstream schema evolves, producing false positives that train the team to ignore quality alerts, which trains the team not to notice the genuine quality failures that appear alongside the false positives.

The contract registry is the operational artifact that makes ownership explicit and the quality model auditable. A contract registry can be as simple as a YAML file in the data platform repository that maps each dataset field to its owner, monitoring rule, threshold, and enforcement action — checked by the pipeline at startup and requiring a code review to modify. Or it can be as sophisticated as a data catalog with quality rule annotations, automated rule execution, and lineage tracking that shows which downstream consumers depend on each quality rule. The complexity of the registry should match the complexity of the pipeline: a team running three pipelines with ten downstream consumers can manage quality rules in a YAML file; a team running 300 pipelines with 1,000 downstream consumers needs a more structured system. The decision record should document which approach was chosen and why, so that future engineers know whether a rule that is not in the YAML file is intentionally out of scope or was simply never added.

The database migration strategy is relevant to schema enforcement because database migrations change the schema that the enforcement rules are written against. A migration that renames a column must update the quality rules that reference the old column name, or the quality rules will begin failing immediately after the migration — not because the data quality has degraded, but because the rule definition is stale. The quality rule update process must be integrated with the database migration workflow: a migration that touches a column referenced by quality rules must include a corresponding update to the quality rule definitions, enforced as a migration gate before the migration is promoted to production.

4. Data freshness model and SLA definition

Document the freshness SLA for each dataset (the maximum acceptable age of data in the dataset at the time of consumption, measured from the source event timestamp — not from the pipeline run time), the monitoring query that measures current data age against the SLA threshold, the SLA tier assignment (not all datasets have the same freshness requirement — operational dashboards used for real-time pricing decisions have a different SLA than weekly reporting dashboards), and the SLA breach notification path (who is alerted, expected response time, and what the downstream consumers should do when the SLA cannot be met).

The freshness SLA must account for the distinction between pipeline latency (the time from data generation to data availability in the warehouse) and source event latency (the time from the real-world event to data generation in the source system). A daily ETL pipeline that runs at midnight and completes in 4 hours has a pipeline latency of 4 hours but a source event latency that may be 24 hours if the source system batches events daily. The total freshness latency from real-world event to warehouse availability is the sum of both — up to 28 hours in this case. If a downstream consumer's freshness SLA requires data to be no more than 12 hours old, the daily ETL architecture cannot meet the SLA regardless of pipeline performance improvements, because the source event batching introduces a minimum 24-hour delay. The freshness SLA definition must identify the bottleneck in the total freshness pipeline and document whether meeting the SLA requires architectural changes (streaming ingestion to replace batch ETL, real-time API polling to replace daily export) or whether the SLA must be renegotiated with the downstream consumer to reflect the architectural reality.

The SLA tier assignment creates a priority model for freshness monitoring. Tier 1 datasets (freshness required within minutes — real-time pricing, live inventory, operational dashboards used during active customer interactions) warrant continuous monitoring with immediate paging on SLA breach. Tier 2 datasets (freshness required within hours — daily reporting, end-of-day analytics, batch recommendation model updates) warrant hourly freshness checks with business-hours alerting. Tier 3 datasets (freshness required within days — weekly cohort analysis, monthly financial reporting, historical trend analysis) warrant daily freshness checks with non-urgent alerting. Applying the same monitoring frequency and alert urgency to all datasets regardless of tier wastes on-call attention on non-urgent freshness delays and desensitizes the team to alerts for the truly time-sensitive datasets. The read replica configuration is relevant to freshness because replica lag is an additional source of freshness latency for consumers who query replicas for reporting — the freshness SLA must include replica lag as a component of total data age, not just pipeline latency.

5. Quality failure response model and downstream protection

Document what downstream consumers receive when a quality failure occurs and the pipeline halts or quarantines failing records, how downstream consumers are notified of quality failures that may have already produced incorrect output, the remediation workflow for quality failures (who initiates the investigation, who performs the fix, who validates the corrected data, who approves resumption of downstream consumption), and the decision authority for accepting a known-bad dataset under time pressure (who can make the call to release data with documented quality failures when the cost of delay exceeds the cost of known-bad data).

Downstream consumers have two failure modes when the pipeline delivers bad data: they process it (producing incorrect outputs that may propagate further before detection) or they reject it (using stale data, showing empty results, or failing altogether). A quality failure response model that halts the pipeline at the first detection protects downstream consumers from incorrect data but exposes them to unavailability — the pipeline halt is invisible to consumers who are still querying the dataset and receiving stale data. A model that alerts and continues routes incorrect data to downstream consumers who process it without knowing it is incorrect. Neither outcome is acceptable; the response model must minimize the exposure period for both failure modes and must notify downstream consumers as soon as a quality failure is detected, regardless of whether the pipeline halts or continues.

The downstream notification mechanism must be explicit: which teams are notified, through which channel, with what information (the dataset affected, the quality dimension that failed, the estimated impact on the consumer's use case, the estimated resolution time, and what the consumer should do in the interim). A data engineering team that pages itself when a quality failure occurs but does not notify the analytics team that the revenue attribution reports are based on potentially incorrect data has protected the data pipeline from bad data but has not protected the company from acting on incorrect insights. The downstream notification is as important as the pipeline halt decision, and it should be part of the quality failure response model, not an afterthought in the incident post-mortem.

The decision authority for releasing data with documented quality failures is the most politically sensitive part of the response model. During a holiday campaign's peak hours, the marketing team and the CFO may have stronger opinions about whether the pipeline should resume processing (even with known quality caveats) than the data engineering team does. The quality failure response model must document who has the authority to make that call — releasing known-imperfect data under time pressure — and what the documentation requirement is when that authority is exercised (a logged decision with the known quality issues, the estimated impact, the alternative considered, and the names of the people who approved the release). Without explicit decision authority and documentation requirements, the call is made informally at 3 AM by whoever is on-call, without the context needed to evaluate the trade-off, and without a record that the decision was made so that downstream consumers can evaluate the data with appropriate skepticism. The incident response playbook for data quality failures must include the escalation path and decision authority matrix as a runnable procedure, not a general principle — the engineer on-call at 3 AM needs to know who to call and what question to ask, not a framework for thinking about data governance.