The data pipeline orchestration decision record: why the orchestration model you chose determines your failure isolation surface and your backfill complexity ceiling

Published 2026-07-11 · WhyChose

Data pipeline orchestration tool decisions are made in the first data infrastructure sprint, in the AI session where someone asks "what should we use to schedule and monitor our data pipelines?" The answer names a tool — Apache Airflow, because it is the most commonly referenced in the engineering posts that come up when you search for data pipeline orchestration; Prefect, because the engineer asking has heard from a colleague that it is more Pythonic and easier to test; dbt, because the team is already writing SQL transformations and someone mentions that dbt has scheduling capability. The implementation follows the answer. A requirements.txt file is updated, a Docker Compose file is extended, a GitHub Actions workflow is written, a first DAG or flow is created. The system begins producing scheduled data. The decision is not written down anywhere that would survive the engineer's departure from the company.

The orchestration choice is not a scheduling preference. It is a structural constraint that propagates through every subsequent decision about pipeline design, backfill procedures, failure isolation, and the boundary between the data transformation layer and the orchestration layer. A team that chooses Airflow at a specific version without documenting the choice will, at some point between one and two years later, need a capability that requires a version that is significantly ahead of their current pin — because the data engineering ecosystem moves fast, because features that teams eventually need (dynamic task mapping, the TaskFlow API, native sensor deferral) were added in specific versions after many teams had already pinned an earlier release. The migration will be scoped as a one-week upgrade and will take six weeks, because the version delta has accumulated 23 months of API changes across every DAG they have written, every custom operator they have built, and every third-party plugin they have integrated — none of which was documented as a deliberate version-dependent choice at the time it was introduced.

Two ways pipeline orchestration decisions produce the wrong outcome

The Airflow version pin story

A 35-person data team builds a data platform for a B2B logistics analytics product. The platform serves operations managers who need daily reports on carrier performance, shipment delay rates, and warehouse throughput across a network of third-party logistics vendors. In the founding data infrastructure sprint, the senior data engineer — the only person on the team with prior data platform experience — sets up Apache Airflow 2.1.4 because that was the version referenced in the Medium post she used as a reference implementation when building the first DAG skeleton. The version goes into requirements.txt as a pinned dependency. No upgrade policy is documented anywhere. No note in any engineering document explains why 2.1.4 specifically — not 2.2, not 2.3, not the latest release. The implicit rationale is that 2.1.4 was the version in the tutorial, and the tutorial worked.

Fourteen months later, the data engineering team needs to build a pipeline that processes performance data from a variable number of vendor API endpoints. The exact count of active vendors changes weekly as new logistics vendors are onboarded to the network and underperforming vendors are offboarded. In January there are 23 active vendors; by March there are 31; by August there are 19. Each vendor requires the same sequence of API calls, transformations, and quality checks, but the work must be parallelized across vendors to complete within the nightly window. The natural implementation is dynamic task mapping: at DAG runtime, query the vendor configuration table, retrieve the current list of active vendors, and create one downstream task per vendor. This is exactly the use case that Airflow 2.3's @task.expand() and the TaskFlow API's dynamic task mapping feature were designed for.

The current Airflow version is 2.1.4. The dynamic task mapping feature requires 2.3.0. Upgrading is initially scoped at one week by the engineering manager, who has read the Airflow 2.3 migration guide and seen that it is a minor version bump. The actual elapsed time from the decision to upgrade until the upgraded system is running stably in production is six weeks. The reasons accumulate in layers. First, Airflow 2.3 deprecated the old BaseOperator.pre_execute() hook that three of the team's custom operators relied on for connection setup logic — two operators for proprietary vendor APIs and one for an internal data quality scoring service. Each required rewriting the setup logic into the operator's execute() method in a way compatible with the new lifecycle model. Second, six DAGs used the old TaskInstance.xcom_pull() with key='return_value' pattern in a way that relied on implicit behavior that changed between 2.1 and 2.3; each had to be diagnosed individually and refactored to use the explicit return value pattern. Third, a custom SlackWebhookOperator that the team had pinned from a community plugin repository was last updated in November 2021 — it has no 2.3-compatible release and the maintainer has archived the project. The team rewrites the Slack notification logic from scratch using the requests library wrapped in a PythonOperator, which requires updating the alert routing configuration in every DAG that used the old operator. Fourth, two of the most complex DAGs used the old PythonOperator directly with explicit operator instantiation, a pattern that is incompatible with the @task.expand() dynamic mapping feature they actually need — these required full refactoring into the TaskFlow API. The original decision — why 2.1.4, what the intended upgrade cadence was, what the version selection criteria were for this system's use cases — exists in a GitHub pull request comment that is 14 months old and was written in passing by the engineer who created the repository. Nobody knew to look for it. Nobody did look for it until the six-week migration was already underway and someone wondered whether the upgrade would have been less painful if the version pin had been set differently in the first place.

The two-orchestrator boundary failure

A 20-person analytics team at a mid-market e-commerce company uses dbt for SQL transformations invoked by Airflow DAGs. This is the correct model: one orchestration path. The dbt models run as BashOperator steps in Airflow DAGs, each calling dbt run --select model_name with appropriate environment variables for the warehouse connection. The Airflow scheduler manages the dependency order of pipeline stages: an extraction DAG runs first, loading raw data from Shopify and Stripe into the warehouse staging schema; a transformation DAG runs after it completes, calling dbt to transform the staged data into the analytics schema; a notification DAG fires after transformations complete, sending a Slack message to the analytics channel confirming data freshness. This architecture is entirely visible in the DAG code and the dbt project files, but it is not documented explicitly anywhere as a design decision. There is no document that says "Airflow is the orchestrator; dbt is the transformation layer; no model should be scheduled outside of the Airflow transformation DAG." The architecture is implicitly understood by the two engineers who built it.

Six months after the platform launches, a senior data analyst joins the team from a company that used dbt Cloud for end-to-end scheduling. She is a strong dbt practitioner: she knows how to build incremental models, configure sources and exposures, write data tests, and set up dbt Cloud jobs. She is assigned to improve the morning dashboard refresh workflow. The problem she is solving is real: the Airflow transformation pipeline runs at midnight UTC, and by the time the finance and operations teams arrive at 9 AM, the data is nine hours old. They need fresh data at 9 AM. She sets up a dbt Cloud project connected to the production warehouse, creates a dbt Cloud job that runs a subset of models at 9 AM UTC, and begins delivering fresher morning data. She does not ask about the existing orchestration architecture because it is not visible to her from the dbt project files alone — the Airflow DAGs are in a separate repository that she does not have access to in her first weeks. The 9 AM dbt Cloud job solves the analyst team's problem. No alerts fire. No dashboards show anything wrong.

Three months after the dbt Cloud job is deployed, a data engineer adds a new incremental model: daily_transaction_summary, which uses the last value of updated_at in the summary table as a high-watermark timestamp to process only new transaction rows since the previous run. The model is wired into the Airflow midnight pipeline. Because the model is in the dbt project and the dbt Cloud job runs all models in the analytics schema by default, the 9 AM dbt Cloud run includes daily_transaction_summary. The 9 AM run advances the watermark to 9:05 AM UTC. At midnight, the Airflow pipeline invokes dbt run --select daily_transaction_summary. The model reads the watermark — 9:05 AM from the previous day's dbt Cloud run — finds no transaction rows with updated_at greater than 9:05 AM from the previous day, and exits successfully having processed zero rows. The Metabase dashboard that the finance team uses for daily transaction reconciliation shows "no new data" for three consecutive days. The alert on the Airflow pipeline shows all green because the dbt command exited with code 0. The alert on the dbt Cloud job shows all green because the model ran successfully. The Metabase dashboard alert fires on day three when a finance analyst notices the stale row count and escalates. The incident investigation takes two full days. The root cause is the two-orchestrator boundary: two scheduling systems running the same model on different schedules, with no documented contract between them about watermark ownership. The fix requires removing daily_transaction_summary from the dbt Cloud job, resetting the watermark to the correct position, and — most importantly — writing down the rule that should have been documented when the analyst set up dbt Cloud: dbt is the transformation layer, Airflow is the orchestrator, and no model may be scheduled in both.

Three structural properties that pipeline orchestration determines

The task dependency model and its expressiveness ceiling

The expressiveness ceiling is the highest-complexity dependency pattern the framework supports natively without workarounds. Static DAGs define all tasks and all dependencies at DAG definition time — the complete task graph is known before any pipeline execution begins, before any data is loaded, before any runtime state is queried. This is sufficient for pipelines where the set of tasks is fixed in advance: extract data from source A, extract data from source B, run transformation C, run transformation D, validate output E, publish report F. The task graph is the same on every execution. The expressiveness ceiling for static DAGs is the full range of pipeline patterns where the set of tasks does not change between runs.

It is insufficient for pipelines where the task set is determined by runtime data: process each vendor in the current active vendor list, where the list changes weekly. The pre-2.3 Airflow workaround for this pattern was to generate DAG files programmatically — a Python script that reads the vendor list from a database at DAG folder parse time and writes one DAG file per vendor, or one task object per vendor within a single generated DAG. This approach has a scaling problem: as the vendor count grows, the DAG folder contains more generated files, Airflow's scheduler spends more time parsing them, and the DAG grid in the Airflow UI becomes unwieldy. Alternatively, a single operator can iterate internally over all vendors, hiding the parallelism from the orchestrator's visibility entirely — the task appears as one unit of work, and if one vendor's API call fails, the entire task fails and the task's retry logic retries all vendors from the beginning, including the ones that succeeded.

Airflow 2.3 introduced @task.expand() and the full dynamic task mapping feature, which resolves this limitation cleanly. An upstream task returns a list of vendor identifiers, and the downstream task decorated with @task.expand(vendor_id=upstream_result) creates one task instance per element of the list at runtime. The task graph is fully visible in the Airflow UI, individual task failures are isolated to their vendor, and retry logic applies per vendor. This closes the most common expressiveness gap. But it requires running Airflow 2.3 or later, and a team that pinned 2.1.4 in 2022 and never established an upgrade policy may be blocked from using this feature — not by a conceptual limitation of the framework, but by the accumulated API drift between their pinned version and the version that provides the feature they need.

Prefect's flow-centric model makes dynamic fan-out natural from the foundation: a @flow function calls task_function.map(list_of_inputs) where the list is produced by a preceding task, and Prefect creates parallel task runs for each element. The list is determined at runtime; the number of parallel branches is unknown at flow definition time. This is not a version-gated feature — it is the default execution model. Dagster's asset-centric model shifts the abstraction from task graphs to data assets: software-defined assets declare what data they produce, what data they depend on, and how to compute the transformation. Dynamic partitions, added in Dagster 1.1, enable partition sets whose members are determined at runtime — a partition set of "all active vendors" that is computed fresh on each materialization run. The mental model is different from Airflow or Prefect in a way that can be disorienting for engineers who think in terms of task graphs: Dagster asks "what asset do you want to be fresh and by when?" rather than "what task should run at what time?"

The practical consequence of the expressiveness ceiling: it should be assessed at framework selection time against the known pipeline use cases and the anticipated use cases 12 to 24 months ahead. A team building its first five pipelines for fixed-schema batch loads can use any framework — the expressiveness ceiling is not a constraint for static pipelines. A team that knows it will need event-triggered pipelines, dynamic fan-out over runtime-determined sets, or real-time incremental processing should choose a framework whose ceiling fits those needs without a version gate, or commit explicitly to an upgrade cadence that keeps the version current with the feature they anticipate needing. The data lakehouse decision record determines what table formats these pipelines read and write — the orchestration expressiveness ceiling interacts with the storage format's support for incremental processing patterns.

The backfill model and its operational cost per pipeline type

Three pipeline types exist in most data platforms, and each has a fundamentally different backfill model. Understanding which type a pipeline is — and documenting the recovery procedure for that type — is a decision that must be made at pipeline design time, not at incident response time.

Date-partitioned batch pipelines process all data for a logical execution date — the data that arrived or was generated on a specific calendar day. Airflow's native backfill mechanism was designed for exactly this pattern. Running airflow dags backfill --start-date 2026-01-01 --end-date 2026-01-31 my_dag creates 31 DAG runs, one per day, each with execution_date set to the corresponding date. Each run's tasks use the execution_date template variable to filter their source queries to the appropriate date partition. This produces the correct behavior: the run for 2026-01-15 processes January 15th's data, and the backfill fills in 31 days of data in the correct partition order. The backfill can be run with --reset-dagruns to clear previous run state if the runs already exist with incorrect data. This is the model Airflow's design optimizes for, and it is the model most data engineers assume is universal — which leads directly to the failure mode for the second pipeline type.

Incremental pipelines process only records with a timestamp greater than the previous run's high-watermark. The watermark is typically stored in a pipeline state table — a small table with one row per pipeline that records the maximum processed_at timestamp from the previous successful run. On each execution, the pipeline reads the current watermark, processes all source rows with a timestamp greater than the watermark, and updates the watermark to the maximum timestamp of the rows it just processed. When Airflow's backfill creates historical DAG runs for an incremental pipeline, each run reads the current watermark (which is the present day, because all data through today has already been processed), finds no source rows with a timestamp between the watermark and the run's logical execution date (which is in the past), and exits successfully having processed zero rows. All 31 backfill runs complete with status "success," the monitoring system shows a green calendar, and the historical data gap is entirely untouched. This is the silent failure mode of applying date-partitioned backfill logic to an incremental pipeline — it looks correct and is incorrect. Recovery requires a procedure that is non-obvious and carries risk: manually updating the watermark in the state table to the desired backfill start date, running the pipeline, monitoring that source rows are being processed and that the row counts are consistent with expectations, then verifying that the watermark advances to the end of the backfill range. If the watermark is set to a date that is too early, the pipeline reprocesses rows that were already processed in previous runs, which causes duplicate rows in downstream tables if the pipeline's write pattern is not idempotent. The recovery procedure must be documented per pipeline before the pipeline is deployed to production, because the engineer investigating a data gap at 2 AM cannot safely design the watermark reset procedure from first principles under incident pressure.

Event-driven pipelines are triggered by external events — an S3 file arrival, a Kafka message, a webhook from an upstream system — rather than by a schedule. These pipelines have no concept of a date range backfill. Recovery after a logic bug that produced incorrect output requires replaying the triggering events from the source. If the source is S3, reprocessing is possible by re-triggering the S3 event notification on the relevant files, assuming the bucket still contains them and the pipeline's idempotency model handles re-processing correctly. If the source is a Kafka topic, reprocessing requires a consumer group offset reset to the desired starting offset, which requires sufficient topic retention to have the messages available and enough downstream idempotency to handle message re-delivery. If the source is a webhook from a third-party system, the events may simply be gone — the vendor's webhook delivery is not a replay-capable source. The recovery model must be documented at pipeline design time because it determines what infrastructure decisions must be made: whether the S3 bucket must have versioning and long-term retention enabled, whether the Kafka topic must have a retention period long enough to support the maximum expected investigation-to-recovery window, and whether a webhook must be designed with an event receipt table that enables manual replay from recorded payloads. The observability strategy decision record must include monitoring of the pipeline state table and watermark values as first-class observability signals — a watermark that has not advanced in a window longer than the expected run duration is a pipeline failure that must be distinguished from a pipeline that ran successfully and processed zero rows because no new data arrived.

The orchestration boundary and the two-path failure mode

The orchestration boundary is the explicit definition of what the orchestrator owns and what the transformation layer owns. The orchestrator owns scheduling, dependency management between pipeline stages, failure notification, retry logic with backoff, SLA alerting, and cross-pipeline dependency coordination. The transformation layer owns SQL logic, model dependency ordering within the transformation step, data test execution, documentation, and column-level lineage. When the boundary is clear and written down, adding a new tool that also provides scheduling capability triggers a deliberate evaluation: "this tool provides scheduling; does that mean we are adding a second orchestration path, and if so, what is the coordination contract between the two paths?" When the boundary is not documented, a new team member who brings experience with the scheduling capabilities of the new tool can introduce a second orchestration path without anyone having the organizational context to flag the problem.

The two-orchestrator failure mode follows a consistent pattern across teams that experience it. The failure is not immediately visible because both orchestration paths complete successfully on their own terms — each produces successful run status, each fires success notifications, each dashboard shows green. The failure surfaces when a pipeline behavior that depends on execution order or watermark state produces incorrect results. The incremental model story above is one instance. Another common instance is a pipeline where the midnight Airflow run's output table is expected to be the input to a next-day aggregation pipeline, but the 9 AM dbt Cloud run has already modified the output table — adding rows that belong to the current day's data — and the next-day aggregation processes partially-modified data that does not represent a complete daily snapshot. The investigation is expensive in both cases because both monitoring systems show green, the failure is in the interaction between the two systems rather than within either system, and the engineer investigating must first discover that two orchestration paths exist before they can begin investigating the interaction between them.

The correct model is one orchestration path for production data pipelines. If dbt Cloud scheduling is preferred over maintaining Airflow operators for scheduling convenience or because the analytics team wants to manage their own model refresh schedules, the correct decision is to migrate the full orchestration for those models to dbt Cloud and decommission the corresponding Airflow DAGs — not to run both. If both Airflow and dbt Cloud scheduling are kept for different models, the contract between them must be documented explicitly: which models are exclusively owned by which scheduler, what the coordination mechanism is between them (disjoint schedule windows with a documented no-overlap guarantee, or a distributed lock via a pipeline coordination table), and who owns the state when a conflict occurs. The feature flag decision record is relevant here: feature flags can gate pipeline behavior changes during the migration from one orchestration path to another, allowing the new orchestration path to be deployed and tested against production data before the old path is decommissioned. The SLO and error budget decision record provides the specification that forces the two-orchestrator boundary question into the open: if the finance dashboard has a documented SLA that it must reflect data within two hours of midnight, that SLA makes it impossible to have two orchestration paths that both affect the upstream models without a documented coordination contract, because any ambiguity in which path runs first directly affects whether the SLA is met.

Three AI session types that embed orchestration decisions without documenting them

The first data infrastructure session is where the framework selection and version pin happen. The team is building its first pipeline and asks "what should I use to schedule this data pipeline?" The answer is Airflow — it appears in more data engineering blog posts, has more Stack Overflow answers, and has a larger provider ecosystem than any alternative. The installation guide in the answer specifies a version, because the installation command includes a version constraint. The version goes into requirements.txt. No version selection rationale is written anywhere. No upgrade policy is defined. The document that would make the version pin a deliberate choice — "we chose Airflow 2.1.4 because it was stable at selection time; our upgrade policy is quarterly or when a required feature is blocked; our version selection criteria are security patch availability and provider ecosystem compatibility" — is not created because the session did not produce it and nobody asked for it.

The vendor onboarding session is where static DAGs get embedded as the universal pattern for a use case that will eventually require dynamic task mapping. When the team needs a new pipeline for a new data source, they ask how to add it to their existing Airflow setup. The answer describes a new DAG with a set of tasks for the new source. The DAG is written. Nobody in the session asks "should this DAG use dynamic task mapping if the number of processing units could vary at runtime?" because dynamic task mapping was not mentioned in the answer, and the engineer asking did not know to ask. The new DAG is written as a static DAG because that is the pattern in the answer and the pattern of all the existing DAGs. Six months later the requirement changes — the data source now produces a variable number of output partitions per day — and the static DAG cannot handle it without a full rewrite. The design decision that should have been made explicitly ("this pipeline is static-count because the partitions are always exactly five; this other pipeline should be dynamic because the partition count varies") was never made. It was made by default.

The new tool evaluation session is where the two-orchestrator boundary failure is introduced. A team member asks "can I use dbt Cloud to give the analytics team more control over their model refresh schedules?" The answer explains how to set up a dbt Cloud project, connect it to the warehouse, and configure scheduled jobs. The answer is technically correct. But it does not ask "what is your current orchestration architecture?" and it does not answer "what will be the coordination contract between dbt Cloud scheduling and your existing Airflow DAGs?" The implementation happens. The dbt Cloud job runs. The models refresh. The boundary between the two orchestration systems is not documented because the session that set it up did not produce a boundary document, and everyone involved in the session understood only one side of the architecture. WhyChose's open-source extractor is designed specifically for these sessions — it surfaces the original infrastructure decisions embedded in AI chat histories as structured decision records, because the version pin, the static-vs-dynamic DAG choice, and the tool boundary that was or was not discussed are all in the conversation log, in the form they took when the decision was first made, extractable before the next incident makes them visible in the worst possible way.

The five sections of a data pipeline orchestration decision record

The first section documents orchestration framework selection and version governance. This means recording the framework name and the exact version at selection time, not a range — Airflow 2.3.4, not Airflow 2.x. It means documenting the alternatives that were evaluated and the rationale for the choice: what properties of the selected framework were decisive (operator ecosystem, managed infrastructure, dynamic task support, Python-native flow definition), and what properties of the rejected alternatives were insufficient. It means defining the minimum upgrade cadence — quarterly, biannual, or triggered by specific conditions — and the version pin policy: exact version in requirements.txt or a lock file, never a version range that allows silent upgrades to incompatible versions. The upgrade trigger criteria must be explicit: security vulnerabilities in the pinned version, a required feature blocked by the current version, a framework deprecation announcement, or the framework's stated end-of-life date. The section should name who owns the upgrade process and describe the upgrade procedure at a high level — not because the procedure is static, but because naming an owner and a procedure makes it a managed process rather than a deferred task that accumulates until the version delta is large enough to make the upgrade painful.

The second section documents the transformation layer boundary and orchestrator responsibility. This is an explicit enumeration: what the transformation layer (dbt, Spark, pandas) is responsible for, what the orchestrator is responsible for, and what is explicitly out of scope for each. The central rule must be stated directly: a model or pipeline should be scheduled by exactly one orchestration path, and that path is the production orchestrator. If multiple scheduling tools exist in the organization's technology stack — dbt Cloud, Airflow, Prefect Cloud, Dagster — the models owned by each must be mutually exclusive and explicitly listed. Any new tool that provides scheduling capability, before it is adopted, must be evaluated against this boundary: will it introduce a second orchestration path for any model that is already scheduled by the existing orchestrator? The boundary must have an owner, typically the data platform team lead, who is responsible for evaluating new tool adoptions against the boundary rule and for maintaining the list of which models are owned by which scheduling system.

The third section documents the pipeline type classification and backfill model for every production pipeline. Every pipeline that is deployed to production is classified at design time as one of three types: date-partitioned batch, incremental with a high-watermark, or event-triggered. For each pipeline, the section documents the supported backfill mechanism — the exact command or procedure to use, not "use Airflow backfill" but the specific command with the DAG name and the parameter flags appropriate to this pipeline type. For incremental pipelines, it documents the watermark table name, the column that stores the watermark, the SQL or script used to reset the watermark, and who has the database permissions required to run the reset. It documents the recovery procedure for a logic bug requiring historical reprocessing — including the idempotency model (does reprocessing produce duplicates, and if so, what is the deduplication procedure). This section is a living document: every new production pipeline is added to the registry at design time, not retrospectively after an incident reveals that the recovery procedure was undefined.

The fourth section documents the failure notification model and the on-call integration. It specifies how pipeline failures are surfaced: which failure conditions generate Airflow email alerts, which generate Slack notifications via DAG callback or a monitoring tool, and which generate PagerDuty pages because they affect pipelines with data freshness SLAs. It maps the notification to the business impact: a failure of the midnight transaction pipeline means the finance dashboard will not reflect today's transactions, which means the 9 AM finance reconciliation meeting will use stale data. This mapping turns a technical alert ("DAG transaction_summary failed at 01:47 UTC") into a business-impact statement that an on-call engineer can use to calibrate their response urgency. The section should specify the escalation path for data pipeline failures and reference the on-call rotation design — the on-call rotation design decision record covers whether data engineers are in a separate on-call rotation from platform engineers, what the SLA for response is during off-hours, and what constitutes an escalation-worthy data pipeline failure versus one that can wait until business hours.

The fifth section documents pipeline dependency management and shared table ownership. The foundational rule: a table has exactly one pipeline that writes to it and zero or more pipelines that read from it. When two pipelines write to the same table, the write coordination contract must be documented — mutual exclusion enforced by the orchestrator's dependency graph, append-only with disjoint partition ownership, or upsert with idempotent keys that guarantee correctness under concurrent writes. New pipelines that write to tables must be reviewed against the existing ownership registry before deployment, so that two independently-developed pipelines do not both write to a shared table and discover the conflict in production. Cross-pipeline dependencies — pipeline B depends on pipeline A's successful completion before it can produce correct output — must be modeled as explicit sensor tasks in the orchestrator. An ExternalTaskSensor in Airflow, a flow dependency in Prefect, or a cross-asset dependency in Dagster makes the dependency visible in the orchestrator's execution graph and causes downstream pipelines to wait correctly. Time-based schedule assumptions ("pipeline A always finishes by 3 AM so pipeline B is scheduled at 4 AM") are fragile: they are invisible to the orchestrator's dependency model, they fail silently when pipeline A takes longer than expected, and they are never documented in the DAG code — they exist only in the mind of the engineer who wrote them, which means they are undocumented constraints waiting to produce an incident.

The original data infrastructure planning session is in the engineer's AI chat history. The version pin decision — why 2.1.4 specifically, what the upgrade policy was, what the version selection criteria were — is in that conversation log. The discussion of whether dbt is the transformation layer or the orchestrator, or the absence of that discussion, is in the conversation log for the session where dbt was first introduced to the stack. The vendor onboarding pattern that determined whether static or dynamic DAGs would be the default — and whether the team would recognize the dynamic task mapping ceiling before hitting it — is in the conversation log for the session where the second and third DAGs were built. These logs exist. They are searchable. WhyChose's open-source extractor surfaces them as structured decision records before the next version migration, before the next two-orchestrator boundary failure, before the next backfill incident makes the undefined recovery procedure visible at the worst possible moment. The decisions never written down post covers why orchestration choices are the canonical example of founding infrastructure decisions that accumulate invisible constraints — and the new CTO onboarding problem describes what it costs a new data engineering leader to evaluate pipeline health when none of those decisions were ever recorded.

Further reading