The engineering metrics decision record: why the measurement model you chose determines your improvement feedback loop and your Goodhart's law exposure surface

The DORA dashboard that measured the wrong things for eighteen months

A forty-person B2B SaaS company is preparing for a Series A. The incoming VP of Engineering has worked at a company that used DORA metrics and wants to establish them before the diligence process begins. She asks Claude: "How do I set up DORA metrics tracking for an engineering team?" Claude returns a dashboard implementation: a GitHub Actions workflow that records pipeline run timestamps, a Grafana query that aggregates CI/CD pipeline completions per day, a PagerDuty API integration that pulls incident open and close timestamps, and a hotfix label in GitHub that marks PRs deployed within 24 hours of a failed deployment.

The implementation takes two days. The dashboard goes live. The starting baseline is: deployment frequency 2.1 per day, lead time 8.3 hours, change failure rate 6.8%, MTTR 2.4 hours. The VP sets targets for one year out: 5 per day, 3 hours, 4%, 45 minutes. The dashboard is reviewed in monthly engineering all-hands.

Eighteen months later, the dashboard shows: deployment frequency 4.4 per day, lead time 2.7 hours, change failure rate 4.1%, MTTR 34 minutes. The numbers have moved substantially toward the targets. The VP presents the progress at the Series A pitch. An investor asks: "When you say 4.4 deployments per day — is that to production, or does that include staging and feature environments?" The VP does not know the answer from memory; she pulls up the Grafana dashboard and finds that the query is counting all GitHub Actions workflow runs with a successful completion status — including runs on feature branches, staging deployments, and failed production deployments that were retried successfully within the same day (counted as two successful pipeline runs).

The actual production deployment frequency, when the analyst re-runs the query with a filter for the production deploy job and deduplicated by Git SHA per day, is 1.4 per day. The gap between 4.4 and 1.4 is the accumulation of eighteen months of pipeline runs that were never production deployments: feature branch test suites, staging environment promotions, and retry sequences where a failed production build was fixed and re-run in the same day. None of this was intentional — the CI/CD pipeline had simply grown to run more jobs across more environments, and the original query counted all of them.

The lead time measurement had a similar problem. The workflow recorded the timestamp of the first commit in a pull request as the start time. The company had a practice of committing work-in-progress to feature branches multiple times per day during development. A developer who started a feature on a Monday and opened the PR for review on Wednesday would have a "lead time" that started from Monday — measuring the age of the branch rather than the time from PR open to deploy. The actual review-and-merge latency (the component that process changes could address) was averaging 6.1 hours against the reported 2.7 hours because the early exploratory commits were compressing the numerator in the average. PRs opened on the same day their first commit was pushed had the highest apparent lead time reduction over eighteen months; they had also been the branches with the least exploratory work-in-progress, not the ones where process had actually improved.

The hotfix label was applied retrospectively, in the weekly engineering retrospective, by whoever remembered which deploys had caused issues during the previous week. Three engineers applied the label independently using different criteria: one applied it to all PRs that referenced an open incident ticket, one applied it only to PRs explicitly titled "hotfix," and one applied it to all unplanned deploys regardless of whether the deploy resolved a user-impacting incident. The same incident had been attributed to two different PRs across two weeks during a particularly complex outage. The change failure rate of 4.1% was measuring the consistency of the labeling convention as much as it was measuring deployment quality.

The measurement model had been specified at the implementation level — which API calls to make, which timestamps to record, which Grafana queries to write — but not at the semantic level. The AI chat that produced the dashboard implementation produced working code. The code measured something. Nobody had documented what that something was, whether it accurately represented the underlying capability, or how to handle the ambiguous cases that emerge over eighteen months of a growing engineering organization.

When the analyst re-measured all four metrics with a consistent specification — production deployments by Git SHA per day, lead time from PR-open to production deploy, change failure rate from incidents opened within 24 hours of the causally attributed deployment, MTTR from first monitoring alert to incident resolved status — the corrected baseline from eighteen months ago and the corrected current metrics showed genuine improvement: deployment frequency from 0.9 to 1.4, lead time from 5.2 hours to 3.8 hours, change failure rate from 5.1% to 3.9%, MTTR from 1.9 hours to 42 minutes. Real improvement, just less dramatic than the dashboard had suggested, and diagnostic for different interventions than the dashboard numbers implied.

The MTTR target that taught engineers to close tickets faster than to fix incidents

A developer tools company has an MTTR problem. Incidents are running an average of 2.1 hours from alert to resolution. The engineering managers, working with the CTO, set a target of 45 minutes and include MTTR improvement as a component of quarterly on-call performance reviews. The incentive model is straightforward: engineers who resolve incidents quickly are demonstrating strong incident response skills. The CTO asks Claude for advice on operationalizing this goal. Claude suggests: set the target, track weekly averages by engineer and by incident category, celebrate quick resolutions in team retrospectives, run regular game-days to practice response skills.

Within three months, the team's MTTR drops from 2.1 hours to 47 minutes. Six months later it is at 38 minutes, below the 45-minute target. The CTO presents the improvement at a board meeting as evidence that the investment in on-call tooling and game-day exercises has produced measurable results.

Eight months after the target was set, a senior engineer is reviewing incident history as part of a post-mortem for a recurring database connection exhaustion issue. She is trying to understand why the same root cause has produced nine separate incidents over the previous six months, each resolved in under 20 minutes, when the underlying issue clearly has not been fixed. She pulls the incident timeline from PagerDuty for all nine incidents and finds a pattern: incidents 1, 3, 5, and 7 were resolved by the same on-call engineer with MTTR between 8 and 14 minutes. Incidents 2, 4, 6, and 8 were opened by the same engineer as new incidents between 18 and 72 hours after the preceding one was resolved. Incident 9, the one that triggered the post-mortem, lasted 2.8 hours under a different on-call engineer who spent the full time diagnosing the root cause rather than applying the restart-the-connection-pool workaround that resolved incidents 1-8 quickly.

The engineer who resolved incidents 1, 3, 5, and 7 in under 15 minutes had genuinely resolved the immediate symptom each time: the connection pool was exhausted, the service was restarted, connections recovered, the incident was closed. Creating a new incident ticket when the problem recurred was not incorrect — the recurrence was a new symptom event that triggered a new PagerDuty alert. The practice was technically valid within the PagerDuty incident model. It was also the behavior that optimized the MTTR metric for the engineer's quarterly review while deferring the root cause investigation that would have eliminated the incident category entirely. The engineer was not acting in bad faith. The incentive structure had made fast symptom resolution the individually rational optimization target, and there was no measurement or reward for the work that would reduce incident frequency rather than incident duration.

A second pattern appeared in a different incident category. Eleven incidents over the previous nine months showed MTTR under 8 minutes — but eight of those were incidents where the on-call engineer had added a "monitoring" timeline event in PagerDuty immediately after acknowledging the alert, then changed the incident status to "resolved" five to seven minutes later while continuing to watch the metrics. In three cases, the incident had been reopened the following day when the issue recurred. In five cases, the issue had not actually been confirmed resolved when the "resolved" status was set — the engineer had applied the standard remediation step, seen error rates begin to drop, and closed the incident to keep the MTTR number down before the 15-minute observation window confirmed the fix held. None of this was documented in the incident timeline. The PagerDuty record showed a clean resolution.

The Goodhart's law exposure had been present from the moment MTTR was tied to individual quarterly reviews. The metric was a valid proxy for team incident response capability. When it became an individual performance target, it became optimizable through behaviors that improved the metric without improving the capability: ticket splitting that recorded fast resolutions of recurring symptoms, premature resolution that captured fast MTTR before confirming the fix held, and incident categorization decisions that kept certain slow-resolution incident types below the severity threshold that triggered MTTR tracking. None of these behaviors were documented in any policy or procedure. They were learned individually, applied individually, and visible only in aggregate — the MTTR trend looked like genuine improvement until someone read the incident history at the individual ticket level.

The decision that was never written down was not "should we track MTTR" — that decision was correct. It was: what constitutes a single incident versus multiple incidents for MTTR purposes, what evidence is required before marking an incident as resolved, and what the connection between MTTR and individual performance reviews implies for the behaviors the incentive creates. Those three decisions were made implicitly, in the moment, by whoever configured PagerDuty and whoever designed the performance review criteria. Neither decision was documented in an ADR, a runbook, or a measurement policy. The next engineering leader to inherit the system would see an MTTR of 38 minutes and eighteen months of incident data that appeared to show excellent reliability performance.

Three structural properties the engineering metrics decision determines

1. The measurement model specification and the feedback loop ceiling

Metrics without a measurement specification produce dashboards that cannot be compared across time, teams, or systems. The specification requires four components: the event definition (what constitutes one occurrence of the measured thing), the scope boundary (which events are included and excluded), the aggregation model (how individual events are combined into the reported metric), and the ambiguous case decisions (how specific real-world events that don't fit the standard definition are classified).

For deployment frequency: the event definition must specify "successful promotion of a software artifact to the production environment where it serves end-user traffic." This definition excludes staging deployments, feature branch builds, and failed production deployments that were retried within the same day. A failed deploy that is fixed and redeployed within two hours counts as one deployment event (the eventual success), not two (the failure and the retry), because the countable unit is a change reaching production users, not a pipeline execution. A canary deployment that reaches 5% traffic and is halted due to elevated error rate counts as one deployment attempt with a change failure — not as a deployment that never occurred — because production users were exposed to the change. Blue-green deployments count as one event when traffic is switched to the new environment, not when the new environment is built. A feature flag that enables a new code path for a subset of users without a corresponding deployment event counts as a deployment event at the time the flag is enabled in production, because the change in production behavior occurs at flag enable time regardless of when the binary was deployed. Each of these ambiguous cases must be resolved before the metric can be compared across teams or over time.

For lead time: the start time definition is a hypothesis about which latency is most actionable. "First commit in the branch" measures branch age including exploratory work-in-progress and fails to distinguish long-lived feature branches from quick fixes. "PR opened for review" measures the review-and-merge portion of the delivery pipeline — the stage where process changes (reviewer availability, review queue depth, approval requirements) can move the number. "First commit after PR is opened" measures iteration latency between review feedback and merge. "CI pipeline started after merge" measures automation latency. A team that believes review queue depth is the binding constraint should measure "PR opened" to "deployment." A team that believes test suite runtime is the constraint should measure "merge" to "deployment." Measuring "first commit" to "deployment" conflates three latency categories and makes all three harder to improve because the measurement cannot attribute a change in lead time to a change in any specific stage.

For change failure rate: the failure definition determines what the metric measures. "Incidents opened within 24 hours of a deployment" requires a consistent incident creation policy — who creates incidents, for which severity levels, and how incidents are attributed to deployments (causal attribution based on the incident timeline vs. temporal proximity of the most recent deployment). "Hotfixes deployed within 48 hours" requires agreement on what a hotfix is: a PR that explicitly reverts the failing change, any PR that references an open incident ticket, or any unplanned deployment that follows an incident regardless of causal connection. "Rollbacks within 1 hour" is precise for fast-detection failures but misses delayed failures — a deployment that introduces a subtle data corruption issue detected 72 hours later. Each definition produces a different failure rate for the same incident history, and none is universally correct. The choice is a decision about which failure mode the team most wants visibility into, not about which definition is objectively accurate.

For MTTR: the measurement model must specify the incident start time and the incident end time, the incident identity rule, and the handling of related incidents. Incident start time options: monitoring alert trigger (accurate but limited to failures that monitoring catches before users are impacted), customer report (delayed, measures the time after users were already affected by an average that depends on the severity and visibility of the failure), or on-call engineer observation of anomalous metrics before any alert fires (the most accurate start time but requiring timestamp discipline during incident response when people are focused on resolution). Incident end time options: fix deployed (optimistic — the fix may not have resolved the issue), fix confirmed by monitoring returning to baseline (accurate but requires a defined baseline and a defined observation window), or "all clear" message sent in the incident channel (the most common implementation and the most gaming-susceptible, because the "all clear" is sent by whoever is on call with whatever timing judgement they apply). The incident identity rule — what constitutes one incident versus multiple incidents — is the most consequential specification for preventing ticket splitting. The minimum-integrity rule: incidents with the same root cause within a 24-hour window are the same incident regardless of how many tickets are opened; reopening a resolved incident within 24 hours requires extending the original MTTR rather than starting a new MTTR clock.

2. The Goodhart's law exposure surface and the measurement integrity model

Every metric has a proxy relationship with the underlying capability it is supposed to represent. Deployment frequency is a proxy for the team's ability to deliver working software in small, low-risk batches. Lead time is a proxy for organizational responsiveness to requirements and bug reports. Change failure rate is a proxy for the team's pre-production quality confidence. MTTR is a proxy for the team's combined detection, diagnosis, and remediation capability. When the proxy is optimized directly — particularly when it is tied to team goals or individual performance evaluation — the proxy diverges from the underlying capability in ways that are predictable rather than surprising.

The Goodhart's law exposure is proportional to the gap between what the metric counts and the behavior that would actually move the metric through genuine improvement. MTTR improves through better detection (monitoring coverage, alert quality, MTTD reduction), better tooling (runbooks, deployment rollback automation, observability), better process (clear incident declaration criteria, role-based response, communication protocols), and more resilient systems (circuit breakers, graceful degradation, reduced blast radius per failure). MTTR also "improves" through ticket manipulation (close and reopen, split incidents, defer "all clear" confirmation to create shorter apparent resolution windows), measurement gaming (close tickets quickly based on the standard remediation step before confirming the fix held, observe issues informally before declaring incidents to delay the MTTR start time), and scope reduction (redefine severity thresholds so fewer events qualify as incidents, exclude certain incident types from the MTTR calculation). The metric cannot distinguish between these two paths. The measurement integrity model specifies the audit mechanisms that separate legitimate metric improvement from proxy optimization.

The measurement integrity model has four components. First, the incident identity definition: incidents with the same root cause within a 24-hour window are the same incident regardless of ticket count; the audit mechanism is a monthly review of incidents that were reopened within 24 hours, identifying whether the reopening was for a genuinely independent failure mode or a recurrence of the same root cause. Second, the resolution validation requirement: P1 incidents require a 15-minute observation window after the remediation step is applied before the incident can be marked resolved; the observation window is recorded in the incident timeline as a mandatory "observing" status before the "resolved" status is set. Third, the performance review isolation: MTTR is a team-level and system-level signal, not an individual skill signal; individual on-call engineers should not be evaluated on their personal MTTR average, because the optimization path from individual evaluation produces ticket splitting and premature resolution rather than system-level improvement; individual on-call quality signals should measure runbook contribution, post-mortem quality, and follow-up task completion rate — behaviors that improve the system rather than the metric. Fourth, trend against a consistent baseline rather than point-in-time targets: a target of "MTTR under 45 minutes" creates the Goodhart's law trap the moment engineers discover the 45-minute threshold in their performance criteria; a trend of "MTTR improving by 10% per quarter against the same incident category mix and measurement model" creates a comparison baseline that is harder to game because gaming must be applied consistently across quarters, not just in the quarter where the target must be hit.

The Goodhart's law exposure review should be documented for each metric before the metric is deployed. The documentation asks: what specific behaviors would improve this metric number without improving the underlying capability? For each behavior: how would we detect it in the data? What is the prevalence threshold at which it signals gaming rather than coincidence? Who is responsible for running the audit? Publishing the audit mechanism alongside the metric communicates that the organization measures genuine improvement, not metric performance, and creates a detection capability for the behaviors that would otherwise only surface in post-mortems.

3. The metric scope and the diagnostic value

DORA metrics correlate with high team performance; they are not causal inputs into it. The correlation is well-established in the research: elite performers exhibit high deployment frequency, low lead time, low change failure rate, and low MTTR simultaneously. Measuring deployment frequency does not cause a team to perform at the elite level; it creates a visibility surface on which the team's current delivery performance becomes visible. The diagnostic value of the metric — its ability to tell you where to invest to improve the underlying capability — requires the ability to trace the metric's value to specific process or system properties that can be changed. A deployment frequency of 1.4 per day without a lead time breakdown by stage is a score without a diagnostic signal. A deployment frequency of 1.4 per day with a lead time breakdown showing 0.8 hours in CI build, 1.1 hours in test execution, and 2.4 hours in review is a diagnosis: the review stage is the largest contributor to lead time, and investment in reviewer availability, review tooling, or approval process simplification has the highest expected return.

The diagnostic signal for each metric requires a secondary breakdown that the metric alone cannot provide. For deployment frequency: lead time breakdown by stage (build, test, review, merge-to-deploy) identifies the binding constraint. A team with fast builds and fast tests but long review latency has a process problem, not a tooling problem. A team with fast review and fast merge but long test execution has a test infrastructure problem, not a process problem. For MTTR: MTTD (mean time to detect) and time-to-action (time from alert acknowledgment to first remediation step) are the decomposed signals. An MTTR of 45 minutes with MTTD of 38 minutes and time-to-action of 7 minutes points to monitoring investment as the highest-leverage intervention — if incidents are running for 38 minutes before anyone knows about them, better runbooks will not change MTTR significantly. An MTTR of 45 minutes with MTTD of 2 minutes and time-to-action of 43 minutes points to runbook and tooling investment. For change failure rate: failure mode categorization (infrastructure failures, application logic failures, data migration issues, configuration errors, third-party dependency failures) and point of failure detection (caught by pre-production tests, caught by canary monitoring, caught by production monitoring, reported by customer) identify which prevention layer has the highest gap. A change failure rate with 70% of failures caught only by customer reports points to monitoring coverage investment. A change failure rate with 40% of failures from data migration issues points to database change management investment.

The scope boundary for the metric determines which comparisons are valid. Industry benchmarks from the DORA research program — the elite/high/medium/low tier definitions — conflate different deployment cadences (weekly release cycles for enterprise software, dozens of daily deploys for consumer products), team sizes (a 5-person team with 8 daily deploys vs. a 300-person organization with 800 daily deploys), and metric definitions (the survey self-reports on deployment frequency, which means the survey respondents are applying their own definition of "deployment" rather than a consistent specification). Using industry benchmarks to evaluate a team's performance is only valid when the team's metric definition is confirmed to match the definition implied by the benchmark cohort. Benchmarks are useful for orientation — "are we in the right order of magnitude?" — and invalid for goal-setting, because hitting a benchmark number through any means (including through the proxy optimizations that Goodhart's law predicts) produces a passing grade on the benchmark without improving the underlying delivery capability that made the benchmark cohort high-performing.

The most valuable comparison is the team's own trend over time against a consistent measurement model. A deployment frequency trend from 0.9 to 1.4 per day over six months, measured with the same event definition and scope boundary throughout, is interpretable: what changed during those six months, what interventions were applied, and which of the lead time stage breakdowns moved? The trend comparison is also the comparison most resistant to Goodhart's law gaming, because gaming must be sustained consistently across the measurement period — a spike in gaming in Q3 relative to Q2 is visible in the data as a discontinuity in the trend, not a smooth improvement.

Five ADR sections the engineering metrics decision record needs

1. Metric selection and the improvement hypothesis

Document which metrics the team has selected and the specific improvement hypothesis each metric is testing. The DORA four (deployment frequency, lead time for changes, change failure rate, MTTR) are the standard starting point for software delivery performance. Teams often need additional metrics for capability gaps that DORA does not measure: MTTD (leading indicator for MTTR — if MTTD is high, monitoring investment is the highest-leverage intervention; if MTTD is low and MTTR is still high, runbook and tooling investment is the intervention); mean time between failures (MTBF — the incident frequency complement to MTTR, which is the other half of the reliability picture); cycle time from requirement to first commit (the lead time leading indicator — if cycle time is long but lead time is short, the constraint is upstream of the coding session in the requirements and design process); and on-call burden (pages per engineer per week, which is a leading indicator for team burnout and the attrition that undermines every other delivery metric).

The improvement hypothesis format forces the team to articulate the causal theory behind each metric. "We believe deployment frequency is constrained by our 22-minute test suite runtime, and that reducing it to under 8 minutes will increase deployment frequency from 1.4 to 4+ per day" is a testable hypothesis. If the test suite is reduced to 7 minutes and deployment frequency does not increase, either the hypothesis was wrong (the constraint is elsewhere — perhaps in the review process or the approval gates) or the measurement model is inaccurate (the change in test runtime did not change lead time as measured because lead time was already dominated by the review stage). The hypothesis also defines the intervention scope: the team knows which specific change to make and how to measure whether it worked. Without the hypothesis, a team with a 22-minute test suite and 1.4 daily deployments has a metric and a number but no theory about what to change.

Document the metric selection exclusions alongside the selections. Metrics that were considered and rejected are as important as the ones selected. "We considered tracking code review cycle time but decided against it because our team is too small for the metric to be meaningful (3 engineers, not enough samples for statistical stability); we will revisit when we exceed 8 engineers." "We considered tracking unit test coverage but rejected it because it is highly gameable (easy to write low-value tests that improve the number without improving the quality gate), and the change failure rate with a failure mode breakdown gives us more accurate visibility into pre-production quality." The exclusion rationale prevents a future team member from adding a metric that the team already decided was low-value.

2. Measurement model specification

For each selected metric, document the event definition, scope boundary, aggregation model, and ambiguous case decisions in enough detail that a new engineer can implement the measurement from scratch and produce the same numbers as the existing implementation.

The event definition for each DORA metric has a required precision level. "A deployment" is not sufficient — "a successful promotion of a service artifact to the production environment serving end-user traffic, identified by a unique Git commit SHA that was not present in the previous production deployment of that service" is sufficient. The commit-SHA deduplication prevents retry sequences from counting as multiple deployments. The "production environment" qualifier excludes staging, preview, and canary environments where the artifact serves test traffic or internal users. The "serving end-user traffic" qualifier excludes artifacts deployed to maintenance mode or deployed but behind a feature flag that is not yet enabled for any user.

The scope boundary specifies which services are in scope for aggregate deployment frequency and which are tracked separately or excluded. Core product services (the services that directly handle user-facing requests) are always in scope. Infrastructure services (database migrations, configuration management, infrastructure-as-code changes) may be in scope if changes to them have a direct deployment event in the CI/CD pipeline and are tracked separately to prevent them from diluting the product delivery signal. Internal tooling services (admin dashboards, data export jobs, analytics pipelines) are included if they have an incident impact path to user-facing functionality, otherwise excluded. The GitOps case requires special handling: a GitOps reconciliation loop may produce dozens of "deployment-like" events per day without any change to the desired state; the event definition must specify that only changes to the Git-controlled desired state count as deployment events, not reconciliation runs that apply the same desired state to diverged cluster state.

The aggregation model specifies the statistical summary. Deployment frequency is reported as a rolling 7-day median of daily production deployment counts rather than a daily average, because the median is less sensitive to outlier days (a major incident response day where the team deploys twenty hotfixes, or a holiday week with zero deployments) and because the rolling window smooths the signal without hiding week-to-week trends. Lead time is reported as the 75th percentile rather than the median, because the median can be stable while the tail — the long-running PRs that are blocking — is growing; the 75th percentile is more sensitive to process problems that affect a minority of deploys. Change failure rate is reported as a 30-day rolling rate rather than a weekly rate to provide enough deployment volume for the percentage to be statistically meaningful for teams deploying fewer than 20 times per week.

The ambiguous case decisions table should be published in the ADR and updated when new cases emerge. A representative set: a deployment that passes all CI checks but fails a post-deployment smoke test and is rolled back within 10 minutes counts as a deployment event (the artifact reached production) with a change failure (the smoke test failure is the rollback trigger). A database migration that is run as a separate CI job before the service deployment counts as part of the same deployment event as the service deployment that required it. A deployment where a bug is discovered 6 days later and fixed with a forward fix (not a hotfix label, but causally attributable to the deployment) is retroactively classified as a change failure if the causal attribution is documented in the post-mortem and the deployment is updated in the incident attribution record within 7 days of the post-mortem.

3. Goodhart's law exposure review

Before finalizing the measurement model, document each metric's Goodhart's law exposure: the specific behavioral adaptations that would improve the metric number without improving the underlying capability, and the audit mechanism that detects each adaptation.

For deployment frequency: the primary exposure surfaces are inflated pipeline runs (counting staging deployments, failed retries, and non-functional commits), commit inflation (splitting a single logical change across multiple small commits and PRs to inflate the count), and environment scope creep (gradually including more non-production environments in the deployment count as the organization adds environments). The detection mechanism is a quarterly audit of the deployment event composition: what percentage of recorded deployment events correspond to production Git SHA changes vs. other pipeline events? A declining ratio signals scope creep or pipeline inflation. The audit is run by the platform engineering team rather than by the individual teams whose metrics are being measured.

For MTTR: the primary exposure surfaces are ticket splitting (closing and reopening incidents with different symptom descriptions to record multiple short MTTRs instead of one long one), premature resolution (marking incidents resolved before the observation window confirms the fix held), severity underclassification (recording events below the severity threshold that triggers MTTR tracking to exclude slow-resolution incidents from the metric), and MTTD gaming (delaying incident declaration after informal observation to compress the apparent MTTD while the actual user impact continues). The detection mechanisms: ticket splitting is detected through a monthly review of incidents resolved in under 10 minutes that resulted in a same-root-cause incident within 24 hours; premature resolution is detected through a review of incidents where the first customer contact or monitoring alert after resolution is within 30 minutes of the "resolved" status being set; severity underclassification is detected through a quarterly review of all "below threshold" events against the severity criteria, with reclassification authority held by the incident review board rather than the on-call engineer.

For change failure rate: the primary exposure surface is attribution gaming — applying the hotfix label to PRs that fixed issues unrelated to the triggering deployment, not applying the label to PRs that should have it because the engineer believes the deployment wasn't the "real" cause, and reverse-gaming where high-performing teams avoid deploying changes that might fail in order to keep the rate low (also known as the "batch and slow" failure mode where change failure rate looks excellent because the team is not deploying frequently enough to accumulate failures). The detection mechanism is a cross-reference audit between the change failure rate, deployment frequency, and MTTR: a team with very low change failure rate and low deployment frequency may be avoiding deploys to protect the metric. The correlation between deployment frequency and change failure rate should be monitored — in high-performing teams, frequent deployments should be associated with low change failure rate; if a team's change failure rate drops as their deployment frequency drops, the change failure rate improvement may be a selection effect rather than a quality improvement.

4. Feedback loop cadence and diagnostic signal

Document the review cadence and the diagnostic signal that makes each metric interpretable at each cadence level. DORA metrics are most valuable at three timescales: weekly retrospective, monthly trend, and quarterly cross-team. Each timescale has a different audience, a different question, and a different diagnostic signal.

Weekly retrospective: the engineering team reviews the current week's metrics against the rolling 4-week baseline. The question is: what specific events moved the metric this week? A lead time spike on Wednesday should be traceable to a specific PR that had a long review cycle (and the retrospective question is why that PR had a long review cycle — was it unclear, was the reviewer unavailable, did it require multiple rounds of feedback on a complex change?). An MTTR spike should be traceable to the specific incidents that week, and the retrospective question is whether the MTTR reflected a detection gap, a diagnostic gap, or a remediation gap. The weekly retrospective does not require formal dashboards — it requires the team to look at the events behind the numbers, not just the numbers. This is the timescale at which the diagnostic signal is most valuable and most actionable.

Monthly trend: engineering managers review the most recent 4-week metric trend against the prior 4-week period and the prior year's same 4-week period. The question is: are the interventions we applied last month producing the expected metric changes? If the team invested in test parallelization and the lead time's test execution stage has not moved, either the parallelization was not implemented correctly or the binding constraint was not the test execution stage. The monthly cadence is the feedback loop that validates or invalidates the improvement hypothesis. The documentation of the improvement hypothesis is the reference against which the monthly trend is evaluated — without the documented hypothesis, the monthly trend is a number that moved without a theory about why.

Quarterly cross-team: the VP of Engineering or CTO reviews the full DORA suite plus additional reliability metrics (SLO compliance, error budget consumption, on-call burden) across all teams. The question is: which teams need investment, which are operating at a constraint that requires organizational resources to resolve, and where is the organization's aggregate delivery capability moving relative to the prior quarter? The SLO error budget is the reliability companion metric to the DORA stability metrics: change failure rate and MTTR measure the team's response to failures; the error budget measures the cumulative user impact of all reliability events against the organization's reliability commitment. A team with excellent MTTR but a fully consumed error budget is resolving incidents quickly while having too many of them — the quarterly review should surface this pattern.

The diagnostic signal for the quarterly review is the cross-team comparison normalized by service risk profile. A payments processing service with a 0.9% change failure rate and a content delivery service with a 4.1% change failure rate are not directly comparable — the payments service has a higher blast radius per failure, which should rationally produce more conservative deployment practices and lower failure rates. The quarterly review should compare each team's metrics against a baseline appropriate for the risk profile of the systems they operate, not against a single organizational target that treats all services as equivalent.

5. Metric sunset and scope evolution

Document the conditions under which a metric should be retired, replaced, or redefined, and the process for updating the measurement model when the underlying system changes.

Metrics sunset when the improvement hypothesis has been confirmed and the metric is stable above the target threshold with the binding constraint having moved elsewhere. A team that improved deployment frequency from 0.9 to 6.2 per day by reducing test suite runtime and simplifying the review process is no longer learning from deployment frequency as an active improvement target — the constraint has moved to something that deployment frequency doesn't measure. The metric transitions to a health check (alert if deployment frequency drops below 3.0 per week, indicating a regression in the delivery pipeline) rather than an active improvement target. The health check threshold should be set below the achieved performance level with enough headroom to catch meaningful regressions without generating false alarms from normal week-to-week variation.

Metrics become invalid as active trend comparisons when the underlying system changes in ways that break the measurement model. A deployment strategy migration from a monolithic release cycle to per-service independent deployments changes the deployment frequency denominator — aggregate frequency increases not because delivery throughput improved but because the deployment unit got smaller. The measurement model update must specify whether frequency is now reported per-service (allowing per-service trend comparison) or as an aggregate (requiring a rebaseline rather than a trend comparison across the migration boundary). A severity threshold change that reclassifies certain incident types from "below threshold" to P2 will increase the apparent change failure rate and MTTR for the same underlying failure frequency — the historical trend is broken at the threshold change date and comparison requires noting the reclassification. The on-call rotation structure change affects MTTR measurement when the new rotation includes a longer handoff window between the alert and the first responder — if MTTR is measured from the alert timestamp and the new rotation has a 5-minute acknowledgment SLA rather than the previous 1-minute SLA, MTTR will increase by approximately 4 minutes for all incidents regardless of how quickly they are resolved after acknowledgment.

The scope evolution policy specifies which system changes require a measurement model update and which can be absorbed without disrupting trend comparisons. Adding a new microservice to the deployment scope requires updating the deployment frequency scope boundary documentation but does not require rebaselining — the new service's deployments are tracked separately for a 90-day calibration period before being added to the aggregate trend. Changing the CI/CD tooling (migrating from one pipeline platform to another) requires a 30-day parallel measurement period to confirm that the new tool produces the same event count for the same deployments as the old tool before the historical trend comparison is resumed. A new monitoring system that detects incidents more rapidly (reducing MTTD from 35 minutes to 3 minutes) will appear to worsen MTTR if MTTR is measured from the first alert — the team is now declaring incidents 32 minutes earlier than before, so the MTTR clock starts 32 minutes earlier for the same failure events. This is a measurement model change: the scope of what MTTD captures has changed, and the MTTR trend must be interpreted against the MTTD change context rather than as a regression in resolution capability. Document these transitions as they occur, with the date of the change and the expected direction and magnitude of the metric impact, so that future trend analysis can account for them.

The new technical leader inheriting a team with DORA metrics and no measurement model documentation cannot determine: whether the deployment frequency counts production deployments or all CI pipeline runs, what start time is used for MTTR, whether the change failure rate is based on hotfix labels or incident attribution, or whether any of the historical trend represents genuine improvement or proxy optimization. These are the decisions made in the initial metrics adoption session that are invisible in the dashboard configuration if not documented in an ADR. The Grafana query shows numbers. It cannot show why the measurement definition was chosen, how ambiguous cases are classified, what the Goodhart's law exposure audit found, or whether the trend comparison is valid across recent system changes.

Further reading

• CI/CD pipeline decision record — deployment frequency and lead time are measured from CI/CD pipeline events; pipeline architecture determines which events can be attributed to specific production changes and which are environment builds, retries, or non-functional runs
• Deployment strategy decision record — change failure rate is a function of deployment strategy; blue-green, canary, and feature-flag strategies produce different failure modes and different deployment event definitions for DORA measurement
• SLO error budget decision record — the error budget is the reliability complement to DORA's change failure rate and MTTR; error budget consumption measures cumulative user impact while DORA metrics measure per-incident response capability
• On-call rotation decision record — MTTR starts when an incident is declared; on-call structure and acknowledgment SLA determine the latency between alert and first human response that precedes any resolution work
• Chaos engineering decision record — controlled chaos experiments produce MTTR measurements under known failure conditions; experiment results are the clearest diagnostic signal for where MTTR is constrained between detection, diagnosis, and remediation
• Test strategy decision record — change failure rate is a function of pre-production validation coverage; the test strategy decision maps which failure modes are caught before production and which remain in the change failure rate numerator
• Documentation strategy decision record — measurement models require the same documentation rigor as operational runbooks; an undocumented measurement model diverges in implementation across teams within 6–12 months, making cross-team metric comparisons invalid
• GitOps decision record — GitOps provides an audit trail of desired-state changes that is more precise for deployment frequency measurement than CI pipeline events, which may include reconciliation runs that apply the same desired state without any underlying change
• The new CTO onboarding problem — inheriting a team's DORA dashboard without the measurement model: "4.4 deployments per day" is uninterpretable without knowing whether it counts all environments, all CI pipeline runs, or only successful production promotions of changed artifacts
• Decisions never written down — the measurement model specification, the ambiguous case decisions, the Goodhart's law exposure review, and the metric scope boundary are made in the initial metrics adoption session and invisible in the dashboard configuration if not documented in an ADR