What is the difference between a feature flag and a canary deployment?

A feature flag and a canary deployment both enable gradual exposure of new behavior to a subset of users, but they operate at different layers and have different rollback mechanics. A feature flag is a runtime code branch: the application evaluates the flag on each request (or per session, or per user) and executes different code paths based on the flag's value. Flag-off is near-instantaneous — the flag service returns false, the old code path executes, no deployment is required. The rollback happens in seconds to minutes (depending on flag evaluation cache TTL) without touching the deployment pipeline. A canary deployment is an infrastructure-level pattern: a subset of the fleet runs the new code version while the rest runs the old version. Traffic is split at the load balancer or service mesh level — some requests are routed to canary instances (new code), others to stable instances (old code). Canary rollback requires rebalancing traffic away from the canary instances, which takes minutes to tens of minutes depending on the deployment system, and may require a full deploy to drain the canary. Feature flags and canary deployments are complementary, not competing. A common pattern is to deploy the new code to the entire fleet (so all instances are running the same binary and there is no version skew) and then use a feature flag to control which requests exercise the new code path. This gives you the infrastructure simplicity of a single-version fleet with the traffic control of canary routing, and rollback is a flag change (fast) rather than a fleet rebalance (slower). The distinction matters for the rollback mechanics section of the progressive delivery ADR: if rollback means flag-off, the rollback time is seconds; if rollback means canary drain, the rollback time is minutes; if rollback means full deploy revert, the rollback time is 10–30 minutes. Documenting which mechanism is appropriate for each type of regression and how each mechanism is executed is the core of the rollback mechanics section.

What should the promotion criteria include at each rollout tier?

Promotion criteria must specify four things at each tier transition. First, the signal metrics: the specific measurements that are expected to change if the feature has a regression — error rate, p99 latency, conversion rate, payment processing latency, bounce rate, or whatever is most directly affected by the feature's behavior. These must be identified before the rollout begins, not inferred after a regression is observed. A metrics list that says 'general system health' is not actionable. A list that says 'checkout conversion rate (baseline 3.2%), payment error rate (baseline 0.3%), checkout p99 latency (baseline 840ms)' defines exactly what to measure and what normal looks like. Second, the minimum observation window: how long the feature must run at each tier before the signal metrics are considered stable enough to evaluate. The window is a function of traffic volume at the tier and the effect size you want to detect. At 1% of 10,000 DAU, you get 100 events per day. Detecting a 5% relative increase in error rate (from 0.3% to 0.315%) at 95% confidence with 100 events per day requires approximately 12–14 days of observation — 'looks clean after 24 hours at 1%' provides almost no statistical power to detect modest regressions. At 10% of 1,000,000 DAU, you get 100,000 events per day, and a 24-hour window provides meaningful signal. Document the minimum window per tier as both a time duration and a minimum event count. Third, the acceptable degradation threshold: the maximum degradation at each signal metric that is acceptable for promotion. The threshold should be expressed as an absolute or relative change from baseline — 'error rate must be below 0.5%' or 'p99 latency must not increase more than 10% above baseline.' The threshold should reflect what is actually acceptable for the product and the SLO, not what is convenient for the release schedule. If a 10% increase in p99 latency will breach an SLO error budget, the promotion threshold must be tighter than 10%. Fourth, the approval process: who can approve promotion at each tier, and whether promotion at some tiers can be automated (automated at 1%→5% if metrics are within threshold after minimum window, requires team lead approval at 50%→100% because the blast radius changes qualitatively at full rollout).

What is a flag cleanup commitment and why does it matter?

A flag cleanup commitment is a documented agreement about when and how a feature flag will be removed from the codebase after it reaches 100% rollout. It matters because flags that reach 100% without a cleanup commitment tend to remain in the codebase indefinitely, accumulating as technical debt that becomes progressively more expensive to remove. The cleanup commitment has four components. First, the cleanup deadline: a specific date or event trigger after which the flag removal work must begin. Common triggers are 'N weeks after reaching 100%' or 'before the next major release.' A deadline without enforcement is not a commitment — it must be tracked in the team's task management system at the time the flag reaches 100%, not added retroactively when the flag is discovered during a cleanup sprint. Second, the dead code definition: an explicit list of the code paths, functions, branches, and components that execute when the flag is false (the old behavior). After 100% rollout with no flag-off events in production, these code paths are dead — they will never execute in production unless someone deliberately flag-offs. The dead code definition is the specification for what gets deleted during cleanup. Without it, engineers removing the flag must reconstruct the off-path from the code, which is the archaeology that makes flag cleanup feel risky — 'I'm not sure what will break if I remove the else branch.' Third, the production safety check: a verification step before cleanup that confirms the off-path is not reached in production. If the flag service provides per-variant evaluation counts, the false-branch evaluation count should be zero for at least N consecutive days before cleanup begins — this confirms that no production code path is evaluating the flag to false. If the flag service does not provide per-variant counts, the safety check must be constructed from application metrics: the code paths that only execute in the false-branch should show zero executions in monitoring. Fourth, the flag service cleanup procedure: after the code removal is deployed and verified, the flag must be deleted from the flag service to prevent future engineers from accidentally re-enabling it. Leaving a flag in the flag service after the code that evaluated it is removed creates a confusing artifact — the flag appears live in the flag service but has no effect in the code — that new engineers interpret as a live flag they must understand before touching.

How should a progressive delivery ADR handle rollback when the flag and the deploy have different scopes?

The most dangerous gap in most progressive delivery decisions is the assumption that flag-off is a complete rollback when the release contains both flag-controlled and non-flag-controlled changes. The flag scope definition — which behaviors are controlled by the flag and which are deployed unconditionally in the same release — must be explicit in the ADR before rollout begins. A flag that controls a checkout UI component may be deployed in the same release as: a new payment processor API endpoint URL (environment variable change, unconditional), a new database index (unconditional schema change), a change to an API response schema that downstream services have already been updated to consume (the downstream updates are irreversible even if the flag is turned off), or a dependency library upgrade (unconditional, applies to all code paths). For any of these deploy-coupled changes, flag-off does not revert them. The rollback mechanics for a mixed release must document each category of change separately: flag-controlled changes (rollback = flag-off, seconds to minutes), deploy-coupled changes that are reversible (rollback = full deploy revert, 10–30 minutes — and verify which of the deploy-coupled changes this reverts and which it does not), and deploy-coupled changes that are irreversible (there is no rollback path — the ADR must document that this change cannot be rolled back and what the consequence of that irreversibility is if a regression requires rollback of the flag-controlled changes). The safest pattern is to separate flag-controlled changes and deploy-coupled changes into different releases: deploy the unconditional changes first (database migrations, dependency upgrades, API endpoint changes), verify them in production independently, and then deploy the flag-controlled code changes in a subsequent release. This separation ensures that flag-off is a complete rollback for the flag release, because the only changes in the flag release are flag-controlled. The ADR should document which release strategy was chosen (mixed release vs. separated releases) and why, and if a mixed release was chosen, must enumerate all deploy-coupled changes and their individual rollback status.

2026-07-25 · ~22 min read · Decision records

The progressive delivery decision record: why the rollout model you chose determines your blast radius control and your rollback latency

Progressive delivery decisions are made during three sessions that never document the strategy — the flag creation session that picks a rollout percentage without documenting the promotion criteria or the rollback threshold, the graduated rollout session that advances through tiers based on "looks good" judgments without recording the observation window or which behaviors are flag-controlled versus deploy-controlled, and the flag cleanup session that either removes the flag years later with no dead-code definition or abandons it at 100% indefinitely. What none of these sessions produce is the blast radius model (which users are affected at each rollout tier and the maximum impact of a regression at that tier), the promotion criteria (which metrics must reach which thresholds after how long an observation window before advancing), the rollback mechanics (what rollback means when the flag scope and the deploy scope are different), or the cleanup commitment (which code paths are dead after full rollout and when they get deleted) — the four gaps that turn a 5-minute flag-off into a 47-minute incident and a KYC flag into a 14-month archaeology project.

The checkout flag where flag-off didn't stop the payment errors

A SaaS company ships a redesigned checkout flow. The new flow restructures how the checkout form collects payment information, removes an intermediate confirmation screen, and integrates a new inline address validator that calls a third-party API during form submission. It is built behind a feature flag so that the rollout can be controlled and aborted quickly if conversion rate drops or payment error rate climbs. The flag is configured in the flag service with a 1% initial rollout. The team watches the dashboard for 48 hours at 1%, promotes to 5%, watches for 24 hours, promotes to 10%, and then — at the end of a Thursday — promotes to 50% to capture the weekend traffic with the new experience. At 50%, the payment error rate on the checkout endpoint spikes from 0.3% to 4.1% within 20 minutes.

The on-call engineer pages in, sees the error rate spike, and flags off the checkout feature flag within 4 minutes of the alert firing. The flag service confirms the flag is now globally off. Payment errors do not drop. They stay elevated at 3.8%. A second alert fires: the API latency on the payment processor webhook endpoint is also degraded. The on-call escalates to the backend team. Over the next 43 minutes, the team works through the payment processor logs and traces the 3.8% error rate to a configuration mismatch: the payment processor's API endpoint URL had been changed from the sandbox environment URL to the new production endpoint URL in the same release as the checkout flag. The URL change was an environment variable update applied unconditionally — not flag-controlled. The new checkout flag controlled the frontend form fields and the address validator, but the payment processor initialization used the environment variable, which was the new production URL regardless of whether the flag was on or off.

With the flag off, users were being routed to the old checkout UI while the payment processor was still initialized with the new production API endpoint. The old checkout UI was calling the old payment processor API (configured in the codebase), but the environment variable override was pointing the processor client to the new production URL, which expected a slightly different request format than the old checkout UI was sending. The 3.8% error rate (down from 4.1% — the flag-off had removed the new checkout form, which was sending the most malformed requests) was entirely from the old checkout UI sending old-format requests to the new-format production endpoint. Fixing the payment error rate required a full deployment to revert the environment variable to the original URL, which took 28 minutes to build, test, and promote through the staging environment. The total incident duration was 47 minutes from the first alert to payment error rate returning to baseline — despite a flag-off within 4 minutes.

The post-mortem identified the root cause as "the flag scope didn't cover the environment variable change." The remediation item was "ensure flag scope is documented before rollout." No remediation item produced an ADR capturing the blast radius model, the promotion criteria that the team had followed informally, or the flag scope definition that would have flagged the environment variable change as deploy-coupled rather than flag-controlled. The decision that was never written down was not just the flag scope — it was the entire rollout model: which metrics the team was watching, what thresholds they were using, what they intended to do if metrics degraded, and what "rollback" meant given that the release contained both flag-controlled and deploy-coupled changes.

The KYC flag at 100% for 14 months

A fintech company's "enhanced_kyc_flow" flag is created in Q3 2024 to gate a new know-your-customer verification flow that adds biometric identity verification to the account signup process. The old flow uses document upload only; the new flow adds a live selfie capture and a liveness check via a third-party biometric API. The flag is created, rolled out to internal users, then to 1%, 5%, 10%, 25%, 50%, 75%, and 100% over three months. At 100%, the conversion rate of the enhanced flow is slightly lower than the old flow (the selfie step has friction) but the fraud rate on verified accounts is 34% lower — an acceptable trade. The flag reaches 100% in Q4 2024. The team moves on to other projects.

In Q1 2026 — 14 months after the flag reached 100% — the fintech company's flag service provider announces deprecation and end-of-life, requiring migration to a new SDK within 90 days. The engineering team audits all active flags to plan the migration. "enhanced_kyc_flow" appears in 23 code locations across 4 services: the web signup service (8 locations), the mobile API service (6 locations), the identity verification service (5 locations), and the fraud scoring service (4 locations). In all 23 locations, the flag is evaluated the same way — if enhanced_kyc_flow is true, take the biometric path; if false, take the document-only path. In all 23 locations, the flag has been returning true for 14 months. The false (document-only) branch has not been executed in production for 14 months.

The migration engineers need to understand three things before they can migrate or delete the flag: what the false branch does (is it dead code that can be deleted, or is it a fallback path that must be preserved), whether removing the false branch will break anything (is any code path in production reaching the false branch through a mechanism not visible in the flag evaluation logs), and whether the flag service contract for enhanced_kyc_flow on unavailability is fail-open (return true, biometric path) or fail-closed (return false, document-only path), because this determines whether a flag service outage during migration could expose users to the document-only flow during what is supposed to be a biometric-only period. None of this is documented. The 14 months of "the flag is on and working" have generated zero written records of the off-behavior, the cleanup intent, or the availability contract. The engineers must read 23 code locations across 4 services and interview the original engineers (2 of whom have left the company) to reconstruct the off-path. The archaeology takes 3 weeks of a senior engineer's time and still ends with uncertainty about whether the document-only fallback path is safe to delete in the fraud scoring service, which performs different risk calculations for biometric-verified vs. document-only accounts and has side effects on the fraud score in the database.

The engineering leader inheriting this system faces a flag that 14 months ago represented a well-considered decision about fraud rates and conversion trade-offs, and now represents an opaque archaeological site whose removal requires the same level of effort as its original implementation. The flag lifecycle decision — cleanup deadline, dead code definition, availability contract — was the four-line entry in an ADR that would have made the 3-week archaeology a 30-minute cleanup task. It was never written down.

Three structural properties the progressive delivery decision determines

1. The blast radius model and the rollout tier definition

The blast radius at a given rollout tier is the maximum number of users exposed to a regression and the maximum impact of that regression per user during the observation window at that tier. The abstract "1% rollout" communicates a fraction; the blast radius model communicates consequences. A 1% rollout for a product with 10,000 DAU exposes 100 users to any regression. A 1% rollout for a product with 1,000,000 DAU exposes 10,000 users. If the feature affects the payment flow and a regression causes payment failures at a rate of 5% of payment attempts, the blast radius at 1% of 1,000,000 DAU is 10,000 users × average payment attempts per day × 5% failure rate = a concrete number of failed payments per hour. This translation from fraction to impact is the blast radius model, and it determines whether a 1% tier is a meaningful safety gate or a gesture.

The rollout unit determines whether a user receives a consistent experience across sessions and requests. User-based rollout evaluates the flag once per stable user identifier and caches the result for the user's lifetime in the system — the user always sees the same variant, in every session, until the rollout percentage changes or the user identifier changes. User-based rollout requires a stable user identifier at flag evaluation time, which means unauthenticated users (visitors who haven't logged in) cannot be targeted by user ID. If the feature affects pages that unauthenticated users can reach, user-based rollout either excludes them (they all see the control) or requires a separate identifier (a persistent cookie or device fingerprint) with its own stability guarantees and privacy implications. Session-based rollout re-evaluates the flag at session start and caches the result for the session duration — a user can see different variants in different sessions if the rollout percentage changes between sessions. Session-based rollout is appropriate for features where session-to-session consistency is not important (background infrastructure changes, algorithm changes where the user doesn't directly observe the variant). Request-based rollout re-evaluates on every request and is almost never appropriate for user-facing features, because a user can see the new variant on one page load and the old variant on the next, which produces confusing UX for any feature that changes visible behavior.

The targeting population before percentage rollout is the risk management lever that the blast radius model most underutilizes. Internal users first — a feature exposed only to the company's own employees has a blast radius limited to the internal team, which can absorb regressions and report them directly. Beta users or early-access cohorts next — a small, self-selected population that is actively monitoring for changes and is more likely to report issues than to churn. Geographic segment next — one region before global rollout, which limits blast radius geographically and creates a natural control group in other regions for comparison. Percentage rollout last. Each targeting phase has its own blast radius calculation and its own promotion criteria, and the phase transitions are the highest-leverage checkpoints in the progressive delivery process. A feature that ships to 5% globally has a larger blast radius than a feature that ships to all users in a single low-traffic region — the percentage may be lower but the exposure is less controlled.

The flag service infrastructure decision determines which rollout units are available: some flag services support stable user-based targeting with consistent hashing; others support only request-level evaluation; others support complex rule-based targeting (users in this org, users on this plan tier, users in this country). The rollout unit and targeting population documented in the progressive delivery ADR must be achievable with the flag service chosen.

2. The promotion criteria and the rollback threshold

Promotion criteria without explicit thresholds and observation windows are social conventions, not safety gates. "Looks good at 1% for 24 hours" is a social convention. "Error rate below 0.4% (baseline 0.3%) after minimum 72 hours and 10,000 checkout events at the 1% tier" is a promotion criterion. The difference is that the first can be met by not looking at the metrics, and the second cannot. The checkout incident in the first story involved promotions from 1% to 5% to 10% that each "looked good" for 24 hours — but at 1% of that product's traffic, 24 hours may not have produced enough payment events to distinguish a 0.3% baseline error rate from a 0.4% early-regression signal at any meaningful confidence level. The promotion from 10% to 50% was the first tier where the traffic volume was high enough to detect the regression quickly — and it did, within 20 minutes.

The minimum observation window at each tier is a function of event frequency and effect size. For a feature in the checkout flow of a product with 1,000,000 DAU and a 3.2% checkout conversion rate: at 1% rollout, the feature is active for approximately 10,000 users per day, of whom approximately 320 reach checkout. Detecting a 10% relative increase in checkout error rate (from 0.3% to 0.33%) at 95% confidence requires approximately 2,000 checkout events in the new variant — at 320 events per day, that is 6 days of observation minimum. "24 hours at 1%" is not a meaningful gate for detecting modest checkout regressions at this traffic level. The minimum observation window must be calibrated: calculate the expected event count per day at the tier's rollout fraction, determine the effect size you want to reliably detect (10% relative increase? 50%? 100%?), and use a power analysis to determine the minimum sample size at that effect size and confidence level. This analysis takes 30 minutes and prevents the "we watched for 24 hours and it looked fine" failure mode.

The rollback threshold is deliberately more sensitive than the promotion threshold. If the promotion criterion for the 5%→10% transition is "error rate below 0.5% after 48 hours and 5,000 events," the rollback threshold at the 5% tier should be "error rate exceeds 0.45% at any point after 1,000 events." The tighter rollback threshold — below the promotion threshold — means the system will rollback on signals that might be noise before confirming promotion on a longer observation. The cost of a false-positive rollback is a minor delay in the rollout; the cost of a missed regression at 5% that is not caught until 50% is proportionally larger blast radius. The asymmetry in the threshold reflects this cost asymmetry: it is cheaper to be conservative about rollback than to be conservative about promotion.

The rollback threshold should trigger automated action, not manual notification. A monitoring alert that pages the on-call when the error rate exceeds the rollback threshold puts the rollback decision in the hands of an engineer who may be asleep, may be unfamiliar with the feature, and who will need to read documentation under time pressure to understand the rollback procedure. An automated rollback trigger — the monitoring system sets the flag to off via the flag service API when the rollback threshold is breached — executes the rollback in seconds, before a human can even acknowledge the alert. The observability platform must support the metrics required by the promotion criteria, and the flag service must provide an API that the monitoring system can call to change flag state automatically for the automated rollback trigger to work.

3. The rollback mechanics and the flag scope definition

Rollback for a feature flag release means at least three different things, and which meaning applies depends on the type of regression and the architecture of the release. Flag-off rollback sets the flag to its off state in the flag service; all flag evaluations after the cache TTL expires return false; the old code path executes for all users. Flag-off rollback is fastest (seconds to minutes, depending on cache TTL) and requires no deployment. It is complete only if all behaviors that need to revert are flag-controlled. Deploy-rollback reverts the code changes in the release to the previous deployed version; all code changes — flag-controlled and deploy-coupled — revert simultaneously. Deploy-rollback takes 5–30 minutes for a typical CI/CD pipeline and reverts everything, including any unconditional changes in the same release. It is the correct rollback when deploy-coupled changes (not just the flag-controlled ones) are contributing to the regression. Database-state rollback undoes schema migrations, data backfills, or external service configurations that were applied alongside the release. Database-state rollback takes hours to days and may require a maintenance window; for some changes (adding a NOT NULL column that has been populated, or migrating data to a new format that downstream services have already consumed), there is no practical rollback path.

The flag scope definition is the document that makes rollback decisions unambiguous during an incident. It lists, for the specific release, every behavior that is flag-controlled (the UI components, the code paths, the API call sites, the configuration keys that the flag gates) and every change that is deploy-coupled (deployed in the same release but not behind the flag, taking effect for all users regardless of flag state). For the checkout incident, the flag scope definition would have listed: flag-controlled — the checkout form field layout, the address validator API call, the confirmation screen removal; deploy-coupled — the PAYMENT_PROCESSOR_API_URL environment variable change. With that definition in place, when the error rate spiked at 50%, the on-call would have known within seconds that flag-off would not revert the payment processor configuration and that a deploy-rollback was required — saving 43 minutes of investigation.

The flag service availability contract is the specification of what happens when the flag service itself is unavailable. If the flag service cannot be reached at evaluation time, the flag client returns either the cached last-known value (if the cache is warm and within TTL), the configured fallback value (true or false, specified at flag creation time), or an error (if the client is configured to throw on unavailability rather than use a fallback). For a payment flow feature at 1% rollout, the correct fallback is almost certainly false (old behavior) — a flag service outage during a 1% canary should not expose the canary feature to 100% of users. For a feature that has been at 100% for six months, the correct fallback is true (new behavior) — a flag service outage should not revert 100% of users to a deprecated old flow that the engineering team has stopped monitoring. The availability contract must be set at flag creation time based on the intended rollout trajectory, and it must be re-evaluated at each major tier promotion, because the correct fallback at 1% (fail-closed) may be the wrong fallback at 100% (fail-open). The incident response playbook for a flag service outage must specify which flags have fail-open vs. fail-closed contracts and the implications of each for user experience and data integrity during the outage.

The database migrations that accompany a progressive delivery rollout require special handling because migrations are irreversible in ways that flag changes are not. A migration that adds a new column used only by the new code path behind the flag is safe: the new column is ignored by the old code path (which never reads it), and dropping the column during rollback is a routine operation. A migration that removes or renames a column used by both the old and new code paths is not safe: the old code path (executed when the flag is off) will fail if the column it relies on is gone. A migration that changes data format or semantics — converting a JSON blob to a structured column — may be irreversible if the new code path has already written data in the new format that cannot be automatically converted back to the old format. The deployment strategy for progressive delivery must ensure that database migrations are backward-compatible with both flag states for the entire rollout window: the new code path can use the new schema, but the old code path (the flag-off path) must also work with the same schema for as long as the flag may be off.

Five ADR sections the progressive delivery decision record needs

1. Rollout model and blast radius per tier

Document the rollout unit (user-based, session-based, or request-based), the targeting population sequence before percentage rollout (internal → beta → geographic segment → percentage-based), the rollout tiers and their specific user fractions, and the blast radius at each tier expressed as the concrete maximum user impact of a regression, not as an abstract percentage.

The blast radius per tier requires knowing the total affected user count, the feature's position in the user journey (how many of those users reach the feature), and the maximum impact per user if the feature regresses (payment failure, data loss, session error, conversion drop). For a checkout feature at 1% of 1,000,000 DAU with a 3% checkout rate and a worst-case "payment fails for all exposed users" regression: 10,000 users exposed × 3% checkout rate × 1 payment attempt per checkout session = 300 failed payments per day at 1%. At 50%: 500,000 × 3% = 15,000 failed payments per day. The difference between tiers is not just scale — it determines whether the team discovers the regression during the canary phase or at the 50% spike that produces an incident. Document the tier at which the blast radius crosses a threshold that the team is unwilling to absorb without additional validation (the "hard stop" tier), and what additional validation is required before crossing that threshold.

The rollout unit documentation must specify the identifier used for user-based targeting (user_id, account_id, org_id — choose the one that represents the entity whose experience should be consistent), how anonymous/unauthenticated users are handled (always control, or targeted by persistent cookie with documented privacy implications), and the cache TTL for the flag evaluation result (how long after a flag change does the new value take effect for an active user session). The cache TTL is the gap between "flag set to off" and "all active sessions executing old code" — a 5-minute cache TTL means 5 minutes of continued new-path execution after a flag-off rollback.

2. Flag scope definition

List explicitly every behavior that is controlled by this flag — code paths, functions, UI components, API call sites, configuration keys — and every change in the same release that is not flag-controlled (deploy-coupled). For each deploy-coupled change, document the rollback mechanism (deploy revert, manual configuration change, irreversible), the estimated rollback time, and any limitations on rollback (a database migration that cannot be reversed, an external service configuration that requires a support ticket to change).

The flag scope definition format is: "this flag controls: [explicit list]. The following changes in this release are NOT flag-controlled and require a deploy-rollback (or have no rollback path) if they contribute to a regression: [explicit list with rollback mechanism per item]." If the explicit list of non-flag-controlled changes is empty (the release contains only flag-controlled changes), say so explicitly — "all changes in this release are flag-controlled; flag-off is a complete rollback." The checkout incident's flag scope definition would have read: "this flag controls: checkout form layout, address validator API call, confirmation screen removal. Not flag-controlled: PAYMENT_PROCESSOR_API_URL environment variable change — rollback requires full deploy, estimated 28 minutes." That one sentence would have saved 43 minutes of incident investigation.

The flag scope definition must be produced before rollout begins, not reconstructed during an incident. It requires a code review of the release changes with the specific question "which of these changes are gated by the flag" — a question that the code reviewers are well-positioned to answer at review time and poorly positioned to answer under incident pressure at 2am. The flag scope definition is the output of that review question, written down and accessible from the incident runbook. The CI/CD pipeline can enforce flag scope documentation as a release gate: flag releases require a flag scope definition in the release notes before promotion to production.

3. Promotion criteria

For each tier transition, document: the signal metrics and their baseline values (the specific measurements that change if the feature regresses, with their current production values before rollout begins), the minimum observation window as both a time duration and a minimum event count for statistical significance at the tier's traffic volume, the acceptable degradation threshold at each signal metric for promotion (absolute or relative bounds), and the approval process (who can approve promotion at each tier, and whether any tier transitions can be automated based on metric thresholds).

The signal metrics must be identified before the rollout, not inferred after a regression. The team that built the feature knows what it changes — which metrics are expected to move if it regresses. "General system health" is not a signal metric. "Checkout conversion rate (current baseline: 3.2% ± 0.2%), payment error rate (current baseline: 0.29% ± 0.05%), checkout p99 latency (current baseline: 840ms ± 60ms)" are signal metrics. They specify what to measure and what normal looks like. With baselines established before rollout, a change from 0.29% to 0.35% payment error rate is distinguishable from noise (baseline ± 0.05% puts the normal range at 0.24%–0.34%), whereas without a baseline, "0.35% looks acceptable" is a judgment made without reference to what was normal before the change.

Automated promotion (the flag service advances to the next tier automatically when the metrics are within threshold after the minimum observation window) is appropriate for low-blast-radius tier transitions (1%→5% where a regression affects hundreds of users) and reduces the time a release spends at low-exposure tiers where the statistical power is low. Manual approval is appropriate for high-blast-radius transitions (10%→50%, 50%→100%) where the human review adds genuine risk assessment value — someone with context about the feature reviews the metrics and decides whether to accept the blast radius increase. The approval process should specify who can approve each transition and what the approval consists of (sign-off in the release tracking system, explicit comment in the monitoring dashboard, or a scheduled meeting for high-stakes releases).

4. Rollback criteria and rollback mechanics

Document the rollback threshold at each signal metric and each tier (tighter than the promotion threshold), the rollback trigger mechanism (manual on-call action, automated flag service API call from monitoring, or explicit rollback runbook step), the rollback procedure for each category of change (flag-off procedure with flag service dashboard/CLI/API command, deploy-rollback procedure with estimated time and verification step, and any irreversible changes that have no rollback path), the flag service availability contract (fail-open or fail-closed on service unavailability, and whether the contract changes at each major tier), and the flag evaluation cache TTL (the time between a flag change and the new value taking effect for all active sessions).

The rollback procedure must be a runbook step, not a concept. "Flag off the feature" is a concept. "Log into the flag service dashboard at [URL], navigate to enhanced_kyc_flow, set the rollout percentage to 0%, click Save, and verify the change by checking the flag service API: GET [API endpoint] — the response should show percentage: 0 within 30 seconds. After 5 minutes (the cache TTL), verify that the payment error rate in [monitoring dashboard URL] has returned to the baseline range of 0.24%–0.34%." That is a runbook step. It can be executed by an on-call engineer who has never touched this flag, at 3am, in under 5 minutes. The payment processor integration specifically requires explicit rollback runbook steps because payment errors have direct revenue impact and the on-call does not have the context to improvise the correct rollback action under time pressure.

The rollback threshold asymmetry should be calibrated at rollout design time rather than ad hoc. A reasonable default: set the rollback threshold at 1.5x the upper bound of the baseline's normal range for error-rate metrics (if the normal error rate range is 0.24%–0.34%, rollback if error rate exceeds 0.51%), and set the rollback threshold at 90% of the lower bound for conversion-rate metrics (if the normal conversion rate range is 3.0%–3.4%, rollback if conversion rate drops below 2.7%). Calibrate these defaults against the product's SLO error budget: if the SLO requires 99.9% payment success and a 0.1% error rate increase will burn the week's error budget in 4 hours, the rollback threshold must be tighter than 0.4% error rate. The billing architecture context is relevant here — features that affect billing or payment flows should have tighter rollback thresholds than features that affect UI presentation, because the blast radius of a billing regression includes failed revenue capture, not just degraded UX.

5. Cleanup commitment

Document the cleanup deadline (a specific date or event trigger set at the time the flag reaches 100%, tracked in the team's task management system), the dead code definition (an explicit list of the code paths, functions, branches, and components that execute when the flag is false — these are deleted at cleanup), the production safety check before cleanup (the verification that confirms the off-path has zero evaluations in production for at least N consecutive days, expressed as a specific monitoring query or flag service metric), and the flag service cleanup procedure (remove flag evaluation from code → deploy → verify no evaluation errors → delete flag from flag service).

The cleanup deadline is the most commonly omitted element. A flag that reaches 100% on Monday with no cleanup deadline is a flag that is still at 100% 14 months later. The deadline must be set at the moment of full rollout and must be tracked in a system that will surface it: a Jira ticket, a GitHub issue, or a calendar reminder with a direct link to the flag and the dead code definition. The deadline should be short enough that the original engineers still remember the context (2–6 weeks after full rollout is typical) and the off-path code is still recent enough that deleting it doesn't feel like archaeology. A flag cleanup that happens 6 weeks after full rollout is a 30-minute code deletion. A flag cleanup that happens 14 months after full rollout is a 3-week archaeology project.

The dead code definition is the answer to "what gets deleted." It lists the specific code sections that execute when the flag is false: function bodies, conditional branches, entire files if the feature was fully encapsulated. The dead code definition makes the cleanup decision a mechanical deletion task rather than a judgment call. Without it, engineers face "which of these else branches are safe to delete" — a question that requires understanding the original intent of each branch, which is reconstructed from the code context (slow), the git history (incomplete), or the original authors (unavailable). The test strategy for the feature should also document which tests exercise only the old path (to be deleted at cleanup) and which tests exercise the new path (to be retained), so the cleanup does not accidentally delete tests that were covering the feature and leaving the post-cleanup code path untested.

The production safety check before cleanup must confirm that the off-path is not reached in any production code path before the code deletion. Even a flag that has been at 100% for 6 months can have a production caller that evaluates it to false: a scheduled job that runs with a hardcoded flag override for testing, an admin endpoint that forces the old behavior for a specific customer segment, or a regional configuration that sets the flag to off for a compliance reason that was added after the global rollout. The safety check — verifying that the false-branch evaluation count is zero for at least 14 consecutive days — catches these cases before the code deletion. Deleting code that is still called in production (even on a rare path) produces a runtime error on that path, which may be silent (caught by a broad error handler), visible (a 500 error on a low-traffic admin endpoint), or catastrophic (a crash in a background job that processes compliance-sensitive data). The safety check is the insurance that makes the code deletion safe rather than optimistic.