The SLO and error budget decision record: why the reliability target you chose determines your feature velocity trade-off and your incident response ceiling

· WhyChose

SLO targets are set in a single engineering conversation. A founding engineer reads the Google SRE book, notes that consumer products typically fall between 99.9% and 99.99%, picks 99.95% because it sounds appropriately ambitious without being unreachable, configures it in the monitoring dashboard, and moves on. Nobody calculates what 99.95% implies about the monthly budget available for incidents. Nobody calculates how much of that budget the team's deployment cadence will consume. Nobody writes down what the error budget policy is — what happens when 50% of the budget is gone, or 100% of it. Nobody asks whether the SLI being measured actually reflects the user experience being promised.

Three years later, the same team is investigating why feature velocity has dropped to near zero despite no change in team size. The error budget dashboard shows a recurring pattern: the budget exhausts within the first two weeks of every month, triggering a freeze that the team routes around by classifying every deployment as a critical fix. The SLO target is simultaneously too strict to maintain (given the deployment cadence) and too loose to be meaningful (the SLI has not been updated since the founding sprint and does not measure the user-facing behaviors users actually care about). The reasoning behind the original 99.95% choice is gone — it was made in an AI chat session or a Slack thread that nobody can locate now — so the question "should we change this?" is answered by opinion rather than analysis.

The SLO decision record exists to prevent this. It captures not the target itself — the monitoring dashboard captures that — but the reasoning: why this target, what it implies about deployment budget and incident budget, what the SLI measures and what it cannot detect, and what happens when the budget is exhausted. The reasoning is what allows the target to be changed when the context changes: when traffic grows, when deployment frequency increases, when the product adds a tier that requires a stricter commitment, or when the SLI definition is found to be missing a critical user-journey signal. Without the reasoning, every target change is a debate about the right number. With it, target changes are analyses of whether the current constraints still fit the current context.

Two things that happen when the decision is not written down

The invisible SLO violation: eleven months of reported flawless uptime while background jobs silently fail

A 38-person B2B SaaS company built a project management tool for design and marketing teams. Their SLO was set in the founding sprint: 99.9% availability, measured as "HTTP 200 responses from the health check endpoint per Pingdom uptime monitoring." Eleven months after setting the SLO, the CTO reviewed the error budget dashboard before a board meeting and reported that the service had achieved 99.97% availability — better than the committed SLO target with ample budget remaining. The board received a reliability section that described the infrastructure as performing above expectations.

Two weeks after the board meeting, a customer success manager escalated an enterprise customer who had been reporting for three months that "task notifications don't always arrive." The team had logged the complaint as an intermittent issue and closed three support tickets with "please let us know if it happens again." When the investigation was opened properly, it took four hours to find the root cause: the async notification worker — a separate process responsible for sending email and in-app notifications when tasks were assigned, commented on, or completed — had been crashing and restarting intermittently due to a memory leak introduced in a deploy seven months earlier. The crash rate increased with usage volume. By the time the enterprise customer escalated, the worker was restarting approximately every 40 minutes, with a 3-4 minute restart window during which all notifications were queued but not delivered.

The postmortem produced an uncomfortable finding: the SLO monitoring had recorded zero error budget consumption during seven discrete periods in the prior eleven months totaling 34 cumulative hours of background job processing failure. The Pingdom uptime check measured connectivity to the /health endpoint, which checked database connectivity and returned 200. The /health endpoint had no knowledge of the async worker — it ran in a separate Heroku dyno with its own memory allocation and process lifecycle. When the worker crashed, the API tier continued responding normally to all HTTP requests. Users could log in, create tasks, and view the dashboard. They could not receive notifications. The SLI measured the API tier's HTTP availability. The SLI said nothing about the notification delivery behavior that users paid for.

The correct SLI for this product was not "HTTP 200 rate from the API." It was "task notification delivered within 5 minutes of trigger event, measured across all notification types." This SLI required instrumentation that was not in place: a notification delivery event logged at trigger time, a delivery confirmation event logged at delivery time, and a monitoring query that calculated the ratio of confirmed deliveries to trigger events within the 5-minute window. Building this instrumentation was a two-week project. The project revealed that the notification delivery success rate had been below 95% for four of the prior eleven months — a period when the error budget dashboard had shown the service running at 99.97% availability.

The SLI had been chosen in the founding sprint because Pingdom was already set up for infrastructure monitoring and adding a URL to the uptime check took five minutes. Nobody asked: "if this SLI shows healthy, does that mean every user experience we are promising is working?" The answer was no. The question was not asked until eleven months of incorrect reliability reporting had accumulated. The original SLO decision was not written down as a decision — it was a configuration in a monitoring tool that carried no reasoning, no definition of what was being measured, and no record of what it could not detect. When a customer success manager discovered the gap, the team had no documented baseline against which to evaluate whether the SLI needed replacement or augmentation. They did not know whether the original SLI definition was intentional ("we chose to measure API availability, not end-to-end notification delivery, because building the deeper instrumentation was out of scope at founding") or accidental ("we set up what the monitoring tool made easy and never thought about whether it was the right measurement"). Without the decision record, both the reasoning and the scope were lost.

The velocity freeze: a deployment cadence that exhausts the error budget before the first incident of the month

A 32-person consumer fintech operated a subscription payments platform. At founding, the engineering lead had set a 99.95% SLO — 22 minutes of allowed "outage-equivalent" budget per month. The reasoning, recalled from memory eighteen months later, was: "99.9% felt too loose for a fintech handling subscription payments; 99.99% felt unreachable; 99.95% was the number in the middle." No one had written down the reasoning at the time. No one had calculated what 99.95% implied about deployment budget at the team's planned deployment frequency.

For the first fourteen months, the target felt achievable. Traffic was modest, incidents were infrequent, and the rolling Kubernetes deployment completed in 3-4 minutes with acceptable error rates during pod replacement. The error budget dashboard was a reassuring shade of green at the end of most months.

At month 22, traffic had grown from 80,000 to 2.4 million monthly active users. The deployment pattern had also evolved: the team was deploying 4 times per day to support an aggressive feature roadmap. The Kubernetes rolling deployment configuration had been updated six months earlier when a performance engineer added a readiness probe that waited for the in-memory object cache to warm before marking a pod as ready. The warmup phase took approximately 90 seconds. During the warmup phase, the pod served requests from the database directly (cache miss path), which produced approximately 12% higher latency on affected requests and a 3% elevated error rate for cache-miss-sensitive operations. The readiness probe kept pods in the unready state during this warmup window, but the warmup window itself produced the elevated error rate as pods transitioned from warm to warm+ready to receiving traffic.

The full rolling deployment cycle — pod replacement, warmup, traffic onboarding, repeat across 14 pods — took 8-9 minutes end-to-end. During this window, at any given moment, 1-2 pods were in the warmup state receiving traffic at the elevated 3% error rate. The effective fleet-level elevated error rate during a deployment was approximately 3% × (2/14) = 0.4% above baseline, sustained for 9 minutes. Per deployment, the budget consumed was: 9 minutes × 0.004 elevated error rate above the SLO = 0.036 equivalent minutes of budget. But the calculation the team was missing was the aggregate: 4 deploys per day × 30 days × 0.036 minutes per deploy = 4.32 equivalent minutes of budget consumed by deployments alone per month.

But this was not the number that broke them. The number that broke them was the large-incident contribution. In month 22, the team experienced two incidents: a 14-minute database connection pool exhaustion event at a 60% error rate (14 × 0.60 = 8.4 equivalent minutes), and an 18-minute payment processing delay at a 40% error rate (18 × 0.40 = 7.2 equivalent minutes). Total: deployment budget (4.32 min) + incident budget (8.4 + 7.2 = 15.6 min) = 19.92 minutes. Against 22 minutes allowed. The month ended with 2.08 minutes of budget remaining and no freeze.

In month 23, three incidents occurred: a 12-minute Redis failover at 25% error rate (3.0 min), a 9-minute deploy gone wrong that required rollback at 40% error rate (3.6 min), and a 31-minute CDN configuration error at 15% error rate (4.65 min). Total: deployment budget (4.32) + incidents (3.0 + 3.6 + 4.65 = 11.25) = 15.57 minutes. Still under budget. Month 23 ended green.

Month 24: Two incidents totaling 28 minutes of equivalent budget. Budget exhausted on day 19 of the month. Freeze triggered. The engineering manager announced that all non-critical deployments were paused for the remaining eleven days of the month. Four features that were blocked on deployments slipped to the following month. Product asked why the features had slipped. The engineering manager explained the error budget policy. Product asked why the error budget had exhausted. The engineering manager explained the incidents. Product asked what could be done to avoid this happening every month. The engineering manager could not answer — because the analysis required to answer it (how much budget are we consuming on deployments vs incidents, what would the budget look like if we improved deployment reliability, what would the budget look like at a different SLO target) had never been done. The SLO target of 99.95% had been set without any of these calculations, and no document existed that contained the reasoning from which the calculations could start.

The investigation that followed took three weeks. The findings: at the team's current deployment cadence and incident rate, the 99.95% SLO required either (a) reducing deployment frequency to 2×/day (losing half the feature velocity), (b) investing in zero-downtime deployment engineering to reduce the per-deploy budget consumption to near zero, or (c) revising the SLO target to 99.9% (44 minutes/month budget) which would accommodate the current pattern with 5-10 minutes of margin. None of these options were obvious from the error budget dashboard. They required the analysis that should have been done when the SLO target was first chosen — the analysis that would have been documented in an SLO decision record. The original selection of 99.95% was made in an AI chat session where the engineer had asked "what SLO should a fintech subscription payments platform target?" and received a recommendation based on industry norms without any calculation of the team's specific deployment profile or incident history.

Three structural properties that are set at SLO decision time

The SLI definition and its user-experience validity surface

An SLI is a measurement. The measurement has a subject: what property of the system's behavior it captures. The measurement also has a blind spot: the set of user experiences that are absent from the SLI measurement and will not appear in the error budget even when they are degraded. The blind spot is the SLI's validity surface — the boundary of what the measurement can and cannot detect.

The most common SLI blind spot is the tier gap: measuring availability at the API tier while the user experience depends on a background processing tier that runs separately. Email notifications, PDF generation, data export, billing event processing, webhook delivery, and push notification dispatch are all background operations that run outside the API request-response cycle. A user who submits a PDF export request receives an HTTP 200 immediately — the API tier is healthy — and then waits for a background job that will never complete because the job processing queue is saturated or the worker is crashed. The API tier SLI never records the failure. The user experience is fully broken. This is the same pattern as the project management tool's notification worker: the failure mode is structurally invisible to an API-tier SLI.

The second SLI blind spot is the percentile gap: measuring average or median latency while the user experience is determined by tail latency. A service with a 200ms average latency and a 3,200ms p99 latency will pass a "median latency under 500ms" SLI while 1% of requests — which at 1,000 requests per second is 10 requests per second, or 864,000 requests per day — are experiencing 3.2 seconds of latency. This SLI is technically green while a meaningful fraction of users experience unacceptable response times. The latency SLI should specify the percentile at which the threshold applies: "99th percentile latency under 1,000ms" catches the tail latency problem that a "50th percentile latency under 500ms" SLI misses entirely. This connects to the observability strategy decision record: the instrumentation required to compute percentile latency SLIs — histogram metrics with appropriate bucket boundaries — must be designed into the monitoring stack at observability setup time. Teams that set a p99 latency SLO after the fact discover that their current metrics cannot compute it accurately because the histogram bucket granularity was chosen for alerting average latency, not tail latency.

The third SLI blind spot is the synthetic path gap: measuring a synthetic request to a known-good endpoint rather than measuring real user request success rates. Synthetic monitoring verifies that the service can respond to a controlled request; it does not verify that the service is correctly handling the diversity of real user requests, which include edge cases, large payloads, and permission patterns that the synthetic request never exercises. The synthetic /health endpoint check that caused the project management tool to miss eleven months of notification failures is one example. A checkout flow synthetic that submits a fixed test order with a pre-approved payment method is another: it will pass while the real checkout fails for users with subscription renewals, international cards, or specific promotional codes.

The SLI decision record must document each SLI by its subject (what behavior it measures), its instrumentation (how the measurement is produced — from which component, at which granularity, aggregated over which window), and its blind spots (what failure modes would not appear in this measurement). The blind spot documentation is the most important part and the most commonly omitted: it forces the team to articulate what the SLI cannot see, which reveals whether additional SLIs are needed to cover the gaps. A team that can enumerate the blind spots of each SLI knows what it knows and what it doesn't. A team with a single uptime-check SLI and no blind spot documentation does not know whether its SLO reflects user experience or only one tier's HTTP availability.

The error budget and the deployment velocity ceiling

The error budget is simultaneously a reliability metric and a deployment velocity constraint. This duality is frequently described in the abstract — "the error budget lets the team balance reliability work against feature work" — but is rarely calculated concretely at SLO definition time. The calculation that the fintech should have performed before setting their 99.95% target is: given our planned deployment frequency and our current rolling deployment error profile, how many minutes of budget does deploying consume per month? If the answer is more than the monthly budget, the target is incompatible with the deployment cadence without investment in zero-downtime deployment first.

The calculation requires two inputs. First, the per-deploy budget consumption: deployment window duration in minutes multiplied by the effective elevated error rate during the window above the SLO threshold. For a 9-minute rolling deployment with a 3% elevated error rate during pod replacement (and a 0.05% SLO error rate threshold), the elevated error rate above threshold is 3% − 0.05% ≈ 3%, and the per-deploy budget consumption is 9 × 0.03 = 0.27 minutes of equivalent outage budget. Second, the monthly deployment count: deployment frequency per day multiplied by 30. At 4 deploys/day, monthly deployment count is 120. Monthly deployment budget consumption: 120 × 0.27 = 32.4 equivalent minutes.

Against a 99.95% SLO (22 minutes/month budget), 32.4 minutes of deployment budget consumption means the team starts every month already 47% over budget from deployments alone, before a single incident. The SLO target is inconsistent with the deployment cadence. This is not a calculation that requires advanced mathematics — it requires knowing the deployment duration, the per-deploy error rate, the deployment frequency, and the monthly budget. All four numbers are available to the engineering team. None of them were used when the SLO target was set.

The deployment velocity ceiling is not the only way the error budget constrains engineering behavior. The budget also constrains the acceptable incident rate. At 99.9% (87.6 minutes/month budget), a team can absorb one 60-minute major incident and still have 27.6 minutes for smaller incidents and deployment overhead. At 99.95% (22 minutes), a 20-minute major incident and two deployments with minor error rate elevation can exhaust the budget. The choice between 99.9% and 99.95% is not just a number change — it changes the incident absorption capacity and the deployment risk tolerance fundamentally. Neither change is described by the number alone. The SLO decision record makes these consequences explicit at definition time, so that the target is chosen with understanding of what it implies, not discovered after the fact through velocity freeze.

The deployment velocity ceiling also changes with traffic growth, in a counterintuitive direction. At higher traffic, each deployment affects more user requests in absolute terms, but the fractional budget consumption remains the same: the budget is a fraction of total requests, the deployment elevated error rate is a fraction of total requests during the deployment window, and the ratio stays constant as traffic scales. What changes with traffic growth is the absolute number of user requests affected per incident — not the budget fraction, but the business impact of each budget unit consumed. This connects to the capacity planning decision record: at higher traffic, the same error rate during a deployment affects more users in absolute terms, which raises the visibility of deployment reliability issues even when the fractional budget consumption is unchanged. A 3% elevated error rate during a 4-minute deployment at 50 requests/second affects 360 total requests; the same rate at 5,000 requests/second affects 36,000 requests. The budget math is the same; the customer impact is 100× larger. The SLO target may need to become stricter as traffic grows — not because the math changes, but because the absolute user impact per budget unit increases.

The burn rate model and the alert timing problem

Error budget alerting has a fundamental timing problem: if you alert when the budget is exhausted, the alert fires after the budget is already gone. A team with 22 minutes of monthly budget that alerts at 100% exhaustion has spent their full reliability allowance before anyone is paged. The alert is informative but not actionable — there is no budget to protect.

Burn rate alerting solves this by alerting based on the rate at which the budget is being consumed, not the absolute amount consumed. A burn rate of 1.0 means the error rate is exactly equal to the SLO threshold — the budget will be fully consumed at the end of the measurement window at the current rate. A burn rate of 14.4 means the error rate is 14.4× the SLO threshold — at this rate, the monthly budget would be exhausted in 1/14.4 of a month, or approximately 50 hours. The burn rate number is derived from the ratio of the current error rate to the SLO error rate.

The two-window burn rate model — recommended by the Google SRE workbook and commonly implemented in Prometheus alerting rules — uses a fast-burn alert and a slow-burn alert. The fast-burn alert triggers when the burn rate exceeds 14.4× over a 1-hour window: this means the service is degrading severely enough to exhaust the monthly budget in approximately 50 hours, and the 1-hour window confirms the burn rate has been sustained for an hour (reducing false positives from brief spikes). The slow-burn alert triggers when the burn rate exceeds 6× over a 6-hour window: this means the service is degrading at a rate that would exhaust the monthly budget in approximately 5 days, and the 6-hour window confirms the elevated error rate has been sustained for 6 hours. Together, the two alerts catch both acute severe outages (fast burn) and persistent lower-severity degradations that would not trigger the fast-burn threshold but would still exhaust the budget within the month (slow burn).

The burn rate thresholds are derived from the SLO target. For a 99.9% SLO, the allowed error rate is 0.1%. A 14.4× burn rate means a 1.44% error rate — approximately 15 times the normal error rate. For a 99.95% SLO, the allowed error rate is 0.05%. A 14.4× burn rate means a 0.72% error rate — the alert triggers at a lower absolute error rate, which requires more precise SLI instrumentation and more carefully tuned alert conditions to avoid false positives at low traffic volumes where natural statistical variation can produce apparent burn rates from small request counts. This is a concrete SLO target consequence that should be documented in the SLO decision record: at low traffic, burn rate alerting on strict SLO targets produces alert noise that requires either traffic volume minimum gates (only evaluate burn rate when request count exceeds a threshold) or longer evaluation windows. The burn rate alert model must be designed alongside the SLO target, not added as an afterthought after the target is already set.

The dead zone below 6× burn rate is a structural property of the two-window model: a persistent 5× burn rate (error rate at 5× the SLO threshold) would exhaust the monthly budget in approximately 6 days, but triggers neither the fast-burn nor the slow-burn alert. This is not an oversight — the two-window model is a practical compromise between alert sensitivity and alert fatigue. The dead zone means that a persistent low-severity degradation that is bad for the SLO may not page anyone before significant budget is consumed. Teams that want coverage in the dead zone add a budget depletion alert at 50% consumption (a ticket, not a page) and a 10× slow-burn alert on a 24-hour window. The SLO decision record should document which alert conditions are active and why, not just the SLO target number — because the target number is meaningless as a reliability guarantee unless the alerting model catches violations before the budget is exhausted. This connects to the incident response playbook decision record: the burn rate alert severity (P0 page vs P1 ticket vs Slack notification) must be documented alongside the incident response runbook, because the engineer who receives the burn rate alert needs to know what action to take and at what urgency — information that lives in the incident response playbook, not in the alerting rule configuration.

Five sections the SLO and error budget decision record should address

1. SLI selection and user-experience validity

Document each SLI as a triplet: the subject (what property of the system behavior is measured), the instrumentation (where the measurement comes from and how it is computed), and the blind spots (what failure modes will not appear in this measurement). The blind spot documentation is the hardest to write and the most valuable to have: it requires the team to reason through failure scenarios and check whether each scenario would be visible in the SLI. The project management tool's notification delivery failure would have been caught by this exercise — it would have forced someone to answer the question "if the notification worker crashes, what does this SLI show?" The answer ("it shows green, because the SLI measures the API tier") reveals the gap before the gap becomes eleven months of incorrect reporting.

For request-success-rate SLIs, document which HTTP response codes count as failures (typically 5xx; sometimes 4xx for specific operations where a client error indicates a service problem), which request paths are included and excluded (health check endpoints are almost always excluded; some synthetic monitoring paths may be excluded), and whether the SLI is computed on the server side (from application metrics) or on the client side (from load balancer access logs). Server-side and client-side error rates can differ meaningfully: a server-side SLI may not record errors that occur before the request reaches the application (TLS handshake failures, connection timeouts at the load balancer layer); a client-side SLI measured at the load balancer may record errors that the application considers correctly handled (intentional rate limit 429s, authentication 401s). The measurement source must be documented because it determines which failure modes are visible and which are not.

For latency SLIs, document the percentile, the measurement point (time from first byte of request to last byte of response, measured at the load balancer or at the application layer), and the threshold. A "p99 latency under 1,000ms" SLI measured at the load balancer is different from the same SLI measured at the application: the load balancer measurement includes network time between client and load balancer, which the application cannot control and which varies by geographic region. Document which percentile was chosen and why — p99 catches the tail that p95 misses, but p99 at low traffic volumes requires careful minimum request count thresholds because a single slow request in a 100-request hour is p99 by definition. If background job SLIs are added (notification delivery success rate, export completion within window), document the trigger-to-completion latency distribution that the threshold is based on, and the monitoring infrastructure required to compute it — this is typically a non-trivial instrumentation investment that must be planned alongside the SLI definition. The observability strategy decision record covers the monitoring infrastructure choices; the SLO decision record should reference it for the specific instrumentation requirements of each SLI.

2. SLO target and error budget model at projected scale

Document the SLO target as a specific number with the reasoning: what considerations drove the choice, what alternatives were considered, and what the choice implies about the monthly budget in concrete terms. The reasoning should include the calculation of budget implications at current traffic and at projected traffic in 12 and 24 months. For a request-success-rate SLO, the budget can be stated in both absolute request failures (monthly traffic × (1 − SLO)) and equivalent outage minutes at full-failure rate (monthly minutes × (1 − SLO)). Both representations are useful: request failures connect to user impact (number of users who experienced an error), and outage minutes connect to incident response (how long can a complete outage last before the SLO is violated).

The deployment budget calculation is the most commonly missing element of SLO target documentation. Before finalizing the target, calculate: (a) the rolling deployment window duration from CI/CD pipeline metrics or deployment history, (b) the effective elevated error rate during the deployment window from monitoring data, (c) the per-deploy budget consumption in equivalent outage minutes, and (d) the monthly deployment budget consumption at the planned deployment frequency. If the monthly deployment budget consumption exceeds 30-40% of the total monthly budget, either the deployment process needs to be improved before setting the strict target, or the target needs to be adjusted to account for the deployment overhead. Document this calculation explicitly so that future target revisions can start from the same inputs rather than requiring a new investigation. This is one of the decisions that originates in an AI chat session — "what SLO should we target for a B2B SaaS?" — and receives a contextless answer based on industry norms, without any calculation of the team's specific deployment profile. The decisions never written down pattern applies directly here: the SLO choice is made conversationally, influenced by a recommendation from an AI session, and never written down with the reasoning that would allow it to be evaluated or changed. The WhyChose extractor can surface those original SLO discussion sessions from ChatGPT or Claude conversation history — the sessions where the team debated 99.9% vs 99.95%, the incident retrospective where someone said "we need to revisit our SLO targets" — all of which contain the context that should have been captured in the SLO decision record but was instead buried in chat history.

Document the SLO tier if the product has multiple user tiers with different reliability expectations. An enterprise tier may require a stricter SLO (99.99%) while the standard tier uses the baseline (99.9%). The multi-tier SLO model requires separate error budget accounting per tier, separate SLI instrumentation that can distinguish enterprise traffic from standard traffic, and separate alerting rules. The decision to add a stricter enterprise tier SLO is itself an architectural commitment — it requires the instrumentation and alerting infrastructure to support it. This connects to the new-CTO onboarding problem: a new engineering lead who inherits an SLO setup without the decision record cannot determine whether the current targets reflect deliberate choices, industry defaults, or inherited configurations from a previous monitoring setup. The decision record answers "why is our enterprise SLA 99.95% and our standard SLA 99.9%?" in a way that no dashboard ever will.

3. Error budget policy and velocity trade-off thresholds

The error budget policy is the most operationally important section of the SLO decision record and the most commonly underdocumented. The policy answers: what does the team do when the budget is partially consumed? What happens when it is exhausted? Who can override a deployment freeze and under what conditions?

A complete error budget policy has three levels. The first level is the budget health threshold — the point at which the team begins additional reliability investment without freezing feature work. A common threshold is 50% budget consumption in the first two weeks of the month: if half the budget is gone before the month is half over, the team adds reliability work to the sprint without stopping feature work. This gives early warning without triggering a freeze that interrupts momentum. The second level is the freeze threshold — typically 100% budget consumption (or 80%, for teams that want a margin). At the freeze threshold, new feature deployments require explicit approval from a named decision maker. The freeze is a signal that the service needs reliability investment before additional feature surface is added. The third level is the override process — who can approve a deployment during a freeze, under what conditions, and what the approval must be recorded against. Security fixes, incident remediations, and pipeline health restorations are the canonical override categories. Document the override approver by role, not by name — names change, roles persist.

The policy should also address budget reset: what happens to the error budget at the start of a new measurement window? A rolling 30-day window means the budget resets gradually as old events age out — there is no clean "start of month" reset that creates a psychological fresh start. A calendar-month window resets cleanly on the first of each month but creates an incentive to defer risky work to early in the month and a pressure to "use" remaining budget at the end of the month. Document which window model is used and why, because the choice affects team behavior in ways that are not obvious from the alerting configuration. This connects to the CI/CD pipeline decision record: the deployment mechanism and deployment frequency are inputs to the error budget calculation. If the team migrates from rolling deployments to blue/green deployments — which can achieve near-zero per-deploy error rate by shifting traffic only after the new version is fully healthy — the per-deploy budget consumption drops dramatically, freeing budget for incidents and higher deployment frequency. The SLO decision record should reference the CI/CD pipeline decision record for the deployment mechanism that the error budget calculation assumes, so that future CI/CD changes trigger a review of the budget calculation.

4. Burn rate alerting thresholds and incident response tiers

Document the specific burn rate alert rules: the window duration, the burn rate threshold, the severity, and the intended response. For each alert condition, document what the alert means in concrete terms at the team's current traffic level: "this alert fires when the error rate has been 14.4× the SLO threshold for at least one hour, meaning the monthly error budget would be exhausted in approximately 50 hours at the current rate." The concrete translation is necessary because burn rate numbers are abstract until connected to a specific traffic level and SLO target. An engineer who receives a "1h burn rate 22.3" alert without documentation of what that means has to calculate the implication on the fly during an incident — which is the worst possible time for arithmetic.

Document the alert conditions that are not covered by the two-window model — the dead zone below 6× burn rate — and the team's policy for that zone. Options include: accepting that the dead zone represents a level of degradation that is slow enough not to require immediate response; adding a 24-hour budget depletion ticket when 50% of the monthly budget is consumed; or adding a third alert on a longer window (24 hours at 3× burn rate). Document which option was chosen and why. The choice depends on the team's operational capacity (frequent low-urgency tickets are noise if the team cannot act on them promptly) and the product's tolerance for the failure modes that fall in the dead zone.

Document the relationship between burn rate alerts and the incident response severity classification. A P0 (page immediately, all-hands response) should correspond to the fast burn alert: the budget is being consumed at a rate that requires immediate response. A P1 (page within the hour, investigate in business hours) should correspond to the slow burn alert. A P2 (ticket, investigate in the next sprint) should correspond to the 50% budget depletion notification. The incident response runbook for each severity should reference the SLO burn rate alert that triggers it, so that on-call engineers who receive an alert can immediately locate the relevant runbook section. This is the connection to the incident response playbook that makes both documents more useful: the burn rate alert fires with severity X, the runbook for severity X describes the first three diagnostic steps and the escalation path. Without this connection, the burn rate alert and the incident response runbook are separate documents that each engineer must manually connect under pressure.

Document the traffic volume minimum for burn rate alerts. At low traffic volumes, burn rate calculations are unstable: a service processing 10 requests per minute that has 2 failures in a one-hour window has a 0.33% error rate against a 0.1% SLO threshold — a 3.3× burn rate — from 2 errors. Whether this is a real problem or statistical noise from low sample counts is ambiguous. Most monitoring platforms support minimum request count conditions on burn rate alert rules — the rule fires only if the calculated burn rate exceeds the threshold AND the request count in the evaluation window exceeds a minimum. Document the minimum and the reasoning: too low a minimum produces alert noise at off-peak hours; too high a minimum misses real incidents during traffic valleys. This matters most for teams with significant daily traffic variation (consumer products with night-time traffic valleys) and for teams running multiple SLOs on different endpoints with different traffic volumes.

5. SLO review cadence and target evolution protocol

Document the cadence at which the SLO targets, SLI definitions, and error budget policies are reviewed for continued appropriateness. The SLO is not a permanent commitment — it is a current best estimate of the reliability level the team can sustain while maintaining feature velocity, given the current architecture, deployment practices, and incident history. Several events should trigger a review outside the regular cadence: significant traffic growth (traffic doubles or increases by an order of magnitude), a change in deployment frequency or deployment mechanism, the addition of a new tier with different reliability expectations, a repeated pattern of budget exhaustion that signals the target and the deployment/incident reality are misaligned, or a compliance or contractual requirement that specifies a minimum availability guarantee.

Document the process for changing an SLO target. Changing a target in the monitoring dashboard is a configuration change; changing the error budget policy that governs team behavior is a team agreement. The process for changing the target should be more deliberate than a Terraform merge: it should require a documented analysis (what was wrong with the current target, what the new target implies about budget, how the deployment budget calculation changes), a review by the engineering lead and the product lead (because the target change affects feature velocity), and an update to the SLO decision record explaining the reasoning. A target change without documented reasoning produces the same problem as the original undocumented target: the next engineer who encounters the target cannot evaluate whether it is correct for the current context.

The SLO review cadence must be connected to the recovery time objective documented in the disaster recovery decision record. The DR RTO — the maximum time from incident start to service recovery — must fit within the error budget implied by the SLO target. A 99.9% SLO (87.6 minutes/month) with a DR RTO of 4 hours means that a single DR event exhausts the monthly budget more than twice over. This is not necessarily wrong — DR events are rare — but the relationship should be explicit. The disaster recovery decision record should reference the SLO decision record for the budget implications of the RTO target. If the DR RTO is 4 hours and the SLO budget is 22 minutes (99.95%), the implication is that any DR event will exceed the monthly budget — which means the team's SLA to customers either excludes DR events explicitly or is automatically violated by any DR scenario. This tradeoff must be documented and communicated to customers in the SLA terms, not discovered during the first major DR event. The feature flag decision record is the third connection: feature flags allow the team to decouple the deploy event from the release event. A deployment that ships code but keeps the feature behind a flag has a lower per-deploy error risk than a deployment that immediately exposes the new feature to all users — because the flag can be turned off within seconds if the deployed code produces elevated errors before the flag is opened, without a full rollback. Teams that use feature flags for high-risk changes typically have lower per-deploy error rates and therefore more error budget available for incidents and high-velocity deployment cadences. The SLO decision record should document whether feature flag adoption is assumed as part of the error budget model or whether the budget calculation assumes all deployments are direct releases.

Finally, document the SLO's relationship to the database reliability decisions that drive the majority of error budget consumption in most services. Database incidents — connection pool exhaustion, replication lag exceeding read routing thresholds, index bloat causing query timeouts, failover events — are the largest single category of error budget consumption for most data-driven services. The database vendor decision record should document the expected MTBF and MTTD of database incidents for the chosen vendor and configuration; the SLO decision record should reference the database vendor's reliability characteristics in the budget model. A team that picks a database vendor with a known failover pattern (RDS Multi-AZ automatic failover takes 60-120 seconds) should include the expected failover contribution to the error budget in the SLO target analysis. A 60-120 second failover at 100% error rate consumes 1-2 minutes of budget per event. If the team experiences two failover events per year (a reasonable estimate for routine maintenance and the occasional unexpected event), the annualized failover contribution to the monthly budget is approximately 0.2-0.4 minutes per month — small relative to a 22-minute budget, but visible relative to a 4-minute budget (99.99% SLO). The ADR template provides the structure for the SLO decision record: decision (the specific SLI definition and SLO target), rationale (the reasoning behind the target, including the budget calculations), consequences (the deployment velocity ceiling, the incident absorption capacity, the alert model derived from the target), and review date (when the target will be revisited). Without this document, every SLO-related conversation starts from scratch. With it, every SLO-related conversation starts from an analyzable baseline that can be updated rather than rebuilt.

Further reading