The API rate limiting decision record: why the rate limiting model you chose determines your abuse surface and your fair-use ceiling

Published 2026-07-14 · WhyChose

API rate limiting decisions are made in the security hardening session, the API design session, or the infrastructure scaling session — whenever the team first needs to protect a service from request flooding, automated abuse, or a single misconfigured client consuming disproportionate capacity. The AI session that adds rate limiting is thorough and practically grounded: it evaluates algorithm options (token bucket, leaky bucket, sliding window, fixed window counter), selects a storage backend for the rate limit counters (in-process memory, Redis, or a managed service), wires the middleware into the request path so that callers exceeding the limit receive a 429 Too Many Requests response with a Retry-After header, and verifies that the throttle works correctly for the test cases used in development. The session ships a working rate limiter. At founding-environment scale — a handful of known callers, predictable traffic patterns, a single server process — the rate limiter does what it is supposed to do.

What the AI session does not produce is the rate limiter's second half. The session answers "which algorithm and which counter backend?" and delivers a working throttle. It does not ask: what is the consumer isolation model — are requests attributed to the caller's IP address, their authenticated user identity, their API key, their tenant organization, or some combination of these identities, and how does the attribution model interact with enterprise customers behind corporate NAT gateways where hundreds of individual employees share a single public IP address? What is the distributed enforcement consistency guarantee — when rate limit counters are stored in Redis and the Redis primary instance fails over to a replica, does the enforcement system fail open (allowing all requests until Redis recovers, resetting counters to zero in the process) or fail closed (blocking all requests until Redis recovers), and is a counter reset during failover documented as an accepted risk or an unacknowledged vulnerability? What is the bypass and exemption policy — which internal service-to-service calls are exempt from rate limiting, which external callers have negotiated higher limits in enterprise contracts, and how is a new exemption added without creating a bypass vulnerability that silently removes protection from a surface the team believed was rate-limited? What is the burst allowance model — does the token bucket's accumulated capacity allow a caller who has been quiet for an hour to immediately send a burst equal to the per-hour limit, and is that burst the desired behavior for the use case the rate limiter is protecting, or does it simply move the abuse vector from sustained high rates to single large bursts? Each of these questions has an answer that is not derivable from "we use token bucket with Redis and return 429s" as the team frames the algorithm choice. Each answer determines whether rate limits protect infrastructure without punishing legitimate heavy users, or become the mechanism by which the largest paying customers are blocked during peak usage, internal callers bypass protection quietly, and a Redis migration produces an unthrottled traffic window from automated integrations that the team believed were throttled throughout. The answers exist in the AI sessions. They are the operational commitments behind the algorithm choice. They are almost never written down.

Two ways API rate limiting decisions produce the wrong outcome in production

The enterprise NAT gateway failure

A B2B SaaS startup adds rate limiting to their REST API during a security sprint in their second year. The decision is well-justified and technically sound: the team has observed two incidents in the past quarter where a misconfigured customer integration sent automated requests at several hundred per second, saturating the database connection pool and causing elevated error rates for all other customers. The founding session selects a token bucket algorithm — appropriate for the bursty-but-low-average-rate access patterns typical of developer integrations — and implements the rate limiter as Express middleware using a Redis-backed token bucket with a limit of 1,000 requests per IP address per hour. The per-IP identity model is chosen because the API is not yet fully authenticated: the search and catalog endpoints accept unauthenticated requests, and the authenticated endpoints use session cookies set per-browser rather than per-calling-application. Per-IP is the only identity signal available across both authenticated and unauthenticated access patterns. The rate limiter ships within two days, the integration incidents stop, and the security sprint closes as a success.

Fourteen months later, the startup closes their largest contract: an 800-person professional services firm that uses the API to power an internal tool that queries the product catalog for their consultants. The customer's IT department uses a corporate network gateway that routes all outbound internet traffic through a single NAT gateway with a single public IPv4 address. The first week after the enterprise contract activates, the customer success team receives an escalation: the internal tool is completely unusable during business hours. Requests are returning 429 errors on the first attempt, before any individual caller has sent a burst. The engineering team investigates and finds the cause immediately: with 800 consultants using the internal tool throughout the business day, the corporate NAT gateway produces an aggregated request rate that exceeds 1,000 requests per hour starting within the first fifteen minutes of each business day. The per-IP rate limit designed to protect against a single misconfigured integration is firing on the enterprise customer's legitimate usage of a paid contract.

The immediate fix is a manual whitelist entry: the enterprise customer's NAT gateway IP address is added to a Redis SET of exempted IPs that the rate limiting middleware checks before applying the token bucket logic. The fix ships within three hours of the escalation. But the exemption mechanism reveals the larger structural gap. The whitelist is implemented by a direct Redis SET command in the staging environment, verified to work, then repeated in production — there is no code change, no code review, no test coverage, and no documentation. Three months later, a new engineer on the infrastructure team initiates a Redis maintenance operation that includes a flush of development-environment keys that have been leaking into the shared Redis instance. The key naming pattern used by the rate limiting middleware's exemption whitelist uses a prefix the new engineer does not recognize as production-critical. The flush command runs, the enterprise customer's IP is removed from the exemption set, and the 429 errors return for the enterprise customer's 800 users at the start of the next business day. The customer success team is paged at 8:47 AM. The on-call engineer investigates, cannot find documentation of the exemption mechanism (it was never committed to the codebase or written in any runbook), and escalates to the rate limiting middleware author, who is on vacation. Two hours of downtime for the enterprise customer result from a maintenance operation that deleted a production configuration artifact that was invisible to the infrastructure team because it existed only in Redis and was never documented.

The founding session that chose per-IP token bucket did not address: what is the consumer isolation model for enterprise customers whose organizational access pattern is fundamentally different from the per-individual-developer pattern the rate limiter was designed for? What is the exemption management mechanism — where are exemptions stored (Redis ephemeral state, application configuration committed to source control, or a database table with an audit log), who is authorized to add an exemption, and what is the process for adding an exemption safely without requiring a production Redis command from an engineer who understands the key naming schema? What is the migration path from per-IP rate limiting to per-API-key or per-authenticated-user rate limiting, which would solve the NAT gateway problem by attributing requests to individual callers rather than the network address they arrive from? Three questions that were deferrable at founding-API scale — a small number of known API consumers with simple access patterns — but that became critical constraints as the customer mix grew to include enterprise organizations whose network topology is invisible to a per-IP rate limiter.

The Redis counter reset and the unthrottled traffic window

A developer tools company adds distributed rate limiting to their API in their third year when they expand from a single-server deployment to a multi-server setup behind a load balancer. The founding session correctly identifies that the in-memory token bucket from the previous implementation cannot enforce rate limits across multiple server processes — a caller can make the full per-minute limit against each server process and receive no throttling, because each process sees only the requests it handled and has no visibility into the requests that other processes handled for the same caller. The session selects Redis as the shared counter backend, implements a Lua script that performs an atomic increment-and-check operation in Redis on each request (incrementing the counter and returning whether the result exceeds the limit in a single atomic operation to prevent race conditions), and deploys the distributed rate limiter with the existing token bucket limits. The session tests distributed enforcement against a load test that sends requests to a round-robin load balancer and verifies that the aggregate rate across all server processes is correctly throttled. The implementation is correct and the distributed enforcement works as designed.

Eight months after deployment, the company's Redis instance undergoes a planned migration from a self-managed Redis Sentinel setup on EC2 instances to a managed Redis service. The migration includes a scheduled maintenance window: the managed Redis instance is seeded with a snapshot from the existing Redis, the application's Redis connection string is updated, and the managed instance begins accepting traffic. What the migration plan does not account for is that the managed Redis instance's snapshot was taken at 11:30 PM on a Sunday — the lowest-traffic window of the week — and is being promoted to production at 2:15 PM on a Tuesday. The rate limiting counters in the snapshot are from Sunday at 11:30 PM: essentially empty, because the API had negligible traffic at that time. The 2:15 PM Tuesday migration cutover clears all rate limiting counter state accumulated during Monday and the first half of Tuesday. Every caller, including the eight automated integration partners that the company's enterprise customers have built on top of the API, begins the 2:15 PM window with a fresh empty counter — as if they had made zero requests all day.

The most active automated integration partner — a CI tooling vendor that calls the API approximately 1,800 times per minute during business hours — immediately absorbs an unthrottled burst: having consumed its rate limit allowance earlier in the day, it is now presented with a full empty counter and sends 14,000 requests in the eight minutes before its per-minute rate limit accumulates enough state to begin throttling it again. Three other integration partners with slightly lower request rates similarly burst in the first few minutes after the migration cutover. The aggregate unthrottled traffic spike is 4× the normal API request rate for those eight minutes, producing elevated database connection pool utilization and elevated error rates across the API. The post-incident review identifies the root cause as a gap in the migration plan: no one documented that rate limiting counters are not cache entries that can be populated from a week-old snapshot, but application state that must be either migrated from the current Redis instance at migration time or accepted as an over-allowance window during the cutover.

The founding session that chose Redis-backed distributed rate limiting did not address: what is the counter persistence model — are rate limiting counters considered ephemeral state (correct for per-second and per-minute windows where a restart cleanly begins a new window) or application state (required for per-day and per-month allowances where a counter reset mid-window resets a caller's entire daily or monthly quota)? What is the migration policy for Redis counter state during a backend migration — should counter state be exported from the old Redis instance and seeded into the new instance as part of the migration plan, and if not, what is the maximum traffic impact of starting with empty counters during the migration cutover window? What is the documentation requirement for the counter state model such that an engineer performing a Redis migration three years after the original implementation understands the rate limiting system's dependency on counter state continuity, and can plan the cutover timing and seeding strategy to minimize the enforcement gap? Three operational commitments that were correct in isolation but whose interaction with a planned maintenance operation produced an unplanned abuse window that the engineering team had no prior signal to anticipate.

Three structural properties that API rate limiting decisions determine

The consumer identity model and the attribution surface

The rate limiting consumer identity model determines what entity the rate limit counter is attributed to — which property of an incoming request is used as the key for the rate limit bucket. The identity model choice has a higher impact on rate limiting correctness than the algorithm choice: a correctly implemented sliding window counter with the wrong identity model will produce false positives (blocking legitimate callers who are incorrectly grouped with high-rate callers) and false negatives (allowing abusive callers who spread their requests across multiple identities) at a rate that no algorithm tuning can correct. The identity model must be chosen as an explicit decision, not as a default inherited from the rate limiting library's example configuration.

Per-IP rate limiting uses the request's source IP address as the counter key. It is the default in most rate limiting library examples because it requires no authentication infrastructure — the source IP is present on every request regardless of whether the caller is authenticated. Per-IP rate limiting is correct for protecting publicly accessible endpoints against unauthenticated abuse from individual actors operating from a single IP address, and for general DDoS mitigation where the attack source is a cluster of IPs that can be blocked at the network layer. Per-IP rate limiting produces incorrect results in three deployment scenarios: enterprise customers behind corporate NAT gateways (documented above); legitimate callers using cloud-based CI or serverless execution environments where requests originate from shared IP pools rotated across many tenants of the cloud provider (an AWS Lambda function calling the API uses AWS's outbound IP range, which may be shared with dozens of other Lambda-using companies); and callers using residential VPN services that route traffic through shared proxy IPs (a legitimate user accessing the API through a commercial VPN shares the VPN endpoint IP with all other users of that VPN endpoint).

Per-API-key rate limiting uses an API key included in the request header or query parameter as the counter key. It correctly isolates each caller's allowance to their specific credential, regardless of the network topology they use to reach the API. Per-API-key rate limiting requires that all callers include an API key — anonymous access must either be prohibited or rate-limited by IP as a fallback, with an explicit acknowledgment that the per-IP fallback has the NAT gateway failure mode for anonymous enterprise access. The API key identity model also requires that the key issuance and lifecycle management policy is documented as a dependency: the rate limit is only as granular as the key provisioning model. If the API key issuance model issues one key per company rather than one key per developer or one key per integration, per-API-key rate limiting is equivalent to per-company rate limiting — a subtle distinction invisible in development where a single tester uses a single key, and critical in production where a company with 50 integrations shares one key and exhausts the limit across all of them collectively. The secrets management decision record governs API key issuance, lifecycle, and revocation, and the rate limiting identity model must be consistent with the key granularity that the secrets management model provides.

Per-authenticated-user rate limiting extracts the user identity from a session token, JWT claim, or authentication header and uses it as the counter key. It provides the finest-grained isolation for human users interacting through a browser or mobile client, and correctly handles the NAT gateway scenario because each user has a distinct identity regardless of the network address their requests arrive from. Per-authenticated-user rate limiting breaks for unauthenticated access (requiring the same per-IP fallback) and for service-to-service API access where the "user" identity is a service account shared across all calls from a given microservice — the service account identity groups all API calls from that service into a single rate limit bucket, which may be the correct behavior (a per-service rate limit) or incorrect (a high-rate service account exhausting its limit and blocking all inter-service calls that use that identity).

Per-tenant rate limiting applies a shared limit at the organizational or account level, with individual user and API key rate limits nested within the tenant limit. The tenant limit is typically negotiated as a contractual term for enterprise customers — "your organization is entitled to 50,000 API calls per day" — rather than derived from infrastructure capacity. The per-tenant model requires that the rate limiting infrastructure can resolve a request's tenant identity from the authentication context, which requires the tenant association to be present in the request's authentication token (a JWT claim, a session attribute, or an API key prefix that encodes the tenant ID). The per-tenant limit and the per-user limit must be documented as separate enforcement layers: a single high-rate user can exhaust the tenant limit and block all other users in the tenant, which may be intended behavior (shared quota that individual users consume from) or an unintended consequence of a rate limiting model designed for individual caller isolation.

The decision record must document the identity model as a hierarchy: the primary identity used for rate limit attribution (API key, user JWT claim, or tenant ID), the fallback identity used when the primary identity is absent (source IP for unauthenticated access), and the rate limit applied at each layer (per-key limit nested within per-tenant limit). The hierarchy must be consistent with the API's authentication model, and the exemption inventory must list every caller for which the standard identity model does not apply and why.

The algorithm and the burst surface

The rate limiting algorithm determines the mathematical relationship between the request rate a caller sends and the rate at which the rate limiter returns 429 responses. The algorithm choice has two independent properties that must be evaluated separately: the burst behavior (how the algorithm handles short-duration bursts that exceed the sustained rate limit) and the enforcement precision (how accurately the algorithm tracks the caller's request rate over the relevant time window).

The token bucket algorithm maintains a bucket with a maximum capacity equal to the burst size, which refills at a constant rate equal to the sustained limit. A caller's requests consume tokens from the bucket; when the bucket is empty, requests are rejected until tokens refill. The critical property of the token bucket is that it allows accumulated capacity: a caller who makes no requests for an extended period fills their bucket to the maximum capacity, enabling them to immediately send a burst of requests equal to the bucket capacity when they resume activity. For a token bucket with a capacity of 1,000 tokens and a refill rate of 1,000 tokens per hour, a caller who was quiet for two hours can immediately send a burst of 1,000 requests — the maximum capacity, not 2,000, because the bucket is capped at its maximum capacity regardless of how long the caller has been inactive. This burst allowance is the primary design difference between the token bucket and the sliding window algorithms: the token bucket rewards callers who have historically been under the limit with accumulated capacity for a future burst; the sliding window applies the limit uniformly regardless of the caller's historical request pattern.

The burst surface of a token bucket is bounded by the bucket capacity, which is typically set equal to the per-window rate limit. A bucket with 1,000 tokens and a per-hour refill rate allows a burst of up to 1,000 requests in a single second, followed by a 3.6-second refill wait per token — a rate that is 3,600× the sustained limit for the first second and then restricted to the sustained rate thereafter. Whether this burst is the correct behavior depends on the caller population and the downstream resource being protected: a burst of 1,000 database queries in one second may saturate the connection pool even if the hourly query rate is within infrastructure capacity, making the burst-allowance property of the token bucket a vulnerability rather than a feature for database-backed endpoints. The rate limiting decision record must document the bucket capacity as a separate parameter from the refill rate, with an explicit statement of whether the burst-equals-limit design is intended or whether a lower burst capacity (such as 10% of the hourly limit) is required to protect downstream resources from burst spikes.

The sliding window counter algorithm tracks the number of requests in a rolling time window. The counter value at the moment of a new request is the count of requests that arrived within the past N seconds. The sliding window prevents burst accumulation: a caller who was quiet for the past hour has a counter value of zero when they resume, but cannot make more than the per-window limit in the next window period regardless of how long they were inactive. The sliding window is more computationally expensive than the token bucket to implement in Redis: an accurate sliding window requires storing a sorted set of request timestamps (one entry per request, with the timestamp as the score) and using ZRANGEBYSCORE to count entries in the current window on each request. For high-rate endpoints, the per-request overhead of a ZADD + ZREMRANGEBYSCORE + ZCARD operation is higher than the single INCR of a fixed window counter, and the sorted set memory consumption grows with the number of requests in the retention window.

The fixed window counter algorithm maintains a single counter that resets at a clock boundary. It is the simplest to implement (a single Redis INCR with an EXPIRE set on the first request in each window) and requires minimal Redis memory and computation per request. The fixed window's failure mode is the boundary burst: a caller can make a full window's requests in the last second of one window and a full window's requests in the first second of the next window, producing a burst of twice the per-window limit within a two-second span. Whether the boundary burst is an acceptable failure mode depends on the protection goal. For protecting against sustained high-rate abuse, the boundary burst is typically acceptable — the abusive caller still cannot exceed the average rate limit over any multi-window period. For protecting downstream resources whose capacity is bounded by instantaneous throughput (a database connection pool, an external API with per-second billing), the boundary burst can produce a 2× instantaneous throughput spike that exceeds the downstream resource's capacity even though the per-window average remains within limits. The decision record must document whether the boundary burst is evaluated and accepted, with a statement of the maximum instantaneous burst and a verification that downstream resources can absorb it.

The leaky bucket algorithm processes requests at a strictly uniform output rate regardless of input rate. Requests are added to a queue and processed at the output rate — if the queue exceeds its maximum size, new requests are dropped. The leaky bucket prevents bursty output: regardless of how many requests arrive in a short period, the downstream resource sees a uniform request rate. The leaky bucket is rarely used for synchronous HTTP API rate limiting in practice because it requires maintaining per-caller queues and introduces queueing latency for requests that wait for the output rate to process them — a caller who sends 100 requests in one second when the output rate is 10 per second experiences a 9-second delay for the last request in the burst, rather than receiving an immediate 429. The leaky bucket is appropriate for internal job queues where work should be smoothed rather than discarded, but is rarely the correct choice for synchronous APIs where clients expect an immediate response.

The distributed enforcement model and the consistency surface

Distributed enforcement is required whenever the API is served by more than one server process — which is the case for any load-balanced deployment, any multi-instance container deployment, and any serverless function deployment where multiple concurrent invocations serve the same API. In a single-process deployment, per-process in-memory counters correctly enforce rate limits because all requests go through the same counter state. In a distributed deployment, per-process counters produce incorrect enforcement: each server process sees only the fraction of the caller's requests that the load balancer routed to it, so the per-process counter for a caller sending 1,000 requests per minute across 10 server processes shows only 100 requests per minute per process — one tenth of the actual rate — and no rate limiting fires even if the caller exceeds the documented limit by 10×.

Shared counter enforcement stores rate limit counters in a centralized system (Redis, Memcached, a database table, or a managed rate limiting service) that all server processes access on each request. The shared counter enforces rate limits correctly across all server processes at the cost of a network round trip to the counter backend on each request. The round trip adds latency to every request in the hot path — for a Redis counter backend co-located in the same availability zone, the round trip is typically 0.5–2ms, which is acceptable for most APIs but non-trivial for high-throughput low-latency endpoints where the round trip represents a 10–50% increase in p99 response time. The decision record must document the counter backend round trip latency as part of the rate limiting design, with a measurement from the target deployment environment and a comparison against the API endpoint's latency budget.

Approximate local enforcement uses per-process counters with periodic synchronization to a shared backend, trading enforcement accuracy for reduced counter backend load and zero per-request network overhead. Each server process maintains a local counter and periodically (every 100ms, 500ms, or 1s) posts the local counter increment to the shared backend and reads back the global counter value. Between synchronization intervals, the local process enforces based on the last-known global counter value plus the local increment since the last sync. The accuracy error is bounded: at a 500ms synchronization interval, a caller can exceed the rate limit by at most (number of server processes) × (requests per 500ms at the limit rate) before all processes have synchronized and the global limit is enforced. For most rate limiting scenarios this over-allowance is acceptable. For scenarios where the accuracy requirement is strict (per-request quota enforcement where the contract specifies an exact count), approximate enforcement is not appropriate.

The enforcement consistency model during infrastructure events — Redis failover, network partition, deployment rollout — must be documented as a specific decision, not left as an implicit consequence of the library's error handling defaults. The three properties to document are: the fail-open versus fail-closed policy when the counter backend is unavailable; the counter state recovery model when the backend resumes after unavailability (counter state preserved if the backend had persistence, counter state reset to zero if the backend was ephemeral or replaced by a migration); and the mixed-version enforcement model during rolling deployments where some server processes use an old rate limiting algorithm and some use a new one. Each of these consistency events produces a specific enforcement behavior that may or may not align with the team's protection goals, and the decision record must document which behavior was evaluated and accepted for each event type. The multi-cloud strategy decision record is relevant when rate limiting counters are distributed across cloud providers — cross-cloud Redis replication adds network latency and replication lag to the counter consistency model that a single-cloud Redis deployment does not have, and the counter state during a cross-cloud network partition requires a documented partition policy separate from the single-region failover policy.

Three AI session types that embed API rate limiting decisions without documenting them

The security hardening session is where rate limiting is first added to a previously unthrottled API. The session is motivated by a concrete incident — an integration sending too many requests, a security audit finding that identifies missing rate limiting as a vulnerability, or a production incident where an unthrottled caller saturated a downstream resource — or by a proactive checklist completion before a compliance review. The session selects an algorithm (typically token bucket because the rate limiting library's example uses it), selects a counter backend (Redis if the team already has Redis for caching or session storage, in-memory otherwise), configures the rate limit values (typically matching what the team believes their free and paid tiers should allow, without a load test to verify that the downstream resources can sustain those limits under concurrent caller traffic), and ships a working rate limiter. The session confirms that the throttle fires correctly for a test caller exceeding the limit. What the session does not produce is the consumer identity model documentation (why per-IP was chosen over per-API-key, what the failure mode is for enterprise customers behind NAT, and what the migration path is to a higher-granularity identity model), the bypass exemption policy (which internal callers are exempt and how that exemption is managed and audited), or the distributed enforcement consistency model (what happens during Redis failover or migration). Each of these is visible in the AI session as a context note or an assumed-but-unstated constraint — "we'll add per-user limiting when we have better auth infrastructure," "internal services won't hit the limits," "we'll migrate Redis if we need to" — that the session does not convert into a documented decision with explicit trade-offs and a trigger threshold for revisiting the assumption. The open-source extractor surfaces these founding security hardening sessions from AI chat history, recovering the algorithm rationale, the identity model assumptions, and the infrastructure constraints that were implicit in the team's shared context when the rate limiter was first deployed.

The API productization session is where rate limits transition from an internal infrastructure protection mechanism to a product feature that differentiates paid tiers. The session is motivated by a business requirement: the team wants to offer a free tier with a low rate limit and a paid tier with a higher rate limit, using the rate limit difference to drive upgrade conversion. The session implements rate limit enforcement that varies by subscription tier — typically by looking up the caller's subscription level from a database or authentication token claim and selecting the appropriate rate limit configuration. What the session does not address is the consumer experience of encountering a rate limit: what error response body does a 429 carry beyond the status code and Retry-After header, does the error response include the caller's current counter value and the time until the counter resets so that callers can implement intelligent retry logic rather than polling every second, and does the API provide a non-request-consuming endpoint that returns the caller's current rate limit status so that clients can check remaining capacity without consuming allowance? The API's rate limit discoverability model — the X-RateLimit-Limit, X-RateLimit-Remaining, and X-RateLimit-Reset headers that inform callers of their current limit status on every response — is not implemented in most founding rate limiting sessions because the session is focused on enforcement (returning 429s when the limit is exceeded) rather than transparency (informing callers of their limit status before they exceed it). The absence of rate limit headers is invisible in development, where the developer knows the limit and tests against it deliberately, and visible in production as integration support tickets from external developers who received unexpected 429s with no context about their current counter value or when the counter resets. The API gateway decision record is the companion document: many API gateways provide rate limiting as a managed service that includes rate limit header injection as a built-in feature, and the decision to implement rate limiting in the application layer versus the gateway layer determines whether rate limit headers are available without additional implementation effort.

The scaling session is where rate limiting is first distributed across multiple server processes. The session is motivated by a horizontal scaling event: the team adds a second server process, moves to a container orchestration platform with multiple pod replicas, or migrates to a serverless architecture where each invocation is a separate process. The session identifies that the existing per-process in-memory counters cannot enforce rate limits correctly across multiple processes and replaces them with Redis-backed shared counters. The session tests the distributed enforcement under load and verifies that the aggregate rate limit is correctly enforced across the server process fleet. What the session does not address is the operational model for the shared counter backend: what is the Redis replication model (Standalone, Sentinel, Cluster), what is the data persistence model (in-memory only, RDB snapshots, AOF), what is the failover behavior and the counter state during failover, and what is the documentation artifact that records these infrastructure decisions so that the next engineer who performs Redis maintenance, migration, or replacement understands the rate limiting system's dependency on counter state continuity. Three months after the scaling session, the Redis instance is treated as a general-purpose cache by the operations team and subjected to the same maintenance procedures used for the session cache — including periodic flushes to reclaim memory — without knowledge that the rate limiting counters in the same Redis instance are not cache entries that can be evicted without consequence, but rate limit state whose loss produces an enforcement gap. The real-time architecture decision record is relevant for APIs that apply rate limiting to subscription or streaming connections: the rate limiting model for long-lived connections (WebSocket subscriptions, Server-Sent Event streams) requires enforcement logic that is different from per-request HTTP rate limiting, because a long-lived connection that was established before the rate limit fires cannot be rejected with a 429 and must instead have its event delivery rate throttled or the connection terminated with a protocol-level close.

The five sections of an API rate limiting decision record

The first section documents the consumer identity model and the attribution hierarchy. The primary identity for rate limit attribution must be named explicitly — "requests are attributed to the API key included in the X-API-Key header" — rather than implied by the library configuration. The rationale for the chosen identity must address the three failure modes: corporate NAT gateways (why per-IP was rejected or what the documented failure mode is for enterprise customers whose requests aggregate behind a shared IP), cloud execution environment IP sharing (why per-IP produces false positives for callers using shared outbound IP pools), and shared API keys (why per-key attribution is correct given the key granularity that the secrets management model provides). The fallback identity for unauthenticated requests must be documented separately from the primary identity, with an explicit acknowledgment of the fallback's failure modes: "unauthenticated requests are attributed to the source IP as a fallback; this produces false positives for NAT gateway scenarios and is an accepted limitation for unauthenticated access because requiring authentication for all callers eliminates the failure mode and is the migration target for the following quarter." The per-tenant rate limit must document the tenant identity resolution mechanism (JWT claim, API key prefix, or a database lookup that adds per-request database overhead), the shared quota allocation model (whether individual users within a tenant draw from a shared pool or have independent quotas), and the contract override mechanism (how a tenant's contractual rate limit is stored, enforced, and updated without requiring a code deployment). The new CTO onboarding problem in a rate-limiting-heavy API is an identity model problem: an incoming technical leader finds a rate limiting system attributing limits to IP addresses, a manually maintained Redis SET of IP exemptions not reflected in any code or configuration, an enterprise customer whose rate limit was negotiated in a contract but enforced only as a Redis key set by an engineer who left two years ago, and no documentation of which identity model governs which endpoint. Formalizing the identity model in a decision record before the third enterprise contract is signed is far less expensive than reconstructing the model from Redis key inspection and on-call engineer memory after the fourth.

The second section documents the algorithm selection and the burst model. The algorithm must be named with specificity — "token bucket with a per-key capacity of 1,000 tokens refilling at 1,000 tokens per hour, enforced via a Redis Lua script that atomically decrements the counter and checks whether the remaining count is negative" is more evaluatable than "we use a token bucket algorithm." The burst model must document: the maximum burst size (the bucket capacity), whether the burst capacity is intentionally equal to the per-window limit (allowing a caller to consume their entire hourly allowance in a single second) or intentionally lower (a burst capacity of 100 tokens for a 1,000 tokens/hour limit, restricting bursts to 10% of the hourly allowance), and the downstream resource impact assessment (does the maximum burst exceed the downstream database's connection pool size, the downstream API's per-second limit, or the endpoint's compute capacity under concurrent execution?). The window size must be documented with the rationale for the chosen window duration — per-second limits protect against instantaneous bursts but require fast Redis counters; per-minute limits protect against minute-scale abuse; per-day limits support contractual quota enforcement but accumulate counter state for the window duration. Multiple window tiers are common ("100 requests per minute AND 5,000 requests per day") and must be documented as separate enforcement layers with an explanation of how they interact: a caller who has consumed their daily quota but not their per-minute quota is throttled by the daily limit, and the per-minute counter continues incrementing (consuming from a quota that is already exhausted) until the daily window resets. The window boundary burst vulnerability of the fixed window algorithm, if chosen, must be documented explicitly: "a caller can send up to 2× the per-window limit in a burst around the window boundary; this is evaluated and accepted because downstream resources have been load-tested at 2× normal traffic and can sustain the burst without capacity constraint."

The third section documents the distributed enforcement model and the infrastructure consistency requirements. The counter backend must be named (Redis Standalone, Redis Sentinel, Redis Cluster, Memcached, a managed rate limiting service) with the rationale for the selection. The Redis configuration must document the replication model and its rate-limiting-specific implications: Redis Standalone (no replication; a single instance failure resets all counters and produces an enforcement gap equal to the failover time); Redis Sentinel (automatic failover with replication, with counter state continuity dependent on whether AOF persistence is enabled; failover takes 30–120 seconds during which the fail-open policy applies); Redis Cluster (horizontal sharding across multiple nodes, with each rate limit key assigned to a specific shard — the Lua script must be written to operate within a single hash slot to avoid cross-slot operations that Redis Cluster does not support). The persistence model must be documented with the rate-limiting-specific consequence: "Redis is configured without AOF persistence; rate limit counters are ephemeral and lost on Redis restart or migration; for per-minute and per-hour counters this is acceptable because the window resets naturally; for per-day and per-month counters this requires AOF persistence, database-backed counters, or counter export before maintenance operations." The fail-open versus fail-closed policy must be documented with the rationale, the maximum duration of the enforcement gap (the expected failover time), and the monitoring requirement: an alert that fires within 60 seconds of rate limiting enforcement becoming degraded, so that the engineering team is notified during a Redis incident before an abuse window opens. The migration policy for planned maintenance must document whether counter state must be exported and imported as part of the migration plan, and what the impact is if it is not — the counter reset produces an over-allowance equal to every active caller receiving a full fresh allowance at migration time, which must be within the infrastructure's absorption capacity during the migration window.

The fourth section documents the bypass and exemption policy. The internal caller exemption list must be enumerated explicitly — "the following caller identities are exempt from rate limiting: the internal health check endpoint caller (exempted because health check probes from the load balancer must not consume rate limit capacity), the monitoring agent (exempted because synthetic monitoring calls must not consume the monitoring account's rate limit and trigger false-positive 429 alerts), and the data pipeline import service (exempted because bulk import operations are authorized to exceed the standard per-minute rate limit under controlled conditions and use a separate per-day allowance instead)." The exemption mechanism must be documented — where the exemption list is stored (source-controlled configuration, a database table with an audit log, or Redis) — with a specific prohibition on ephemeral Redis storage for production exemptions unless the Redis instance has a documented recovery procedure for restoring the exemption list after a counter reset. The exemption addition process must specify who is authorized to add an exemption, what review and approval steps are required before a new exemption is active in production, and what documentation is required for each exemption (the caller identity, the reason for the exemption, the date added, and the review date — to prevent exemptions from accumulating indefinitely without review of whether they are still required). Enterprise contract rate limit overrides must be documented in the same exemption framework: the difference between an exemption (a caller not rate limited at all) and an override (a caller with a different rate limit than the standard tier) must be explicit, because treating a contracted override as a blanket exemption removes protection that should still apply even for high-entitlement callers. The GraphQL subscription decision record is relevant when rate limiting applies to subscription operations: subscription rate limiting (limiting the number of active subscriptions per caller, or the event delivery rate per subscription connection) has a different exemption policy from HTTP request rate limiting because subscription connections are long-lived and the exemption must apply for the connection's lifetime, not per-request. The exemption for internal subscription consumers (monitoring subscriptions, data pipeline subscriptions) must be documented as a connection-level exemption, not a per-request exemption, and the mechanism for identifying an internal subscription consumer at connection time must be specified.

The fifth section documents the observability model and the tuning cadence. The rate limiting system's observability must provide three views: the enforcement view (how many 429 responses were returned in the last period, broken down by identity and endpoint, to verify that the rate limiter is firing at the expected rate and not silently failing to enforce); the utilization view (what fraction of each consumer's rate limit allowance is consumed in a typical period, to identify callers near the limit who will likely experience throttling during traffic spikes, and callers far under the limit who would benefit from a higher entitlement without infrastructure risk); and the identity distribution view (what fraction of requests arrive with each identity type — authenticated with API key, authenticated with user identity, unauthenticated falling back to IP — to monitor whether the identity model is correctly attributing requests and to detect shifts in the caller population that may require identity model updates). The 429 response must include machine-readable response headers that inform callers of their current limit status: X-RateLimit-Limit (the limit value for the current enforcement tier), X-RateLimit-Remaining (the number of requests remaining in the current window), X-RateLimit-Reset (the Unix timestamp at which the current window resets and the counter returns to zero), and Retry-After (the number of seconds until the caller can retry). These headers must also be included on successful responses, not only on 429s, so that callers can implement proactive backoff before hitting the limit rather than reactive retry after receiving a 429. The tuning cadence must specify how often the rate limit values are reviewed: monthly reviews of the utilization distribution to identify whether the standard limit is set correctly for the caller population (a limit that is too low produces many callers near the limit and a high volume of legitimate 429s; a limit that is too high provides minimal protection because abusive callers can consume significant infrastructure capacity before being throttled); quarterly reviews of the identity model to verify that attribution remains consistent with the API's authentication model and the customer base's network topology; and trigger-based reviews after any production incident where rate limiting behavior was a contributing factor. The decisions never written down in a rate limiting system are the exemption inventory (which callers bypass the rate limiter and why), the burst model validation (whether the algorithm's burst allowance was tested against downstream resource capacity), and the consistency model during infrastructure events (what happens to enforcement during the Redis incidents and migrations that occur every 12–24 months and are guaranteed to interact with the rate limiting counter state in ways that the founding session's developer did not foresee).

The security hardening session that added rate limiting, the API productization session that differentiated tiers by rate limit, and the scaling session that distributed enforcement to Redis each produced API rate limiting decisions whose long-term operational cost — a 429-blocked enterprise customer behind a NAT gateway discovered via an account escalation two weeks after the contract signed, because the per-IP identity model was never evaluated against the enterprise network topology the sales team was actively pursuing; or an unthrottled traffic burst during a Redis migration that allowed automated integration partners to send 4× their contractual quota for eight minutes, because the counter state continuity requirement was never documented and the migration was planned using the same procedure as a general cache migration; or a silent internal service exemption that removed rate limiting from a microservice's API calls for three years after the engineer who added it left the company, because the exemption existed only in Redis and was never reflected in code, configuration, or documentation — exceeds what a consumer identity model document, a burst model validation, and an exemption policy would have cost at the time the founding decisions were made. The decisions are in the AI sessions: the per-IP model rationale, the in-memory counter that was "good enough for now" before the second server process was added, the Redis key that was set manually to exempt the enterprise customer's NAT gateway IP because "we'll formalize it later." WhyChose's open-source extractor surfaces these founding rate limiting sessions as structured decision records before the next NAT gateway escalation, the next Redis migration counter reset, or the next exemption-list key deletion in a maintenance flush reveals the undocumented operational commitments as an unplanned engineering sprint or an enterprise customer incident. The decisions never written down in a rate limiting deployment are the consumer identity model, the bypass exemption inventory, and the distributed enforcement consistency guarantee — the three operational commitments that determine whether rate limits protect infrastructure while preserving the fair-use ceiling for legitimate heavy users, or become the mechanism by which the largest paying customers are blocked during peak usage and the team discovers, in an incident post-mortem, that the rate limiting system they believed was protecting them had been silently bypassed for eighteen months by a Redis key that nobody knew existed.

Further reading