What is the difference between a shallow health check and a deep health check, and when should each be used?

A shallow health check confirms that the server process is running and accepting connections without verifying any application-level functionality. A TCP health check confirms that the server is accepting connections on the designated port. An HTTP shallow health check sends a request to a lightweight endpoint — commonly /health or /healthz — and confirms that the server returns a 200 status code; the endpoint typically returns a fixed response body without executing any application logic. Shallow health checks have low false-positive rates — they fail when the process is down, which is a genuine failure condition — and they add negligible load to the server. Their limitation is that they cannot detect application-level failures where the process continues running but the application cannot serve requests correctly: a database connection pool that is exhausted and causing all request handlers to block, a cache connection whose reconnection logic has stalled, an internal queue that is full, or a configuration error introduced by a deployment that causes a specific endpoint class to return 500 errors without crashing the process. A deep health check adds active verification of the application's critical dependencies: the health endpoint executes a lightweight database query such as SELECT 1, confirms that the result is returned within a threshold time (indicating the connection pool has available connections and the database is responding), and optionally checks that the connection to other critical dependencies such as a cache or a message broker is available. Deep health checks have higher false-positive rates — a transient database timeout during peak load can cause a healthy backend to be temporarily removed from the pool — which is why the failure threshold (number of consecutive failures before removal) is typically set higher for deep checks than for shallow checks. The failure threshold must be calibrated against the expected transient failure duration: if temporary database connection interruptions during peak load last up to three seconds and the health check interval is ten seconds, a deep health check with a three-second timeout set to a failure threshold of two consecutive failures requires twenty seconds of continuous failure before a backend is removed — long enough to absorb most transients while still detecting genuine application-level failures that persist.

What is session affinity (sticky sessions) and when does it create operational problems?

Session affinity, also called sticky sessions, is a load balancer configuration that routes requests from the same client to the same backend instance consistently across requests within a session. Session affinity is required when the application stores session state locally on the backend — in the process's memory, in a local file, or in an in-process cache — and cannot serve a request that was initiated on a different instance because the session data is not accessible from instances other than the one that created the session. Without affinity in this configuration, a user whose second request is routed to a different backend than the first request finds no session data at the new backend and is treated as a new anonymous session, requiring the user to authenticate again. Session affinity solves this problem by ensuring that all requests from the same client within a session are routed to the same backend. Session affinity creates operational problems in three distinct scenarios. First, traffic imbalance from IP-based affinity: when affinity is implemented by routing all requests from the same source IP address to the same backend (IP-hash), corporate NAT appliances that assign a single egress IP to hundreds or thousands of internal users cause all of those users to be routed to the same backend instance simultaneously, while other backends receive a fraction of the traffic. Second, affinity loss during backend removal: when a backend is removed from the pool for deployment, a health check failure, or maintenance, all sessions that were affinitized to that backend are redistributed to other backends — but the session data stored locally on the removed backend is inaccessible from the new backend, so those users are effectively logged out. Cookie-based affinity mitigates the IP-hash imbalance problem by tracking affinity at per-client granularity rather than per-IP, but does not solve session loss on removal if session data remains stored locally. Third, migration difficulty: eliminating session affinity requires migrating all session state from local storage to a shared backing store accessible from all backends — an operation that typically requires either a maintenance window to migrate active sessions or accepting a burst of session invalidations during the migration. The migration path from local to shared session storage must be documented as part of the original affinity decision, or the affinity model becomes an unacknowledged constraint that persists long after the team has the shared session infrastructure that would make affinity unnecessary.

What is connection draining and what happens without it during a rolling deployment?

Connection draining is a load balancer behavior that allows in-flight requests to complete gracefully when a backend is removed from the pool before severing the connection. When a backend is removed — during a rolling deployment, after a health check failure, or for planned maintenance — there are typically in-flight requests that the backend has already accepted but not yet completed: a user's request executing a database query, a file upload in progress, an API call to an external service waiting for a response. Without connection draining, removing the backend from the pool immediately closes the connection between the load balancer and the backend, dropping all in-flight requests. The load balancer returns a 502 Bad Gateway or 503 Service Unavailable error to the caller, and any work already performed by the backend for those requests — a database row inserted, a payment initiation sent to a third-party gateway, a file partially written to object storage — is abandoned without a response being returned to the caller. During a rolling deployment, where backends are taken out of the pool one at a time to deploy a new version and then re-added after the new version starts accepting traffic, deploying without connection draining means that every deployment cycle drops some in-flight requests. In a system receiving 1,000 requests per second per backend with an average request duration of 200 milliseconds, removing a backend from the pool at any moment has an expected 200 in-flight requests outstanding. Without draining, all 200 are dropped. With a 30-second drain timeout, the backend is removed from the pool for new requests but continues serving the 200 in-flight requests until they complete; by the time the drain window expires, all normal-duration requests have completed. Long-running requests — file uploads, report generation, async task initiation — that exceed the drain timeout are still dropped when the timeout elapses, which is why the drain timeout must be calibrated against the longest expected request duration in the application. A default 30-second drain timeout that is not reviewed against the application's actual request duration distribution drops all requests that exceed 30 seconds, which may be acceptable or may not, depending on which endpoints have long-running request patterns.

The load balancer decision record: why the health check model you chose determines your failover speed ceiling and your traffic distribution surface

Published 2026-07-16 · WhyChose

Load balancing decisions are made when the team first deploys to production with more than one backend instance — typically when the application outgrows a single server, when a reliability requirement mandates more than one instance for redundancy, or when a deployment strategy requires running the new version alongside the old version briefly to switch traffic. The AI session that configures the load balancer is practically grounded and reaches a working state quickly: it selects a load balancer — nginx upstream, an AWS Application Load Balancer, HAProxy, a Kubernetes ingress controller, or a cloud provider's managed load balancing service — adds the backend instances to the pool, configures a balancing algorithm (round-robin or least-connections), sets a health check to remove backends that are not accepting connections, and verifies that requests are distributed across the backend pool and that bringing down one backend instance causes the load balancer to stop routing traffic to it. The session ships a load balancing configuration that works correctly in the test environment and that the team observes distributing traffic and failing over when a backend is taken down manually.

What the AI session does not produce is the load balancer's second half. The session answers "which backends, which algorithm, and how do we confirm traffic is distributed?" and delivers a working configuration. It does not ask: what is the health check specification — what does the health check endpoint actually verify, and what failure modes does the check detect versus what failure modes does it not detect, and if a backend's application layer is failing to serve requests correctly (database connection exhausted, ORM query timing out, cache disconnected) but the process itself continues running and returning 200 to the health endpoint, how long will the load balancer continue routing production traffic to the broken backend before that failure is visible? What is the session affinity model — does the application store session state locally on the backend process, and if so, are requests from the same client consistently routed to the same backend, and what client identity signal determines affinity (source IP address, a cookie injected by the load balancer, or a consistent hash of a user identifier embedded in the request), and what happens when a client whose requests have been affinitized to a specific backend connects from behind a corporate NAT appliance that makes many clients appear as the same source IP? What is the connection draining policy — when a backend is removed from the pool for a deployment, a health check failure, or planned maintenance, how long does the load balancer continue serving in-flight requests to the departing backend before severing the connection, and what happens to a file upload that is sixty percent complete or a payment transaction that has been initiated but not confirmed when the drain timeout elapses? What is the active-active failover model — if the load balancer itself is a single point of failure, what is the redundancy mechanism for the load balancer layer, and if the load balancer runs in an active-passive configuration, how long from the active node's failure to the passive node promoting itself and handling traffic, and what is the connection state during that window? What is the N-1 capacity model — if one backend is removed from the pool during peak traffic, do the remaining backends have sufficient capacity to handle the full load without exceeding their resource limits, and if not, what is the expected degradation and how is it monitored? Each of these questions has an answer that is not derivable from "we configured a round-robin load balancer with an HTTP health check on /health returning 200" as the team frames the founding configuration. Each answer determines whether the load balancer routes production traffic to a backend for thirty-four minutes while that backend's ORM is returning 500 errors to every caller, whether an enterprise customer's 400 employees are all routed to the same backend instance because a corporate NAT presents them as a single IP to the IP-hash affinity model, and whether every rolling deployment silently drops the in-flight requests of the users whose requests happen to land on a backend during the precise moment it is removed from the pool. The answers exist in the AI sessions. They are the operational commitments behind the load balancer configuration. They are almost never written down.

Two ways load balancing decisions produce the wrong outcome in production

The shallow health check and the application-level failure that never trips it

A fintech API platform deploys its first multi-instance production configuration in its third year of operation. Until this point, the API ran on a single application server behind an nginx reverse proxy, and any downtime meant an outage. As the user base grows and uptime requirements become part of enterprise contract SLAs, the team adds two additional application server instances and configures an AWS Application Load Balancer in front of all three. The founding deployment session configures the ALB: the target group health check sends an HTTP GET to /health on port 443 every thirty seconds, marks a backend unhealthy after two consecutive failures (60-second detection window), and marks it healthy again after three consecutive successes (90-second re-admission window). The /health endpoint is a one-line route handler: it returns HTTP 200 with a JSON body of {"status":"ok"} without executing any application logic, checking database connectivity, or verifying that the ORM or connection pool is in a functional state. The session tests the configuration by stopping one backend instance and confirming that the ALB stops routing traffic to it within sixty seconds. The configuration ships.

Eight months after the ALB is deployed, the team runs a database schema migration as part of a product release. The migration changes the column type of a frequently-queried field from VARCHAR to TEXT — a compatible change at the database layer — but the team's ORM version has a known bug where the column type change causes the ORM's internal type coercion to fail silently, returning malformed query objects that the database rejects with a "column type mismatch" error on every query that touches the affected field. The migration runs successfully; the ORM bug is not caught in the staging environment because staging uses a separate database instance where the column type change was applied three days earlier and the ORM type coercion cache had since been rebuilt. In production, the migration is applied, the new application version is deployed to all three backends in a rolling update, and all three backends come up successfully — the process starts, the Node.js event loop is healthy, and the /health endpoint returns HTTP 200. But the first production POST /transactions request that reaches any backend triggers the ORM bug: the query to write the transaction record fails with "column type mismatch," the ORM returns a 500 Internal Server Error to the caller, and every subsequent POST /transactions request on all three backends produces the same error.

For thirty-four minutes, the ALB routes POST /transactions requests to all three backends, all three return HTTP 500, and the health check returns HTTP 200 on all three. The monitoring alert fires when the per-endpoint P99 error rate threshold is exceeded at the ten-minute mark, the on-call engineer investigates, identifies the ORM error in the application logs, and initiates a rollback. The rollback takes twenty-four minutes to complete across three instances in the rolling update. The health check's failure detection mechanism never triggered during the incident: the check was configured to detect process-down failure, which did not occur, and was not configured to detect ORM query failure, which did. The post-incident review identifies that the /health endpoint specification was never documented: there was no written record of what the endpoint was expected to verify (process health only) versus what failure modes it would not catch (application-layer errors that do not crash the process), no discussion of whether a deep health check that executed a database query would have been appropriate given the team's SLA requirements, and no acknowledgment that the gap between "process up" and "application healthy" is a deliberate operational trade-off that must be documented and covered by an alternative detection mechanism (error rate monitoring, synthetic transaction monitoring, application performance monitoring) rather than discovered during an incident review.

The IP-hash session affinity and the enterprise customer behind a corporate NAT

A B2B SaaS collaboration platform launches with session state stored server-side in the filesystem — each user's active session is written to a file in /tmp/sessions/ on the application server that handled the login request. The founding deployment session adds a second application server for redundancy and configures nginx as a load balancer with IP-hash session affinity: all requests from the same source IP address are consistently routed to the same backend by computing a stable hash of the source IP and mapping each hash range to a specific backend. The rationale is practical: the team cannot change to shared session storage (Redis is not yet in the stack) without a larger engineering investment, and IP-hash affinity means that each user's requests consistently reach the backend where their session file was written. The test environment shows two test accounts routing to different backends, both maintaining sessions correctly. The configuration ships.

Fourteen months after launch, the platform signs its largest enterprise customer — a 400-person professional services firm. Within two days of onboarding, the support team receives escalating reports from the enterprise's users: the platform is "extremely slow" compared to what the team uses for internal testing. The engineering team investigates and discovers that the enterprise customer's corporate network routes all 400 users through a single NAT appliance with one external IP address. The IP-hash algorithm maps that one IP address to Backend A. Backend A is handling all 400 concurrent sessions from the enterprise customer plus its proportional share of the remaining user base. Backend A's CPU utilization is at 97%, its filesystem I/O is saturated from concurrent session file reads and writes, and its application response time has degraded to six to eight seconds per request. Backends B and C are at 15% CPU utilization, handling the remainder of the user base whose source IP addresses hash to those backends. The load balancer shows an 83% / 8% / 9% traffic distribution across three backends that were intended to share load equally.

The team cannot correct the imbalance without addressing the session affinity model, and they cannot change the session affinity model without first migrating session state from filesystem storage to a shared backing store. The migration requires building the Redis session store integration (not currently in the codebase), deploying Redis, migrating the session serialization format, and handling the transition window during which some sessions are in the filesystem and others are in Redis. The migration is planned for the next quarterly release — eight weeks away. During those eight weeks, the enterprise customer's 400 users continue experiencing degraded performance because the founding session affinity decision did not document: what the session storage model is, why IP-hash affinity was chosen, what the traffic imbalance risk under a corporate NAT would be (the current test environment uses residential IP addresses with natural IP diversity, masking the NAT scenario), and what the migration path to stateless routing would require. The decision was correct given the founding constraints (no Redis, filesystem sessions, need for session persistence across requests) but was never written down as a documented constraint with a known trade-off and a defined migration trigger.

Three structural properties that load balancing decisions determine

The health check model and the failure detection ceiling

The health check model determines the maximum speed at which the load balancer can detect a backend failure and the minimum set of failure modes that trigger removal from the pool. Health checks operate at three levels of depth, each with different detection coverage and false-positive risk. A TCP health check confirms that the server process is accepting connections on the configured port — it detects process crashes, port binding failures, and complete network connectivity loss, but nothing at the application layer. An HTTP shallow health check sends a request to a designated endpoint (commonly /health or /healthz) and confirms a 200 status code — it detects process crashes plus routing failures, but nothing beyond the ability of the server to accept and respond to a minimal HTTP request without executing application logic. An HTTP deep health check sends a request to an endpoint that actively verifies the application's critical dependencies — executing a database query and confirming a response within a threshold latency, checking the connectivity and responsiveness of a cache or message broker, confirming that an internal queue is not backed up beyond a configured depth — and returns a non-200 status code if any dependency check fails. The correct depth is a function of the failure modes the team wants the load balancer to handle automatically versus the failure modes they will handle through other detection mechanisms (error rate monitoring, synthetic transactions, application performance monitoring). A deliberate shallow health check — chosen because the team accepts the false-negative risk for application-layer failures and will rely on APM to detect those — is a documented trade-off. A shallow health check that is shallow because no one discussed the depth question is an undocumented constraint.

The health check timing model — interval, failure threshold, success threshold, and timeout — determines the detection and recovery speed. An interval of thirty seconds with a failure threshold of two consecutive failures produces a maximum detection window of sixty seconds from the first failed check to backend removal. A ten-second interval with the same failure threshold produces a thirty-second maximum detection window. But reducing the interval also increases the health check traffic to each backend: a thirty-second interval with ten backends produces twenty health check requests per minute across the pool; a ten-second interval produces sixty. For deep health checks that execute database queries, this matters: 60 database queries per minute from health checks is not negligible on a heavily loaded backend, and the health check itself can contribute to the database load the team is trying to monitor. The success threshold — the number of consecutive successful checks required before re-adding a recovered backend to the pool — must be documented alongside the failure threshold. A success threshold of three at a thirty-second interval means a backend that has recovered from a transient failure is held out of the pool for ninety seconds while it accumulates consecutive passing checks. During those ninety seconds, the backend is fully healthy and capable of serving traffic but is not doing so, which matters if the backend pool is running at or near capacity. The failure and success thresholds encode a directional bias: a low failure threshold (fail fast, remove early) prioritizes not sending traffic to a broken backend at the cost of more frequent false-positive removals; a low success threshold (re-add fast) prioritizes recovering pool capacity at the cost of re-adding a backend that may not be fully stable. The decision record must document both thresholds with the rationale: "we use failure threshold 2 because sixty seconds of sending traffic to a broken backend is our maximum acceptable detection latency; we use success threshold 3 because we have observed that recovered backends need ninety seconds to stabilize before they should receive production traffic, based on the recovery pattern from the 2025-Q3 database failover incident." The service mesh decision record is the complementary upstream document: in systems where a service mesh handles east-west traffic between microservices, the mesh's health check model and the external load balancer's health check model must be consistent in what they verify — a service mesh that marks a sidecar healthy based on the Envoy proxy's TCP acceptance while the application inside the pod is failing ORM queries will route internal traffic to a broken service the same way an undocumented shallow health check does.

The session affinity model and the traffic distribution surface

The session affinity model determines whether and how the load balancer routes requests from the same client to the same backend instance, and what the operational consequence is when that routing guarantee cannot be maintained. Session affinity is required by applications that store mutable state locally on the backend — in-process memory, filesystem session files, local SQLite databases, or in-process LRU caches — and cannot reconstruct that state from a shared store for requests that arrive at a different instance. The three most common affinity mechanisms differ in their per-client granularity and their sensitivity to network topology. IP-hash affinity routes all requests from the same source IP to the same backend: the source IP is hashed and the hash is mapped to a backend, so the same IP always reaches the same instance. IP-hash affinity requires no application or client changes and produces stable routing for clients with consistent source IPs. Its failure mode is the corporate NAT: a NAT appliance that assigns a single egress IP to many internal clients collapses those clients' traffic onto one backend instance, producing traffic imbalance proportional to how concentrated the user base is behind NAT appliances. Cookie-based affinity instructs the load balancer to inject a stickiness cookie (AWSALB, PHPSESSID-LB, or a provider-specific name) into the response to the first request from each client; subsequent requests carry the cookie and are routed to the same backend. Cookie-based affinity provides per-client granularity rather than per-IP granularity, eliminating the corporate NAT problem, but does not work for API clients that do not preserve cookies between requests. A mobile API client that calls the load balancer from a native SDK and does not implement cookie storage will not maintain affinity across requests. The decision record must specify which client types are expected to call each load-balanced endpoint and whether each client type preserves cookies.

The session affinity model must also document the planned migration path away from affinity, because IP-hash or cookie-based affinity is almost always a temporary accommodation for local session storage rather than a permanent architecture choice. The migration path is: move session storage from the local backend to a shared backing store (Redis for in-memory sessions, a database table for persistent sessions) accessible from all backends, update the session read and write paths in the application to use the shared store, verify that sessions survive backend removal and that any backend can serve any request, then remove the affinity configuration from the load balancer. Each step has a cost and a planned timeline. The decision record must name the session storage migration as a tracked technical debt item with a trigger condition — "we will migrate from filesystem sessions to Redis sessions when we have three or more backend instances in a single pool, or when a corporate NAT traffic imbalance is confirmed in production monitoring." A trigger condition converts the affinity decision from an indefinitely deferred assumption into a managed constraint with a defined resolution path. The container orchestration decision record is the upstream document in environments where the load balancer is implemented as a Kubernetes Service and ingress controller: Kubernetes's default Service routing model is stateless round-robin with no session affinity; enabling sessionAffinity: ClientIP on a Service activates IP-hash affinity at the kube-proxy level, but this affinity model has the same corporate NAT vulnerability as any IP-hash implementation, and it is not visible in the standard Kubernetes load balancing documentation as a capacity-limiting constraint.

The connection draining policy and the failover speed ceiling

The connection draining policy determines what happens to requests that are in flight when a backend is removed from the load balancer pool. In the absence of a documented draining policy, the default behavior depends on the load balancer implementation: some load balancers immediately sever connections on backend removal, returning 502 or 503 errors to in-flight callers; others drain passively by not sending new requests to the removed backend but allowing existing connections to close naturally; others enforce a configurable drain timeout. The consequence of immediate connection severing during a rolling deployment is that every deployment drops some fraction of in-flight requests — a fraction that is proportional to the system's requests per second per instance multiplied by the average request duration. For a system with 500 requests per second per backend instance and an average request duration of 150 milliseconds, the expected in-flight request count at any moment is 75 per backend. Each rolling deployment that removes a backend without draining drops those 75 requests. Across a three-instance rolling deployment that removes and re-adds each backend sequentially, 225 requests are dropped per deployment. If deployments happen daily, 225 dropped requests per deployment is a known, measurable, avoidable service degradation that the team can quantify and decide to prevent or accept — but only if the draining policy is explicitly designed and documented rather than implicitly determined by the load balancer's default behavior.

The drain timeout must be calibrated against the application's request duration distribution. A 30-second drain timeout is sufficient if all requests complete within 30 seconds, which is true for most synchronous web requests. But applications with long-running operations — file upload endpoints where large files take minutes to transmit, report generation endpoints that execute expensive queries across large datasets, async job submission endpoints that wait for a job to be acknowledged by a queue broker — have a subset of requests whose duration exceeds any fixed drain timeout. The decision record must classify requests by expected duration and specify the drain timeout alongside a statement of what the timeout implies for long-duration request classes: "30-second drain timeout; file upload requests to /api/uploads may take up to 300 seconds for large files; uploads in progress when a backend is removed will be dropped at 30 seconds; this is accepted because upload clients implement automatic retry for connection failures and resume from the last successful chunk." If long-duration requests are not retried automatically and cannot be safely abandoned mid-execution — a payment transaction that has been sent to a third-party gateway but not confirmed, a database migration that must complete atomically — the drain timeout must cover the maximum expected duration of those operations, or the deployment process must drain those specific endpoints before removing the backend from the pool. The connection draining policy interacts with the active-passive failover model: when a backend fails due to a health check failure rather than a graceful removal, there is no drain window — connections are transferred to remaining backends immediately. In this scenario, the question is not how to drain but how to design operations that are safe to retry or idempotent when transferred mid-execution to a different backend, which connects to the database read replica decision record: the replicas' replication lag at the moment of a backend failure determines whether a request that is retried on a different backend reads stale data from a replica rather than the freshly written primary.

Three AI session types that embed load balancing decisions without documenting them

The initial production deployment session is where the load balancer first appears. The session focuses on a concrete goal: route production traffic to multiple backends and confirm that removing one backend does not drop requests. The health check is configured to the minimum viable definition of "backend is healthy" — the backend is accepting HTTP connections and returning 200 — because the test scenario is "backend instance crashes or is stopped" and the chosen health check detects that scenario correctly. The session affinity model is configured based on the first constraint encountered: if the application has local session state, the session adds IP-hash affinity to preserve session consistency without examining what the corporate NAT risk model looks like or what the migration path to stateless routing requires. Connection draining is not configured explicitly, because the test deployment does not generate enough traffic to make dropped in-flight requests visible — in the test environment, draining without a configured timeout produces zero observable dropped requests because there are no concurrent users. The load balancer configuration that emerges from the founding deployment session is correct for the test scenario (single developer, no concurrent load, no corporate NAT clients, no long-running requests) but carries implicit decisions that are not documented as operational commitments: the health check model is shallow by default, the session affinity mechanism was chosen based on the first mechanism that worked rather than the mechanism appropriate for the expected client population, and the draining behavior defaults to whatever the load balancer's default is. The open-source extractor surfaces these founding deployment sessions from AI chat history, recovering the load balancer configuration rationale, the health check configuration choices, and the session affinity mechanism that was selected in the founding session before the team needed to understand the corporate NAT failure mode or the in-flight request drop behavior during rolling updates.

The scaling session is where the load balancer's implicit decisions surface as observed problems but are often corrected without being documented. The session is triggered by a capacity problem — the existing backend instances are at high utilization during peak load, and the team adds more instances to distribute the load. The session focuses on adding instances to the pool and verifying that traffic distributes across the expanded pool. If IP-hash affinity is configured, the team may observe that the traffic distribution across the expanded pool is uneven — some backends receive significantly more traffic than others. The on-call engineer investigates and discovers that specific source IPs are driving the imbalance, either because of NAT appliances or because the IP-hash algorithm produces an unequal hash distribution across the specific backend count (IP-hash is not perfectly uniform, and adding a backend changes the hash boundaries for all existing IP addresses, redistributing affinity assignments). The team may adjust the backend count to improve the distribution or switch to cookie-based affinity — but neither change is documented as a policy decision explaining why the change was made, what the previous behavior was, and what the remaining risks of the new configuration are. The scaling session fixes the observable symptom and moves on. The health check model is not revisited in the scaling session unless a backend failure has been recently observed that the health check did not catch.

The reliability hardening session is triggered by an incident: a deployment that dropped requests, a backend failure that the health check detected too slowly or not at all, a traffic imbalance that caused one backend to exhaust its resources while others were underloaded. The session adds deep health checks if the incident was an application-level failure that the shallow check missed, adjusts the drain timeout if the incident was dropped requests during a rolling deployment, or investigates and reconfigures the session affinity model if the incident was an imbalance from a NAT-concentrated user population. Each corrective action is implemented as a point fix for the incident — the deep health check endpoint is defined to detect the specific failure mode that the incident exposed, the drain timeout is set to approximately the duration of the longest request observed in the incident, the session affinity model is changed to cookie-based after the IP-hash imbalance is confirmed. What the reliability hardening session almost never produces is a decision record that documents the complete load balancer configuration as a system: the health check model as a specification of what is and is not detected and why, the session affinity model as a documented choice with a migration trigger, the drain timeout as a calibrated value with the request duration distribution analysis that supports it. The fixes are documented in incident post-mortems but not consolidated into a living architectural document. The next engineer who joins the team reads the incident post-mortems and learns what went wrong historically, but does not have a single document that describes what the load balancer configuration is committed to doing and what failure modes it is intentionally not designed to catch. The new CTO onboarding problem in a system with an undocumented load balancer configuration is the incident history archaeology: an incoming technical leader reads three post-mortems about health check gaps, two post-mortems about rolling deployment request drops, and one post-mortem about NAT-driven traffic imbalance, and must reconstruct the current configuration's intended behavior from the series of reactive fixes rather than from a document that states the intended behavior and the trade-offs it encodes.

The five sections of a load balancer decision record

The first section documents the health check specification. The check type must be named (TCP, HTTP shallow, HTTP deep) alongside the explicit statement of what the check verifies and what it does not verify. For an HTTP shallow check: "the /health endpoint returns HTTP 200 if the Node.js process is accepting connections; it does not verify database connectivity, ORM query success, or cache availability; application-layer failures that do not crash the process are detected by the error rate alert on the /transactions and /auth endpoint classes, not by the health check." For an HTTP deep check: "the /healthz endpoint performs a SELECT 1 query against the primary database replica, confirms the response in under 200 milliseconds, and performs a GET request against the cache cluster's designated health key; a check failure indicates loss of database connectivity or cache connectivity; transient ORM query failures under 200 milliseconds during peak load are expected to not trigger the health check failure threshold because the query timeout is 200 milliseconds and the failure threshold is two consecutive failures — a 400-millisecond transient does not accumulate two failures at a 30-second check interval." The health check timing parameters must all be documented as a set: interval, timeout, failure threshold, success threshold, and the implied detection and recovery latencies they produce. The test procedure for verifying the health check behavior must be specified: "health check failure detection is verified by stopping the database connection pool on a non-production instance and confirming that the instance is removed from the ALB target group within 75 seconds (2 consecutive failures × 30-second interval + 15-second propagation margin)." A health check specification that states only the parameters without the test procedure is incomplete — the parameters can be misconfigured and only the test procedure reveals it.

The second section documents the session affinity specification. The affinity mechanism must be stated with the rationale: "IP-hash affinity is configured because the application stores session data in filesystem session files; IP-hash is preferred over cookie-based because the initial client population is primarily residential users without NAT concentrations." Or: "no session affinity is configured because the application stores all session state in Redis and any backend can serve any request." If affinity is configured because of local session storage, the migration path must be stated: "IP-hash affinity will be replaced with no-affinity routing when the session storage migration from filesystem to Redis is complete; the migration trigger is any confirmed NAT-driven traffic imbalance exceeding 20% deviation from equal distribution, or any backend pool expansion beyond four instances." The client type coverage must be addressed: "cookie-based affinity is not effective for the mobile SDK clients that call /api/v1/*; those clients do not preserve cookies; the mobile SDK clients are stateless (session tokens are sent in the Authorization header, not in session cookies) and do not require affinity — affinity is only required for browser-based clients using /app/* routes." The section must document the consequence of affinity loss during backend removal: "when a backend is removed from the pool, IP-hash redistribution causes approximately 1/N of all existing affinitized sessions to be routed to a new backend; those sessions have filesystem session data on the removed backend that is inaccessible from the new backend; affected users will be logged out and must re-authenticate; this is accepted as a known consequence of filesystem session storage and is mitigated by the Redis session migration timeline."

The third section documents the connection draining specification. The drain timeout must be stated as a value with the calibration rationale: "30-second drain timeout, calibrated against the P99 request duration of 18 seconds observed in production metrics; the 30-second drain timeout covers all requests at the P99 duration with a 12-second margin; long-running requests on /api/reports (report generation, up to 5 minutes) are not protected by the 30-second drain timeout; report generation requests interrupted by a drain timeout respond with a 503 error; the report generation client retries automatically on 503 with a unique report_request_id parameter that allows the new backend to resume the generation, so interrupted reports are retried rather than lost." For each endpoint class whose maximum request duration exceeds the drain timeout, the decision record must state whether the behavior during drain (request drop, 503, redirect to a different backend) is acceptable and what the caller's recovery mechanism is. The drain trigger must be documented: "draining is triggered by the ALB target group deregistration API call; it is called automatically by the deployment pipeline before each backend is replaced; it is not triggered by health check failures — health check failures cause immediate removal from the pool without a drain window; requests in flight on a health-check-failed backend may be dropped." The distinction between graceful removal (deployment, planned maintenance) and health-check-triggered removal (failure response) must be explicit because they have different in-flight request behaviors.

The fourth section documents the failover model. If the load balancer layer itself is redundant (active-active or active-passive load balancer nodes), the failover mechanism, failover time, and connection state during failover must be documented: "two ALB nodes in active-active configuration; both nodes share the virtual IP via VRRP; if one ALB node fails, the surviving node handles all traffic; there is no failover window because both nodes are active; the single-node capacity of 50,000 connections per second is sufficient to handle peak production traffic of 18,000 connections per second." The N-1 capacity analysis must be documented for the backend pool: "the backend pool runs four instances at 40% CPU utilization under peak load; removing one backend during peak load increases the remaining three instances to 53% CPU utilization; this is within the acceptable operating range (target: below 70% to preserve burst headroom); the deployment drain window may temporarily result in three backends sharing traffic intended for four, which is within capacity." If the N-1 capacity analysis shows that removing one backend would push remaining backends above capacity, the decision record must state the mitigation: "the backend pool during a peak-load deployment must be temporarily expanded to five instances before removing a backend for update, to maintain four active backends throughout the rolling deployment; the deployment pipeline implements this by adding a wait-for-scale step before initiating the rolling update." The CDN decision record is the upstream document for the failover model in systems where the CDN and the load balancer operate in sequence: CDN failover (routing to a different CDN edge PoP) and load balancer failover (removing a backend from the pool) interact — a CDN failover that routes to a different edge PoP where the session affinity cookie is not recognized produces a session affinity miss, which may cause a logout for users with local session storage.

The fifth section documents the observability model. The load balancer's observability must provide five views that together characterize whether the configuration is providing the intended protection and whether any implicit assumptions are being violated in production. First, the per-backend request distribution: the fraction of total load balancer traffic that each backend receives, measured over rolling 5-minute windows; a distribution that deviates significantly from equal (for round-robin or least-connections) or from the expected IP-hash distribution (for IP-hash affinity) indicates a session affinity concentration from a NAT appliance, a routing configuration error, or an asymmetry in the backend pool's capacity or health check status. The alert threshold must be documented: "alert when any backend's traffic share deviates by more than 25% from the expected equal share (e.g., one backend receives more than 37.5% of traffic in a three-backend pool where equal share is 33%)." Second, the per-backend error rate: the fraction of requests routed to each backend that return non-200 status codes, broken out by backend instance; a backend whose error rate is significantly higher than its peers indicates a deployment failure, an application-level failure, or a resource exhaustion that the health check has not detected. An alert that fires when a single backend's error rate exceeds the pool average by more than 15 percentage points is the detection mechanism for application-level failures that shallow health checks do not catch. Third, the health check success and failure rate per backend: the count of health check successes and failures per backend per minute; a backend that is passing health checks but showing an elevated failure rate (flapping — passing most checks but failing occasionally) indicates intermittent connectivity issues or a resource exhaustion that is transient but recurring; a flapping backend that never accumulates consecutive failures sufficient to trigger removal may contribute to user-visible errors on the fraction of its health-check failure intervals when in-flight requests encounter its transient unavailability. Fourth, the connection count per backend: the number of active connections to each backend; a backend whose connection count is significantly higher than its peers may indicate a session affinity concentration (all connections from a NAT-dense user population are routed to one backend), a connection leak (connections are not being released after request completion), or a slow backend whose requests are taking longer than average and accumulating as outstanding connections while faster backends turn over connections more quickly. Fifth, the drain window duration during rolling deployments: the measured time from drain initiation to the point where in-flight request count reaches zero for each backend removed during a deployment; a drain window that consistently reaches the drain timeout (rather than completing early) indicates that requests are regularly being dropped at the timeout boundary, and the drain timeout should be extended or the endpoint with long-duration requests should be examined. The decisions never written down in a load balancer deployment are the health check depth specification (what the check does and does not verify), the session affinity model and its migration path, and the connection draining policy with its request duration calibration — three properties that together determine whether the load balancer detects application-level failures and routes around them, distributes traffic proportionally across the backend pool, and removes backends without dropping the in-flight requests of the users whose traffic happens to be in progress at the moment of removal.

The initial production deployment session that configured an HTTP shallow health check because the test scenario was "backend process crashes" and no other failure mode was tested; the scaling session that switched from IP-hash to cookie-based affinity after a NAT-driven traffic imbalance was observed but documented the change only in the incident post-mortem; and the reliability hardening session that set the drain timeout to thirty seconds based on the P99 of the incident's dropped requests without examining the tail of request durations above the P99 each produced load balancing decisions whose long-term operational cost — thirty-four minutes of an application-level ORM failure routing production traffic to three broken backends because the health check was shallow by default and the application-layer error rate monitoring alert threshold had not been configured; a 400-person enterprise customer's users all routed to a single backend because IP-hash affinity was still in place from the founding session and no NAT-driven imbalance had been observed in the prior user base; file uploads being dropped at thirty seconds during rolling deployments because the drain timeout was not calibrated against the upload endpoint's actual request duration distribution — exceeds what a health check depth specification, a session affinity migration trigger, and a drain timeout calibration analysis would have cost at the time the founding decisions were made. The decisions are in the AI sessions: the health check configuration that "adds /health returning 200," the IP-hash affinity that "solves the session problem for now," the drain timeout that "defaults to 30 seconds." WhyChose's open-source extractor surfaces these founding deployment sessions as structured decision records before the next application-layer failure routes production traffic to a broken backend for the duration of the monitoring alert threshold, the next enterprise customer's corporate NAT concentrates their entire team on one backend instance, or the next rolling deployment drops in-flight file uploads because the drain timeout was set to the load balancer default rather than calibrated against the application's actual request duration distribution at the tail. The decisions never written down in a load balancer deployment are the health check depth specification, the session affinity model with its migration path, and the connection draining policy with its request duration calibration — the three operational commitments that determine whether the load balancer is a reliable traffic distribution layer that detects failures promptly and handles backend transitions gracefully, or becomes the mechanism by which the founding deployment session's implicit assumptions about failure detection, client network topology, and request duration produce outages, imbalances, and data loss that are disproportionate to the simplicity of the configuration that started it all.