2026-09-19 · ~22 min read · Decision records

The health check design decision record: why the probe semantics you configured determine your cascading restart exposure and your false availability signal surface

Liveness probe scope, readiness probe depth, and startup probe timeout are container health check decisions that are almost never made explicitly — they emerge from the default configuration of the first Kubernetes deployment and accumulate their consequences as service count, dependency graph complexity, and deployment frequency grow. Three failure patterns: the team whose liveness probe database query converts a recoverable database slowdown into a pod restart storm that extends an 8-minute degradation into a 23-minute outage; the team whose deep readiness probe takes the entire application offline when a non-critical external payment processor runs a maintenance window, blocking 97% of user actions that have no dependency on the payment processor; and the team whose startup probe timeout was calibrated against a lightly loaded staging node and begins blocking rolling deployments when production nodes reach sustained high CPU utilization during a quarterly reporting cycle.

A 31-person SaaS company built a project management and documentation platform for architecture and infrastructure teams — a tool that tracked engineering projects, stored technical documentation, managed team assignments, and generated quarterly progress reports for engineering leadership. The platform ran on Kubernetes with three application services: a core API service, a background job runner, and a notification dispatcher. The team had configured health checks for all three services during the initial Kubernetes migration two years earlier. For the API service, they had configured a liveness probe that made an HTTP GET request to a /health endpoint. The /health endpoint executed a SELECT 1 query against the PostgreSQL database and returned HTTP 200 if the query completed within five seconds and HTTP 500 if the query failed or exceeded the timeout.

The configuration made immediate sense when it was written. The team had experienced connection pool exhaustion bugs in their previous non-Kubernetes deployment where the application process was alive but no longer able to reach the database, and a restart had been the fix each time. The liveness probe was designed to catch that condition automatically. The logic was: if the service cannot reach its database, it cannot serve requests, so it should be restarted.

On a Tuesday afternoon, a background analytics job triggered an unintentional full table scan on the primary PostgreSQL instance. The job was generating a monthly project summary report, querying a table that had grown from 50,000 rows during development to 4.2 million rows in production, using a filter column that had no index. The query had run without incident in staging — where the table had 12,000 rows — and had completed in under a second in the last two production runs when the table was smaller. At 4.2 million rows, the query took 14 seconds. During those 14 seconds, the database CPU spiked to 94% and query response times across all other connections increased to between 3 and 7 seconds. The SELECT 1 queries from the liveness probe, which normally completed in under a millisecond, began taking 4 to 6 seconds, exceeding the 5-second probe timeout on approximately 60% of probe attempts.

Three of the five API service pods had their liveness probes fail. Kubernetes marked them as failed and initiated restarts. The three pods were killed and began their restart sequence. When the replacement pods came up and attempted their first liveness probe checks, the database was still under load from the analytics query. The first SELECT 1 from the restarted pods also exceeded five seconds. Two of the three restarted pods failed their liveness probes before they could pass even once and entered CrashLoopBackOff. Kubernetes applied a 10-second backoff delay before the next restart attempt. Meanwhile, the two pods that had not failed their initial probes began receiving all incoming API traffic. Under the concentrated load, their own database query response times increased. Their liveness probe checks, competing with production API queries for database connection pool slots, began timing out as well. Within four and a half minutes of the analytics job starting, all five API service pods were either in active restart cycles or failing their liveness probes.

The analytics query completed in 14 seconds. PostgreSQL performance returned to normal. But the pods that had been killed during the degradation were now cycling through exponential restart backoff — the pods killed once were waiting 10 seconds, the pods killed twice were waiting 20 seconds, the pods killed three times were waiting 40 seconds. The liveness probe timeout that had been set to catch permanent connection pool failures was now catching nothing at all — the database was healthy — but the pods were still in backoff cycles started during the degradation window. Full service recovery took 23 minutes from the start of the analytics job. The root cause of the 23-minute service disruption was not the database performance issue — which was self-resolving and had lasted 14 seconds — but the liveness probe configuration that converted a transient database slowdown into a cascade of pod restarts that took far longer to recover from than the original condition.

The probe configuration had been set during the Kubernetes migration using the official Kubernetes documentation's health check example, which showed a liveness probe calling an HTTP endpoint. The team had built a "thorough" health endpoint that checked database connectivity because they associated database unreachability with the connection pool bug they had solved with manual restarts before. The decision that included a database query in the liveness probe — which implicitly accepted that any database performance degradation severe enough to exceed a 5-second timeout would trigger pod kills — had never been written down, never reviewed against the question of what conditions a pod restart would actually fix versus what conditions a pod restart would make worse, and never examined against the team's history of production incidents. The liveness probe semantics — what the probe's pass/fail result was supposed to mean, and what Kubernetes would do when the probe failed — had been accepted at face value from a getting-started example without being evaluated as a decision about pod lifecycle management under degraded conditions.

A 38-person SaaS company built an e-signature and document workflow platform — a tool that allowed businesses to create signature requests, route documents to signatories, track completion status, and archive signed contracts for compliance. The platform integrated with four external services: a PDF rendering service for document generation, an email delivery provider for signature request notifications, a payment processor for subscription management, and a cloud object storage provider for document archiving. The engineering team had designed a "comprehensive" readiness probe endpoint at /health/ready that verified the application's connectivity to all integrated services. The endpoint made HTTP requests to each external service's health or status endpoint and returned HTTP 200 only if all four checks passed within a 2-second timeout per check. The readiness probe on all three application pods was configured to call this endpoint with a failureThreshold of 2 and a periodSeconds of 10.

The design intent was defensive: if the PDF rendering service was down, signature requests requiring PDF generation would fail with a user-visible error; if the email service was down, signature request notifications would not be delivered and users would not know they had documents pending signature. The team reasoned that a pod incapable of completing a signature request workflow should be removed from the load balancer. Removing it from the load balancer would prevent the user from receiving an error response.

On a Thursday morning, the payment processor ran a scheduled maintenance window that the team had not been informed of. The maintenance window lasted approximately 8 minutes. During the maintenance window, the payment processor's external status endpoint returned HTTP 503. The /health/ready endpoint on all three pods made its routine payment processor connectivity check, received a 503, and returned HTTP 500. The Kubernetes readiness probe failed on all three pods. Kubernetes removed all three pods from the service's endpoint list. The application became inaccessible — the load balancer had no healthy endpoints to route traffic to.

During those 8 minutes, the platform's core functions — creating signature requests, viewing documents, tracking completion status, downloading signed contracts — were completely unavailable. None of these functions involved the payment processor. Payment processor calls were only made during subscription checkout and plan management, which accounted for approximately 3% of user sessions and zero of the platform's core document workflow operations. The readiness probe had taken 100% of user actions offline to protect against failures in a dependency that was relevant to 3% of user actions, while simultaneously blocking the 97% of user actions that had no dependency on the payment processor at all.

The customer support team received 47 tickets during the 8-minute window. The engineering team's incident response began six minutes after the application went offline, when the on-call alert for "no healthy endpoints in the API service" fired. The investigation identified the payment processor 503s as the cause in about three minutes. The payment processor maintenance window ended before the incident response team had finished documenting the cause. Service was restored automatically when the payment processor's status endpoint began returning 200 and the readiness probes passed. Total user-visible outage: 8 minutes plus the time for two successful readiness probe cycles to restore the endpoints (approximately 20 seconds). Forty-seven support tickets for a maintenance window in a payment processor the application had not been using during the outage.

The readiness probe design had never been reviewed against the dependency graph of the application's actual request types. The team had listed its four external integrations and checked all four in the readiness probe, treating each integration as equally critical to the pod's ability to serve requests. The dependency graph analysis — which integrations are required for what fraction of user actions, and which integrations affect readiness in the sense that their failure means the pod should not receive traffic — had never been performed. The decision about health check depth was made implicitly when the health endpoint was first written: "check all the things we integrate with, comprehensive is better than shallow." The tradeoff between false negatives (missing a real availability failure) and false positives (marking the pod unavailable when most of its functions work fine) had never been articulated as the design question the readiness probe needed to answer. The decision had been made as a code choice — writing a health endpoint that called all four external services — without being labeled as a decision about what "ready to receive traffic" meant for this service and this dependency graph.

A 45-person B2B SaaS company built a data processing and reporting platform for enterprise clients in the financial services sector. The core service was a Spring Boot Java application that performed ETL transformations on client datasets, applied configurable business rules, and generated scheduled compliance reports. The service had substantial initialization overhead: it loaded the Spring application context (including a significant number of application beans, several database repositories, and a set of scheduled task definitions), established connection pools to two PostgreSQL databases, fetched and cached a 400 MB configuration dataset from object storage that was used by the business rule engine, and ran a startup health check that verified the configuration data was internally consistent before marking the application ready. On a well-provisioned node with available CPU, this startup sequence completed in approximately 35 seconds. The team had configured a startup probe with failureThreshold: 10 and periodSeconds: 3, providing a 30-second window for the startup probe to pass before Kubernetes would kill the pod and restart it. When they had set this value, the 30-second window seemed conservative — they had observed startup times of 20 to 28 seconds in testing and staging.

The observation had been made under development and staging conditions. The staging environment ran each pod on a dedicated node with no co-located workloads. Development laptops had 16 cores available to the JVM. The 20 to 28 second startup time reflected the best-case startup path: ample CPU for JVM class loading, no contention for memory or I/O during the 400 MB configuration fetch, no competing load on the connection pool endpoints during the database connection establishment phase.

Production Kubernetes nodes ran at a higher co-location density. Each node typically hosted between 8 and 14 pods across the platform's services, background job runners, and monitoring sidecars. During normal business hours, nodes ran at 40 to 65% CPU utilization. During quarterly reporting cycles — when the platform generated compliance reports for all enterprise clients simultaneously over a 3-day window — nodes ran at 75 to 90% CPU utilization. The increased CPU contention during quarterly reporting directly affected the startup time of new pods scheduled on these nodes: the JVM class loading phase, which is CPU-intensive and requires significant parallel I/O to load class files from the container filesystem, took 25 to 30 seconds longer under 85% node CPU utilization than under 10% utilization. The 400 MB configuration fetch competed with ongoing ETL worker I/O for object storage bandwidth. The startup sequence that completed in 35 seconds on a well-provisioned node took 55 to 70 seconds on a loaded production node during a quarterly reporting cycle.

A configuration change deployment was scheduled during the quarterly reporting cycle — a routine update to the business rule engine's parameter set that had been tested and approved through the team's change review process. The deployment triggered a rolling update. The first replacement pod was scheduled on a node running at 87% CPU utilization. The pod began its startup sequence. After 30 seconds, the startup probe had not received a passing response from the application — the Spring context initialization was at approximately 70% completion and the configuration data fetch had not yet begun. Kubernetes recorded the startup probe failure and killed the pod. The second startup attempt started on the same node. The node was now handling the resource release from the first killed pod in addition to its existing load. The second startup attempt also exceeded 30 seconds and was killed. Kubernetes applied a 20-second backoff delay before the third attempt. The rolling update was blocked — Kubernetes would not proceed to replace the next pod until the first replacement pod was healthy.

The on-call engineer noticed the deployment had stalled 40 minutes after it was initiated. The investigation identified the startup probe failures and CrashLoopBackOff state. Resolving the immediate issue required cordoning the heavily loaded node so the replacement pod would be scheduled on a less-loaded node, allowing startup to complete within 30 seconds. Once the pod was healthy on the less-loaded node, the rolling update could continue. The entire unblocking process — diagnosing the failure cause, cordoning the node, waiting for the new pod to schedule and start, completing the rolling update — took two hours. The configuration change that should have taken 15 minutes of automated rolling update deployment required two hours of manual intervention.

The startup probe timeout had been set at what the team considered a comfortable margin above the observed startup times. The observation window had been narrow — a staging environment that did not replicate production node density, with startup times measured under conditions that represented the best-case scenario rather than the worst-case scenario the probe timeout needed to accommodate. The decision about startup probe timeout — which implicitly defined "startup is taking too long and constitutes a failure" — had not been reviewed against the range of node load conditions the service would encounter, and had not been connected to the deployment risk of a probe timeout that would be exceeded under ordinary production load conditions during the platform's own planned high-utilization events. Nobody had asked: at what node CPU utilization does our startup time exceed the 30-second startup probe window, and does that CPU utilization occur regularly in production? The answer to both questions was yes. The startup probe timeout had been set without examining the question it was designed to answer.

Structural properties set by the health check decision

Three structural properties are determined when an engineering team establishes — or fails to establish — a health check design decision record: how the pod lifecycle responds to external dependency failures and transient degradations, how accurately the readiness signal reflects the pod's actual ability to serve requests across its full range of request types, and how robustly the startup probe accommodates the variance in startup time across the full range of node load conditions the service will encounter in production. None of these are labeled as decisions during the initial Kubernetes deployment — they emerge from the probe endpoints that were written during a migration sprint, the getting-started examples that informed the initial configuration values, and the conditions that were present during the initial testing that were not representative of the conditions the probes would encounter at scale.

Property 1: The liveness probe scope and the cascading restart exposure. A liveness probe answers one question: is this process stuck in an unrecoverable state that only a restart can fix? The canonical examples of states a restart fixes are: a deadlocked goroutine pool that is not processing work, a memory leak that has exhausted the heap with no recovery path, a corrupted in-memory state from a bug that requires a clean initialization to reset. The canonical examples of states a restart does not fix are: an external database that is temporarily slow, a downstream service that is temporarily unavailable, a network partition that will resolve in minutes. When a liveness probe checks external dependency connectivity, it converts external degradation events — which may be self-resolving — into pod kill events that are not self-resolving during a degradation window, because newly restarted pods encounter the same degraded external condition and fail their own liveness probes before they can pass. The cascade is self-amplifying: each killed pod concentrates traffic on the remaining pods, increasing their load, increasing their response times, making it more likely that their liveness probes will also exceed the timeout. The decisions never written down in the health check domain frequently include the liveness probe scope: the team that added a database query to the liveness endpoint had a reasonable motivation (they associated database unreachability with a bug that required manual restarts) but never wrote down that motivation as a decision record that connected the problem being solved (stale connection pools) to the failure mode being avoided (restart storms from transient external degradation) and evaluated whether the liveness probe was the correct mechanism for the first problem without introducing the second. The container orchestration decision record connects at the Kubernetes lifecycle semantics layer: the orchestrator's behavior — kill the pod when liveness fails, remove from load balancer when readiness fails, wait for startup probe before applying liveness — is determined by the orchestration platform choice and must be understood before the probe semantics can be configured correctly; a team that sets up probes without understanding the lifecycle consequences of probe failure is making configuration decisions without knowing what those decisions do.

Property 2: The readiness probe depth and the false availability signal. A readiness probe answers whether the pod should receive traffic from the load balancer. The answer to that question depends on the dependency graph of the pod's request types: for each category of incoming request, which dependencies are required to fulfill it? A dependency that is required for 100% of request types belongs in the readiness probe — if the pod cannot reach it, the pod cannot serve any traffic. A dependency that is required for 3% of request types does not belong in the readiness probe — its unavailability means 3% of requests will fail, which is a degraded service state that should be surfaced in monitoring and alerts, but not a state where the remaining 97% of requests should be blocked. The tradeoff is explicit: a shallow probe (checking only hard dependencies) produces false negatives — pods that are marked ready but fail some portion of requests because a soft dependency is unavailable. A deep probe that includes external third-party services produces false positives — pods that are marked not ready and removed from the load balancer because a non-critical external service has a transient failure. In practice, false positives in a readiness probe cause service outages that are indistinguishable from genuine availability failures and generate support tickets and incident response effort proportional to the blast radius of the false positive. False negatives from shallow probes produce elevated error rates for the subset of requests affected by the unavailable dependency, which is a more proportionate response to the actual failure scope. The observability strategy decision record connects at the monitoring signal layer: a deep health check endpoint — accessible at a separate path from the readiness probe, for use by monitoring dashboards and alerting rather than by the Kubernetes readiness probe — can report the status of all dependencies including external third-party services without affecting traffic routing; the distinction between "health information visible to monitoring" and "health signal that controls traffic routing" is an architectural choice that resolves the tension between wanting comprehensive dependency status visibility and wanting traffic routing decisions to be based only on the pod's core serving capability. The SLO and error budget decision record connects at the availability measurement layer: false positive readiness failures — applications marked unavailable when they could serve the majority of requests — consume error budget and affect SLO calculations in ways that are disproportionate to the actual user impact of the underlying dependency failure; the error budget accounting should distinguish between genuine availability failures and readiness probe false positives, and the health check design should minimize false positives to preserve error budget for genuine service degradation. The logging strategy decision record connects at the incident diagnosis layer: when a readiness probe false positive takes the application offline, the logs from the health endpoint must clearly identify which specific dependency check failed, with the HTTP status code and response time from the failed check, so that the on-call engineer can immediately identify the external service causing the false positive rather than spending incident time diagnosing which of several checked dependencies is the cause.

Property 3: The startup probe timeout and the deployment risk surface. A startup probe's pass/fail threshold defines the boundary between "startup is taking longer than expected due to load conditions or initialization variance" and "startup has hung and the pod needs to be killed and restarted." The boundary must be set at a value that correctly classifies the worst-case legitimate startup time as a pass, not a failure. If the boundary is set below the worst-case legitimate startup time — because the worst-case startup time was not observed during the calibration measurement, which was taken in a less-loaded environment — then deployments to production under high load conditions will fail, pods will enter CrashLoopBackOff, and rolling updates will block. The consequence of an incorrectly calibrated startup probe timeout is not visible in the steady state: pods that were started when nodes were lightly loaded will have passed their startup probes without issue. The consequence only surfaces during deployments, which is precisely the most sensitive operational window, when a deployment block requires manual investigation, node management, or configuration changes to resolve. The CI/CD pipeline decision record connects at the deployment validation layer: the pipeline can include a startup probe validation step that deploys the service to a staging environment with production-representative node density and CPU co-location, measures the startup time under load, and fails the deployment if the measured startup time is within a specified margin of the configured startup probe timeout — catching the calibration gap before a production deployment, rather than during one. The capacity planning decision record connects at the node resource modeling layer: the startup probe timeout calibration depends on the worst-case node CPU utilization the service will encounter during a deployment, which depends on the capacity model for peak utilization during planned high-load events (quarterly reporting cycles, batch processing windows, product launches) — the capacity plan that predicts the maximum expected node CPU utilization under sustained load also defines the worst-case startup time that the startup probe timeout must accommodate. The WhyChose extractor finds the health check configuration decisions in your AI session history — the initial Kubernetes migration session where the startup probe timeout was set using a value from the staging environment, the post-incident retrospective where a pod restart storm was diagnosed but the liveness probe scope was not identified as the root cause, and the quarterly planning session where someone said "we should review our probe timeouts before the Q3 reporting cycle" and the follow-up became stale before any action was taken.

The health check ADR: five sections

Section 1: Liveness, readiness, and startup probe scope definitions. Specify the scope of each probe type explicitly — what each probe checks, what conditions each probe is designed to detect, and what the Kubernetes lifecycle action that follows a probe failure is expected to accomplish. For the liveness probe: define it as checking only local process state — does the process have CPU, memory, and thread availability to handle incoming requests? The canonical liveness check is an HTTP GET to a health endpoint that returns 200 unconditionally if the process is able to receive and respond to HTTP, and 500 if the process is deadlocked, out of memory, or otherwise in a state where restarting is the correct recovery action. Liveness must not check external database connectivity, external service availability, or any I/O operation that can be affected by conditions outside the process. For the readiness probe: define it as checking whether the pod is in a state to receive and successfully process traffic — including warm-up completion, connection pool availability to hard dependencies, and the absence of any maintenance mode or graceful shutdown state. For the startup probe: define it as checking whether the application has completed its full initialization sequence including all startup-time I/O (config fetches, model loading, database migration runs, cache warming). Connect this section to the incident response playbook decision record: the playbook should describe what liveness probe failures indicate (the process itself is stuck, restart is appropriate) versus what readiness probe failures indicate (the process cannot serve traffic but is not stuck — investigate the dependency or wait for recovery before assuming a restart is needed), so that on-call engineers respond to probe failures with actions that match the probe semantics rather than treating all probe failures as restart-requiring failures.

Section 2: Readiness probe depth and dependency inclusion policy. Define the dependency inclusion policy that determines which of the service's dependencies should be checked in the readiness probe. The policy should classify dependencies into three tiers. Tier 1 — hard synchronous dependencies: services or data stores that the pod must reach to serve any incoming request at all, including the primary database, required caches, and internal services whose availability is required for every request path. Tier 1 dependencies are included in the readiness probe. If any Tier 1 dependency is unavailable, the pod should not receive traffic because every request will fail. Tier 2 — soft synchronous dependencies: services whose unavailability affects some request types but not others, including external APIs for non-core features, soft caches that have fallback paths, and integrations that serve a subset of request types. Tier 2 dependencies are not included in the readiness probe. Their unavailability should be surfaced through application-level error handling, metrics-based alerts, and a separate comprehensive health endpoint (at a path like /health/deep) that the monitoring system calls. Tier 3 — external third-party services: services operated by external vendors whose availability is not under the team's control, including payment processors, email providers, external APIs, and any service with a published SLA below 99.9%. Tier 3 dependencies are never included in the readiness probe, regardless of how the application uses them. Transient failures and maintenance windows in Tier 3 services must not produce application-level readiness failures. Document the tier classification for each dependency in the decision record, with the reasoning for the classification, so that when new dependencies are added they are explicitly classified at integration time rather than defaulting to inclusion in the readiness probe.

Section 3: Probe timeout and failure threshold configuration. Specify the configuration values for each probe type and the methodology used to derive them. For the startup probe: measure the 95th percentile startup time under the highest sustained CPU utilization the production nodes experience during planned high-utilization events (quarterly reporting cycles, batch processing windows, peak user load periods), and set the startup probe deadline — the product of failureThreshold and periodSeconds — to at least 1.5 times that measured value. Document the measurement conditions: the node CPU utilization at measurement time, the co-location density, the dataset size for any startup-time data loads, and the date of measurement (so the configuration can be reviewed when any of these conditions change). For the liveness probe: set failureThreshold to at least 3 and periodSeconds to at least 10, requiring a minimum of 30 seconds of consecutive probe failures before a pod kill is triggered. This provides time for transient conditions — GC pauses, brief network hiccups, momentary database slowness — to resolve before the kill threshold is reached. For the readiness probe: set a timeoutSeconds that is shorter than the application's p95 request latency, so that a degraded dependency is detected before the pod begins serving failed requests to users; and set a successThreshold of at least 2 before a pod is returned to the load balancer after a readiness failure, to prevent flapping when a dependency is intermittently available. Connect this section to the observability strategy decision record: probe failure events should produce log entries with the probe type, the failed check, and the response time, so that the frequency distribution of probe failures is visible in the monitoring system and can be used to detect calibration issues before they produce incidents.

Section 4: Graceful shutdown and readiness state transitions. Specify the graceful shutdown sequence that ensures the pod is removed from the load balancer before it begins rejecting new connections, and that in-flight requests are given sufficient time to complete before the process exits. The sequence has three phases. Phase 1: on receipt of SIGTERM, the application sets an internal shutdown flag and the readiness endpoint begins returning HTTP 503 immediately. This causes the readiness probe to fail and Kubernetes to remove the pod from the service's endpoint list. Phase 2: the application waits for a propagation window — typically 15 to 30 seconds — during which kube-proxy updates the endpoint list and the load balancer stops routing new requests to the pod. This window is implemented as a preStop lifecycle hook that sleeps for the propagation duration. Phase 3: the application drains in-flight requests — refusing new connections while allowing active requests to complete — and then exits. The terminationGracePeriodSeconds value must be set to encompass all three phases: Phase 1 is nearly instantaneous, Phase 2 takes 15 to 30 seconds, Phase 3 takes at most the p99 request duration plus a buffer. A total terminationGracePeriodSeconds of 60 seconds is appropriate for most web services. Document the expected Phase 2 duration from the network configuration (kube-proxy update propagation delay plus load balancer endpoint propagation delay) and the expected Phase 3 duration from the observed p99 request latency, so that if either changes, the terminationGracePeriodSeconds configuration can be updated accordingly.

Section 5: Health check testing and deployment validation. Specify the test coverage for the health check endpoints and the validation procedure for probe configuration changes. Unit tests should verify that the liveness endpoint returns 200 unconditionally when the process is running, that the readiness endpoint returns 503 when the shutdown flag is set, and that the readiness endpoint returns the correct response for each Tier 1 dependency state (healthy, unreachable, degraded). Integration tests in a staging environment with production-representative node density should verify that the startup probe deadline is not exceeded at the 90th percentile of startup times measured under the staging environment's load conditions — these tests should run as part of the deployment pipeline for any change that affects the service's initialization path (new dependencies added, configuration dataset size changes, database migration runs added to startup). Probe configuration changes — changes to failureThreshold, periodSeconds, timeoutSeconds, or the logic of the health endpoint itself — should require a load test in a staging environment that reproduces the worst-case startup time before the change is applied to production, and the load test result should be documented in the change review record. Connect this section to the CI/CD pipeline decision record for the specific gates in the deployment pipeline where health check validation runs and what failure in those gates means for the deployment proceed/block decision.

FAQ

What is the difference between liveness, readiness, and startup probes in Kubernetes?

Each probe type answers a different operational question. A liveness probe answers: is this process stuck in an unrecoverable state that requires a restart to fix? Kubernetes kills and restarts any pod that fails its liveness probe, so liveness should only check conditions that a restart will actually resolve. The canonical liveness probe is a trivial HTTP check to a health endpoint that returns 200 if the process can receive HTTP connections and 500 if the process is deadlocked, out of memory, or otherwise unable to process requests — not if an external dependency is unavailable. A readiness probe answers: should this pod receive traffic from the Kubernetes service and load balancer right now? Kubernetes removes any pod that fails its readiness probe from the service's endpoint list, so traffic stops being routed to it, but the pod is not killed and restarted. Readiness failure is recoverable without a restart — when the pod becomes ready again, Kubernetes adds it back to the endpoint list. Readiness probes check whether the pod is in a state to serve requests, including warm-up state after startup and graceful degradation states during rolling deploys. A startup probe answers: has the application finished its initial startup sequence? A startup probe replaces the liveness probe during the startup window — Kubernetes does not apply the liveness probe until the startup probe has passed, preventing liveness from killing a slow-starting pod that has not yet finished initialization. Startup probes are essential for services with significant startup time (JVM applications, services that load large models or datasets, services that run long startup migrations) where the liveness probe's failure threshold would be exceeded during normal startup.

What should a readiness probe check and what should it not check?

A readiness probe should check the internal application state that determines whether the pod can serve requests, plus connectivity to hard dependencies that are required for every incoming request type. For hard dependencies where every request type requires the dependency (a primary database, an internal cache that backs all read paths), a readiness check that verifies connectivity with a timeout shorter than the average request timeout is appropriate — if the pod cannot reach the dependency, it will fail every request. A readiness probe should not check external third-party services whose availability is not under the team's control (payment processors, email providers, external APIs), soft dependencies that only affect a subset of request types, or anything that involves network I/O to a service that is expected to have intermittent availability. The test for whether a dependency should be in the readiness probe is: if this dependency is unavailable, does the pod fail 100% of incoming requests? If yes, include it in readiness. If no, handle the degradation in the application layer and report it through a separate comprehensive health endpoint and monitoring alerts rather than through the readiness signal. A separate internal health endpoint at a different path (for example /health/deep) can check all dependencies including external ones and report partial availability without affecting traffic routing.

How do you choose the right timeout values for startup and liveness probes?

Startup probe timeout should be set to the worst-case startup time observed under the worst production load conditions, with a safety margin added on top. The worst case is not the typical startup time in development or staging — it is the startup time on a heavily loaded production node, after a cold pod eviction, during the peak load event the team regularly runs (quarterly reporting cycles, batch processing windows). Practically: measure startup time in production under the highest CPU utilization the nodes regularly see (not peak-of-all-time, but the 90th percentile of normal operating conditions), and set the startup probe deadline — the product of failureThreshold and periodSeconds — to 1.5 to 2 times that measured value. For a Java service that typically starts in 35 seconds on a lightly loaded node and 60 seconds on a loaded node, a startup probe with failureThreshold 20 and periodSeconds 5 (100-second window) provides an adequate safety margin. Liveness probe timeout should be generous enough to avoid false positives from transient slowness, with a failure threshold that requires multiple consecutive failures rather than a single failure. A liveness probe with timeoutSeconds 5, failureThreshold 3, and periodSeconds 10 requires three consecutive probe timeouts (30 seconds of consecutive probe failures) before triggering a restart — this prevents a single transient event (a GC pause, a brief network hiccup) from triggering an unnecessary pod kill. For most services, requiring 30 to 60 seconds of consecutive liveness probe failures before triggering a kill is the right calibration.

How should health checks interact with graceful shutdown?

When a pod receives a SIGTERM signal, it should immediately begin failing its readiness probe so that Kubernetes removes it from the load balancer endpoint list before the pod starts refusing new connections. Without this, there is a gap between when the pod starts shutting down and when Kubernetes updates the endpoint list, during which the load balancer may route new requests to a pod that is mid-shutdown. The implementation requires the health endpoint to track the pod's shutdown state: when SIGTERM is received, a shutdown flag is set, and the /health/ready endpoint begins returning 503 immediately. A preStop lifecycle hook that sleeps for 15 to 30 seconds then provides the propagation window — the time for Kubernetes to process the readiness probe failure, for kube-proxy to update the endpoint list, and for the load balancer to stop routing new requests to the pod. After the propagation window, the application drains in-flight requests and exits. The terminationGracePeriodSeconds value must be long enough to include the full propagation window plus the drain window for in-flight requests to complete. A common starting point is terminationGracePeriodSeconds set to 60 seconds, with a 15-second preStop sleep for propagation. The preStop sleep runs concurrently with SIGTERM delivery — terminationGracePeriodSeconds should be set to the full desired shutdown window, not preStop duration plus request drain time.

Further reading