2026-08-01 · ~24 min read · Decision records

The service discovery decision record: why the discovery mechanism you chose determines your stale endpoint surface and your rolling deployment failure rate

Service discovery decisions are made in three founding sessions that never document the consequences — the "extract to microservices" session that picks DNS-based service discovery without specifying the DNS TTL contract or the client resolver caching behavior, the "add health checks" session that wires health probes without specifying the deregistration procedure or the acceptable deregistration latency, and the "debug the rolling deployment 503s" session that discovers the client services are bypassing the Kubernetes Service by caching individual pod IP addresses that no longer route to running pods. What none of these sessions produce is the service discovery mechanism specification, the health check deregistration latency contract, the client DNS resolution policy, or the load balancing model that determines whether inter-service traffic routes to dead endpoints during deployments and how long each dead endpoint stays reachable.

A 40-person platform engineering team at a B2B payments company ran their inter-service communication across twelve microservices through Kubernetes Services, relying on the Kubernetes cluster DNS to resolve service names to cluster IP addresses. The deployment worked correctly for the first fourteen months — rolling updates to any individual service completed without errors, and the cluster IP abstraction meant that pod replacement during updates was invisible to calling services. In month fifteen, the team migrated their payment processing service from a Python Flask application to a Go gRPC server. The gRPC migration changed the inter-service protocol from HTTP/1.1 to HTTP/2 multiplexed over long-lived TCP connections. Three other services called the payment processor: the order service, the invoicing service, and the reconciliation service. All three were JVM-based Spring Boot applications that had been running since the platform's initial deployment.

The first rolling update to the new gRPC payment processor completed in eleven minutes. Within seconds of the update completing, the on-call engineer's Slack received six simultaneous alerts: payment processing was failing at a 32% error rate. The errors were connection refused, not gRPC status codes — the gRPC clients in the three calling services were attempting to connect to IP addresses that no longer had a running process behind them. The Kubernetes cluster IP for the payment processor service appeared healthy in the Kubernetes dashboard: the Service resource showed three running pods, all passing readiness probes, all receiving traffic according to the Service's load balancer. But the three calling services were not sending traffic through the cluster IP. They had resolved the payment processor's DNS name to individual pod IP addresses during startup — eighteen weeks earlier — and had maintained persistent HTTP/2 connections to those specific pod IPs ever since. The cluster IP's virtual routing was not in the traffic path. When the rolling update replaced the old pods with new pods running at different IP addresses, the calling services' persistent connections to the old pod IPs received connection refused. The gRPC client retry logic attempted to reconnect — to the same old pod IPs, because the resolver had not been asked to re-resolve the name. The connections failed permanently.

Diagnosis took 45 minutes. The Kubernetes Service appeared healthy because it was healthy — from the cluster's perspective, three pods were passing readiness probes and the Service was routing correctly. The traffic path that was broken was not visible in the Kubernetes dashboard: the persistent HTTP/2 connections bypassing the cluster IP by going directly to pod IPs were connections to external-to-Service infrastructure that Kubernetes had no visibility into. The root cause was that the gRPC client library the team had chosen resolved DNS once at client construction time and held the resolved IP addresses in the connection pool without re-resolving. The DNS name had resolved to the pod IPs of the headless Service variant used by the gRPC default configuration rather than the cluster IP. The headless Service returned the individual pod IPs; the gRPC client connected to those pods directly; the pods were replaced during the rolling update; the connections failed. None of this was documented in the founding "extract to microservices" AI chat sessions that picked the architecture. The headless Service vs. cluster IP distinction had been resolved by the gRPC client's default configuration, not by a deliberate architectural choice. The DNS TTL behavior — that gRPC's default DNS resolver caches resolutions indefinitely by default — had not been a decision. It had been an omission.

A 22-person infrastructure startup operated a hybrid deployment across AWS EC2 and on-premise hardware, using Consul as a service registry. Each service instance registered itself with Consul on startup, advertising its IP address, port, and a set of tags indicating the service version and environment. Health checks were configured as Consul TTL checks: each service instance was responsible for calling the Consul API at a regular interval to renew its TTL, signaling to Consul that the instance was still alive. The TTL was set to 60 seconds, and each service instance called the renewal API every 20 seconds. If Consul did not receive a renewal within 60 seconds, the check transitioned to the critical state and Consul removed the instance from the set of healthy endpoints returned to service discovery queries.

In month nine, the company began running compute-intensive workloads on AWS Spot Instances to reduce infrastructure cost. One of the Spot Instance pools ran backend API instances. On a Tuesday afternoon, three Spot Instances in the pool received a 2-minute interruption notice — the standard AWS Spot Instance interruption signal. The application code on those instances did not handle the interruption signal (SIGTERM from the EC2 spot interruption handler). The instances continued running for 90 seconds after the interruption notice, then were terminated by AWS without completing a graceful shutdown. The Consul deregistration call that each instance would have made during a graceful shutdown did not execute. The three instances stopped calling the Consul TTL renewal API at the moment of termination. Consul's 60-second TTL meant that the three instances would remain in the healthy endpoint set for up to 60 seconds after the last TTL renewal — which was up to 80 seconds from the time of termination (20 seconds had elapsed since the last renewal when the interruption occurred, leaving 40 seconds of TTL; the instances ran for 90 seconds after the interruption notice, so the last renewal happened 40 seconds before termination, leaving 20 seconds of TTL after termination). In practice, the three instances stayed in the healthy endpoint set for between 20 and 60 seconds after being unreachable.

During that window, the API gateway's Consul-backed load balancer continued routing approximately 25% of requests (three of twelve registered instances) to the dead instances. Each request to a dead instance produced a TCP connection timeout after the client's 5-second timeout period. For 20 to 60 seconds, one in four API requests was failing with a 5-second timeout visible to end users. The total error count during the incident was approximately 600 timed-out requests across an average request rate of 100 requests per second and a deregistration window of 40 seconds for the three instances combined. The on-call engineer identified the root cause quickly — Consul showed the three instances transitioning to critical within 60 seconds of the interruption — but the incident's user-visible impact had already occurred. The fix was straightforward: configure the EC2 Spot interruption handler to call the Consul deregistration API before the instance terminated, reducing the deregistration window from 60 seconds to near-zero for graceful spot interruptions. But the requirement — that Spot Instance interruptions must trigger immediate Consul deregistration — had never been documented in the service discovery decision record, because no service discovery decision record had been written. The Consul TTL duration, the TTL renewal interval, and the interaction of those parameters with the spot interruption signal had all been determined by defaults and by the engineer who set up the initial Consul integration, without a discussion of the deregistration latency that the company's SLA could tolerate.

Structural properties set by the service discovery design decision

Three structural properties are determined when a team decides how services locate each other. None appear explicitly in the founding session that picks a service discovery mechanism — they are the operational consequences of choices made under the pressure of getting the microservices architecture to work before the first production deployment.

Property 1: The discovery mechanism and the stale endpoint surface. A service endpoint is stale from the moment the instance it points to stops handling requests. The stale endpoint surface is the window during which a client holds a reference to a stale endpoint and may route requests to it. The stale endpoint surface is determined by the discovery mechanism and by the client's caching behavior.

In Kubernetes DNS-based discovery using a cluster IP Service (not headless), the stale endpoint surface for the client is near-zero: the client always sends to the stable cluster IP, and Kubernetes updates the cluster IP's routing to exclude unhealthy pods within seconds of a readiness probe failure. The stale endpoint surface exists inside the Kubernetes data plane — there is a propagation delay between a pod failing a readiness probe and the cluster's kube-proxy updating the iptables or eBPF routing rules on each node — but this propagation delay is typically under 5 seconds and is not visible to the client. In Kubernetes DNS-based discovery using a headless Service (clusterIP: None), the DNS record resolves to individual pod IPs, and the stale endpoint surface is determined by the DNS TTL configured on the Service and by the client's DNS resolver TTL. Kubernetes sets the DNS TTL for pod A records to 30 seconds by default in CoreDNS; a client that caches the DNS resolution for 30 seconds may route to a terminated pod for up to 30 seconds after termination. A client that ignores the DNS TTL and caches indefinitely (the JVM default with networkaddress.cache.ttl=-1) may route to a terminated pod until the connection fails and the client's connection pool is flushed.

In Consul-based discovery with a client-side load balancer, the stale endpoint surface is the health check deregistration latency — the time between the instance failing its health check and being removed from the list of healthy endpoints returned by Consul's catalog API. This is typically the health check interval multiplied by the failure threshold. In AWS App Mesh or Istio with a sidecar proxy, the stale endpoint surface is the xDS update propagation delay — the time between the control plane updating the endpoint set and the sidecar proxy receiving and applying the update. This is typically under 5 seconds for a small cluster. The container orchestration decision record documents the deployment platform; the service discovery ADR must specify which discovery mechanism is used and document its stale endpoint surface, because the stale endpoint surface determines the health check configuration required to bound error exposure during instance failures.

The stale endpoint surface also determines the interaction with rolling deployments. A rolling deployment replaces service instances one by one (or in small batches), with a configurable rollout speed. If the client's stale endpoint surface exceeds the time between when an old instance is terminated and when the client refreshes its endpoint list, the client will route to terminated instances. The rolling deployment configuration — maxUnavailable, maxSurge, and terminationGracePeriodSeconds in Kubernetes — must be chosen to ensure that old instances remain alive for longer than the maximum stale endpoint window. The deployment strategy decision record documents the rollout approach; the interaction between the rollout speed, the terminationGracePeriodSeconds, and the service discovery stale endpoint surface must be specified in both documents, because a change to the rollout configuration that shortens the instance lifetime may create a stale endpoint window that the clients cannot absorb.

Property 2: The health check model and the deregistration latency. A service instance can fail in three ways: graceful shutdown (the process receives a signal and handles it, calling deregistration before exiting); crash (the process exits unexpectedly without handling the signal); or degraded availability (the process is running but not able to handle requests successfully — out of memory, database connection exhausted, thread pool full). The health check model must detect all three failure modes and trigger deregistration within the acceptable latency window.

Graceful shutdown deregistration is the easiest case: the process handles SIGTERM, calls the service registry's deregistration API or fails its Kubernetes readiness probe, waits for in-flight requests to complete, and exits. The deregistration happens immediately, the stale endpoint window is the propagation delay from the registry to the clients. The health check is the fallback for graceful shutdown failures — if the SIGTERM handler does not complete (the process is killed with SIGKILL, the handler times out, or the spot interruption terminates the instance before the handler completes), the health check detects the failure and triggers deregistration through its normal path. For crashes, the deregistration latency is health_check_interval × failure_threshold: the health check runs at its configured interval, and an instance is marked unhealthy after a configured number of consecutive failures. For degraded availability — the process is alive but not serving requests — the health check must differentiate between "alive and unhealthy" (TCP check passes because the port is open, but HTTP check fails because the application is returning 503) and "alive and healthy" (both checks pass). A TCP-only health check will not detect degraded availability if the port is still open; an HTTP health check against a purpose-built health endpoint will detect it if the health endpoint reflects the application's actual ability to handle requests.

The deregistration latency must be specified in the service discovery ADR and compared to the client timeout. If the client timeout is 5 seconds and the deregistration latency is 30 seconds, a client will experience 6 timeouts per dead instance per 5-second window before the instance is deregistered. At 10 requests per second per client, a 30-second deregistration window for one dead instance produces 300 timeout errors per client before the instance is removed from the healthy set. The observability strategy decision record documents the alerting infrastructure; the health check failure alert — fired when an instance transitions to the unhealthy state in the registry — must be a separate alert from the application error rate alert, because a health check failure may be self-correcting (the instance recovers before the deregistration latency expires) or may indicate a systemic failure (multiple instances failing simultaneously), and the response procedures differ. The alert on simultaneous multi-instance health check failures is the higher-severity event and must fire within the deregistration latency window to allow the on-call engineer to intervene before the error count reaches the SLA threshold.

Property 3: The load balancing model and the traffic distribution consistency. Once a client knows which endpoint addresses are healthy, it must decide how to distribute requests across them. The load balancing model determines whether the distribution is consistent, how quickly it adapts to endpoint changes, and what happens to requests in flight when an endpoint is removed.

Server-side load balancing — the client sends all requests to a stable proxy address (an AWS Application Load Balancer, an HAProxy instance, an Envoy proxy), and the proxy distributes requests across the backend pool — centralizes the endpoint list maintenance in the proxy. The proxy knows the current healthy endpoint set, distributes traffic according to a configured algorithm (round robin, least connections, IP hash for session affinity), and handles endpoint removal by draining existing connections to the endpoint before fully removing it from the pool. The client's view of service discovery is simple: one address. The cost is the additional network hop through the proxy and the single-point-of-failure risk of the proxy itself (mitigated by running the proxy in an HA configuration). The load balancer decision record documents the proxy infrastructure; the service discovery ADR must specify how the proxy's backend pool is populated and updated — whether from a static configuration, from a Consul catalog query, from the Kubernetes Endpoints API, or from another source — because the proxy's backend pool update mechanism determines the deregistration latency at the load balancer level.

Client-side load balancing — the client process holds a list of healthy endpoint addresses and distributes requests itself — eliminates the proxy hop and allows the client to implement load balancing algorithms that are aware of request-level semantics (gRPC subchannel pick policies, consistent hashing for cache affinity). The cost is that each client must maintain an up-to-date view of the healthy endpoint set. In gRPC, the client-side load balancer uses a resolver to obtain the initial endpoint list and watches for updates; the default DNS resolver re-resolves at the DNS TTL interval; the xDS resolver subscribes to a control plane's streaming API and receives updates within seconds. The choice of gRPC resolver determines the stale endpoint surface on the client side. The service mesh decision record documents whether the team uses a service mesh; if a sidecar proxy (Envoy, Linkerd) is deployed, the sidecar can intercept outbound connections and provide client-side load balancing with xDS-based endpoint updates without requiring any change to the application code, eliminating the need for application-level resolver configuration.

Connection affinity is the hidden load balancing problem in HTTP/2 and gRPC deployments. HTTP/2 multiplexes multiple requests over a single long-lived TCP connection. If the client maintains one TCP connection to each backend and all backends are healthy, requests are distributed at the connection level during connection establishment — but once the connections are established, the distribution is sticky to the connections. A client that opens one connection to each of three backends distributes approximately one-third of requests to each backend. A client that opens one connection to one backend and then adds two more backends distributes new requests to the new backends but all in-flight requests on the first connection continue to go to the first backend. The load balancing algorithm applies at connection establishment time, not at request time, unless the client implements request-level subchannel picking (which gRPC's round-robin policy does). The service discovery ADR must specify whether the team's inter-service protocol uses short-lived connections (HTTP/1.1 without keep-alive) or long-lived connections (HTTP/2, gRPC), and how the load balancing model handles the connection affinity property of long-lived connections.

What the founding session records and what it omits

Service discovery is implemented across multiple phases: an initial phase when the team extracts the first microservice and needs the calling service to find it, a second phase when the team adds health checks after the first production incident involving a dead endpoint, and a third phase when the team debugs inter-service errors during rolling deployments and discovers the underlying stale endpoint problem. None of these phases produces a service discovery ADR. Each phase addresses the immediate symptom, leaving the underlying design — the discovery mechanism choice, the DNS TTL policy, the health check deregistration latency, the load balancing model — undocumented and discoverable only through the next incident.

Three types of AI chat sessions generate these gaps:

The "how do services find each other?" session. The trigger is the first microservice extraction: the team has split one component out of the monolith and the remaining monolith (or other services) needs to call the new service. The engineer asks how to configure inter-service communication in the chosen deployment platform — Kubernetes, ECS, Nomad, or a VM-based setup. The session explains how to create a Kubernetes Service resource, how to reference it by name, how the DNS resolution works at a high level, and how to configure the calling service's HTTP client with the service name. What the session does not ask or answer: whether the team should use a headless Service or a cluster IP Service, and what the implications of that choice are for gRPC or other HTTP/2 protocols; what the DNS TTL on the Service's DNS record is, and whether the client's DNS resolver respects that TTL; whether the client HTTP library maintains a persistent connection pool and whether that connection pool re-resolves DNS on connection failure or on a time interval; what the expected behavior is when the service is updated and old pods are replaced with new pods — specifically, whether clients that were successfully communicating with old pods will automatically start communicating with new pods without errors, and over what time window; and whether the team will need cross-cluster or cross-environment service discovery (services in one Kubernetes cluster calling services in another cluster, or services on EC2 calling services on-premise) that Kubernetes cluster DNS does not support. The microservices vs. monolith decision record documents the architectural decision to decompose; the service discovery mechanism is the first operational consequence of that decision and must be specified in the same decision record or an immediately following one, because the discovery mechanism choice is a foundational constraint on every subsequent microservice extraction.

The "add health checks" session. The trigger is typically a production incident where a service instance crashes, is replaced, or becomes degraded, and the calling services route traffic to the bad instance for longer than expected — producing errors that the on-call engineer attributes to "a brief service disruption" without investigating the deregistration latency that extended the disruption. The engineer asks how to add health checks to the service so that bad instances are detected and removed from load balancing quickly. The session explains how to configure Kubernetes liveness and readiness probes, or how to configure a Consul health check, or how to configure an AWS ALB target group health check. What the session does not ask or answer: what the acceptable deregistration latency is for the service, given the service's SLA and the client timeout configuration; how to calculate the maximum number of client errors produced by a dead instance during the deregistration window, and whether that number is within the error budget; what the difference between a liveness probe and a readiness probe is in Kubernetes — specifically, that a failed readiness probe removes the pod from the Service's Endpoints list (which is what prevents traffic routing to a degraded pod) while a failed liveness probe restarts the pod (which is what triggers recovery from a crash), and that the team almost certainly wants a readiness probe, not a liveness probe, for degraded availability detection; what the health check endpoint should check — whether it should check the database connection, the downstream service connections, and the application's critical internal state, or just return 200 if the process is running; and how the health check interacts with graceful shutdown — specifically, that in Kubernetes, a pod that receives SIGTERM must immediately fail its readiness probe to be removed from the Endpoints list, and that the terminationGracePeriodSeconds must be long enough for the traffic drain to complete before the process exits. The circuit breaker and resilience decision record documents the client-side error handling policy; the health check configuration and the circuit breaker configuration must be designed together, because the circuit breaker's error threshold and open time determine how quickly the client stops routing to a dead endpoint independently of the health check deregistration latency — a circuit breaker that opens after 5 consecutive errors provides a client-side safeguard that limits error exposure even when the health check deregistration window is long.

The "services can't find each other during deployments" session. This session occurs after the first rolling deployment that produces inter-service errors — connection refused, connection timeout, or failed DNS resolution — that do not appear during normal operation. The engineer searches for "Kubernetes rolling update 503 errors" or "microservices DNS caching deployment" and asks the AI to explain why the deployment is causing errors. The session may explain the DNS TTL caching problem, the gRPC DNS resolver behavior, or the headless vs. cluster IP distinction — but the explanation is reactive, in response to the specific error observed, not proactive documentation of the properties that prevent the error class. What the session does not produce: a documented DNS TTL policy for all services — the specific TTL configured on each service's DNS record, the specific resolver TTL configured in each client, and the test that verifies the client honors the resolver TTL; a documented rolling update procedure that specifies the minimum terminationGracePeriodSeconds for each service based on the maximum stale endpoint window of its callers; a documented gRPC resolver policy — whether all gRPC clients in the codebase use the default DNS resolver (which does not re-resolve on connection failure by default) or a configured resolver that re-resolves at the DNS TTL; and a documented test for rolling deployment correctness — a staging environment test that runs a rolling deployment of each service while its callers are under load and verifies that the error rate during the deployment does not exceed the acceptable threshold. The CI/CD pipeline decision record documents the deployment pipeline; the service discovery rolling update correctness test must be part of the staging deployment gate in the CI/CD pipeline, not a post-incident remediation, because the failure mode is silent under low load (errors are rare or retried successfully) and only becomes visible under production load during a deployment.

The WhyChose extractor surfaces the "how do services find each other," "add health checks," and "deployment errors" sessions from AI chat history. The service discovery ADR converts the implicit choices in those sessions — using a headless Service by default, using the JVM's default DNS caching policy, using a TTL check without checking the TTL duration against the client timeout — into a documented discovery mechanism specification that identifies which discovery model is in use and what its stale endpoint surface is, a documented health check deregistration latency target and the check configuration required to meet it, a documented DNS TTL policy that includes the client-side TTL configuration for each language runtime in the stack, and a documented rolling update procedure that specifies the terminationGracePeriodSeconds for each service based on its callers' maximum stale endpoint window.

The five sections of a service discovery ADR

Section 1: Discovery mechanism and registry model. Document the discovery mechanism as a specific choice from the set of available options for the deployment platform, with the reasoning for that choice over the alternatives. In Kubernetes, the choice is between cluster IP Services (stable virtual IP, server-side load balancing via kube-proxy, recommended default), headless Services (DNS returns individual pod IPs, client-side load balancing, required for stateful applications like Kafka and Cassandra that need clients to address specific instances), external name Services (DNS alias for external services), and third-party registry integration via the ExternalDNS controller or a service mesh. Document which mechanism is chosen for each service class: stateless HTTP services (cluster IP Services with a sidecar proxy for gRPC), stateful data services (headless Services with client-side DNS-aware discovery), and external services (ExternalDNS or hardcoded external names with explicit configuration). The infrastructure-as-code strategy decision record documents how infrastructure is managed; the service registry configuration — whether Consul agent configs, Kubernetes Service manifests, or AWS Cloud Map namespace definitions — must be codified in the IaC repository so that adding a new service requires creating a service definition file that captures the health check configuration, the DNS TTL, and the discovery mechanism, rather than applying an ad-hoc configuration that is not reviewed or version-controlled.

If the team uses a service registry (Consul, AWS Cloud Map) in addition to or instead of platform-provided DNS, document the registry's consistency model and the propagation latency from the registry to the clients. Consul uses the Raft consensus algorithm for consistency and provides eventual consistency for catalog reads — a service registration may take up to a few seconds to propagate to all Consul servers and be visible to catalog API queries. The propagation latency from Consul registration to the load balancer routing traffic to the new instance is: Consul consensus latency + Consul catalog API poll interval on the load balancer side. If the load balancer polls the Consul catalog every 10 seconds, a new instance may take up to 10 seconds to receive traffic after registering. Document the expected registration propagation latency and the expected deregistration propagation latency separately, because the deregistration path — health check failure → Consul marks critical → load balancer removes from pool — has a different latency profile from the registration path and the acceptable latency differs: a new instance that is slow to receive traffic is an availability reduction (fewer instances handling traffic); a dead instance that is slow to be removed is an error producer (requests to dead instances).

Section 2: DNS TTL configuration and client resolver caching behavior. Document the DNS TTL configured on each service's DNS records and the DNS resolver TTL configured in each language runtime in the stack. Kubernetes configures the DNS TTL for Service DNS entries via the --min-ttl flag on CoreDNS; the default is 5 seconds for most CoreDNS configurations, though some distributions configure 30 seconds. Verify the actual TTL by running dig +nocmd my-service.my-namespace.svc.cluster.local @kube-dns and reading the TTL from the answer section. Document the TTL value and the CoreDNS configuration that produces it, so that changes to the CoreDNS configuration are understood to affect the stale endpoint window for all services.

For each language runtime in the stack, document the DNS resolver TTL behavior and the configuration required to make it honor the DNS record's TTL. JVM: the default is to cache DNS resolutions indefinitely (networkaddress.cache.ttl = -1 in the JVM security policy). Set networkaddress.cache.ttl = <N seconds> in the application's JVM security policy to make the JVM re-resolve at the DNS TTL. The value should be equal to or slightly less than the DNS record's TTL. Go: the standard library's net.Resolver honors the DNS TTL by default, but the HTTP transport's connection pool maintains open connections that bypass DNS resolution after the initial connection. Set DisableKeepAlives: true in the HTTP transport for services where connection reuse is less important than endpoint freshness, or configure DialContext to use a fresh resolver per connection. Node.js: the dns.lookup function (used by default by the Node http module) uses the OS resolver cache and may cache indefinitely depending on the OS configuration; use dns.resolve4 with explicit TTL tracking if honoring the DNS TTL is required. Document the test that verifies each client's resolver TTL behavior: a test that starts a mock DNS server returning a TTL of 5 seconds, has the client make a connection, changes the mock DNS server's response to a different IP, waits 6 seconds, has the client make another connection, and asserts that the second connection goes to the new IP.

For gRPC specifically, document the resolver policy. gRPC's name resolution is handled by a resolver plugin separate from the OS DNS resolver. The default gRPC DNS resolver re-resolves when a subchannel fails or when the client explicitly calls ResolveNow(); it does not re-resolve on a TTL-based timer by default. If the gRPC client needs to refresh the endpoint list on a time-based schedule (because the cluster IP may change, or because the team uses a headless Service that returns pod IPs), configure the service_config with a loadBalancingConfig that specifies a resolver that refreshes on the DNS TTL. Alternatively, use the xDS-based gRPC resolver that subscribes to a control plane's streaming API — this provides sub-second endpoint updates without relying on DNS TTL at all, at the cost of requiring a running xDS control plane (Envoy's management server, Istio pilot, or a custom xDS server). The DNS architecture decision record documents the DNS infrastructure; the service discovery DNS TTL policy must be reviewed alongside the DNS architecture because changes to the DNS serving infrastructure (migrating from a custom CoreDNS configuration to a managed DNS service, for example) may change the TTL behavior for service DNS records.

Section 3: Health check protocol and deregistration procedure. Document the health check protocol for each service class. For HTTP services: an HTTP GET health check against a purpose-built /healthz or /readyz endpoint that returns 200 if the service is ready to handle requests and a non-2xx status code if it is not. The endpoint must check the service's ability to handle requests — not just that the process is running, but that the database connection pool has available connections, that any required downstream services are reachable, and that the application's critical internal state (job queue not full, cache not exhausted) is within normal operating parameters. The endpoint must respond within the health check timeout; a response that takes longer than the timeout is treated as a failure by most health check implementations. For TCP services: a TCP connection check that verifies the port is open and accepting connections. For TTL-based checks (Consul): document the TTL duration and the renewal interval, and verify that the renewal interval is less than half the TTL to provide a buffer against transient renewal delays.

Document the deregistration procedure for each failure mode. Graceful shutdown: the process handles SIGTERM, fails its readiness probe immediately (returns 503 from /readyz), waits for the Kubernetes endpoint slice propagation delay (typically 5–10 seconds), then begins draining in-flight requests and waiting for completion, then calls the service registry deregistration API if using a sidecar or Consul agent, then exits. The terminationGracePeriodSeconds in the Kubernetes pod spec must be set to: endpoint propagation delay + maximum request duration + deregistration call latency + a buffer (typically 5–10 seconds). If a service has request handlers that can run for up to 30 seconds (long-lived uploads, slow database queries), the terminationGracePeriodSeconds must be at least 50 seconds. Document the terminationGracePeriodSeconds for each service and the basis for that value. Crash or out-of-band kill: the health check detects the failure within health_check_interval × failure_threshold. Document the maximum deregistration latency for crashes and the calculation that produces it. For Spot Instance or preemptible VM scenarios: document the interruption signal handling — the process must handle the spot interruption notification (EC2 instance metadata endpoint, GCP preemptible VM shutdown notice) and deregister immediately, treating the interruption notice as an early SIGTERM with a defined deadline. The secrets management decision record documents credential management; the service registry token or certificate used by the service instance to authenticate its deregistration call must be a short-lived credential that is rotated per-instance and revoked when the instance deregisters, rather than a long-lived credential that is shared across instances and cannot be individually revoked.

Section 4: Load balancing model and connection management. Document the load balancing model as a choice between client-side and server-side, with the reasoning for that choice given the service class and the protocol. For HTTP/1.1 services: server-side load balancing via a Kubernetes Service with kube-proxy (iptables or eBPF) load balancing is the recommended default. The load balancing is connection-level (each new TCP connection is load-balanced), which distributes requests evenly across backends for short-lived connections. For gRPC and HTTP/2 services: server-side load balancing at the TCP level does not distribute requests evenly because HTTP/2 multiplexes multiple requests over one connection; a single long-lived HTTP/2 connection that goes to one backend routes all subsequent requests to that backend regardless of the server-side load balancer. Use a proxy that load-balances at the HTTP/2 stream level (Envoy, Linkerd, or an Istio ingress gateway) rather than at the TCP connection level; or use client-side load balancing with a gRPC round-robin subchannel pick policy that distributes requests across multiple subchannels, each to a different backend. The caching strategy decision record documents the caching layer; if the services use consistent hashing for cache affinity (directing requests for a given cache key to the same backend that has that key in memory), document the consistent hashing ring configuration and how it interacts with endpoint additions and removals — adding an endpoint changes the hash ring and redirects some keys to the new endpoint, which may cause cache misses during the transition period.

Document the connection pool configuration for each service. For HTTP/1.1 connection pools: MaxIdleConns (maximum number of idle connections maintained in the pool), MaxIdleConnsPerHost (maximum idle connections per backend host), IdleConnTimeout (how long an idle connection is kept before being closed). For HTTP/2: MaxConcurrentStreams (maximum number of concurrent requests per connection), the KeepAlive timeout, and the connection health check (HTTP/2 PING frames) interval. Document the connection draining behavior when an endpoint is removed from the load balancer's pool: whether in-flight requests on connections to the removed endpoint are allowed to complete, and for how long (the drain timeout). A drain timeout of 0 means connections to the removed endpoint are immediately closed, dropping in-flight requests. A drain timeout of 30 seconds means in-flight requests have 30 seconds to complete before the connection is forced closed. The drain timeout must be longer than the maximum expected request duration for the service.

Section 5: Cross-service authentication and mTLS integration. Document how services authenticate to each other, because service discovery and service-to-service authentication are coupled: a service that has discovered a healthy endpoint must also be able to authenticate that the endpoint is the legitimate service instance it expects, not a rogue process that has registered under the same service name. In Kubernetes without a service mesh, services inside the cluster trust each other by default — any pod can call any Service without authentication. This is appropriate for small, low-risk deployments where the cluster boundary is the trust boundary, but it means that a compromised pod can call any service in the cluster. For inter-service authentication: mutual TLS (mTLS) requires each service instance to present a certificate during the TLS handshake; the calling service verifies the certificate is signed by the cluster's internal certificate authority (CA) and that the certificate's common name or SAN matches the expected service name. This prevents a rogue process from accepting traffic as a legitimate service. A service mesh (Istio, Linkerd) can provide mTLS automatically via the sidecar proxy, with certificate rotation handled by the control plane. Without a service mesh, mTLS requires each service to manage its own certificate lifecycle — requesting a certificate from the cluster CA (Kubernetes has a built-in CertificateSigningRequest API), renewing it before expiry, and distributing the CA certificate to all services that need to verify peer certificates. The VPC network topology decision record documents the network-level security boundaries; the service-to-service authentication model must be specified in the service discovery ADR or a companion zero-trust networking ADR, because the authentication model determines whether the discovery mechanism is also the authorization boundary or whether authorization is handled at the application level. The zero trust network access decision record documents the network access policy; if the team implements zero trust, every inter-service request must be authenticated and authorized regardless of whether the requester is inside the cluster network, and the service discovery ADR must specify how service identity is established (SPIFFE/SPIRE identities, Kubernetes service account tokens, mutual TLS certificates) and how that identity is passed to the authorization layer.

Further reading