What is the difference between using environment variables and a secrets manager for runtime configuration, and when should a team switch?

Environment variables are the default configuration mechanism for twelve-factor applications: each service reads its configuration from the process environment at startup, and the environment is set by the deployment platform (container orchestration, shell scripts, or CI/CD tooling). Environment variables are universally supported, require no infrastructure beyond the deployment platform, and are simple to reason about. A secrets manager (AWS Secrets Manager, HashiCorp Vault, Google Cloud Secret Manager) is a dedicated service for storing secrets — credentials, API keys, TLS certificates — with additional capabilities that environment variables do not provide: access control at the secret level (which services can read which secrets), audit logging of every secret access (who read which secret and when), automatic secret rotation (the manager rotates credentials on a schedule and notifies consuming services), and versioning (multiple versions of a secret can exist simultaneously, allowing gradual rollout of a new credential before the old one is revoked). The transition point from environment variables to a secrets manager is typically driven by one of three conditions: (1) a security audit requires audit logging of credential access — environment variables are set at deployment time and not logged per-access; (2) the team needs to rotate a credential without redeploying all services that use it — environment variables require a restart or redeployment to pick up new values; (3) the number of secrets has grown to the point where managing them as environment variables in deployment configuration files creates a risk of secrets leaking into version control or CI/CD logs. Teams that start with environment variables and switch to a secrets manager later typically face a migration that requires updating every service to add the secrets manager SDK, updating deployment configuration to grant secrets manager access instead of setting raw environment variables, and deciding whether the existing environment variable names will be preserved as the names of secrets in the manager or whether a new naming convention will be adopted. The migration cost is proportional to the number of services and the number of secrets per service. The configuration management decision record that documents the choice of environment variables over a secrets manager should include the criteria for when the team will switch — the audit logging requirement threshold, the rotation frequency threshold, or the secrets count threshold — so that the migration is triggered by a known condition rather than by a security incident that makes the absence of the secrets manager costly.

What is environment parity and why does configuration drift between staging and production cause production-only bugs?

Environment parity is the property that staging and production (and development and any other environment in the deployment pipeline) are configured identically except for the values of environment-specific secrets and endpoints. A high-parity environment means that a bug reproduced in staging is caused by the same code path as the equivalent bug in production, and that a feature that works in staging will work in production under equivalent load. Configuration drift is the divergence of environment configurations over time — environment variable names that differ between environments, optional features that are enabled in production but disabled in staging, or infrastructure dependencies (Redis, a message queue, a third-party API) that are mocked or absent in staging but live in production. Configuration drift causes production-only bugs when the application behavior depends on a configuration value that is present in production but absent or different in staging. The category of bug is particularly costly because it cannot be reproduced in staging without first identifying that a configuration difference exists — which requires comparing the full configuration surface across environments, a task that is not straightforward when configuration is spread across environment variables, deployment manifests, and infrastructure-as-code files. The most common form of configuration drift is variable naming inconsistency: a variable that was named one way in the original staging environment and renamed in production during a migration, or a variable that was added to staging under one convention and added to production independently under a slightly different convention. The configuration management decision record that establishes a variable naming convention, a required set of variables for each environment, and a parity verification step in the deployment pipeline prevents most configuration drift. Without the decision record, each engineer who adds a new configuration value to the application makes an independent choice about the variable name, and the choice is made in whatever environment the engineer is working in at the moment — which may not be production. The drift accumulates until a production-only behavior reveals it.

Should an application fail to start if a required configuration value is missing, or start in a degraded mode?

An application should fail to start if a required configuration value is missing in almost all cases. The fail-fast-at-startup property means that a misconfiguration is detected at deployment time rather than at the moment a user request exercises the code path that reads the missing configuration value. The alternative — starting in a degraded mode when configuration is missing — defers the error to request time and produces a runtime error that may be difficult to attribute to a missing configuration value rather than a code bug. The degraded mode approach is appropriate in a narrow set of cases: when the application has a meaningful degraded state that it can communicate to users (for example, a caching layer that is unavailable falls back to reading from the database, and the application can continue serving requests without the cache at reduced performance), and when the degraded state is explicitly designed and documented as an intended operating mode rather than as an error state. The key distinction is between a missing optional configuration value — where the application has a documented default behavior that is appropriate for production — and a missing required configuration value, where the application cannot fulfill its core function without the value. A database URL is a required value: the application cannot serve requests without it, and starting without it will produce a runtime error for every request that accesses the database, which is worse than a clean startup failure that blocks deployment and alerts the deploying engineer immediately. An optional metrics endpoint URL is an optional value: the application can run without emitting metrics, and the appropriate behavior is to log a warning at startup and continue. The configuration management decision record should explicitly enumerate which configuration values are required (missing value causes startup failure) and which are optional (missing value produces a startup warning and a documented default behavior), and should specify the validation mechanism — a startup check function that reads all required values before the HTTP server starts accepting connections, a schema validation library like envalid or joi-env, or a custom validation step in the application initialization path. Without this enumeration, each engineer who adds a new configuration value makes an independent decision about whether to require it or default it, and the decision is rarely documented, producing a fleet where some missing values cause startup failures and others cause silent runtime errors depending on which engineer wrote the configuration-reading code.

How should a team implement zero-downtime secret rotation for a database credential?

Zero-downtime database credential rotation requires the application to support either hot reloading of the credential without a process restart, or a two-phase rotation protocol where the new credential is valid alongside the old credential for a transition window. Hot reloading means the application periodically re-reads the credential from the configuration source (secrets manager, Kubernetes secret, or environment variable set via a sidecar) and updates its connection pool with the new credential, replacing connections authenticated with the old credential as they are recycled. The connection pool manager must support credential refresh without draining all connections simultaneously — a pool drain causes a burst of new connections that can exceed the database's connection limit if all instances refresh at the same time. The hot reload approach requires the application to be designed for credential refresh from the beginning: the credential must be stored in a mutable reference that the connection pool reads on each new connection, not in an immutable value read once at process startup. If the application stores the database URL as a startup constant, hot reloading requires a code change before the first rotation. The two-phase rotation protocol avoids the hot reload requirement by using the database's ability to have multiple valid credentials simultaneously: in phase one, a new credential is created alongside the old credential (the database user is granted a new password, but the old password remains valid); in phase two, after all application instances have been restarted or have refreshed their connection pools with the new credential, the old credential is revoked. The transition window — the time between phase one and phase two — must be long enough for all application instances to complete the refresh. For a rolling restart on a container orchestration platform, the transition window is the time for a full rolling restart of the largest deployment in the fleet. For a hot reload approach, the transition window is one credential refresh interval (typically 60 to 300 seconds). The configuration management decision record should document which rotation approach is supported, the required transition window, the steps for executing a rotation (who creates the new credential, how instances are updated, how the old credential is revoked, and how the rotation is verified), and the monitoring check that confirms all instances are using the new credential before the old one is revoked.

The configuration management decision record: why the runtime configuration strategy you chose determines your secret rotation latency and your environment parity ceiling

July 9, 2026 · WhyChose

Runtime configuration starts with environment variables because they are the obvious choice when launching the first service. The twelve-factor application model specifies environment variables as the configuration mechanism, every deployment platform supports them, and they require no additional infrastructure. The application reads its database URL, API keys, and feature flags from the process environment at startup. The strategy works, and it is not documented as a decision — it is the default. The first time the strategy becomes a decision is when the security team schedules a quarterly database credential rotation and asks the platform engineer how long the rotation window will take and whether any user requests will fail during it.

The answer depends on how the application reads its database credential. If the credential is read once at process startup and stored in an immutable connection pool configuration, then rotating the credential requires restarting every application instance — a rolling restart that takes between 45 seconds and 4 minutes depending on the fleet size and the container orchestration platform's restart strategy. During the rolling restart, instances starting with the new credential and instances still running the old credential coexist, and the load balancer routes requests to both. The rotation itself is not atomic; there is a window where some fraction of connection pool errors will appear in monitoring if the old credential is revoked before all instances have restarted. If the team has not rehearsed the rotation, they discover the restart dependency during the first quarterly window — not during the initial configuration decision. A configuration management decision record that documented the credential reading strategy and the rotation procedure would have identified the startup-read constraint as a rotation latency factor before the team committed to a quarterly rotation schedule that assumes zero downtime.

The configuration management decision record is missing from most service architectures because environment variables feel too simple to require documentation. The decision is a platform-level choice made once and then inherited by every subsequent service. It propagates by convention: new services copy the environment variable reading pattern from existing services, and the pattern is never revisited until a constraint surfaces — a credential rotation window, a staging bug that does not reproduce in production, a new environment that must be provisioned from scratch, or a compliance audit that requires an audit log of every secret access. By that point the pattern has been applied across twenty services, and changing it requires coordinating across the entire fleet rather than making a single decision at the beginning. The configuration management decision record should be written when the first service is built, documenting the variable naming convention, the required versus optional variable enumeration, the rotation procedure, and the environment parity enforcement mechanism. Without it, each of those decisions is made implicitly, per-service, by the engineer building that service at that moment — and the implicit decisions accumulate into constraints that are only visible when a security audit, a production incident, or a new environment addition forces them into the open.

Two things that happen when the decision is not written down

The database credential rotation that required a maintenance window

A 60-person e-commerce company ran their backend services as Node.js applications deployed on Kubernetes. Configuration was managed through environment variables set in Kubernetes Deployment manifests, which were stored in a git repository and applied via their CI/CD pipeline. The database URL for their PostgreSQL primary instance was stored as the environment variable DATABASE_URL, injected into each pod's environment at deployment time from a Kubernetes Secret. The application used a connection pooling library that initialized the pool with the database URL read from process.env.DATABASE_URL once when the application module was first loaded. Once the pool was initialized, the application held connections authenticated with that credential for the life of the pod.

Fourteen months after the initial deployment, the company's new security lead conducted an infrastructure audit and identified that the primary database credential had never been rotated. The company's security policy, which had been written for compliance purposes but not previously enforced, required quarterly credential rotation for all production database accounts. The security lead requested that the platform team schedule the first rotation. The platform engineer responsible for the database infrastructure reviewed the process: updating the DATABASE_URL value in the Kubernetes Secret and rolling out updated pods. She estimated the rolling restart at approximately ninety seconds for the three-replica deployment, noting that during the restart there would be a brief period when the old credential would need to remain valid while the new pods came online.

The rotation was scheduled for a Sunday morning at 4am UTC, which was the company's lowest-traffic window. The platform engineer executed the rotation: she updated the PostgreSQL user's password, updated the Kubernetes Secret with the new DATABASE_URL, and triggered a rolling restart. During the restart, two old pods were running with the old credential and one new pod came online with the new credential. A second new pod started. At the point where the third old pod was terminated and its replacement came online, the old credential was still valid — the new password had replaced the old one in PostgreSQL, but the old credential had not been explicitly revoked, so old pods continued to function until their connections were recycled or until the pods were terminated. After the rolling restart completed in 87 seconds, all three pods were running with the new credential. The rotation was functionally successful, but monitoring showed a spike of 23 connection errors during a 12-second window when two pods were terminating simultaneously and new connections from the incoming replacement pods raced against the termination of the old pools.

The post-rotation review identified two undocumented assumptions that the rotation procedure had depended on. First, the old credential must remain valid until all pods using it have terminated — the rotation procedure only works if the database user account supports a brief period of having both the old and new credentials valid simultaneously, which PostgreSQL does not natively support for a single user (changing a password immediately invalidates the old password). The zero-error rotation had succeeded because the timing worked out: all connections from pods using the old credential happened to be in a quiet state during the termination window. A rotation during higher traffic would have produced authentication failures for in-flight requests on terminating pods. Second, the application had no mechanism to refresh the database credential without a pod restart, which meant the rotation procedure would always require a rolling restart and would always have a potential error window proportional to the restart duration and the concurrent connection count. The engineering lead asked why neither assumption had been documented. The answer was that the environment variable approach had been adopted from a template at company founding and no one had written a configuration decision record that covered secret rotation, because rotation had not been a concern at the time and the template did not prompt for it. A configuration management decision record written at founding that included a rotation policy section would have identified the startup-read constraint, specified whether the application was required to support hot credential reload, and documented the zero-downtime rotation procedure — or documented that zero-downtime rotation required a two-phase credential handoff that was beyond the team's current operational scope and would be revisited before the first rotation was required.

The environment variable drift that bypassed caching in production for eight months

A 35-person SaaS company built their platform with a Redis cache layer for expensive API responses. The application used Redis for two purposes: session storage (keyed by session token, 30-minute TTL) and API response caching (keyed by request signature, 5-minute TTL). The Redis connection was configured through an environment variable. The original platform engineer who set up the infrastructure in early 2024 used a single REDIS_URL variable for both staging and production, following the Redis client library's convention for a single connection string: redis://localhost:6379 in staging and redis://10.0.1.45:6379 in production.

Eight months later, the team provisioned a new microservice to handle an analytics pipeline. The engineer building the service was new to the team and set up the Redis connection by following the documentation for the Redis client library, which showed an alternative configuration style using separate REDIS_HOST and REDIS_PORT environment variables. She set up the staging environment with REDIS_HOST=localhost and REDIS_PORT=6379, and the service worked correctly in staging. For production, she added REDIS_HOST=10.0.1.45 and REDIS_PORT=6379 to the production deployment manifest. She did not add REDIS_URL to the production deployment manifest because the service used REDIS_HOST and REDIS_PORT, not REDIS_URL.

Three months after that, the team began work on a feature that required two of the existing services to share cached data with the new analytics service. The feature required the existing services to read from the same Redis cache namespace that the analytics service wrote to. A junior engineer implementing the feature added a new cache read in one of the existing services. Following the convention in the existing codebase, she configured the Redis connection using REDIS_URL. In staging, the connection worked. In production, the connection also appeared to work — the service started successfully, the Redis client reported a successful connection, and no errors were logged. What the monitoring dashboard did not show clearly was that the cache hit rate for the new feature's cache key was zero in production. The Redis client was connecting successfully to Redis, but the cache keys written by the analytics service (which used REDIS_HOST and REDIS_PORT) and the cache keys the new feature was reading (using REDIS_URL) were going to different Redis connections in the production configuration: the analytics service was correctly connecting to the production Redis instance, but the new feature's cache reader was falling back to a default localhost connection because the REDIS_URL was undefined in the production analytics service deployment manifest, and the Redis client silently used a localhost default instead of throwing a connection error, because localhost Redis was actually running as a sidecar for an unrelated purpose in that pod.

The investigation that uncovered the configuration drift took four days and involved a platform engineer, the original service author, and a database performance engineer who was initially called in to explain why one set of queries was showing elevated latency in production but not in staging. The root cause was traced by comparing the full environment variable set across the staging and production deployments for each service — a comparison that was possible only because the team had recently migrated deployment manifests to a structured YAML format that supported diffing. The team found that REDIS_URL was defined in the staging environment for every service but was absent from the production environment of the analytics service, whose author had used the alternative variable naming convention without knowing that the organization had a standard. The fix required updating the production deployment manifest for the analytics service to add REDIS_URL and a full regression test pass to verify that the Redis connections were correct across all services and environments. The post-incident review could not establish when the inconsistency had been introduced with certainty, but the git history suggested it had existed since the analytics service's initial production deployment eight months earlier — meaning the data that should have been shared between services via Redis had been going to separate cache namespaces for the full eight months the feature had been in production. A configuration management decision record that established REDIS_URL as the organizational standard, documented the naming convention, and required a parity check between staging and production environment variable sets before deployment would have blocked the inconsistent configuration before it reached production.

Three structural properties that are set at configuration strategy selection time

The secret access model and the rotation latency ceiling

Every configuration strategy has a secret access model — the mechanism by which the application reads secret values, how often it re-reads them, and what happens when a secret value changes while the application is running. The rotation latency ceiling is the minimum time required for a secret value change to be reflected across all running instances of all services that use the secret. The rotation latency ceiling is set at configuration strategy selection time and cannot be reduced below the ceiling without changing the strategy or adding a hot-reload mechanism to every service that uses the secret.

Environment variables read once at process startup have the highest rotation latency ceiling: the ceiling is the time required to restart every service instance that uses the secret, which is the maximum rolling restart duration across the fleet. For a fleet with twenty services deployed across three availability zones with rolling restart strategies, the rotation latency ceiling may be 10 to 20 minutes for a secret shared across all services. A quarterly rotation schedule that requires a 20-minute maintenance window for every secret rotation is operationally acceptable for many teams, but the requirement for the maintenance window should be known at strategy selection time — not discovered during the first rotation. A secrets manager with a hot-reload mechanism has a lower rotation latency ceiling: the ceiling is the re-read interval configured for the secrets manager SDK, typically 60 to 300 seconds. The trade-off is that every service must be built to re-read secrets at the interval, which requires the connection pool, HTTP client, and any other credential-consuming component to support credential refresh without a restart. The connection pool refresh requirement is the most significant implementation constraint — most connection pool libraries support credential refresh only if the credential is stored as a mutable reference rather than as a string constant used to initialize the pool. Documenting this constraint in the configuration management decision record means that every engineer who adds a new database connection knows that they must use the credential refresh pattern from the beginning, not refactor the connection initialization when the first rotation reveals the constraint.

The secret access model also determines the security property of secrets in the runtime environment. Environment variables are readable by any process running in the same operating system context — they appear in /proc/[pid]/environ on Linux, can be read by child processes, and can be logged accidentally if a debugging tool dumps the process environment. A secrets manager accessed via API call stores the secret value in the secrets manager service and delivers it to the application on request, which means the secret value is not present in the process environment at all times — only when the application explicitly fetches it. The security model difference is significant for high-sensitivity secrets: a database password stored as an environment variable is present in the process environment for the full lifetime of the process; the same password fetched from a secrets manager via API is present in application memory only during the window between the fetch and the use. The configuration management decision record should document the security model explicitly — not to prescribe a specific choice, but because the choice has implications for the secrets management tooling required in each deployment environment and for the compliance audit evidence that needs to be produced when auditors ask how secrets are protected at runtime.

The environment parity model and the configuration drift surface

Environment parity is maintained through a combination of naming conventions, required variable enumeration, and automated parity verification. The configuration drift surface is the set of configuration values that can diverge between environments without producing an immediate, visible error — the set of silent differences that accumulate over time and surface as production-only bugs. The drift surface is determined by three factors: the naming convention (whether it is enforced by tooling or by convention), the required variable list (whether it is checked at deployment time or discovered at runtime), and the parity verification step (whether it runs in CI before deployment or is checked manually ad-hoc). A configuration strategy that relies entirely on convention and manual verification has a large drift surface; a strategy that enforces naming and required variable sets in the deployment pipeline has a small drift surface.

Naming conventions are most effectively enforced by a schema definition that all environments must conform to. A schema definition specifies the required variable names, the type of each variable (string, integer, boolean, URL, one-of-enum), and which variables are required versus optional. Tools like envalid for Node.js, pydantic-settings for Python, or viper with environment validation for Go can enforce the schema at application startup — an application that starts with a variable name that does not match the schema fails immediately with a clear error identifying the mismatched name. A schema-enforced variable set prevents the naming drift that produced the Redis incident: a new service that uses REDIS_HOST and REDIS_PORT instead of REDIS_URL fails the schema validation at startup in any environment where the schema requires REDIS_URL, blocking the deployment before the inconsistency reaches production. The schema also serves as the source of truth for the required variable list — a new environment can be provisioned by starting from the schema and providing values for each required variable, with the confidence that a schema-valid environment is a configuration-complete environment.

Parity verification in the deployment pipeline compares the environment variable sets across environments as part of the deployment process. The comparison can be as simple as a CI step that diffs the environment variable names (not values) between staging and production manifests and fails if any name present in production is absent in staging or vice versa. The values are environment-specific and should differ; the names should be identical. A parity check that runs on every deploy prevents drift from accumulating: the engineer who adds a new environment variable to staging in one PR and adds it to production in a subsequent PR fails the parity check in the second PR if the name differs. This connects to the CI/CD pipeline decision record: the environment parity check is a quality gate in the same category as test coverage and linting, enforced at the deployment pipeline level rather than the application level. Without the pipeline gate, environment parity is a social norm — maintained by engineers who remember to check both environments when making a configuration change, violated by engineers who configure staging first and intend to configure production next but forget, or by engineers who are not aware that the convention exists.

The drift surface is also influenced by the infrastructure dependency set for each environment. A staging environment that mocks external dependencies — an in-memory database instead of PostgreSQL, a local Redis instead of a managed Redis cluster, a stub for a third-party payment API — has a configuration difference from production that is deeper than variable values: the environment itself behaves differently. The configuration management decision record should document which infrastructure dependencies are mocked in which environments and the known behavior differences that the mocking introduces. A staging environment that uses an in-memory database will not reproduce production bugs caused by database query plan instability, connection pool exhaustion, or replication lag. A staging environment that uses a payment API stub will not reproduce production bugs caused by the payment provider's rate limits, webhook delivery latency, or error response formats. Documenting the mock boundaries explicitly — not to eliminate them, but to establish a shared understanding of what staging does and does not replicate — prevents the class of production-only bugs that are caused by an engineer's assumption that a behavior verified in staging is guaranteed to hold in production. This connects to the microservices vs. monolith decision record: teams that run multiple services locally often mock inter-service calls because running the full service fleet locally is impractical, and the mock boundary must be documented or the engineer testing a feature locally does not know which services are real and which are mocked in their local environment.

The configuration validation surface and the fail-fast versus degraded-gracefully tradeoff

Configuration validation determines whether a missing or malformed configuration value causes the application to fail at startup (fail-fast) or to start and fail at the moment the missing value is first used (fail-at-runtime). The fail-fast approach is almost always the correct default for required configuration values: a startup failure is detected at deployment time, before users are affected, and the error message can identify exactly which variable is missing or malformed. A runtime failure is detected when a user request exercises the code path that reads the missing value, which may be minutes or hours after deployment, and the error is typically a NullPointerException or undefined-is-not-a-function error that does not directly identify the missing configuration variable without further debugging.

The validation surface is the set of configuration values that are validated at startup. A wide validation surface — all required variables validated before the HTTP server starts — catches all missing and malformed values at deployment time. A narrow validation surface — only the most critical values validated, with others read lazily when first used — catches only the validated values at deployment time and discovers the remaining values at runtime. The decision about what to include in the validation surface is rarely made explicitly: engineers add configuration reads to the codebase as they need them, typically using the same pattern as the existing reads, and validation is added when a missing value causes a production incident that makes the validation requirement obvious. The configuration management decision record should specify the validation requirement — that all required configuration values must be enumerated in the startup validation check and that the startup check must run before the HTTP server starts accepting connections — and the pattern for adding a new variable to the validation check. With the requirement documented, an engineer who adds a new configuration variable to the application knows to add it to the startup validation check as part of the same change, rather than discovering that the validation requirement exists only when their PR is reviewed by an engineer who knows the convention.

The degraded mode alternative is appropriate for optional infrastructure dependencies where the application has a meaningful fallback behavior. A caching layer is the canonical case: if the application can serve requests without the cache at reduced performance, and if the reduced performance does not cause requests to time out or to fail in ways that users cannot recover from, then a missing or unavailable cache endpoint can be handled by logging a warning at startup and continuing without caching. The configuration management decision record should enumerate which infrastructure dependencies are optional (the application can function without them) and which are required (the application cannot function without them), and should specify the fallback behavior for optional dependencies. Without this enumeration, each engineer who adds a new infrastructure dependency makes an independent decision about whether it is required or optional, and the decision is reflected implicitly in the code — a missing required dependency either throws an uncaught exception at startup (which happens to be fail-fast behavior) or silently produces incorrect results (which is the worst outcome, as with the Redis case above where the client connected to the wrong Redis silently). The explicit enumeration in the decision record makes the dependency classification a deliberate policy choice rather than an implicit result of the connection error handling code that happened to be written.

Five sections the configuration management decision record should address

1. Configuration storage and access model selection

Document the configuration storage mechanism chosen for each category of configuration value: environment variables for all configuration, environment variables for non-sensitive values with a secrets manager for secrets, or a secrets manager for all configuration values. The rationale for the choice should include the factors that were evaluated: the operational complexity of managing a secrets manager versus the simplicity of environment variables, the rotation latency requirement given the security team's rotation schedule, the audit logging requirement given the compliance framework, and the language and framework support for the chosen secrets manager SDK. If a secrets manager is used, document the specific service (AWS Secrets Manager, HashiCorp Vault, Google Cloud Secret Manager, Azure Key Vault), the authentication mechanism the application uses to access the manager (IAM role, service account, AppRole), and the re-read interval for hot reload if supported. If environment variables are used for sensitive values, document the rotation procedure explicitly — including the maximum acceptable downtime window for a rotation, the restart mechanism, and whether the rotation requires a maintenance window or can be executed via rolling restart.

The access model selection also determines the operational tooling for managing configuration values in each environment. Environment variables managed through deployment manifests in a git repository require a process for updating the manifest, having the change reviewed, and deploying the updated manifest. A secrets manager accessed through a web console or CLI requires access controls on the manager itself and a process for granting access to engineers who need to update secrets. Document the access control model for configuration updates — which engineers or roles can update configuration values in production, what review process applies to configuration changes, and whether configuration changes require the same deployment pipeline as code changes or can be applied out-of-band. The out-of-band configuration change path is a common source of undocumented changes: if an engineer can update a production secret directly through the secrets manager console without going through the deployment pipeline, that change is not tracked in the git history and is not reviewed by a second engineer. The configuration management decision record should specify whether out-of-band changes are permitted and what audit trail they produce.

2. Secret lifecycle management and rotation policy

Document the rotation policy for each category of secret: which secrets are rotated, how frequently, who triggers the rotation, what the rotation procedure is, and what the expected impact on application availability is. The rotation policy should be specific: "database credentials are rotated quarterly by the platform team via a rolling restart procedure with a maximum 120-second restart window" is a rotation policy; "we rotate credentials regularly" is not. Include the two-phase rotation procedure if applicable — the steps for creating the new credential, validating the new credential is accepted by the database before the old one is revoked, updating the application instances with the new credential, and revoking the old credential only after all instances have confirmed they are using the new one. The validation step before revocation is the safety mechanism that prevents a failed credential update from locking all application instances out of the database simultaneously: if the new credential is validated by one instance before the old credential is revoked, then a failed credential update leaves all instances still connected with the old credential, allowing the rotation to be aborted cleanly.

Include the secret rotation test in the staging environment. A rotation procedure that has never been rehearsed is a rotation procedure that may fail during a live production rotation. The staging environment should have its own credentials that are rotated using the same procedure as production, on the same schedule, before the production rotation is executed. A successful staging rotation confirms that the procedure works and produces the expected restart behavior. A failed staging rotation identifies a problem with the procedure before it reaches production. The rotation test in staging should be included in the quarterly rotation schedule rather than being an optional step that is skipped when the team is under time pressure. This connects to the CI/CD pipeline decision record: the staging rotation test is a pre-production gate for the rotation procedure in the same way that the test suite is a pre-production gate for code changes. The configuration management decision record should document the staging rotation test as a required step in the rotation procedure rather than a recommended best practice.

3. Environment parity enforcement and configuration drift prevention

Document the variable naming convention — the format for variable names (SCREAMING_SNAKE_CASE, the required prefix for service-specific variables, the required suffix for URL-type variables), the canonical list of shared infrastructure variable names (the exact name for the Redis connection string, the database URL, the message queue endpoint, the external API base URL), and the process for adding a new variable name to the canonical list. The canonical list is the organizational standard — a new service that uses a shared infrastructure dependency reads the canonical variable name for that dependency, not a service-specific variant. The canonical list prevents the drift that requires an investigation to resolve: when every service uses REDIS_URL for the Redis connection, a Redis configuration change requires updating one variable name in one place; when each service uses its own variable name (REDIS_URL, REDIS_HOST, CACHE_URL, SESSION_REDIS), a Redis configuration change requires finding every service's variable name and updating each one, with a risk of missing one.

Document the parity verification step: the mechanism that prevents staging and production configuration from drifting. The minimum viable parity check is a CI step that lists the environment variable names defined in the staging deployment manifest and the production deployment manifest and fails if any name is present in one but absent in the other. The check runs on every PR that modifies a deployment manifest. The check should be extended to cover all environments when additional environments (pre-production, canary, disaster recovery) are provisioned. Include the process for intentional environment-specific variables — variables that are legitimately present in one environment but not another (for example, a DEBUG_MODE variable that is defined in staging but explicitly undefined in production). The parity check should have an allowlist for intentional differences, documented with the reason for each difference, so that the check does not produce false positives that engineers learn to ignore. A parity check with too many false positives is not enforced — engineers learn to click through the failing check and treat it as noise. The allowlist keeps the false positive rate low enough that a real parity violation produces a signal that engineers respond to.

4. Configuration validation and startup gate policy

Document the startup validation requirement: which configuration values must be validated before the application starts accepting requests, what the validation rules are (type, format, required presence), and the library or pattern used to implement the validation. The validation should be implemented as a single startup function that reads all required variables, validates each one against its rule, and either starts the application (all values present and valid) or exits with a non-zero status and a clear error message listing each missing or invalid variable (any value absent or invalid). The exit-on-validation-failure behavior ensures that the deployment platform (Kubernetes, Docker Compose, systemd) detects the failure, marks the instance as unhealthy, and does not route traffic to it. An application that exits during startup with a non-zero status code triggers the platform's unhealthy instance policy — typically preventing the rolling deployment from completing, which alerts the deploying engineer before the invalid deployment reaches full rollout.

Include the required variable list in the decision record, or include a pointer to the schema file that enumerates it. The required variable list is the document that a new engineer or a new environment provisioning process uses to configure an application from scratch. It should include: the variable name, a one-line description of the value, the expected format or type, an example value (redacted for sensitive values), and whether the variable is required or optional with the default behavior if optional. A new environment can be provisioned by starting from the required variable list, providing a value for each required variable, and running the application — if the startup validation passes, the environment is correctly configured. Without the required variable list, environment provisioning requires reading the codebase to discover all configuration reads, which is a time-consuming and error-prone process that misses variables read in infrequently executed code paths. This connects to the developer experience decision record: the required variable list and the startup validation check together determine how quickly a new engineer can get the application running in their local development environment. A startup failure with a clear error message identifying a missing variable is a better onboarding experience than a runtime failure with an obscure error when a code path is first exercised during a development session.

5. Observability instrumentation for configuration failures

Document the monitoring and alerting required for configuration-related failures. The minimum instrumentation is a structured startup log that records all configuration values (with sensitive values redacted to their presence or absence) at application startup. The startup log provides the evidence that an application started with the correct configuration, and it is the first artifact checked when a configuration-related incident is suspected — before checking the deployment manifest, before comparing environments, before reading the code. A startup log that records "REDIS_URL: defined (redis://***)" and "DATABASE_URL: defined (postgres://***@***:5432/***)" confirms that the application read the expected variables; a startup log that records "REDIS_URL: undefined (using localhost default)" would have immediately identified the Redis incident described above as a configuration problem rather than as a caching behavior difference that required four days to trace.

Include the metric instrumentation for configuration reload events if the application supports hot reload. A metric for credential refresh attempts (labeled by secret name and result: success, error, no-change) provides the signal that the hot reload mechanism is functioning. An alert on a failed credential refresh — a refresh that returns an error or returns a credential that fails validation against the target service — is the early warning for a rotation that did not propagate correctly to the secrets manager before the application tried to fetch the new value. Without this metric, a failed hot reload is invisible until the next request that uses the rotated credential produces an authentication error. The metric and alert convert a silent failure into a monitored event that the on-call engineer can respond to before user-facing errors appear. Document the metric names, label sets, and alert thresholds in the configuration management decision record and reference the observability platform decision record for the monitoring infrastructure. The configuration management decision record's observability section specifies what to monitor; the observability platform decision record specifies where the metrics are stored and how alerts are routed. Both must be present for the monitoring to be complete.

Include the process for auditing configuration drift after the fact. Even with a parity check in the CI pipeline, configuration changes that bypass the pipeline — manual updates through a secrets manager console, emergency changes applied directly to the production environment, or changes made by a deployment automation tool that does not go through the same pipeline — can introduce drift that the CI check does not catch. A weekly or monthly automated audit that reads the current configuration from all running instances (via the startup log, a configuration dump endpoint, or the deployment platform's current environment report) and compares it against the expected configuration from the canonical manifest is the backstop. The audit catches drift that accumulated since the last deploy — configuration that was valid at the last deployment but has since diverged due to out-of-band changes. Document the audit schedule, the comparison mechanism, and the escalation path for a drift alert. An audit that finds drift without a documented escalation path produces a finding that no one acts on because no one knows whether they are responsible for resolving it. The escalation path should identify the team responsible for configuration management, the severity threshold at which an audit finding becomes an incident, and the timeline for resolving drift once it is identified.

What to do with the configuration management decision record

The configuration management decision record's primary audience is the engineer who adds the first new configuration value to an existing service, the engineer who provisions a new environment, and the platform engineer who executes the next credential rotation. Each of these engineers is interacting with a configuration strategy that was chosen before they arrived, with constraints that are invisible unless documented. The engineer who adds a new variable needs to know the naming convention and the process for adding the variable to the startup validation check. The engineer who provisions a new environment needs the required variable list. The platform engineer who executes a rotation needs the rotation procedure, the staging rehearsal requirement, and the monitoring check that confirms the rotation is complete. The configuration management decision record is the document each of them reads before making the change — and the absence of the decision record means each defaults to their best understanding of the convention, which differs by engineer and by the convention they came from before joining the team.

If the configuration management decision record was not written when the first service was built, the most useful reconstruction starts with three questions: what is the rotation procedure for the most sensitive credential in the production environment (if the answer is "we would need to restart all the services," document that and the maximum restart window and decide if it is acceptable), what is the required variable list for each service (read it from the codebase rather than from a document, then write it into the decision record so the next environment provisioning does not require the same code read), and what is the variable naming convention (infer it from the existing variable names, identify any inconsistencies, document the standard, and plan the cleanup). Those three answers, written in a single document alongside the deployment manifests in the infrastructure repository, are the configuration management decision record that the next engineer who touches configuration will find most valuable — more valuable than any post-incident report from a rotation that required a maintenance window or a cache bypass that ran for eight months.

The ADR template provides a starting structure for the decision record. The WhyChose extractor can surface configuration-related decisions from ChatGPT or Claude conversations during the initial service design — the discussion where environment variables were chosen over a secrets manager, or where the rotation schedule was decided, may contain reasoning about the latency tolerance and the compliance requirements that was never written into a document and is now recoverable from the chat export. The decisions never written down essay covers why configuration management choices are among the decisions whose absence is least visible day-to-day — the application reads its variables at startup, starts successfully, and serves requests correctly in the happy path — and most expensive to discover when a rotation window reveals a startup-read constraint, an eight-month cache bypass reveals an environment parity gap, or a security audit asks for the audit log of every credential access. The authentication strategy decision record and the access control model decision record together with the configuration management decision record form the complete credential governance surface for a production service — three documents whose combined scope covers how users are authenticated, how permissions are enforced, and how the credentials that underpin both are stored, rotated, and monitored.