The secrets rotation decision record: why the rotation procedure you chose determines your consumer update window failure surface and your compromised credential response time

A 28-person B2B SaaS company stored all credentials in AWS Systems Manager Parameter Store and maintained a rotation runbook in Confluence linked from their incident response playbook. The quarterly database credential rotation procedure was four steps: generate a new password locally, run ALTER USER app_user PASSWORD '...' against the RDS PostgreSQL instance, update the Parameter Store value with the new password, and trigger rolling restarts of the application services that read from Parameter Store at startup. When the procedure was written, three services connected to the database. The procedure listed them by name in the restart section with the order they should be restarted — the stateless API services first, the background job service last, each restart verified by checking that the service's health endpoint returned HTTP 200 before proceeding.

The procedure ran without incident for six months and two rotation cycles. Over those six months, the engineering team shipped two features that required new services: a webhook processing service for incoming Stripe events, and a data sync service for a third-party CRM integration. Both services connected to the same PostgreSQL database, both read the database credential from Parameter Store at startup, and both were deployed and running in production. Neither was added to the rotation runbook. The engineer who built the webhook service assumed someone else would update the runbook. The engineer who built the sync service didn't know the runbook existed. The runbook itself didn't have a prompt or checklist item reminding the team to update it when new database consumers were added.

The third rotation cycle ran two months later. The on-call engineer executed the procedure: generated the new password, ran the ALTER USER statement, updated Parameter Store, restarted the three services listed in the runbook. All three restarted cleanly and passed their health checks. The on-call engineer marked the rotation complete and went back to their regular work. Twenty minutes later, the error rate on the user-facing API began climbing. Database connection failures appeared in the logs. The webhook processing service and the data sync service — both still running with the old database password cached from their last startup — began failing when they tried to open new connections. PostgreSQL does not invalidate existing connections when a user's password changes; existing connections continue working until they close. But when the connection pool exhausted connections and attempted to open new ones, those new connections authenticated with the old cached password and failed. Users received 503 errors on any request that touched the data sync or webhook processing paths. The on-call engineer identified the cause 9 minutes into the incident, restarted the two missing services, and the error rate returned to zero 3 minutes later. The total count: 847 user-facing request failures, 12 minutes of degraded service, and an incident retrospective that produced a checklist item added to the runbook — "before rotating, list all services that connect to this database by searching the codebase for the Parameter Store path." The founding session that established the quarterly rotation procedure documented "quarterly database credential rotation" — it did not specify that a consumer inventory is a prerequisite for the rotation procedure, that the inventory must be regenerated before each rotation rather than maintained as a static list, or that the restart sequence is incomplete if any consumer is missing from the inventory.

A 42-person SaaS platform provided a REST API to both their own internal services and to enterprise customers who built integrations on top of it. Internal services authenticated using API keys stored in AWS Secrets Manager; each service fetched its key at startup and refreshed it every 60 seconds using the Secrets Manager SDK's caching layer. External customers received API keys through the developer portal — a customer could create up to five API keys and manage their own lifecycle. Over time, the platform team recognized that customer-managed keys created operational risk: if a customer's key was compromised, the customer was responsible for rotating it, and some customers hadn't rotated their keys in over a year. The platform team decided to add automatic server-side rotation for all API keys, including customer keys, on a 90-day cycle. The rotation design: the rotation Lambda creates a new key version, registers it as valid in the API gateway's key store, marks the old key version for decommissioning 24 hours later, and sends the new key to the Secrets Manager entry so that internal services would auto-refresh within 60 seconds. The 24-hour grace period was calculated to cover the maximum refresh delay for internal services: 60 seconds for the automatic SDK refresh, plus a 30-minute buffer for services that happened to restart during the window, plus additional time for any service whose startup sequence was slow. The design was sound for internal services. The founding session that designed the automatic rotation documented "90-day API key rotation with 24-hour grace period." It did not specify how customer keys would be rotated under this model — specifically, how enterprise customers who held the old key in their own configuration management systems would be notified of the rotation and given the new key value.

The implementation treated customer keys the same as internal keys: they were stored in Secrets Manager (the customer's key was stored in a Secrets Manager entry that only the customer's designated integration principal had access to), subject to the same 90-day rotation Lambda, and covered by the same 24-hour grace period. What the implementation did not account for was that customers did not have their integrations configured to auto-refresh from Secrets Manager. Enterprise customers had built their integrations before the platform offered Secrets Manager storage — they had copied their API keys into their own systems months or years earlier. The API key was hardcoded in a configuration file, set as an environment variable in a deployment pipeline, or stored in a third-party secret manager that the customer's team managed. None of those storage locations had a mechanism to automatically fetch a new value from the platform's Secrets Manager when rotation happened.

The 90-day rotation ran on a Tuesday at 2:00 AM UTC. The rotation Lambda created new key versions for all 340 active API keys, including those belonging to enterprise customers. At 2:01 AM, the old keys were marked for decommissioning. At 2:00 AM Wednesday, the old keys were invalidated. By 2:00 AM Wednesday, all internal services had refreshed their keys from Secrets Manager and were using the new values. But 23 enterprise customers had not updated their configurations — none of them had been notified that rotation was happening, and none of them had any mechanism to auto-refresh from the platform's Secrets Manager. At 2:00 AM Wednesday, their integrations began receiving 401 Unauthorized responses. The failure was not immediately visible: most enterprise customers ran their integrations on business-day schedules, and 2:00 AM Wednesday was early morning in most time zones. The first enterprise customer opened a support ticket at 9:14 AM, reporting that their nightly reconciliation job had failed with authentication errors. By 11:00 AM, 14 enterprise customers had opened tickets. The most severely impacted customer had a nightly reconciliation job that ran 11,400 API calls against the platform between midnight and 4:00 AM every night. The job had run at midnight, used the old key, received 11,400 consecutive 401 errors, wrote no output, and logged the failure to a monitoring system that was checked the following business morning. The 24 hours of reconciliation data for that customer were missing and unrecoverable from the platform's API — the platform didn't support historical backfill of reconciliation data. The founding session documented "90-day API key rotation with 24-hour grace period" — it did not specify the external consumer registry (a list of all enterprise customers holding a credential, separate from the internal secrets store), the notification procedure for external consumers before rotation, the extended grace period for external consumers relative to the internal grace period, or the process for customers who did not acknowledge the notification before the external grace period expired.

A 35-person analytics SaaS company ran daily ETL jobs that extracted data from a third-party market data provider. The provider issued OAuth 2.0 service account credentials in the form of JSON key files with a 365-day validity period. Credential renewal required the platform team to log into the provider's portal, generate a new credential file, download it, upload it to AWS Secrets Manager, and update the ETL job's startup configuration to fetch the new credential version. The procedure was documented as an annual task in the team's operational calendar, timed to run 30 days before the credential's expiry date to allow adequate time for the credential to be in place before the old one expired. The founding session that established this procedure documented "annual credential renewal, 30 days before expiry." For three years, the procedure ran without incident: the platform engineer renewed the credential in early December for the January expiry, verified the new credential worked against the provider's test endpoint, uploaded it to Secrets Manager, and updated the ETL job's startup configuration to fetch the new credential path. The old credential expired in January. The new credential had been in Secrets Manager for 30 days by then. All jobs that started after the upload used the new credential. All jobs that started before the upload had completed by the time the old credential expired. The buffer was adequate.

In the fourth year, the platform engineer responsible for the annual renewal left the company in October. The operational calendar reminder went to their archived email address. The operational calendar itself was owned by the departing engineer and was not transferred to anyone else. The new platform engineer who joined in November was not briefed on the annual renewal requirement — it was not in the service's documentation, not in the incident runbook, not in the secrets management policy, and not in any file that the onboarding process directed new engineers to read. The credential expiry date was stored in the provider's portal and in the Secrets Manager secret's metadata, but no alert or monitoring job watched for approaching expiry dates. The credential expired on January 4th at midnight UTC without any prior notification to the team.

The daily ETL job ran every day at 11:50 PM UTC. On January 4th, the job started at 11:50 PM and fetched the service account credential from Secrets Manager as part of its startup sequence. At 11:50 PM on January 4th, the credential was still valid — it did not expire until midnight. The startup sequence completed at 11:53 PM. The job began processing data. At midnight UTC on January 5th, the credential expired at the data provider. At 12:04 AM on January 5th, the ETL job made its first API call to the provider after midnight and received a 401 Unauthorized response. The job's error handling for 401 responses was to log the error and abort — it was not retried, because the assumption encoded in the job's error handling was that a 401 meant the credential was wrong, not expired, and retrying a wrong credential was not useful. The job aborted with the 401 error logged at 12:04 AM. By 12:04 AM, the job had been running for 14 minutes. In those 14 minutes, it had processed 34% of the day's data volume before the first API call to the provider returned a 401. The partial output — 34% of the daily data — was written to the data lake. The remaining 66% was not written. The ETL job does not resume from a checkpoint; it restarts from scratch. Restarting the job after obtaining a new credential would process 100% of the day's data again, duplicating the 34% already written. The data team's options were: purge the partial output and rerun the job from scratch (losing the 14 minutes of processing and potentially producing inconsistencies in downstream reports that had already read from the partial output), or leave the partial output and accept that January 4th's data in the data lake was 34% complete. They chose the latter, flagging January 4th's data as incomplete in their data quality tracker. The founding session documented "annual credential renewal, 30 days before expiry" — it did not specify the expiry monitoring requirement (an automated alert when any credential's expiry date is within 60 days), the credential hold time for long-running jobs (the ETL job holds the credential from startup until the first post-expiry API call, which in this case was 14 minutes after expiry), the rotation buffer requirement (the new credential must be in place and the old credential must remain valid for at least `max_job_duration` after the new credential is issued), or the mid-job abort behavior for 401 errors that should distinguish between a wrong credential and an expired credential.

Structural properties set by the secrets rotation decision

Three structural properties are determined when a team designs their secrets rotation procedure. None appear explicitly in the session that establishes the initial rotation cadence, the session that configures the grace period, or the session that schedules the annual renewal — they are the operational consequences of design choices made under the assumption that rotating a secret and updating the secrets store are the complete rotation procedure.

Property 1: The consumer inventory and the update window. Secrets rotation is zero-downtime only if every consumer of the old credential updates to the new credential before the old credential is invalidated. The consumer inventory is the authoritative list of every process, service, and external party that holds a copy of the credential — not just services that fetch it from the secrets store at startup, but also services that cache it in memory and re-fetch on a schedule, batch jobs that capture it at job start and hold it for hours, CI/CD pipelines that embed it as a build secret, monitoring agents that use it for synthetic checks, and integration partners who received it through a developer portal or a support ticket months ago. The consumer inventory must specify for each consumer: its fetch mechanism (how it gets the credential — auto-refresh from the secrets manager SDK, environment variable set at container startup, startup-time fetch with no auto-refresh), its credential hold time (the maximum time it holds one credential version before fetching a new one), and its update procedure (what must happen to make it use the new credential — automatic on next SDK cache expiry, rolling restart, deployment pipeline run, manual configuration update). The rotation grace period — the time the old credential remains valid after the new credential is issued — must be at least as long as the maximum credential hold time across all consumers in the inventory, plus a safety buffer. A consumer inventory that is not regenerated before each rotation is worse than no inventory: a stale inventory creates confidence that all consumers are covered when the newly-added consumer that will cause the incident is not in it. The inventory must be generated by searching for the credential's identifier in the codebase (every path, every environment variable name, every reference to the secrets manager key) rather than maintained as a static list, because static lists drift from reality the moment the next service is added without updating them. The secrets management decision record documents the secrets store selection that determines the automation capability for rotation — AWS Secrets Manager's rotation Lambda model versus HashiCorp Vault's dynamic secrets model versus manual Parameter Store updates — and the SDK caching model that determines each consumer's credential hold time based on the store they fetch from. The audit log decision record documents the access log requirements at the secret store level — the credential access log that makes it possible to enumerate every service that fetched a specific secret in the past 90 days, which is the authoritative source for consumer inventory reconstruction when a static list is unavailable.

Property 2: The external consumer gap. Internal consumers are discoverable: a search of the codebase for the secrets manager path finds every service that fetches the credential, and the secrets store's access log finds every principal that fetched it in the past N days. External consumers are not discoverable from either source. An enterprise customer who received an API key through the developer portal 14 months ago, copied it into their deployment pipeline's secret management system, and has been using it in every nightly reconciliation job since then is invisible to the codebase search and invisible to the secrets store access log — their calls appear as authenticated API requests, not as secrets store fetches. The external consumer gap is a property of every credential that is ever distributed out-of-band to a party who stores it in their own systems. The gap affects API keys issued to customers, client certificates issued to integration partners, shared secrets issued to third-party monitoring tools, and any credential whose lifecycle is managed by the issuing team but whose storage is managed by the consuming party. The rotation procedure that works correctly for internal consumers — update the secrets store, wait for the internal grace period, invalidate the old credential — fails for external consumers because external consumers have no mechanism to detect that rotation is happening, no access to the secrets store to fetch the new credential, and no automated process to update their own configuration when a new credential is issued. The decision record must specify the external consumer registry — a maintained list of every external party that holds each credential, with their contact information, the credential version they currently hold, and the date they last confirmed their integration — and the external rotation procedure: advance notification at least 14 days before the rotation date, the new credential value sent to each external consumer through a secure channel (email with the value, or a developer portal page where they can retrieve it after authenticating), a confirmation requirement before the overlap window closes, and an overlap window for external consumers that is distinct from and longer than the internal grace period. The API gateway decision record documents the API key lifecycle management model at the gateway layer — the key registration, validation, and invalidation mechanics that determine how long the overlap window can be held open for external consumers without requiring coordination with the upstream service's authentication model. The zero-trust network access decision record documents the short-lived token model as an architectural alternative to long-lived rotating secrets for external consumers — an OAuth 2.0 client credentials flow where external consumers request tokens that expire in hours rather than months, eliminating the external consumer rotation problem by making the credential's lifetime short enough that rotation is continuous rather than periodic.

Property 3: The long-lived process and the mid-job credential expiry surface. A credential fetched at process startup is valid at the moment it is fetched. For most processes — short-lived request handlers, medium-lived worker processes that restart every few hours — the credential fetched at startup will remain valid for the entire lifetime of the process, and the rotation grace period can be set short enough that the old credential is invalidated before the process's next startup. But for long-lived processes — batch jobs that run for hours, scheduled jobs that run overnight, stream processing jobs that run continuously — the credential hold time is the job duration, and the rotation grace period must account for the worst case: a job that starts one second before the new credential is issued and runs for the job's maximum duration before the old credential can be safely invalidated. The rotation procedure must classify every credential consumer by its maximum hold time and specify the rotation accommodation for each class. For long-lived batch jobs, two accommodations are possible. The rotation buffer: the rotation procedure must not invalidate the old credential until a time after the new credential is issued that is at least equal to the maximum job duration plus the time between job starts (a job that runs for 8 hours and starts every 24 hours requires a rotation buffer of 32 hours, because the worst-case job starts at the moment the new credential is issued and runs for 8 hours, and the next job cannot start until 24 hours later, at which point it will fetch the new credential). The mid-job re-fetch: the batch job periodically polls the secrets manager for a new credential version and reconnects to the upstream service without restarting the job — which requires the upstream service to concurrently support both the old and new credential during the re-fetch window, the job's processing loop to be interruptible at a safe checkpoint, and the reconnection to be idempotent. The rotation buffer is simpler to implement but extends the effective credential lifetime; the mid-job re-fetch is more complex but allows shorter rotation cadences. The decision record must specify which accommodation is used for each job type, because the answer depends on the upstream service's concurrent credential validity model — a service that immediately invalidates the old credential on rotation cannot support the rotation buffer without modification. The data pipeline decision record documents the batch job checkpoint and restart model that determines whether mid-job re-fetch is architecturally feasible — the ETL job that writes to a partitioned output path and can resume from the last committed partition is a different problem from the ETL job that writes to a single output file and must restart from scratch. The multi-region deployment decision record documents the secret replication model across regions — the propagation lag between primary and replica regions that adds to the effective hold time for consumers in non-primary regions and must be included in the grace period calculation.

What the founding session records and what it omits

The founding secrets rotation session typically records the rotation cadence (the time interval between rotations — 30 days, 90 days, annually), the secrets store that holds the credential (AWS Secrets Manager, HashiCorp Vault, AWS Parameter Store), the credential type being rotated (database password, API key, service account JSON key), and sometimes the rotation trigger (manual procedure, automated Lambda, expiry-driven renewal). It may record the rationale for the chosen cadence — a 90-day cycle is a reasonable baseline for internal service credentials, an annual cycle is appropriate for credentials issued by a third-party provider whose issuance process requires manual portal steps. What it does not record is the consumer inventory: the complete list of every service, job, pipeline, external customer, and third-party tool that holds a copy of the credential, organized by fetch mechanism and maximum credential hold time. What it also does not record is the grace period specification: the concrete number of minutes or hours the old credential must remain valid after the new credential is issued, derived from the maximum hold time in the consumer inventory. And what it does not record is the external consumer procedure: if the credential has ever been distributed to a party outside the codebase — a customer through a developer portal, a monitoring tool through a manual setup, an integration partner through a support ticket — the founding session almost never documents that fact or specifies how those external consumers will be handled during rotation.

The consumer inventory omission compounds with time. When the rotation procedure is first written, the consumer inventory is accurate by construction — the engineer writing the procedure knows all the services that connect to the database because they just built the system. The procedure's restart sequence reflects the current consumer list. But the procedure is static and the codebase is dynamic. Every new service that connects to the database adds an entry to the consumer inventory without adding an entry to the rotation runbook. The drift between the runbook and the actual consumer list is invisible until a rotation exposes the gap — at which point the missing consumer's failure is the first signal that the runbook is incomplete. The correct design is a consumer inventory that is regenerated at rotation time by searching the codebase for references to the credential's identifier, not maintained as a static list between rotations. The infrastructure-as-code strategy decision record documents the Terraform variable and module structure that centralizes secrets references — when every consumer declares its dependency on a specific secrets manager path as a Terraform variable, a search for that variable name finds every consumer in a way that is always current with the codebase. The CI/CD pipeline decision record documents the pipeline secret injection model — whether secrets are fetched at pipeline start, injected as environment variables at container startup, or fetched per-step — which determines the pipeline's credential hold time and whether the pipeline appears in the consumer inventory as a consumer that needs to be included in the grace period calculation.

The external consumer omission is a structural gap that requires a separate process to close, because external consumers are definitionally outside the internal audit trail. The secrets store's access log records when the platform's own services fetched the credential; it does not record when the credential was issued to a customer six months ago through the developer portal. The codebase records how internal services fetch and use the credential; it does not record which enterprise accounts have integrated using the credential and stored it in their own Vault instance. The external consumer registry is the only mechanism that bridges this gap — a document (or database table) that is updated every time a credential is issued to an external party, recording the party's identity, the credential version issued, the date issued, the contact information for the technical team responsible for the integration, and the date of last confirmed use. The registry is maintained by the team that issues credentials to external parties, which is usually the platform or developer-experience team rather than the team that owns the credential's underlying resource. The external consumer rotation process — advance notification, new credential delivery through a secure channel, confirmation collection, extended overlap window — cannot be automated without the registry, because the registry is the source of truth for who must be notified. The incident response playbook decision record documents the runbook entry for authentication failures that must include credential rotation as a cause — rotation-caused failures appear as a sudden increase in 401 Unauthorized responses with no code change, which is the same symptom as a compromised credential being revoked, and the runbook must help the on-call engineer distinguish between the two causes. The observability strategy decision record documents the metric-based alerting on authentication failure rate — the alert that fires when the 401 rate for a specific credential or service exceeds a threshold, which is the operational signal that triggers the rotation investigation rather than a customer support ticket arriving hours after the fact.

The long-lived process omission is the most technically subtle of the three because it requires reasoning about the credential hold time of processes that are architecturally invisible to the rotation procedure. A rotation procedure that specifies "wait 24 hours before invalidating the old credential" is written with services in mind — services restart every few minutes during a rolling deployment, have TTL-based SDK caches measured in seconds or minutes, and will certainly update within 24 hours. But the same procedure is applied to the credentials used by batch jobs that run for 8 hours, stream processors that run continuously, and scheduled jobs that start once a day. For these consumers, 24 hours may not be sufficient — a job that starts at 11:50 PM and holds the credential for 8 hours will still be holding the credential at 7:50 AM, and if the old credential was issued before the job started and is invalidated 24 hours after the new credential was issued at midnight, the job started before the new credential was issued and its held credential is the one expiring. The rotation buffer calculation must be: new credential issued at T=0, maximum job duration = H hours, next job start = J hours from T=0 (where J is the job's schedule interval), hold-time = H, expiry of old credential must be no earlier than T=H (the job that started at T=0 runs until T=H), and the next job that starts at T=J will start after T=0 and fetch the new credential if J is greater than the grace period. For a job that runs 8 hours and starts every 24 hours, the minimum grace period is 8 hours (for the job that starts at T=0); a 24-hour grace period is sufficient with margin. For a job that runs 8 hours and starts every 6 hours, the minimum grace period is 8 hours (for the job that starts at T=0 and runs to T=8), but the next job starts at T=6, which is before T=8, and the job that starts at T=6 will fetch the new credential if it starts after the new credential is issued at T=0 and runs to T=14. The grace period must cover both T=8 (end of the T=0 job) and T=14 (end of the T=6 job) — in this case 14 hours. The WhyChose decision extractor finds the founding rotation sessions in your ChatGPT and Claude export — the "how should we handle credential rotation?" infrastructure setup session, the "let's automate the 90-day API key rotation" automation design session, the "we need to renew the provider credential before it expires" annual renewal session, the "why are our enterprise customers getting 401 errors this morning?" incident retrospective session. It extracts the rotation cadence, the grace period, and the consumer update procedure from the founding session and surfaces the consumer inventory process, the external consumer registry, and the long-lived process classification that the session documented versus the ones it omitted.

The five ADR sections for a secrets rotation decision

Section 1: Credential classification and rotation cadence. Specify every credential type in the system, grouped by rotation cadence and consumer type. Credential types commonly include: database passwords (rotate every 90 days for production, every 30 days for databases storing PCI-scoped or HIPAA-scoped data), API keys for internal service-to-service authentication (rotate every 90 days), API keys issued to external customers (rotate every 90 days for internal versioning, with a 60-day external overlap window from notification to invalidation), third-party provider service account credentials (rotate according to the provider's expiry policy, with a minimum 60-day pre-expiry renewal window), TLS client certificates (rotate every 12 months or 30 days before expiry, whichever comes first), and CI/CD pipeline secrets (rotate every 90 days, with each rotation triggering pipeline restarts in the affected environments). For each credential type, specify the rotation trigger: time-based (automated rotation on a fixed interval), expiry-driven (automated alert and renewal process when the credential's expiry date is within a specified number of days), event-driven (immediate rotation when a compromise is suspected or confirmed), or manual (rotation on demand via an approved runbook). Specify the tooling that executes each trigger type and the team responsible for each credential class. The secrets management decision record documents the secrets store and rotation automation model — the AWS Secrets Manager rotation Lambda, the HashiCorp Vault dynamic secrets engine, or the manual Parameter Store procedure — that constrains which rotation cadences and trigger types are operationally feasible without building custom tooling.

Section 2: Consumer inventory process and grace period specification. Specify the consumer inventory generation process for each credential: the search query used to find every reference to the credential in the codebase (the secrets manager path, the environment variable name, the configuration key name), the access log query used to find every principal that fetched the credential in the past 90 days (Secrets Manager provides this via CloudTrail, Vault provides it via the audit log), and the external consumer registry query that finds every external party issued the credential through out-of-band channels. Specify the mandatory pre-rotation steps: run the inventory generation queries, verify the output against the previous inventory and investigate any discrepancies (a new consumer that wasn't in the last inventory but is now is a service added without updating the runbook — it must be classified and added before rotation proceeds), confirm that every consumer in the inventory has a documented update procedure and a known credential hold time. Specify the grace period for each credential type in minutes or hours, derived from the maximum credential hold time across all consumers in the inventory: the grace period is `max(consumer_hold_times) × 1.3` (30% safety buffer) for internal consumers, and a separate longer value for external consumers. Specify the rotation abort condition: if any consumer in the inventory has an unknown hold time or an update procedure that cannot be completed within the grace period, the rotation must be postponed until the consumer is updated or its hold time is characterized. The infrastructure-as-code strategy decision record documents the Terraform variable structure that makes secrets references discoverable by path — the module interface that centralizes secrets manager path references into Terraform variables, making the codebase search for consumer inventory generation accurate and complete.

Section 3: External consumer registry and notification procedure. Specify the external consumer registry: the data store (a database table, a spreadsheet with access controls, or a secrets management platform feature) that maintains one record per credential-per-external-consumer, containing the credential identifier, the consumer's organization name, the consumer's technical contact email and webhook notification endpoint, the credential version currently in use, the date the credential was issued, the date the consumer last confirmed their integration (either through a test call using the credential or through an acknowledgment response to a rotation notification), and the date the consumer is expected to rotate to the next version. Specify the registry update process: every time a credential is issued to an external party through any channel (developer portal, support ticket, sales engineering engagement), the registry must be updated within one business day of issuance. Specify the external rotation notification procedure: 30 days before the rotation date (or 14 days for credentials with short rotation cycles), send each external consumer in the registry a notification containing the rotation date, the new credential value delivered through a secure channel (a time-limited signed URL to a portal page where the consumer can retrieve the new value after authenticating, or a PGP-encrypted email if the consumer has provided a public key), and a deadline by which the consumer must confirm they have updated to the new credential. Specify the confirmation collection mechanism and the action for consumers who do not confirm: escalate to the account team for manual follow-up rather than automatic invalidation, because an unannounced credential invalidation for an enterprise customer is a service level agreement breach. Specify the external overlap window separately from the internal grace period: the old credential remains valid for external consumers until the end of the external overlap window (30 days for standard integrations, 90 days for enterprise customers with formal change management processes). The API gateway decision record documents the key validation model at the gateway — whether the gateway validates API keys against a local cache or makes a remote call to the key store, which determines how quickly the external overlap window can be closed after the last confirmed consumer updates without needing to coordinate with the gateway team.

Section 4: Long-lived process classification and rotation buffer. Classify every credential consumer by its maximum credential hold time. Short-lived processes (hold time under 5 minutes): request handlers, short-running scripts, and any process that restarts more frequently than the internal grace period — these require only that the rotation grace period exceeds 5 minutes, which any reasonable grace period will satisfy. Medium-lived processes (hold time 5 minutes to 4 hours): connection-pooled services with a secrets manager SDK cache and a deployment rolling restart that completes within the medium-held window — these are covered by a grace period of 4 hours plus a buffer. Long-lived processes (hold time over 4 hours): batch jobs, stream processors, and scheduled jobs whose duration approaches or exceeds 4 hours — these require explicit rotation accommodation. For each long-lived process, specify the rotation accommodation type. Rotation buffer: the old credential must remain valid for at least `max_job_duration + next_job_start_from_T0` hours after the new credential is issued. Specify the calculation: `max_job_duration` (the P99 job duration measured over the past 90 days), `schedule_interval` (the time between job starts), and the resulting `minimum_grace_period = max_job_duration × ceiling(minimum_grace_period / schedule_interval)`. Mid-job re-fetch: specify the polling interval (how often the job checks the secrets manager for a new credential version), the re-authentication mechanism (how the job reconnects to the upstream service after fetching the new credential without restarting the job), the safe checkpoint definition (the point in the job's processing loop where re-authentication can interrupt without producing inconsistent output), and the re-authentication idempotency requirement (a reconnection that fails after establishing the new connection must leave the job in the same state as before the reconnection attempt). The data pipeline decision record documents the batch job checkpoint model — the partition-level write and commit structure that makes mid-job re-fetch feasible by defining the safe interrupt points in the processing loop.

Section 5: Compromised credential response procedure and expiry monitoring. Specify the expiry monitoring configuration: an automated check that runs daily against every credential's expiry date (extracted from Secrets Manager secret metadata, from the credential itself if it contains a validity field, or from an expiry calendar maintained by the team) and fires an alert when any credential's expiry date is within the defined renewal lead time. The alert must go to a team-owned distribution list or on-call rotation, not to the individual engineer who set up the credential — individual ownership means the alert goes dark when the engineer leaves or changes roles, which is precisely the scenario that allows a credential to expire silently. Specify the compromised credential response procedure separately from the scheduled rotation procedure: a compromised credential must be rotated immediately rather than at the next scheduled interval. The emergency rotation procedure must be executable without the grace period — the moment a compromise is confirmed, the old credential must be invalidated as soon as possible, and the acceptable impact is the consumer downtime that results from breaking the consumer update window. The procedure must specify who approves emergency invalidation (the on-call security engineer or the credential owner's team lead), how to notify internal teams of an emergency rotation so they know to expect authentication failures and restart their services, and how to notify external consumers of the emergency (a breach notification template sent to all external consumers in the registry within 2 hours of confirmed compromise, regardless of business hours). Specify the rotation audit log requirements: every rotation event must be logged with the credential identifier, the initiating principal, the rotation timestamp, the new credential version identifier, the grace period start and end time, and the count of consumers confirmed updated before grace period expiry. The audit log is the evidence that rotation happened when required and that the grace period was observed — it is the documentation required for SOC 2 Type II audits that include credential management as a control. The observability strategy decision record documents the metric-based alerting on authentication failure rate — the monitoring that detects rotation-caused failures before a customer support ticket surfaces them, by alerting when the 401 rate for a specific service exceeds its baseline by more than two standard deviations over a 5-minute window, which is the signal that a rotation either missed a consumer or completed prematurely.