The encryption-at-rest decision record: why the key management model you configured determines your backup data exposure surface and your key rotation migration failure
Encryption-at-rest is configured once, early — during the GDPR sprint, the PCI DSS assessment, or the HIPAA readiness review — against a data volume and team size small enough that the consequences of each configuration choice are invisible. The algorithm is correct. The key size meets the standard. The fields are covered. What the founding encryption session does not specify is where the key is stored relative to the data it protects, who in the organization can invoke the decryption operation, and what the plan is for rotating the key when the data volume has grown to a scale that makes rotation expensive. Three failure patterns develop from those omissions: the company that discovered its field-level encryption was architecturally sound and completely ineffective because the key lived in the same database as the encrypted data and was therefore in every backup that exposed that data; the company that discovered a supply chain attack had given an attacker full decryption access to customer payment data because the KMS key policy that was supposed to be tightened after the initial sprint was never tightened; and the company that discovered a mandatory HIPAA key rotation required an eleven-hour live migration that blocked the patient data pipeline for three hours because the rotation had never happened and the data volume had grown to a scale that made the migration plan impossible to write quickly.
A 25-person B2B SaaS that built workflow automation tools for human resources and operations teams had added field-level encryption to meet GDPR compliance requirements in its second year. The GDPR sprint covered eight columns across three tables — employee name, email, phone number, national identification number, payroll account details, home address, date of birth, and a free-text personal notes field — using AES-256-GCM with a 256-bit symmetric key. The implementation was technically correct: the encryption used an authenticated mode that detected tampering, generated a unique initialization vector for each encrypted value, and stored the IV alongside the ciphertext. The engineering team wrote tests confirming that the plaintext could not be recovered from the ciphertext without the key. The GDPR compliance checkbox was marked green.
The encryption key was a 32-byte value generated during the initial setup and stored in a config table in the same PostgreSQL database alongside other application configuration — the Stripe webhook secret, the SendGrid API key, the Slack notification token. The table was used by the application to read configuration at startup rather than relying on environment variables, which the founding engineer had found difficult to manage across staging and production. The key was stored as a hex-encoded text value in a row with the key name encryption_master_key. It was not rotated after creation. No access policy distinguished it from other config rows.
The company's backup process took a nightly pg_dump of the entire database and uploaded it to an S3 bucket. The S3 bucket had been created during the initial infrastructure provisioning with public access disabled. In the company's second year, an engineer reconfigured the CloudFront distribution that served the product's static assets and, in the process of setting public read on the assets bucket, accidentally applied the same configuration to the backups bucket in an adjacent step. The misconfiguration was not detected by the company's configuration drift monitoring, which was checking for public access on the assets infrastructure but not on the backups bucket. The backups bucket was publicly readable for fourteen days.
A security researcher running an automated scan of publicly accessible S3 buckets found the bucket on the ninth day. The researcher downloaded the most recent pg_dump — a 4.2 GB compressed file — and inspected its contents. The dump contained the complete database schema and all table data, including the config table with the encryption_master_key row. The researcher wrote a sixty-line Python script: load the dump, extract the encryption key from the config table, iterate the encrypted rows in the eight PII columns, apply AES-256-GCM decryption using the extracted key and the stored IV, write the plaintext to a CSV. The decryption completed in four minutes and twelve seconds. The CSV contained complete plaintext PII for 18,400 employee records. The researcher submitted a responsible disclosure report the same day.
The encryption had been architecturally correct. AES-256-GCM is an appropriate algorithm for field-level encryption. The IV uniqueness requirement was correctly implemented. The key size met the NIST recommendation. The GDPR compliance assessment that had greenlit the implementation had tested the encryption scheme in isolation — it had not tested the effective security boundary that resulted from storing the key in the same system as the encrypted data. The founding technical session had documented the encryption algorithm, the covered fields, and the compliance rationale without specifying where the key must be stored or where it must not be stored. The separation requirement — a key must be stored in a system with access controls and backup paths independent of the system storing the encrypted data — was not written down because it felt obvious at the time of the founding sprint, when the key was being configured on a development machine, and only became non-obvious fourteen months later when the backup path, the S3 permissions, and the misconfiguration window had each evolved independently from the key storage decision.
A 34-person fintech SaaS that built payment infrastructure tooling for marketplace platforms had used AWS KMS for column-level encryption of payment data after a PCI DSS assessment identified that storing payment instrument BINs in plaintext was a compliance gap. The engineering team chose a customer-managed KMS key over an AWS-managed key for auditability — customer-managed keys log every API call to CloudTrail and allow the company to set its own key policy and rotation schedule. The choice was the right one. The key policy that was initially written was not.
The first version of the key policy granted kms:Encrypt, kms:Decrypt, and kms:GenerateDataKey to arn:aws:iam::${account-id}:root — the account root principal, which in AWS IAM is the effective way of granting an action to all IAM principals in the account, including every role and user. The engineer who wrote the policy knew it was broader than necessary. The comment in the Terraform configuration said: # TODO: tighten to specific roles once we identify which services need Decrypt — using root for now to unblock the sprint. The sprint completed. The TODO remained. In the fourteen sprints that followed, no one returned to the KMS key policy because the application was working correctly, the compliance assessment had been satisfied, and the key policy configuration — buried in a Terraform module that was touched only when KMS configuration changed — was not part of any regular review cycle.
Eighteen months after the initial PCI DSS sprint, the engineering team upgraded a popular JavaScript utilities library. The library had been compromised in a supply chain attack six weeks earlier: the attackers had gained access to the library maintainer's npm account and published a new version containing a script that, when executed in a CI/CD environment, serialized the available environment variables and temporary IAM credentials to a remote endpoint. The compromised version had been in the npm registry for six weeks before the attack was disclosed. The company's package-lock.json pinned the library at the prior version, but a developer running npm update during a local dependency cleanup session pulled the new version and committed the updated lockfile without reviewing the changelog. The CI/CD pipeline built the next deploy with the compromised library version.
The compromised library executed its exfiltration script during the build. The CI/CD pipeline role had, by necessity, broad IAM permissions: it needed to push Docker images to ECR, update ECS task definitions, invalidate CloudFront caches, and read SSM Parameter Store values for deployment configuration. The exfiltrated temporary credentials included an access key, a secret key, and a session token valid for twelve hours. The attackers received the credentials and began exploring the account. Within ninety minutes of exfiltration, they had identified the KMS key ARN — visible in the application's environment variables and in the CloudTrail events from the last deploy — and attempted a kms:Decrypt call using the exfiltrated CI/CD credentials.
The call succeeded. The KMS key policy granted Decrypt to the account root, which included the CI/CD role. The attackers iterated the payment token columns — the column values were retrievable by any IAM principal with read access to the RDS instance, which the CI/CD role had through a parameter store credential — and called kms:Decrypt for each token. The temporary session lasted twelve hours. The account team's incident response triggered a credential rotation 72 hours after the exfiltration, when a CloudTrail alert for anomalous Decrypt call volume finally surfaced the investigation. During those 72 hours, the attackers had made 340,000 successful Decrypt calls and had the plaintext BINs for all stored payment tokens.
The founding technical session had documented the KMS key choice and the compliance rationale — customer-managed key for audit trail, 256-bit key material, CloudTrail logging enabled — without specifying that the IAM scope for Decrypt operations was a minimum-privilege requirement that had no acceptable temporary exception. The TODO comment that acknowledged the broadness was not a compliance gap flag; it was a debt marker that the team intended to resolve and never did because the compliance assessment did not re-check key policy scope after the initial sprint passed, and no automated tool enforced the minimum-privilege constraint on KMS key policies as a regression check in the CI/CD pipeline.
A 43-person healthcare SaaS that built clinical trial data management tools for pharmaceutical research organizations had implemented encryption for PHI — protected health information — under HIPAA's technical safeguard requirements. The founding engineer had chosen envelope encryption: each patient record in the participant_data table was encrypted with a unique 256-bit data-encrypting key (DEK); each DEK was encrypted by a KMS-managed key-encrypting key (KEK); only the encrypted DEK was stored in the database alongside the encrypted record; the plaintext DEK existed in memory only during the decryption operation and was never written to disk or logged. The architecture was correct and represented genuine defense-in-depth: even if an attacker gained read access to the database, they would need both the encrypted DEK and the KEK to recover any patient record, and the KEK lived only in KMS.
The founding technical session had included a note on key rotation: "DEKs and KEK should be rotated at least annually per NIST SP 800-57 guidance on key lifetime for symmetric encryption keys used for data at rest." The note was accurate. NIST SP 800-57 does recommend periodic rotation for data-at-rest encryption keys, with the specific interval depending on the data sensitivity and the number of bytes encrypted per key. The note did not specify whether "annually" meant automatic KMS rotation, manual rotation, or both. It did not specify which team member was responsible for initiating the rotation. It did not specify the procedure for rotating the KEK — how to re-wrap each stored DEK with the new KEK version while the application continued operating. It did not specify how long the re-wrapping migration was expected to take or what systems would be affected during the migration. It did not specify what the migration plan was if the DEK count had grown to a scale that made a single-pass re-wrap infeasible within a maintenance window.
AWS KMS supports automatic annual rotation for customer-managed keys: when enabled, KMS generates a new key version each year and transparently uses the new version for all new Encrypt calls while retaining the ability to Decrypt data encrypted with prior versions. The founding engineer had considered enabling automatic rotation and had decided against it initially — automatic KMS rotation rotates only the KEK material within KMS, but the stored DEK ciphertexts in the database are still wrapped by the prior key version and must be re-wrapped to realize the security benefit of the rotation; automatic rotation without a DEK re-wrap migration meant the old key material remained the actual protection for all existing records regardless of the rotation setting. The intent was to implement the full rotation procedure — KMS rotation plus DEK re-wrap — as a scheduled annual task. The intent was never formalized into a calendar event, a runbook, or an automated job. KMS automatic rotation was not enabled. The manual rotation procedure was not written. The annual reminder was not set.
Four years and three months after the initial encryption implementation, the company's HIPAA compliance audit included a review of encryption key management practices. The auditors asked for documentation of the last key rotation date and the rotation procedure. The engineering team checked the AWS KMS console: automatic rotation was disabled. The CloudTrail history showed no manual key rotation events since the key's creation date. The stored DEK count in the database was 51,000,000 — one DEK per participant record, accumulated over four years of clinical trial data import. The KEK had encrypted every one of those 51 million DEKs and had never been rotated.
The remediation required re-wrapping every DEK: read the DEK ciphertext from the database, call kms:Decrypt with the current KEK to get the plaintext DEK, call kms:Encrypt with the new KEK version to get the re-wrapped ciphertext, write the new ciphertext back to the database. At two KMS API calls per record and AWS KMS's default transaction limit of 5,500 cryptographic operations per second per key, the theoretical minimum time for 51 million records was approximately 5.1 hours at full throughput — but KMS throttling, network latency, and database write contention produced an actual runtime of eleven hours and seventeen minutes for the batch migration job. During the last three hours of the migration, the patient data import pipeline — which needed to call kms:GenerateDataKey to create DEKs for new records — was blocked because the batch migration was consuming the KMS quota for the key. Clinical trial sites waiting for import confirmations during an active data collection window were unable to submit new records for three hours.
The founding technical session had documented the DEK/KEK architecture correctly — the threat model it addressed, the KMS configuration, the compliance rationale — without specifying an automatic rotation setting, a manual rotation procedure, or a migration plan for the re-wrapping that rotation required. The omission was benign at founding: there were no DEKs yet, the migration cost was zero, and writing a migration plan for zero records felt premature. The omission became structural risk at 51 million records, when the migration cost had accumulated silently over four years with no session or review that looked at the delta between the documented rotation target and the absence of any rotation event.
Structural properties set by the encryption-at-rest decision
Three structural properties are determined when a founding team configures its encryption-at-rest architecture: where the keys live relative to the data they protect, who can invoke the decryption operations those keys enable, and how expensive the rotation migration will be when it is eventually required. None are labeled explicitly in the founding encryption session — they are security properties that emerge from the assumptions the founding decisions embed about what the effective security boundary is, which credential scope is acceptable, and how much data the product will have accumulated by the time the first rotation is forced.
Property 1: The key storage model and the backup data exposure surface. The security boundary that field-level or column-level encryption establishes is only as strong as the separation between the key store and the data store. If the key resides in the same system as the data it protects — the same database, the same S3 bucket, the same secrets store that is backed up to the same location — then the effective security boundary for the encrypted data is the security boundary of the combined system, not the security boundary of the encryption algorithm. A backup of the combined system contains both the key and the ciphertext; any exposure of the backup is a full plaintext exposure. The separation requirement is non-negotiable and has no performance or complexity tradeoff: using a dedicated key management system (AWS KMS, Google Cloud KMS, HashiCorp Vault, Azure Key Vault) means the key is in a different system with different access credentials, a different backup path, and different audit logs; the only cost is two additional API calls per encryption and decryption operation. The structural fix is to specify, in the encryption decision record, that encryption keys must be stored in a dedicated key management system that is never included in the encrypted database's backup scope, and to include a verification step in the backup verification procedure that confirms the backup does not contain key material. The database backup verification decision record connects here directly: the backup verification procedure should include confirming that the key storage system and the application database are backed up to different locations with different access credentials and different audit paths, and that a sample of production backups does not contain rows from any table or file that stores key material. The secrets management decision record covers the closely adjacent case of operational credentials (API keys, service passwords) stored in the same system — the threat model differs but the separation requirement is structurally identical: a secret whose exposure enables access to the protected system must not be stored in that system or in its backup.
Property 2: The IAM scope for cryptographic operations and the compromised credential exposure surface. A KMS key policy that grants Decrypt to a broad principal — the account root, a deployment role, a developer IAM user — means that any credential compromise that affects that principal's scope also compromises the decryption capability for all data encrypted by that key. The minimum-privilege requirement for KMS key policies is categorically stricter than for most other IAM resources because the decryption operation is irreversible in its consequence: once a plaintext is produced from an encrypted record, that production event cannot be undone, and if it was produced by an unauthorized principal, the data has been disclosed regardless of what happens to the principal's credentials afterward. The founding session that configures the KMS key policy for the first production deployment is under pressure to unblock the initial compliance sprint and typically produces a policy that is too broad, with a TODO comment marking the intended tightening. The tightening never happens because there is no system that enforces a key policy scope constraint as a regression test, and the working software provides no signal that the scope has not been tightened. The structural fix is to specify the acceptable principal scope in the encryption decision record as a constraint with no temporary exception: the Decrypt and GenerateDataKey permissions must be scoped to the specific IAM roles that require those operations, listed by ARN, with no wildcards and no account root; the list must be reviewed and re-verified quarterly as a standing item in the security review. The access control model decision record is the upstream context: IAM scope creep on a KMS key policy is a specific instance of the access control model's minimum privilege violation, and the quarterly access review that the access control model specifies should include KMS key policy scope as an explicit audit item. The audit log decision record connects at the detection layer: CloudTrail logs every KMS API call with the calling principal, the key ARN, the timestamp, and the source IP; configuring a CloudWatch metric alert for anomalous Decrypt call volume from unexpected principals or unexpected source IP ranges provides the detection capability that can bound the exposure window when a compromise does occur.
Property 3: The key rotation cadence and the migration accumulation failure. Envelope encryption with a key-encrypting key creates a rotation migration cost that grows linearly with the number of DEKs wrapped by that KEK. At founding, the DEK count is zero and the migration cost is zero. The migration cost accumulates invisibly with each new encrypted record, and the rotation policy — if it specifies an interval without a migration plan — becomes structurally harder to execute each month. The session where a founding engineer documents "rotate at least annually" is typically the same session where the first records are being encrypted, when the migration plan for rotating the KEK is genuinely trivial to write: wrap the new KEK around the current KEK, re-wrap the twenty DEKs that exist at founding, verify the migration, delete the old KEK version. At 51 million DEKs, the migration plan requires a sustained batch job, a KMS throughput budget, a dual-version decryption path in the application, and a production window that does not conflict with the highest-volume data import periods. Writing that plan is not difficult — it is a well-understood engineering problem — but it requires knowing the DEK count at migration time, which is unknown at founding. The structural fix is to specify three things in the encryption decision record: the rotation interval as a concrete date (not a target), the rotation method (AWS KMS automatic rotation enabled by default, manual rotation procedure as a documented runbook), and a migration complexity threshold — a DEK count or data volume at which the quarterly review must also update the migration plan and verify that the plan can be executed within the acceptable maintenance window. The secrets rotation decision record covers the operational credentials case: rotating an API key or service password requires updating an environment variable and deploying, which has a known cost and a straightforward rollback; rotating a KEK under envelope encryption requires a data migration, which has a cost that scales with data volume and a rollback procedure that requires maintaining the old key version until the migration is verified complete. The two rotation decisions must be made independently and should be documented in separate records even if they share a rotation review cadence.
What the founding session records and what it omits
The founding technical session on encryption at rest — typically a one-day sprint during the first compliance review — records the encryption algorithm, the key size, the fields or tables covered, and the compliance requirement being satisfied. If the team has thought carefully about the architecture, it records the choice between symmetric encryption with a single key, envelope encryption with per-record DEKs, or transparent database encryption, and the threat model each addresses. If the team has access to a senior security engineer, it records the key management system chosen and the integration pattern. What it does not record is the key storage separation requirement, the IAM scope constraint for cryptographic operations, or the migration plan for key rotation.
These omissions are structurally similar to the omissions in the rest of this series: they are benign at founding, when the omissions are covered by the founding engineers' shared context. The key storage separation requirement is not an omission risk at three engineers on a single-application architecture, when the founding engineer who configured KMS also configured the backup process and understands intuitively that the key is in KMS and the backup is in S3 and the two are separate. The IAM scope constraint is not an omission risk at three engineers, when every IAM principal is a person the founding team knows and the CI/CD role's permissions were written by the same engineer who wrote the KMS key policy. The migration plan is not an omission risk at zero DEKs, when the founding engineer could re-wrap all DEKs in thirty seconds on a laptop if rotation were needed today.
The failure modes develop at different rates and from different triggers. The key storage co-location failure develops from an infrastructure change — a backup location, a misconfigured permission, a new export path — made by someone who does not know the key is in the system being backed up or exported, which typically happens within one to two years as the team grows and the infrastructure changes are made by engineers who were not part of the founding encryption sprint. The IAM scope failure develops from a credential compromise affecting a principal whose scope was broad from founding — which can happen at any time, but the probability increases with team size, dependency count, and the number of external integrations that require broad IAM permissions for their CI/CD or deployment workflows. The migration accumulation failure develops from the compounding of the rotation non-event over the years since founding — each year the rotation does not happen, the migration cost grows and the motivation to write the migration plan decreases because there is always more urgent work and the non-event is invisible until an audit or a key compromise forces the migration at whatever scale has accumulated.
The encryption-at-rest ADR closes these gaps by documenting the key storage separation requirement, the IAM scope constraint, and the rotation plan — including the migration procedure and the complexity threshold trigger — at the time the encryption architecture is established. The decisions never written down in the encryption domain are not the algorithm or the fields covered — those are always documented as the compliance checkbox items. They are the authoritative key store (which system stores the keys and why that system's access controls and backup path are independent of the encrypted data's system), the IAM minimum scope (which roles require Decrypt access and the maximum acceptable scope at any point in the key's lifetime), and the rotation migration plan (how the re-wrapping will be executed at the data volume the product will have in three to five years, specified now when the plan is easy to write and the migration cost is near zero). The new CTO onboarding problem in the encryption domain is specific: the incoming technical leader finds the encryption algorithm documented in the compliance checklist and the KMS key ARN in the Terraform state, but cannot determine whether the key storage separation requirement was verified against the backup system, whether the IAM scope on the KMS key policy has been tightened from the initial sprint configuration, or whether any rotation has occurred since the key's creation date. The encryption ADR makes those decisions explicit, auditable, and verifiable against current system state. The WhyChose extractor finds the encryption-at-rest discussions in your AI session history — the conversation where the founding engineer chose between AES-GCM and envelope encryption, debated whether to store the key in Parameter Store or KMS, decided what the rotation interval should be, or thought through what the migration would look like — and surfaces those parameters so you can assess which assumptions still hold against the current system architecture, data volume, and team composition.
The encryption-at-rest ADR: five sections
Section 1: Algorithm, scope, and key management system. Specify the encryption algorithm (AES-256-GCM for field/column-level encryption; AES-256-CBC or ChaCha20-Poly1305 are acceptable alternatives with documented rationale), the key size, and the fields, columns, or tables covered. Specify the threat model the encryption addresses: protection against an attacker who has read access to the database (e.g., via a SQL injection vulnerability or a misconfigured database credential) but not to the key management system; protection against a backup exposure where the backup data is obtained but the key management system is separately controlled; protection against a database volume snapshot exposure. Specify which threat models the encryption does not address: it does not protect against an attacker who also has access to the application server's runtime memory, where the plaintext DEK exists during decryption; it does not protect against an attacker who has compromised the key management system itself; it does not substitute for access control — an authenticated application user with legitimate query access can request their plaintext data regardless of at-rest encryption. Specify the key management system and the integration pattern: AWS KMS, HashiCorp Vault, or equivalent; the API call pattern (Decrypt on each read, GenerateDataKey on each write for direct encryption; Decrypt the DEK on each read, GenerateDataKey once per record for envelope encryption); the key ARN or key alias and the process for updating it if the key is rotated to a new key ID rather than a new version of the same key. The multi-tenant data isolation decision record intersects here if the product is multi-tenant: the encryption decision must specify whether each tenant requires separate DEKs (tenant-specific key management, where one tenant's key compromise does not expose another tenant's data) or whether shared DEKs are acceptable given the application-layer tenant isolation; tenant-specific DEKs require a DEK namespace by tenant_id and a KMS key policy that optionally allows per-tenant key grants.
Section 2: Key storage separation and backup verification. Specify the key storage separation requirement explicitly as a constraint: encryption keys must be stored in the designated key management system and must not be stored in any system that is included in the encrypted database's backup scope — not in the application database, not in a config table, not in an application config file stored in the same S3 bucket as database backups, not in the environment variables of a container whose image is pushed to the same ECR repository that backup scripts have access to. Specify the independent backup path requirement: the key management system must have a backup and recovery path that is independent of the encrypted database's backup path — different S3 location, different IAM credentials for backup access, different encryption key for the key backup itself. Specify the backup content verification procedure: a verification step in the database backup verification process must confirm that a sample of production backups does not contain key material — for pg_dump backups, the verification script must grep the dump for known key patterns and table names associated with key storage, and must return an error if any match is found. The backup verification frequency must match the backup verification decision record's schedule for restore testing — the separation check should run at least as often as the restore test. Connect this section to the database backup verification decision record so the separation check is a standing item in the backup verification checklist, not a separate procedure that can fall out of scope when the backup verification cadence is updated.
Section 3: IAM scope for cryptographic operations. Specify the minimum IAM scope required for each KMS operation: which IAM roles require kms:Encrypt (the application write path), which require kms:Decrypt (the application read path), which require kms:GenerateDataKey (the DEK generation path for envelope encryption), and which require kms:ScheduleKeyDeletion and kms:DisableKey (administrative operations requiring a separate break-glass process). Specify that the kms:Decrypt permission must not be granted to the account root principal, any wildcard principal, any IAM user (as opposed to role), or any role whose primary purpose is deployment or CI/CD operations — the application read role and the deployment role must be separate and must have different KMS permissions. Specify the condition constraints applicable to the use case: aws:SourceVpc for services that only decrypt within the VPC, aws:SourceIp for services with a known IP range, aws:RequestedRegion to prevent cross-region key usage. Specify the quarterly review requirement: the KMS key policy must be verified quarterly against the list of authorized principals in this decision record; any principal in the key policy that is not in the authorized list must be removed; any new role that requires Decrypt access must be added to both the authorized list and the key policy through a reviewed change, not through an ad-hoc policy update. Connect this section to the access control model decision record's quarterly access review as a standing agenda item: the KMS key policy review is a specialized form of the access control minimum privilege review, and the two reviews should be scheduled together so that the key policy audit does not become a separate process that can fall behind the access control review cadence. Connect to the audit log decision record for the CloudTrail alert configuration: kms:Decrypt call volume per principal, time-of-day anomalies for Decrypt calls from operational roles, and kms:Decrypt calls from principals not in the authorized list should each produce a P2 or P1 alert with a defined investigation window.
Section 4: Key rotation schedule, procedure, and migration plan. Specify the rotation interval as a concrete schedule: for AWS KMS customer-managed keys, enable automatic annual rotation via the key's rotation setting at the time the key is created (not later, as it is easily deferred); document that automatic rotation rotates the key material within KMS but does not re-wrap existing DEKs, and that a DEK re-wrap migration is required to realize the security benefit of the rotation for existing data. Specify the manual rotation procedure for cases where the automatic rotation setting is not sufficient (key compromise requiring immediate rotation, regulatory requirement for shorter rotation intervals, customer contractual requirement for separate key management): the procedure must include the steps for creating a new key version, updating the application's encryption configuration to use the new version for all new Encrypt calls, running the DEK re-wrap migration for existing records, verifying the migration by spot-checking a sample of records, and retiring the old key version by scheduling its deletion after a retention window that allows the verification to complete. Specify the migration complexity threshold: when the estimated DEK re-wrap migration time crosses two hours (or whichever threshold is acceptable for the product's SLA), the quarterly review must update the migration plan to account for the current DEK count, including the batch job configuration (batch size, sleep interval, KMS quota headroom), the dual-version decryption path required during the migration, and the production window selection criteria. Specify that the migration plan must be tested in a staging environment against a DEK count representative of production before it is run in production — a migration plan that has only been tested against a small staging dataset may fail or block production workloads at production scale for reasons that are not visible at small scale. Connect the migration plan to the secrets rotation decision record's rotation audit cadence: the DEK re-wrap migration is the encryption-at-rest analog of the secrets rotation event, and the two should share a quarterly review so that the migration plan's currency against current data volume is checked at the same time as the secrets rotation compliance is verified.
Section 5: Exposure detection, incident response, and compliance evidence. Specify the monitoring configuration for encryption-at-rest anomalies: a CloudWatch metric alert for kms:Decrypt call volume per minute that exceeds the maximum expected operational volume by a defined multiplier (typically 5x the peak operational rate, to catch bulk decryption without producing false positives from normal traffic spikes); a CloudTrail alert for kms:Decrypt calls from any principal not in the authorized principals list; a CloudTrail alert for kms:ScheduleKeyDeletion or kms:DisableKey calls, which should require a break-glass approval process rather than occurring through normal operations. Specify the incident response procedure for a suspected key exposure: disable the key immediately to stop new decryptions, rotate to a new key, run the DEK re-wrap migration under the new key, reconstruct the exposure window from CloudTrail to identify which principals called Decrypt during the suspected window and which record IDs were decrypted, assess whether the exposed records contain personal data requiring GDPR Article 33 notification or HIPAA Breach Notification Rule reporting. Specify the compliance evidence collection procedure: the quarterly KMS key policy review must produce a documented output (a ticket, a signed review record, or an automated compliance scan result) that demonstrates the key policy was verified against the authorized principals list; the annual rotation event must produce a migration log that records the start time, end time, record count, and verification sample result; these records constitute the compliance evidence for HIPAA, SOC 2, and PCI DSS requirements related to encryption key management. Connect the incident response procedure to the incident response playbook decision record: a key exposure incident has specific response steps beyond the standard incident response playbook, including the key disablement step (which is irreversible once taken and should require a named senior engineer's approval), the regulatory notification assessment, and the customer notification requirement for affected records that may contain personal data under the applicable regulations.
FAQ
What should an encryption-at-rest decision record specify beyond the algorithm and key size?
Four things. First, the key storage separation requirement: where encryption keys are stored, which systems may not store encryption keys (any system that stores the data those keys protect), and what the independent access control and backup/restore path requirements are for the key storage system. Second, the IAM scope for cryptographic operations: which principals are permitted to invoke Decrypt and GenerateDataKey on each key, listed by IAM role ARN, the maximum scope acceptable at any point in the key's lifetime, and the quarterly review procedure for verifying the scope has not drifted. Third, the rotation schedule and migration plan: an explicit rotation interval, the rotation method, and a migration procedure for re-encrypting data or re-wrapping DEKs under the new key — specified at founding when the migration cost is near zero. Fourth, the compliance evidence collection procedure: which audit events (CloudTrail Decrypt calls, key policy reviews, rotation events) constitute the compliance evidence for applicable regulatory requirements, their retention period, and the alert condition for anomalous Decrypt volume from unexpected principals.
How do you ensure encryption keys are not included in database backups?
Three mechanisms. First, key storage separation by system: store encryption keys in a dedicated key management system — AWS KMS, HashiCorp Vault, or equivalent — that is never included in the encrypted database's backup scope; the key management system's backup is encrypted by a different key at a different layer and stored at a different location with different access credentials. Second, backup content verification: add a verification step to the database backup verification procedure that confirms a sample of production backups does not contain key material — for pg_dump backups, grep the dump for known key table names and key identifier patterns; for block-level backups, spot-check the restored database to confirm the key table is absent. Third, separation audit: verify quarterly that the key management system and the application database are backed up to different locations with different IAM policies, different access credentials, and different audit logs, so that an exposure of the application database backup cannot also expose the key backup.
How do you scope KMS key policies to prevent compromised credentials from enabling bulk decryption?
Four constraints. First, restrict by specific role ARN: the KMS key policy's Principal element should list specific IAM roles, not the account root or a wildcard; arn:aws:iam::${account-id}:root grants Decrypt to any principal in the account that can assume any role. Second, separate keys by data sensitivity: use different KMS keys for different sensitivity tiers so that a credential compromise affecting one role's scope does not expose all data categories. Third, add condition constraints: attach aws:SourceVpc or aws:SourceIp conditions to Decrypt permissions if the use case supports them; a compromised credential used outside the VPC cannot invoke Decrypt even with the IAM permission if the VPC condition is enforced. Fourth, alert on anomalous Decrypt volume: configure a CloudTrail metric alert for kms:Decrypt call volume per minute per principal; bulk decryption of large datasets produces a call volume spike detectable against the normal operational baseline well within any reasonable incident response window.
How do you plan a DEK/KEK rotation migration to avoid a multi-hour production window?
Four components. First, measure the migration cost before it is forced: estimate the re-wrap time at current DEK count by running the operation against a sample of 10,000 records and multiplying by the total with a 1.5x safety margin; update this estimate quarterly; if the estimate crosses the acceptable threshold, the rotation cadence must increase or the architecture must change before a compliance audit forces the migration at the current scale. Second, run the migration online: DEK re-wrapping does not touch plaintext data — it only reads the DEK ciphertext, calls KMS Decrypt with the old key, calls KMS Encrypt with the new key, and writes the new ciphertext; this is safe to run concurrently with normal application operations as an idempotent batch job with configurable batch size and sleep intervals, without a maintenance window. Third, implement dual-version decryption during the migration: while the migration runs, the application encounters a mix of old-key-wrapped and new-key-wrapped DEKs; the decryption path must try the new key first and fall back to the old key for records not yet migrated; this dual-version window is the designed operating mode for the migration duration. Fourth, schedule against a low-traffic window: the KMS API call volume during re-wrapping is predictable and substantial; scheduling the batch job during off-peak hours keeps the call rate below the account's service quota and reduces the risk of starving application Decrypt calls during peak usage.