What is the difference between ArgoCD self-healing and the default sync behavior, and why does it matter during an incident?

ArgoCD's sync behavior has two components that are independently configurable and frequently confused: automated sync (which determines whether ArgoCD automatically applies changes from Git to the cluster when a diff is detected) and self-healing (which determines whether ArgoCD automatically reverts direct cluster changes — kubectl apply, kubectl edit, kubectl delete — back to the Git-defined state). Both are disabled by default in ArgoCD and must be explicitly enabled in the Application manifest's spec.syncPolicy. When automated sync is enabled without self-healing, ArgoCD applies new Git commits to the cluster automatically, but if an engineer makes a direct kubectl change, ArgoCD detects the drift (shows the application as OutOfSync in the UI) but does not revert it — the manual change persists until the next automated sync triggered by a new Git commit. When both automated sync and self-healing are enabled — which is the recommended production configuration because it enforces the Git repository as the single source of truth — ArgoCD reconciles on a configurable interval (default 3 minutes, configurable via the application.resync-period setting in ArgoCD's ConfigMap) and during each reconciliation cycle, any resource that differs from the Git-defined state is reverted to match. The operational consequence is that a kubectl patch or kubectl apply made directly to a running cluster during an incident will be silently overwritten by the next reconciliation cycle. The fix disappears without an error message, without a log entry visible to the engineer making the change, and without any indication in the application output that the configuration was reverted. The on-call engineer sees the incident symptoms reappear after 3 minutes, investigates whether the patch took effect, finds the current configuration matches the change they made, investigates whether the change is causing a different issue, and repeats the cycle until a second engineer checks ArgoCD's sync history and sees that ArgoCD applied a sync 3 minutes after the manual change. The correct incident response in a self-healing GitOps environment is to commit the fix to the Git repository and push, rather than applying directly to the cluster. This is the correct procedure when the team knows ArgoCD self-healing is enabled. The problem is that ArgoCD self-healing is frequently enabled in the initial setup session as part of a 'recommended configuration' block in a tutorial or AI session, without documenting it as an operational constraint on incident response — the implication that all production changes must go through Git, including emergency changes during active incidents, is the architectural decision that determines the incident response procedure, and it is not recorded anywhere that the on-call engineer can find during an incident at 2am.

Why is base64 encoding in Kubernetes Secrets not encryption, and what are the GitOps-compatible mechanisms for managing secrets without committing plaintext values to Git?

Kubernetes Secrets store data as base64-encoded values in etcd. Base64 is an encoding, not encryption: base64 converts binary data to an ASCII representation for transmission and storage in text formats, using a character set of 64 printable characters. A base64-encoded value can be decoded to its original form by any system with access to the base64-decode function — no key, no password, no secret is required. `echo 'c3VwZXJzZWNyZXQ=' | base64 --decode` returns `supersecret`. When a Kubernetes Secret YAML manifest is committed to a Git repository with the data field containing base64-encoded values, any party with read access to the repository — or any party with access to the git history after the file is deleted — can decode the secret values immediately. The GitOps workflow's requirement that all Kubernetes resources be defined as YAML manifests in the Git repository creates a tension with the Kubernetes Secret resource, whose data field is base64-encoded and therefore not a safe form for committing to source control. Four mechanisms address this tension. Sealed Secrets: the bitnami-labs/sealed-secrets controller runs in the cluster and exposes a public key; engineers use the kubeseal CLI to encrypt a plain Secret YAML with the public key, producing a SealedSecret YAML with an encrypted ciphertext rather than base64-encoded values; the SealedSecret is committed to Git; the Sealed Secrets controller decrypts it using its cluster-bound private key and creates the plain Secret in the cluster; an attacker with read access to the Git repository sees only ciphertext; the risk is the cluster-private-key, which must be backed up and is the single point of compromise for all sealed secrets. External Secrets Operator: an ExternalSecret custom resource in Git references a secret by path and key in an external secret store (AWS Secrets Manager, HashiCorp Vault, GCP Secret Manager, Azure Key Vault); the External Secrets controller resolves the reference at reconciliation time and creates a plain Kubernetes Secret in the cluster; no secret value is ever committed to Git; the ExternalSecret manifest is safe to commit because it contains only the path and key reference, not the value; rotation is handled in the external secret store, and the ExternalSecret can be configured to refresh the Kubernetes Secret on a schedule; the risk is the IAM role or service account that grants the controller access to the secret store. SOPS with age or GPG: manifests containing secret values are encrypted at the file level using Mozilla SOPS before being committed; Flux natively decrypts SOPS-encrypted manifests using a cluster-bound age private key or a GPG key referenced in the Flux Kustomization; the encrypted file is committed to Git; the secret values are not visible to git history readers because the file is encrypted, not encoded; the risk is the decryption key, which must be stored in the cluster and is the single point of compromise. HashiCorp Vault Agent Injector: secrets are never stored as Kubernetes Secrets; instead, pods annotate themselves to request Vault Agent injection, and a Vault Agent sidecar is injected by a mutating webhook; the sidecar authenticates to Vault using Kubernetes ServiceAccount JWT tokens, retrieves the secrets, and writes them to a shared memory-mapped volume that the application container reads; the application never sees a Kubernetes Secret and the secret value is never stored in etcd; the risk is the Vault server availability and the Vault role binding permissions. Each mechanism has a different secret rotation model, a different blast radius if compromised, and a different recovery procedure if the cluster-bound decryption key is lost. The choice must be made before the first Kubernetes Secret is created in the GitOps repository, because migrating secrets between mechanisms after the fact requires rotating every secret value to ensure that the old mechanism's exposure surface (e.g., Sealed Secrets ciphertext encrypted with a Sealed Secrets private key that has since been rotated) is no longer valid.

What is the GitOps rollback procedure, and how does it differ from kubectl rollout undo or ArgoCD's built-in rollback button?

The GitOps rollback procedure is a git revert or git push to the GitOps repository pointing the application manifest at the previous image tag or configuration state, which the GitOps controller detects as a new commit and syncs to the cluster. The procedure differs from non-GitOps rollback in three important ways. First, kubectl rollout undo resets a Deployment to the previous ReplicaSet state that Kubernetes has retained in its rollout history (default 10 revisions); it does not require a Git commit or code review; it takes effect immediately; but it creates drift between the cluster state and the Git repository because the Git repository still references the broken image tag while the cluster is running the previous image tag. In a self-healing GitOps environment, kubectl rollout undo followed by ArgoCD self-healing produces a loop where the rollback is immediately overwritten by the next ArgoCD sync cycle, re-deploying the broken image. The correct procedure is to push a new commit to the Git repository that updates the image tag to the previous working version, which ArgoCD or Flux detects and syncs to the cluster. Second, ArgoCD has a built-in Rollback button in the UI and a CLI command (argocd app rollback) that reverts the cluster to a previous sync state from ArgoCD's sync history. The ArgoCD rollback sets the application to a specific Git revision from the sync history, deploys the resources from that revision to the cluster, and marks the application as OutOfSync against the current HEAD of the Git branch (because the HEAD still references the broken commit). The ArgoCD rollback is immediate and does not require Git access, making it suitable for incident response when the engineer does not have write access to the Git repository or when the Git repository's CI pipeline takes too long to approve and merge the revert commit. The risk of ArgoCD rollback is that it creates a permanent drift between the cluster state and the Git repository HEAD that persists until a new Git commit is pushed; if the team forgets to push the revert commit after using ArgoCD rollback during an incident, the Git repository continues to reference the broken image tag and the next ArgoCD sync (triggered by an unrelated commit) will re-deploy the broken image. Third, the GitOps rollback procedure for a database schema migration is not the same as the application rollback procedure: if the broken application version also included a database migration, rolling back the application to the previous image tag leaves the database schema in the migrated state, which the previous application version may not be compatible with. The GitOps rollback decision record must specify the procedure for rollbacks that include schema migrations — whether the schema migration is designed to be backward-compatible with the previous application version (expand-migrate-contract pattern), whether a separate schema rollback is required before the application rollback, and whether the database migration and application deployment are gated by the same GitOps Application resource or separate resources that can be rolled back independently. The rollback procedure must be written in the incident runbook as a step-by-step procedure that the on-call engineer can execute during an incident without referring to documentation beyond the runbook itself, because during an active incident the engineer will not have time to read the GitOps documentation and determine the correct rollback procedure from first principles.

How should GitOps repository structure be designed for teams managing multiple environments, and what are the tradeoffs between monorepo and multi-repo approaches?

GitOps repository structure determines the blast radius of a commit pushed to the wrong branch, the access control model for who can deploy to which environment, the complexity of the promotion workflow from development to staging to production, and the auditability of production deployments. Two primary structures are used. Monorepo with environment directories: a single Git repository contains all Kubernetes manifests organized by environment in a directory hierarchy (e.g., environments/development/, environments/staging/, environments/production/); Flux Kustomizations or ArgoCD Applications are configured per environment, each watching a specific directory and a specific Git branch (or the same branch but different paths); a promotion workflow copies manifests from the development directory to the staging directory via a pull request, and from staging to production via a second pull request. The monorepo approach simplifies repository management and allows cross-environment changes to be visible in a single diff, but creates a blast radius where a misconfigured CI pipeline or a mistaken direct push to the main branch can affect all environments simultaneously, and requires fine-grained branch protection rules and CODEOWNERS files to prevent production deployments from being made without review. Multi-repo with one repository per environment: the development, staging, and production environments each have a separate GitOps repository; the application manifests for production are in a separate repository from staging; access to the production repository is restricted to a smaller set of principals; promotion requires opening a pull request in the production repository, which may require additional approval steps beyond what the staging repository requires; a CI pipeline in the application repository pushes image tag updates to each environment repository in sequence after the previous environment's deployment is verified. The multi-repo approach provides strong isolation between environments and a clear access control boundary, but complicates cross-environment visibility and requires automation to keep the environment repositories consistent with each other. A third approach, used by teams with Flux, is a single repository with environment-specific overlays using Kustomize: a base/ directory contains the canonical resource definitions, and overlays/development/, overlays/staging/, and overlays/production/ directories contain environment-specific patches (different replica counts, different resource limits, different image tags, different ConfigMap values); Flux Kustomizations point at each overlay directory and apply the base resources with the overlay patches applied; this approach reduces duplication while maintaining clear environment boundaries, but requires familiarity with Kustomize's patching model and can become complex when overlays diverge significantly from the base. The repository structure decision must specify the branch protection rules for each repository or branch, the PR approval requirements for production environment changes, the automation that triggers promotion from one environment to the next (image update automation via Flux ImageUpdateAutomation, or CI pipeline webhooks that push image tag commits), and the access control model mapping Git repository write access to cluster namespaces and service accounts, because GitOps converts cluster access control into Git access control and the team's production deployment authorization model must be re-expressed as Git repository permissions and branch protection rules.

The GitOps decision record: why the sync model you chose determines your infrastructure drift detection ceiling and your secret management blast radius

Published 2026-07-18 · WhyChose

GitOps decisions are made when the team decides that Kubernetes deployments should be driven from a Git repository rather than from CI pipeline kubectl commands, when an engineer reads a blog post about ArgoCD and sets it up over a weekend to improve deployment traceability, and when a new platform engineer joins and replaces a collection of Helm chart CI jobs with a Flux Kustomization that watches the main branch and syncs to the staging cluster. The AI session that handles the initial GitOps setup is tool-installation-oriented and terminates at the working sync: it installs ArgoCD via the kubectl apply of the install manifest, configures an Application resource pointing at the Git repository and the target namespace, pushes a small manifest change to the repository, observes that ArgoCD detects the change and shows the application as Synced in the UI, and closes. The GitOps tool works. The session succeeds. The session closes.

What the AI session does not produce is the second half of the GitOps decision. The session answers "how do I get ArgoCD to deploy from a Git repository?" and delivers a working ArgoCD installation with a first successful sync visible in the UI. It does not ask: what happens to a direct kubectl apply made during an incident — does ArgoCD's self-healing behavior overwrite the manual change within the next reconciliation interval, and does the engineer making the change know that it will be overwritten; what is the canonical mechanism for managing secrets in the GitOps repository, and has any engineer been told that committing a Kubernetes Secret YAML file with base64-encoded values is equivalent to committing the plaintext value because base64 is not encryption; what is the reconciliation interval, and does the team know that the default ArgoCD resync period means a manual change will be reverted within 3 minutes without producing any error or log entry visible to the engineer who made the change; what is the GitOps rollback procedure for a bad deployment, and how does it differ from kubectl rollout undo (which produces drift that self-healing will immediately overwrite) and from ArgoCD's UI rollback button (which creates persistent drift between the cluster state and the Git repository HEAD); what resources are in scope for GitOps management and which are managed outside of GitOps (Terraform-managed IAM roles, manually applied CRDs from vendor Helm charts, manually configured cluster-level resources), and is the boundary between GitOps-managed and non-GitOps-managed resources documented so that a future engineer does not manage the same resource in both systems and produce a reconciliation conflict; what is the promotion model for environment progression (development to staging to production), and who is authorized to approve a production promotion. Each of these questions has an answer that is not derivable from "an application shows as Synced in the ArgoCD UI" as the session completion criterion. Each answer determines whether an on-call engineer's emergency kubectl patch survives to stabilize the system or is silently overwritten by the next reconciliation cycle while the engineer investigates why the fix isn't taking effect, whether a database credential committed to the GitOps repository as a Kubernetes Secret YAML exposes that credential to every person and system with read access to the repository and its git history in perpetuity, whether a misconfigured manifest pushed to the main branch produces a PagerDuty alert within 5 minutes or goes undetected for hours because reconciliation failure notification was never configured, and whether the rollback procedure in the incident runbook is "git revert the offending commit and push" or "kubectl rollout undo deployment/api" — the second of which is silently overwritten by ArgoCD within the reconciliation interval. The answers are in the AI sessions. They are almost never written down.

Two ways GitOps decisions produce the wrong outcome in production

The reconciliation loop that silently reverted every incident fix for two hours

A developer tools SaaS adopts ArgoCD in month 14 after launch. The team of nine engineers is deploying via a GitHub Actions workflow that runs helm upgrade --install on every push to the main branch, which works but provides no visibility into the current cluster state and no audit trail linking deployed versions to Git commits. A senior engineer spends a Saturday afternoon setting up ArgoCD: installs ArgoCD into the cluster's argocd namespace using the stable install manifest, creates Application resources for the api, worker, and frontend deployments, configures automated sync with self-healing enabled as part of the standard recommended configuration in the ArgoCD getting-started guide, verifies that pushing a change to the Helm values in the repository causes ArgoCD to detect the diff and apply the change within 3 minutes, and updates the GitHub Actions workflow to remove the helm upgrade step. The ADR she writes for the migration covers the tool selection rationale (ArgoCD over Flux based on the team's preference for a UI), the repository structure (a monorepo with a helm/ directory containing the chart and a values/ directory with per-environment values files), and the access control model (ArgoCD syncs using a ServiceAccount with cluster-admin permissions scoped to the application namespaces). The ADR does not document the self-healing behavior, the reconciliation interval, or the implication that direct kubectl changes to the cluster will be overwritten by ArgoCD during the next reconciliation cycle. The session concludes that "ArgoCD is now managing deployments" and does not record that "all cluster changes must now be made via Git commits, including emergency changes during active incidents, because self-healing will revert any direct kubectl changes within 3 minutes."

Seven weeks later, the api Deployment experiences a memory leak in a background job processor that was introduced in a dependency upgrade. The P99 memory usage of the api pods spikes from 320MB to 940MB over a 45-minute window on a Wednesday evening. The on-call engineer receives a PagerDuty alert for pod OOMKills. He investigates the logs, identifies the background job processor as the source of the memory growth, and determines that the fastest mitigation is to set the PROCESS_BACKGROUND_JOBS environment variable to false on the running Deployment — disabling the background job processor on the running pods while the engineering team prepares a proper fix. He runs kubectl set env deployment/api PROCESS_BACKGROUND_JOBS=false -n production. Kubernetes applies the change, the pods restart with the updated environment variable, the OOMKills stop, the alert clears. The engineer updates the PagerDuty incident with the mitigation applied and begins investigating the root cause in the dependency changelog.

Three minutes and twelve seconds later, ArgoCD completes its resync cycle. The ArgoCD controller compares the cluster state (api Deployment with PROCESS_BACKGROUND_JOBS=false) against the Git repository state (Helm values file with no PROCESS_BACKGROUND_JOBS override, which means the environment variable is not set at all). ArgoCD detects a diff and, because self-healing is enabled, applies the Git-defined state to the cluster. The api pods restart with the environment variable removed — equivalent to PROCESS_BACKGROUND_JOBS=true, the default — and the memory leak resumes. ArgoCD logs a sync event with the message "synced successfully" and marks the application as Synced. The engineer's PagerDuty incident, which was marked as resolved after the mitigation, receives a new OOMKill alert 8 minutes later as the memory usage climbs again.

The engineer re-applies the kubectl set env command. ArgoCD reverts it again 3 minutes later. This loop repeats three times over the next 40 minutes. The engineer escalates to a second engineer, describing "a system issue where my kubectl changes are not persisting — the pods keep reverting to the bad configuration." The second engineer checks the Kubernetes event log, the Deployment revision history, and the container environment variables across restarts, none of which show ArgoCD's involvement because ArgoCD's sync applies as a clean Kubernetes apply with no distinguishing annotation. The second engineer checks the ArgoCD UI and sees the application showing as Synced with a sync timestamp of 2 minutes ago. She checks the ArgoCD sync history and finds a pattern: the application synced 3 minutes after the first kubectl set env, 3 minutes after the second, and 3 minutes after the third. The sync is overwriting the manual changes. The mitigation is to push a commit to the Git repository with PROCESS_BACKGROUND_JOBS: false in the Helm values file, which ArgoCD detects and syncs, applying the change persistently because it now matches the Git state. Total incident duration: 2 hours 14 minutes, of which 1 hour 40 minutes was consumed by the investigation of why manual changes were not persisting and the identification of ArgoCD self-healing as the cause. The root cause of the extended incident was not the memory leak — the initial mitigation was correct and took effect in under 2 minutes — but the absence of a documented operational constraint: that self-healing was enabled, what the reconciliation interval was, and that the correct emergency procedure was to commit to Git rather than kubectl apply.

The base64-encoded database password that lived in the GitOps repository for eleven months

A B2B analytics platform adopts Flux GitOps for its Kubernetes deployments six months after launch. The platform engineer sets up a gitops/ repository containing Flux Kustomizations for each application, a base/ directory with shared resources, and environment overlays for staging and production. The setup session installs the Flux CLI, runs flux bootstrap github to configure the Flux controllers and connect them to the repository, creates Kustomization resources pointing at the application directories, and verifies that a change to a Deployment manifest in the repository is applied to the cluster within 60 seconds. The session is focused on the Flux bootstrap configuration and the Kustomization structure. It does not address secret management. The platform engineer notes "secrets handling TBD" in the repository's README and moves on to migrating the application manifests into the repository.

Two weeks later, a backend engineer needs to deploy a new database connection configuration to the staging environment. The database connection string includes a username and password for the staging PostgreSQL instance. The engineer searches the repository for examples of how secrets are handled in the GitOps setup and finds the "secrets handling TBD" note in the README. She searches online for "Kubernetes secret GitOps" and reads a tutorial that creates a Secret resource YAML file with base64-encoded values in the data field. The tutorial note that "Kubernetes Secrets are base64 encoded, not encrypted" does not register as significant — the engineer interprets "base64 encoded" as a form of encoding that is part of how Kubernetes handles secrets internally. She creates a secret.yaml file in the gitops/overlays/staging/ directory containing the database connection string base64-encoded in the data field, commits it to the repository, and pushes. Flux applies the Secret to the staging cluster. The database connection works. The deployment succeeds. The secret.yaml file is committed to the repository.

Four months later, the platform team rotates the staging database password after a security review recommends quarterly credential rotation. A backend engineer updates the secret.yaml file with the new base64-encoded password, commits the change, and pushes. The Secret is updated in the cluster. The old password is no longer valid in the database. The commit history in the gitops/ repository now contains both the old and new password values as base64-encoded strings in consecutive commits. The git log retains both values permanently.

Two months after the password rotation, a new engineer joins the team. During his first week, he clones the gitops/ repository to understand the infrastructure setup. He reads the secret.yaml files in the repository. He knows that base64 can be decoded with a single command. He decodes the values in the current secret.yaml out of curiosity to understand what the encoding looks like — and realizes that what appears as a string of random characters is the plaintext database password. He reports this to the platform team. The platform team investigates the repository history and finds three versions of the staging database password, two versions of a Redis connection string, and one version of a third-party API key, all committed as base64-encoded values at various points over the preceding eleven months. The credentials are rotated immediately. The git history is scrubbed using git filter-repo, which requires all existing clones of the repository to be deleted and re-cloned from the scrubbed origin. The remediation takes two days of engineering time, coordination with all engineers who have cloned the repository, verification that no CI/CD system has a local clone with unrevoked authentication, and a security review to assess whether the credentials were accessed by anyone with repository read access who was not authorized. The initial GitOps setup session's "secrets handling TBD" note was the root cause: there was no documented policy specifying that Kubernetes Secret YAML files must not be committed to the repository in any form — plaintext or base64 — and no mechanism specified for handling secrets in a GitOps-compatible way. Any engineer who needed to deploy a secret did what the first engineer did: searched for examples, found the standard Kubernetes Secret YAML format, and used it.

Three structural properties that GitOps decisions determine

The sync model and the drift detection ceiling

The sync model specifies the reconciliation trigger (continuous poll on an interval versus event-driven sync on Git webhook), the reconciliation interval for poll-based sync, the self-healing behavior when the cluster state drifts from the Git-defined state, the manual change override policy during incidents, and the scope of resources under GitOps management. Without a documented sync model, the team's understanding of GitOps is "the cluster reflects what is in Git" — a model that is accurate for normal operations and silently wrong during incidents, because the team's muscle memory for infrastructure changes (kubectl apply, kubectl edit, kubectl set) produces changes that are overwritten within the reconciliation interval without the engineer being notified. The sync model must specify six properties. First, the reconciliation trigger and interval: ArgoCD polls the Git repository on a configurable interval (default 3 minutes, set by application.resync-period in the argocd-cm ConfigMap); webhook-triggered sync reacts to Git push events via a webhook configured in the Git hosting provider and processed by ArgoCD's API server, producing near-instant sync without polling overhead; the reconciliation interval must be specified in the decision record because it determines the maximum drift window — a cluster change that is not reflected in Git will be reverted within one reconciliation interval, and the engineer making the change must know the interval to predict when the revert will occur. Second, the self-healing behavior: ArgoCD's syncPolicy.automated.selfHeal setting controls whether ArgoCD reverts cluster state to match Git state after a drift is detected; Flux's default behavior is always self-healing (Flux continuously reconciles the cluster to match the Git state without a separate self-healing toggle); the self-healing behavior must be documented explicitly as "any direct kubectl change to a managed resource will be reverted within N minutes" rather than as "ArgoCD automated sync is enabled," because the operational implication of self-healing is different from the operational implication of automated sync, and engineers who have worked in non-GitOps Kubernetes environments will not infer the self-healing behavior from "automated sync enabled." Third, the manual change policy for incidents: the decision record must specify the correct emergency procedure when the cluster needs to be changed immediately and the Git repository CI pipeline approval process would take too long — whether the team suspends ArgoCD's automated sync during the incident (via argocd app set --sync-policy none or the ArgoCD UI's disable auto sync toggle), applies the emergency kubectl change, and re-enables auto sync after confirming the fix is working; or whether the team has a fast-path Git commit and push workflow (a pre-configured emergency branch with auto-merge that bypasses the standard PR review process) that allows commits to reach the cluster within 60-90 seconds; the policy must include the step to subsequently commit the emergency change to the main branch and delete the bypass, because an emergency change that is applied by suspending GitOps and not subsequently committed creates permanent drift. Fourth, the GitOps scope boundary: not all Kubernetes resources are appropriate for GitOps management; cluster-level resources created by managed cloud services (LoadBalancer Services whose spec.loadBalancerIP is assigned by the cloud controller), resources managed by Helm chart operators (CertManager Certificate resources managed by the cert-manager controller), and resources managed by Terraform (the cluster itself, IAM roles, VPC configuration) are not appropriate for GitOps management because a GitOps sync that overrides them creates a conflict with the resource's actual manager; the scope boundary must be documented as the set of resource types and namespaces under GitOps management, with the corresponding manager for resources outside the scope, so that a future engineer does not attempt to manage a cluster-level resource in both the GitOps repository and a Terraform module and create a perpetual conflict where each system reverts the other's changes. Fifth, the drift detection and alerting model: GitOps' primary value proposition is that the cluster state always matches the Git state; drift detection means knowing when the cluster has deviated from the Git state before an incident forces the discovery; ArgoCD's application health and sync status must be monitored with alerts configured to fire when any application enters a Degraded, OutOfSync, or Error state for longer than the reconciliation interval; the alert routing must be specified in the decision record because a reconciliation failure that silently persists for hours (because the alert goes to a Slack channel that is not monitored at 2am) is equivalent to no drift detection; the threshold for escalating a sync failure to PagerDuty versus Slack must be specified. Sixth, the resync and force-sync policy: ArgoCD's hard refresh (which clears the cached manifests and forces a re-evaluation from the Git repository, bypassing the poll interval) and force sync (which applies the Git state to the cluster even if the cluster state is already marked as Synced but may have drifted since the last evaluated diff) are operational tools that the on-call engineer must know exist and know when to use; a force sync is appropriate when a cluster node failure or control plane outage has produced an unknown cluster state; a hard refresh is appropriate when a ConfigMap or Secret that is referenced by a managed resource was changed outside of ArgoCD's awareness and the application health depends on the current value; both operations must be in the incident runbook alongside the rollback procedure. The container orchestration decision record specifies the Kubernetes cluster configuration and the RBAC model; the GitOps sync model's self-healing behavior must be consistent with the cluster's RBAC policy — if the ArgoCD controller's ServiceAccount has cluster-admin permissions (which is the default for many ArgoCD installations), it can revert any resource in any namespace, including resources that the team's access control model designates as restricted; the RBAC permissions granted to the ArgoCD controller must be scoped to the namespaces under GitOps management, not to the entire cluster.

The secret management model and the plain-text exposure surface

The secret management model specifies the mechanism for managing secret values in the context of a GitOps workflow where all Kubernetes resources are defined as YAML manifests in a Git repository, the scope of which secret types use which mechanism, the rotation procedure in the GitOps context, and the recovery procedure if the cluster-bound decryption key (Sealed Secrets private key, SOPS age private key) is lost. Without a documented secret management model, the GitOps repository's convention for secrets is whatever each engineer did when they first needed to commit a secret resource — which, in the absence of a specified mechanism, is the path of least resistance: a Kubernetes Secret YAML with base64-encoded values, which are not encrypted and which expose the plaintext value to anyone with read access to the repository or its git history. The secret management model must specify six properties. First, the canonical secret mechanism for the platform: the choice between Sealed Secrets, External Secrets Operator (ESO), SOPS, and Vault Agent Injector must be made once and applied consistently across all applications and environments; using multiple mechanisms produces a secret management audit surface where some secrets are managed by Sealed Secrets (encrypted ciphertexts in Git, decryptable only by the cluster controller), some by ESO (references to external secret store paths, no values in Git), and some by ad hoc base64-encoded Secret YAMLs (plaintext values in Git), and the security properties of the ad hoc group determine the platform's actual security posture regardless of the controls applied to the other groups. Second, the secret type taxonomy: different secret types have different lifecycle characteristics that affect mechanism selection; long-lived infrastructure credentials (database passwords, TLS private keys, third-party API keys) change infrequently and benefit from the External Secrets Operator's centralized rotation model where a single secret store update propagates to all consuming clusters; application configuration values that are treated as secrets for policy reasons but rarely contain true credentials (feature flag override values, internal service URLs marked as sensitive) may be appropriate for Sealed Secrets because they do not need the full external secret store infrastructure; the type taxonomy must specify which mechanism applies to which category of secret. Third, the rotation procedure in the GitOps context: for ESO-managed secrets, rotation requires updating the value in the external secret store (AWS Secrets Manager, Vault) and the ExternalSecret resource's refreshInterval determines when the ESO controller refreshes the Kubernetes Secret from the store; if the interval is 1 hour, a rotated value in the secret store takes up to 1 hour to propagate to the cluster Secret; the rotation procedure must specify whether the team uses the ESO force-refresh annotation to trigger immediate refresh after rotation or whether the interval-based refresh is sufficient; for Sealed Secrets, rotation requires re-encrypting the new value with kubeseal, committing the updated SealedSecret to Git, and waiting for ArgoCD/Flux to sync; the rotated ciphertext in Git history reveals that the previous ciphertext is no longer valid, but does not expose the previous plaintext value because the ciphertext is encrypted rather than base64-encoded. Fourth, the cluster decryption key backup: Sealed Secrets stores its private key in a Kubernetes Secret in the sealed-secrets namespace; if the cluster is destroyed and the Secret was not backed up, all SealedSecrets in the repository become permanently undecryptable, and every sealed secret must be re-sealed with a new cluster key; the backup procedure for the Sealed Secrets controller key must be documented, scheduled, and tested with a restoration exercise, because a key loss that is discovered during cluster recovery after a catastrophic failure is the worst time to discover that the key was not backed up; similarly, a SOPS age private key that is stored only as a Kubernetes Secret without an external backup produces the same irrecoverable state if the Secret is deleted. Fifth, the secret scope management: Kubernetes Secrets are namespace-scoped; a Secret in the production namespace cannot be referenced by a resource in the staging namespace; in a GitOps monorepo with per-environment overlay directories, the same logical secret (database password for the api service) must be defined separately in the staging and production overlays; the secret management model must specify how the same logical secret is managed across environments — whether the ESO ExternalSecrets reference different paths in the secret store for each environment (preferred because each environment has an independent value and independent rotation), or whether the same value is shared across environments via a shared secret store path (which means that staging and production share the same database password, a security policy violation in most compliance frameworks). Sixth, the secret access audit model: who can read the secret values in the canonical secret store is not the same as who can read the Kubernetes Secret that the ESO controller creates in the cluster; the access audit model must specify both the secret store access control (which IAM roles, Vault policies, or GCP IAM bindings can read which secret paths) and the Kubernetes RBAC bindings that control which service accounts, users, and controllers can read Kubernetes Secret objects in each namespace; in a GitOps environment where the ESO controller has broad secret store read access, a misconfigured ExternalSecret that references a production secret path while being deployed to the staging namespace creates a cross-environment secret exposure that does not require any IAM misconfiguration — the ExternalSecret resource's path reference is the exposure mechanism. The secrets management decision record specifies the secret store technology and the credential lifecycle model; the GitOps secret management model is the integration layer between the secret store and the Kubernetes workloads, and must be specified as a joint concern of the secrets management decision record and the GitOps decision record, because the two decisions determine together whether secret values are ever committed to Git in any form, whether the cluster decryption key is backed up and tested, and whether secret rotation requires a GitOps commit, a secret store update, or both.

The reconciliation failure model and the incident response floor

The reconciliation failure model specifies the alert configuration for GitOps controller failures, the escalation path when a manifest push produces a reconciliation error, the rollback procedure for bad deployments, and the promotion model for environment progression. Without a documented reconciliation failure model, a broken manifest pushed to the GitOps repository produces a silent failure: ArgoCD or Flux enters a Degraded state, stops applying changes from the repository, and waits; the broken application continues running on the previous configuration (which may or may not be the desired behavior during the failure window); no alert fires; the failure is discovered when an engineer checks the ArgoCD UI for an unrelated reason or when a second manifest push also fails and an engineer investigates why the change did not apply. The reconciliation failure model must specify six properties. First, the alert configuration for controller health: ArgoCD exposes Prometheus metrics for application health (argocd_app_info with a health_status label), sync status (argocd_app_sync_total with a phase label), and controller errors; Flux exposes Prometheus metrics for reconciliation success and failure per Kustomization and HelmRelease; both controllers must have alerting configured that fires when any managed application is in a non-healthy or non-synced state for longer than two reconciliation intervals — a threshold that allows for transient git connectivity issues or API server load spikes without paging, while ensuring that a persistent reconciliation failure produces an alert within 10-15 minutes; the alert routing must specify whether the alert goes to PagerDuty (for production environment failures) or Slack (for staging environment failures), and the on-call runbook must include the steps to diagnose a GitOps reconciliation failure. Second, the rollback procedure: the GitOps rollback procedure is a new Git commit that reverts the breaking change — either a git revert that produces a new commit undoing the breaking manifest, or a direct edit of the manifest to reference the previous image tag or configuration — pushed to the GitOps repository and applied by the next reconciliation cycle; this procedure must be in the incident runbook with the exact git commands and the expected outcome; the common mistake — kubectl rollout undo deployment/ — must be explicitly noted as incorrect in a self-healing GitOps environment because it produces cluster state drift that ArgoCD reverts within the reconciliation interval, and the incident on-call engineer at 2am will not know this without the runbook note; the ArgoCD UI rollback as an alternative to git revert must be described with its limitation: it creates persistent cluster-to-Git drift that must be resolved with a subsequent git commit. Third, the emergency change procedure: a fast-path workflow for emergency cluster changes that bypasses the standard PR review process must be documented — the steps to suspend ArgoCD automated sync (argocd app set --sync-policy none), make the emergency kubectl change, verify the change resolves the incident, commit the change to the Git repository, re-enable automated sync (argocd app set --sync-policy automated), and verify that the sync does not overwrite the committed change; the fast-path workflow must specify who is authorized to suspend automated sync and what the expected cleanup steps are after the incident. Fourth, the environment promotion model: in a GitOps workflow, promoting a change from staging to production is not kubectl apply to the production namespace but rather a Git commit that updates the production manifests to reference the new image tag or configuration, followed by an ArgoCD sync that applies the change; the promotion model must specify the workflow — whether promotion is a manual pull request from staging to production with required approval, an automated pipeline that opens a production PR when the staging deployment passes all smoke tests, or a Flux ImagePolicy that automatically updates the production image tag when a new image is pushed to the registry with a stable tag; the automated promotion model reduces time-to-production for validated changes but removes the human review step that catches misconfigured manifests before they reach production. Fifth, the blast radius of a misconfigured manifest: a Kubernetes manifest with a syntax error fails validation at the ArgoCD or Flux level before being applied to the cluster, with no impact; a manifest with a semantic error (a Deployment referencing a non-existent ConfigMap, a resource request exceeding node capacity) fails during cluster apply, producing a reconciliation error that leaves the current cluster state unchanged but prevents future valid manifest changes from being applied until the error is resolved; a manifest with a behavioral error (a correct manifest that specifies incorrect configuration values — wrong replica count, wrong memory limit, wrong environment variable value) is applied successfully and affects running workloads; the blast radius is the set of running pods or services affected by the applied manifest; the decision record must specify the validation gates applied to manifests before they reach the GitOps repository (kubectl validate, kube-score, kubeval, OPA/Gatekeeper policy evaluation in CI) and the configuration tests applied in the staging environment before promotion to production. Sixth, the dependency order for multi-application deployments: in a GitOps repository with multiple applications (frontend, backend API, background worker, database migration job), changes that must be applied in a specific order (database migration before new application version, ConfigMap before Deployment) require explicit orchestration; ArgoCD's sync waves (argocd.argoproj.io/sync-wave annotation) specify the order in which resources are applied during a single sync; the sync wave configuration must be specified in the decision record for applications with ordering dependencies, because a missing sync wave annotation causes ArgoCD to apply resources in an unspecified order that may succeed in one deployment and fail in another based on Kubernetes scheduler timing. The deployment strategy decision record specifies the deployment model (rolling, blue/green, canary); the GitOps reconciliation model is the implementation layer for the deployment strategy, and the rollback procedure must be consistent across both records — a blue/green deployment managed by ArgoCD uses the Rollout resource from Argo Rollouts rather than the standard Deployment resource, and the rollback procedure for a blue/green GitOps deployment is different from both the kubectl rollout undo command and the git revert command.

Three AI session types that embed GitOps decisions without documenting them

The initial GitOps setup session is where the sync model, self-healing behavior, and secret management policy are established without being named as decisions. The session is tool-installation-oriented and terminates at the working sync: install ArgoCD or run flux bootstrap, configure a repository connection, create an Application or Kustomization resource, verify that a manifest change is applied to the cluster, and close. The self-healing configuration is applied as part of the recommended setup — ArgoCD's getting-started guide, the Flux quick-start documentation, and most AI-generated setup guides include automated sync with self-healing as the default configuration because it produces the intended GitOps behavior; but the operational implication — that manual kubectl changes will be overwritten — is never explained to the engineer receiving the setup guide, because the session terminates at "it works" and not at "here is what changes in how your team makes cluster changes." The secret management decision surfaces as a TODO: the initial setup focuses on the GitOps tooling configuration and does not provision a secret management mechanism; the first engineer who needs to commit a secret to the repository encounters the TODO or the README gap and implements whatever mechanism is locally available, which in the absence of a specified mechanism is the standard Kubernetes Secret YAML with base64-encoded values. The initial setup session result is "manifests in Git are applied to the cluster" — not "automated sync is enabled with self-healing every 3 minutes, meaning all emergency cluster changes must be committed to Git; secrets are managed via External Secrets Operator using the production/staging paths in AWS Secrets Manager with the ESO controller's IAM role arn:aws:iam::123456789:role/eso-reader, and no secret values may be committed to the repository in any encoding; the reconciliation failure alert is configured as a PagerDuty Low priority alert firing when any production application is in a non-synced state for more than 6 minutes." The second result requires treating each configuration choice as an operational commitment with documented implications for the team's incident response, secret handling, and deployment workflow, rather than as a technical setup step with a binary working/not-working completion criterion. The new CTO onboarding problem with GitOps is that the incoming technical leader inherits a GitOps configuration — automated sync enabled, self-healing enabled, reconciliation interval undocumented, secret management undocumented — without knowing whether the cluster has ever experienced the "manual fix that disappeared" incident, whether there are base64-encoded secrets in the git history that pre-date the current secret management convention, or whether the reconciliation failure alert goes to a Slack channel that is monitored during business hours only or to PagerDuty that pages at 3am.

The infrastructure migration session embeds GitOps scope decisions as resource-by-resource additions without documenting the cumulative scope or its boundaries. The session is triggered by the need to migrate an existing application from manual kubectl deployments to GitOps management: the engineer exports the existing Deployment, Service, and ConfigMap resources from the cluster using kubectl get -o yaml, cleans up the cluster-assigned fields (resourceVersion, uid, creationTimestamp), commits the cleaned manifests to the GitOps repository, creates an ArgoCD Application resource pointing at the directory, and confirms that ArgoCD shows the application as Synced. The session succeeds in migrating one application. It does not document which resources are now under GitOps management versus which remain managed by other systems (the Helm releases still managed by the previous CI pipeline, the cluster-level resources managed by Terraform, the CRDs managed by the cert-manager and external-secrets Helm charts that were installed separately). The boundary between GitOps-managed and non-GitOps-managed resources grows more complex with each migration session and is never explicitly documented. The consequence is a conflict: an engineer who notices that the cert-manager Certificate resource in the cluster does not match the Certificate resource committed to the GitOps repository (because cert-manager updated the Certificate's status fields and ArgoCD is comparing the full resource including status fields rather than only the spec fields) opens an ArgoCD Application that shows OutOfSync due to the status field difference, triggers a sync that overwrites the Certificate resource, and inadvertently disrupts the cert-manager renewal cycle. The infrastructure decision record specifies what infrastructure is managed by Terraform versus other mechanisms; the GitOps scope boundary is the infrastructure decision complement for Kubernetes resources, specifying which resource types and namespaces are under GitOps management and which are excluded from the scope, so that ArgoCD's sync policies are not applied to resources that have other controllers managing their state.

The incident response session embeds GitOps operational decisions as on-call discoveries that are never written back to documentation. The session is triggered by a production incident where the on-call engineer needs to make an emergency cluster change. The engineer applies the change directly with kubectl, observes that it is reverted (or does not observe this and is confused about why the incident symptoms are persisting), escalates to a second engineer, and together they discover the self-healing behavior and develop an ad hoc workaround — suspending automated sync, applying the fix, re-enabling sync, committing the fix to Git. The workaround works. The incident resolves. The post-mortem produces an action item: "document the emergency change procedure for GitOps environments." The action item may or may not be completed. If it is completed, it is added to a Confluence page or a README file in the GitOps repository that is not linked from the incident runbook for the specific alert type that fired. The next on-call engineer who encounters a GitOps self-healing reversion during an incident finds the same confusion, because the documentation exists but is not accessible in the context of the specific incident. The decisions never written down in a GitOps setup are the self-healing behavior that determines whether emergency kubectl changes persist or are silently reverted, the secret management mechanism that prevents database credentials from being committed to the Git history in a decodable encoding, and the rollback procedure that specifies git revert as the correct emergency action rather than kubectl rollout undo which self-healing will immediately overwrite.

The five sections of a GitOps decision record

The first section documents the GitOps tool selection and repository structure. The tool selection rationale must specify why ArgoCD, Flux, or an alternative was chosen and what was ruled out — ArgoCD provides a rich UI for visibility into application sync status and health, supports multi-cluster management from a single control plane, and has an active ecosystem of integrations (Argo Rollouts for progressive delivery, Argo Workflows for pipeline orchestration); Flux's design philosophy is CLI-first and UI-optional, favoring composable Kubernetes controllers (Kustomize controller, Helm controller, Image Automation controller) over a monolithic operator; the choice affects the operational model (ArgoCD requires running the ArgoCD API server as a dependency; Flux's controllers are lighter-weight but require CLI-based management in the absence of a UI), the multi-cluster support model (ArgoCD has native multi-cluster management from the ArgoCD UI; Flux requires running a Flux controller in each target cluster), and the image update automation model (Flux's Image Automation controller automatically commits image tag updates to the repository when a new image is detected in the container registry; ArgoCD does not have native image update automation and requires a third-party tool like Argo CD Image Updater or a CI pipeline step). The repository structure must specify whether the team uses a monorepo (single repository for all environments and applications) or a multi-repo (separate repositories per environment, per application, or per cluster); the Kustomize base-and-overlay model or the Helm-chart-per-environment model for managing environment-specific configuration; the directory structure for the repository including the naming conventions for environment directories, application directories, and shared resource directories; and the branch model (single main branch with environment directories versus environment-named branches that each represent the desired state of the corresponding environment). The tooling selection must specify the versions of ArgoCD, Flux, and associated CLI tools that are installed and the upgrade policy — ArgoCD and Flux both have active release cadences with breaking changes between minor versions, and the upgrade policy must specify whether version upgrades are applied as part of a quarterly maintenance window, as automated Dependabot or Renovate PRs, or manually after a major release review. The build system decision record specifies the CI/CD pipeline configuration; the GitOps tool selection determines the integration point between the CI/CD pipeline and the cluster deployment — in a GitOps model, the CI pipeline produces a container image and pushes it to the registry, then either commits an image tag update to the GitOps repository or triggers an ArgoCD sync, and the specific integration pattern (image tag commit versus sync trigger) must be specified to avoid parallel workflows where both the CI pipeline and ArgoCD apply changes to the cluster simultaneously.

The second section documents the sync model specification. The sync model must specify all six properties described in the structural properties section above: the reconciliation trigger and interval, the self-healing behavior with its operational implication stated explicitly, the manual change policy for incidents including the exact commands for suspending and re-enabling automated sync, the GitOps scope boundary with the resource types and namespaces under management, the drift detection alert configuration including the threshold and routing, and the resync and force-sync procedures for incident response. The sync model specification is the section of the decision record most likely to be consulted by an on-call engineer at 2am, and must therefore be written with the operational consumer in mind: procedural steps, not architectural descriptions; the exact commands, not a reference to the ArgoCD or Flux documentation; the expected output of each command that confirms the step succeeded; and the rollback step for each procedure in case the step has unintended consequences. The access control model decision record specifies the team's RBAC policy for Kubernetes cluster access; the GitOps model changes the relationship between Git repository access and cluster deployment permissions — in a GitOps environment, cluster deployment requires Git repository write access rather than Kubernetes RBAC cluster-admin; the access control decision record must be updated to reflect this change, specifying which engineers can push to which branches of the GitOps repository (which determines who can deploy to which environment), and what the emergency access procedure is for engineers who need to make direct cluster changes without going through the Git workflow.

The third section documents the secret management model. The secret management model must specify the canonical mechanism (Sealed Secrets, ESO, SOPS, or Vault Agent Injector) with the rationale for the choice, the scope of each mechanism if multiple mechanisms are used (e.g., ESO for database credentials and third-party API keys, Sealed Secrets for application configuration values that are sensitive but not credential-grade), the explicit prohibition on committing Kubernetes Secret YAML files with base64-encoded values to the repository, the rotation procedure for each secret type including the expected propagation time from secret store update to cluster Secret refresh, the cluster decryption key backup procedure and test schedule, the cross-environment isolation policy specifying that staging and production secrets must reference separate paths in the secret store with separate values, and the audit policy for who can read which secret store paths and who can read which Kubernetes Secrets in each namespace. The secret management model must also specify what to do with existing base64-encoded secrets that were committed to the repository before the secret management policy was established — the remediation process (identify all Kubernetes Secret YAML files in the repository history, rotate all committed credentials, remove the files from git history using git filter-repo, require all engineers to re-clone the repository, verify that no CI/CD cache or artifact store has a copy of the repository with the pre-remediation history). The security ADR and threat model decision record specifies the team's threat model; the GitOps secret management model must be evaluated against the threat model by considering the blast radius of each attack surface: compromise of the Git repository read access (who can read the repository, and what information is exposed if they can — with base64-encoded secrets committed, the blast radius is all committed credentials; with ESO references, the blast radius is the ESO controller's IAM role permissions), compromise of the cluster decryption key (Sealed Secrets private key or SOPS age key), and compromise of the ESO controller's IAM role or service account.

The fourth section documents the rollback and environment promotion model. The rollback model must specify the procedure for each rollback scenario: application image rollback (git revert the image tag commit, or argocd app rollback for immediate cluster-to-previous-state with subsequent git revert to resolve the drift), configuration rollback (git revert the configuration change), database migration rollback (the migration rollback procedure if the application version rollback is backward-compatible with the rolled-back schema, or the schema rollback procedure if it is not), and cluster-level resource rollback (restoring the previous version of a CRD or cluster-level configuration). The promotion model must specify the workflow for each environment transition: who initiates the promotion, what validation gates the staged deployment must pass before promotion (smoke tests, load tests, integration tests, manual review), what the PR review requirements are for the production promotion PR, what the expected time from staging deployment to production deployment is under the normal promotion workflow, and what the fast-path promotion procedure is for urgent security fixes that cannot wait for the standard review cycle. The rollback and promotion model must address the interaction between GitOps and database schema migrations: a GitOps deployment that includes a database migration job (Kubernetes Job that runs the migration before the application Deployment is updated) creates a promotion unit where the migration and the application version are applied together; rolling back the application version without rolling back the migration leaves the database in the migrated state that the previous application version may not tolerate; the decision record must specify whether the application version and the schema migration are designed as a backward-compatible unit (the previous application version works with the new schema), whether the database migration is run before or after the application deployment in the sync wave configuration, and what the rollback procedure is if the migration was applied but the application deployment failed. The distributed tracing decision record specifies the OTEL Collector DaemonSet configuration; OTEL Collector configuration changes managed by the GitOps repository — changes to the collector pipeline, the sampling processor policy, the exporter endpoints — are subject to the same GitOps rollback procedure as application changes, and the self-healing behavior that prevents direct kubectl changes to collector configuration must be noted in the OTEL Collector runbook alongside the GitOps emergency change procedure.

The fifth section documents the reconciliation failure model and observability. The observability model must specify the Prometheus metrics exported by ArgoCD or Flux that are scraped and stored in the team's metrics platform, the alert rules configured on those metrics, and the Grafana dashboards or equivalent visibility into the current sync status and health of each managed application. ArgoCD exports metrics including argocd_app_info (labels: name, namespace, dest_namespace, dest_server, project, repo, health_status, sync_status — useful for alert rules that fire when health_status != Healthy or sync_status != Synced), argocd_app_sync_total (sync attempt count with phase label), argocd_app_reconcile_count (reconciliation count per application), and argocd_cluster_api_resource_objects (the number of managed resources per cluster). Flux exports metrics including gotk_reconcile_duration_seconds (reconciliation duration histogram per Kustomization and HelmRelease), gotk_reconcile_condition (reconciliation result with a type label, Ready condition with a status label — False means failure), and gotk_suspend_status (whether a Kustomization is suspended). The alert rules must be specified by condition, threshold, severity, and routing: a production ArgoCD application with health_status="Degraded" for more than 5 minutes is a PagerDuty High alert; a production ArgoCD application with sync_status="OutOfSync" for more than 10 minutes is a PagerDuty Low alert; a staging application with either condition is a Slack alert to the #platform channel. The reconciliation failure runbook must specify the diagnosis steps for each error category: a manifest syntax error (kubectl apply --dry-run=client produces an error message identifying the problematic field); a resource conflict (ArgoCD's UI shows the specific resource that failed to apply with the Kubernetes API error); an image pull error (the pod enters an ImagePullBackOff state, which is visible in argocd_app_info health_status as "Degraded" and diagnosable via kubectl describe pod); a sync timeout (the sync operation exceeded the ArgoCD configurable sync timeout, defaulting to 20 minutes — common for database migration jobs that run during the sync phase). The access control review model must specify the cadence for reviewing who has write access to the GitOps repository, who has access to the GitOps controller's ServiceAccount credentials, and whether any changes to the repository access control model (new engineers joining, contractors given temporary access) require a corresponding review of the scope of access granted. The new CTO onboarding problem with the GitOps observability model is that the incoming technical leader inherits a GitOps setup where ArgoCD is running and applications are showing as Synced in the UI, but the reconciliation failure alert is configured to go to a Slack channel that is no longer monitored, the ArgoCD API server TLS certificate expired two months ago and was renewed by an engineer without documenting the renewal procedure, the ArgoCD admin password was set during the initial setup session and has never been rotated, and the ArgoCD controller's ServiceAccount has cluster-admin permissions that were granted during the setup session's "simplest configuration that works" approach and never scoped to the minimum required permissions.

The initial GitOps setup session that installed ArgoCD with automated sync and self-healing enabled without documenting that manual kubectl changes would be overwritten within 3 minutes, the infrastructure migration session that moved application manifests to the GitOps repository without specifying a secret management mechanism and leaving a README note of "secrets handling TBD" that the first engineer who needed to commit a secret interpreted as permission to use base64-encoded Kubernetes Secret YAML files, and the incident response session where the on-call engineer discovered the self-healing behavior by experiencing it and produced an ad hoc workaround that was never written into the runbook — each produced GitOps decisions whose operational consequences (a 2-hour incident extension caused by manual kubectl mitigations being silently overwritten by the reconciliation loop; eleven months of base64-encoded database credentials visible in git history to every engineer who cloned the repository; reconciliation failure alerts configured to a Slack channel with no after-hours monitoring, producing a window of hours between a bad manifest push and its detection) exceed what documenting the self-healing behavior with its operational implication as an incident response constraint, specifying the secret management mechanism before the first secret was committed to the repository, and configuring reconciliation failure alerts with the same escalation path as application health alerts would have cost to specify at the time the founding GitOps decisions were made. The decisions are in the AI sessions: the self-healing configuration applied as part of the recommended setup without noting that it converts all emergency cluster changes into a Git workflow requirement, the base64-encoding used in the first Kubernetes Secret YAML without the engineer knowing that base64 is not encryption and that the value is readable by any party with repository read access or git history access, the reconciliation interval left at the default without documenting it as the maximum duration for a manual kubectl fix before ArgoCD reverts it. WhyChose's open-source extractor surfaces these initial setup sessions, infrastructure migration sessions, and incident response sessions as structured decision records before an on-call engineer spends two hours investigating why their manual kubectl fix keeps disappearing, before a new engineer decodes the base64-encoded values in a secret.yaml file and discovers that the database password is visible in plain text and in the git history, and before a bad manifest push produces a reconciliation failure that goes undetected for six hours because the alert routing was left at the default Slack channel that nobody monitors at midnight. The decisions never written down in a GitOps setup are the self-healing behavior that determines whether emergency kubectl changes persist or are silently overwritten, the secret management policy that determines whether base64-encoded credentials accumulate in the git history, and the reconciliation failure alert that determines whether a broken manifest push is detected in 5 minutes or in the morning standup when someone notices that last night's deployment did not apply.