Why drift matters more than most teams admit
Every cluster drifts. A late-night fix with kubectl edit, a replica count raised by hand, a ConfigMap patched to unblock a deploy: each one is reasonable at the time and invisible afterwards. Months later the staging cluster behaves differently from production, nobody knows why, and a rebuild from git would produce something that does not match what customers use.
Detecting drift is the main practical benefit of GitOps. Deciding what to do about it is where design matters. This is part of our GitOps, Helm, ArgoCD and Flux service, and it follows from installing the controller.
What good looks like
- Every environment has an explicit, documented sync policy.
- Drift is visible quickly and routed to the people who own the workload.
- Legitimate runtime changes (autoscaling, operator-managed fields) do not raise false alarms.
- Emergency manual changes are possible, tracked and reconciled back into git rather than lost.
- No one is surprised by a controller reverting or deleting something.
The three levers
Sync: automatic or manual
With automated sync, a merged change applies without further action. With manual sync, the controller reports the difference and waits for someone to trigger it. Neither is universally right.
| Environment | Typical policy | Reason |
|---|---|---|
| Dev | Automated, prune, self-heal | Fast feedback, low risk, keep clean |
| Staging | Automated, prune, self-heal | Should mirror production behaviour |
| Production | Automated with sync windows, or manual | Controls timing and blast radius |
Self-heal
Self-heal reverts live changes that disagree with git. It enforces the principle that git is the only way to change things. The cost is that it also reverts a human fix applied during an outage. We pair it with a clear break-glass procedure: pause auto-sync on that application, fix the problem, then commit the same change to git and resume. This is covered further in access control.
Prune
Prune removes cluster objects that no longer exist in git. Without it, deleted manifests leave orphaned resources running forever. With it, a mistaken deletion removes a workload. We enable it, but add protection for stateful resources.
How it works in practice
ArgoCD sync policy
spec:
syncPolicy:
automated:
prune: true
selfHeal: true
syncOptions:
- CreateNamespace=true
- PruneLast=true
retry:
limit: 3
backoff:
duration: 10s
factor: 2
maxDuration: 3m
ignoreDifferences:
- group: apps
kind: Deployment
jsonPointers:
- /spec/replicas
The ignoreDifferences block stops ArgoCD from treating an HPA’s replica changes as drift. PruneLast deletes removed objects after the new ones are healthy, which reduces downtime during restructures. Individual resources can be annotated with argocd.argoproj.io/sync-options: Prune=false to protect them.
Flux reconciliation
Flux continuously corrects drift for resources it manages. A Kustomization sets prune: true and an interval. To pause for an emergency, suspend the object, then resume afterwards:
flux suspend kustomization apps -n flux-system
# fix the issue, commit the change to git
flux resume kustomization apps -n flux-system
Resources can be exempted from reconciliation with an annotation, but we use that sparingly. Choices depend on the tool in your installation.
Managed fields and admission controllers
Mutating webhooks, service meshes and operators add or modify fields after apply. The controller sees the live object differ from git and reports out-of-sync forever. The fix is surgical ignore rules for those exact fields, or server-side apply, which tracks field ownership properly. We prefer server-side apply where the tool supports it because it reflects who owns what.
Detecting drift that GitOps cannot see
A controller only watches what it manages. Resources created outside git and not labelled as managed are invisible to it. To catch these, we add periodic audits listing objects without GitOps ownership labels, and policy engines such as Kyverno or Gatekeeper that deny creating certain objects outside the controller’s identity. Combined with RBAC, this closes most gaps.
Decisions and trade-offs
Strict enforcement or visibility only. Strict self-heal gives consistency but frustrates engineers during incidents. Visibility with alerts gives flexibility but depends on people reconciling afterwards. Many teams start with visibility, then tighten.
Short or long reconciliation intervals. Faster polling detects drift sooner but loads the git server and API server. Webhooks make git changes apply immediately, while the interval remains the safety net.
When not to automate. Databases, long-running migrations and anything with irreversible effects often deserve a manual gate, even in environments that auto-sync everything else.
A worked example: an incident edit
Imagine a payment service degrading at night. The on-call engineer scales a Deployment from four replicas to ten with kubectl scale to buy time. With self-heal on, the controller sets it back to four within moments, and the engineer has to learn that the hard way mid-incident. With self-heal off and only alerts on, the change stays, the application shows as out of sync, and a notification tells the team to reconcile.
Neither outcome is wrong. What matters is that the team chose it. Our usual answer is self-heal in development and staging, and in production either a documented emergency route (a temporary override through git, merged quickly) or an alert-only policy with an owner who resolves the difference by the next working day. The principle is that every difference between git and the cluster ends in one of two ways: git is updated to match, or the cluster is reverted to match git.
Fields that should never count as drift
Some differences are expected and must be excluded so the signal stays clean. Replica counts managed by an HPA, fields written by an operator, and certificates rotated by a controller are common examples. ArgoCD handles this with ignore differences rules, and Flux with server-side apply ownership. Each rule is documented with a reason, because an undocumented ignore rule is how real drift hides.
Common failure modes
- Perpetual OutOfSync. Caused by unignored mutated fields. Fixed with targeted rules.
- Sync loops. Two tools managing the same field fight each other. One owner per field.
- Surprise prune. A refactor deletes manifests and takes live resources with them. We preview with a diff and use prune protection.
- Ignoring too much. Broad ignore rules hide real drift. Keep them narrow.
- Alert fatigue. Everything out-of-sync pages someone. See notifications for routing sensibly.
Delivery and handover
You get documented policies per environment, ignore rules with explanations, a drift dashboard, alerts, and a break-glass runbook. We rehearse an emergency change with your on-call engineers so the process is familiar before it is needed.
How to tell it is working
Out-of-sync is rare and meaningful when it happens. Rebuilding an environment from git produces what was running. Nobody discovers a hand-edited resource months after the fact. To review your current policies, book a consultation.