Detecting and Fixing Terraform Drift in ClickOps Environments

What drift actually costs you

Drift is not a theoretical hygiene issue. It bites in specific ways. An engineer widens a security group during an incident and forgets about it. Months later someone runs a Terraform apply, and either the emergency fix is silently reverted and the outage returns, or the plan is so noisy that nobody dares apply it. Worse, a resource created by hand never appears in code at all, so nobody knows it exists until the bill arrives.

Terraform drift detection is how you turn those surprises into a routine queue of small decisions. It is the practical sequel to moving off ClickOps, and a core part of our Terraform and infrastructure as code service.

What good looks like

• Every stack is checked automatically on a schedule, not only when someone happens to plan.
• The signal is clean: a failing check means something real changed.
• Each finding has an owner, a classification and a resolution date.
• Remediation goes through the same pull request pipeline as any other change.
• Drift trends towards zero over time because the causes are being addressed.

Two kinds of drift

It helps to separate them, because the remedies differ.

Drift in managed resources. Terraform created the resource, and someone changed it afterwards. The plan shows an in-place update that would undo their change.

Unmanaged resources. Nothing in state refers to the resource at all. A plain plan will never show these, because Terraform only looks at what it knows. Finding them needs inventory tools such as AWS Config, resource explorers or an Azure resource graph query, compared against state. Bringing them in is the job of importing existing resources.

How it works in practice

A scheduled read-only check

The simplest dependable detector is a plan with a detailed exit code. In a scheduled workflow, per stack:

terraform init -input=false
terraform plan -detailed-exitcode -lock=false -input=false -out=drift.tfplan
status=$?

if [ "$status" -eq 2 ]; then
  terraform show -no-color drift.tfplan > drift-report.txt
  echo "Drift detected"   # post drift-report.txt to the team channel
fi

Exit code 0 means clean, 1 means an error, and 2 means there are changes. Using -lock=false for a read-only job avoids blocking real work, and the job role should only have read permissions. Remember a plan that shows changes could also mean unapplied code on main, so the report states which commit it ran against.

Reviewing what changed in the real world

When you want to see only what changed outside Terraform, without proposing any fixes, use the refresh-only mode:

terraform plan -refresh-only

This lists real-world differences and, if you apply it, records them into state without changing infrastructure. We use it deliberately, never as a reflex, because accepting drift into state without updating the code just moves the problem.

Triage: revert, codify or accept

Ignored attributes should be few and explained. A long ignore_changes list is a smell that the code and the platform are fighting each other.

Decisions and trade-offs

How often to check. Nightly is a good default for production and weekly for lower environments. More frequent checks cost runner time and produce alerts nobody can act on.

Dedicated drift tools or plain plans. Plain scheduled plans are simple and free. Managed options such as HCP Terraform health checks add dashboards and notifications. Open source drift tools have come and gone, so we check maintenance status before recommending one.

Auto-remediate or alert. Automatically re-applying to revert drift sounds attractive but can undo a legitimate emergency fix. We alert first, and only automate reversion for narrow, well-understood cases such as a tag policy.

When not to chase every difference. In an estate with extensive legacy ClickOps, trying to clear thousands of findings at once stalls the work. We prioritise by risk: identity, network and data first, cosmetic settings later.

Making findings actionable

A report nobody reads is not detection. Each finding should name the stack, the resource address, the attribute that changed and the commit the check ran against. We route reports to the channel of the team that owns the stack rather than one shared channel, and we open a ticket automatically for anything touching identity, networking or data stores. Cloud audit logs then answer the follow-up question of who changed it and when, so the conversation can move straight to whether the change was right.

A useful habit is a short weekly review of open findings, ten minutes long, where each is classified as revert, codify or accept. Findings that sit unclassified for more than a couple of cycles are a sign the process is not owned.

Failure modes we avoid

• Alert fatigue. Noise from autoscaling or provider quirks trains people to ignore the channel. We tune before we alert.
• Silent refresh. Running a bare refresh and moving on hides real changes.
• Drift checks with write access. Detection jobs get read-only roles.
• Fixing symptoms only. If engineers keep editing the console because the pipeline is slow, the pipeline needs fixing. See how we set up fast, safe applies in Atlantis and Terraform Cloud pipelines.

Delivery and handover

You receive the scheduled checks, an initial drift report with classified findings, remediation pull requests for the highest-risk items, and a one-page triage guide for your team. Cloud audit logs (CloudTrail or the Azure Activity Log) are used to trace who made each change, which makes the conversation about causes constructive rather than accusatory.

How to tell it is working

The number of open drift findings falls each month. Console changes in production become rare and are followed by a pull request within a day. Plans before a release are boring. If you suspect your cloud has already moved away from your code, contact us for a baseline.

References

• Manage resource drift, Terraform, HashiCorp Developer
• terraform plan command reference, Terraform, HashiCorp Developer