Terraform Maintenance and State Management Without the Fear

Why state is the part that hurts

Writing Terraform is the easy half. The hard half is the state file, which records what Terraform believes exists. Everything else depends on it: plans, applies, imports and destroys. Teams get into trouble in predictable ways. State sits in a local file on a laptop. Two engineers apply at once. One giant state holds every environment and takes ten minutes to plan. A provider upgrade quietly proposes to replace a database.

Our Terraform maintenance is part of the wider Managed DevOps service. For building a codebase from scratch or converting manual setups, see Terraform and infrastructure as code and our walkthrough of moving from ClickOps to Terraform on AWS.

What good looks like

• Infrastructure as code is the only way production changes: console edits are the exception and are reconciled.
• State is stored remotely, encrypted, versioned and locked.
• State is split along sensible boundaries, such as network, data, platform and per-service stacks.
• Plans are reviewed in pull requests and applied by automation.
• Versions of Terraform, providers and modules are pinned and updated regularly.

How it works in practice

The backend

On AWS the standard backend is an S3 bucket with versioning and encryption, plus native S3 locking in recent Terraform releases or a DynamoDB table in older setups. Azure uses a storage account container and Google Cloud uses a GCS bucket. An example:

terraform {
  required_version = "~> 1.9"

  backend "s3" {
    bucket       = "acme-tfstate-prod"
    key          = "platform/network/terraform.tfstate"
    region       = "eu-west-1"
    encrypt      = true
    use_lockfile = true
  }

  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "~> 5.0"
    }
  }
}

Confirm the options your Terraform version supports; locking behaviour has changed across releases. Bucket access is restricted to the CI role and a small set of administrators, because state can contain sensitive values in plain text.

Splitting state safely

A single state for everything is slow and risky: one bad apply can touch unrelated resources. We split by change frequency and ownership. Network and identity change rarely and affect everything, so they sit in their own state. Application stacks change often and sit separately, reading shared outputs through terraform_remote_state or, better, through data sources and parameter stores that do not couple states tightly.

Moving resources between states uses terraform state mv or moved blocks, then a plan that must show zero changes. We never split by deleting and recreating.

Plan, review, apply

Every change begins as a pull request that runs fmt, validate, a linter such as tflint, a security scan such as Checkov, and plan. The plan output is posted to the pull request. After approval, the pipeline applies it using short-lived credentials through OIDC, as described in our CI/CD ownership page. Tools like Atlantis or standard GitHub Actions both work; the key is that the plan a reviewer approved is the plan that runs.

Drift

Drift happens when someone changes a resource in the console during an incident. A nightly read-only plan with -detailed-exitcode flags it:

terraform plan -detailed-exitcode -lock=false -input=false
echo $?   # 0 means clean, 2 means changes are present (drift or unapplied code)

Each finding is classified. Emergency changes get codified. Accidental ones get reverted. This closes the loop with incident response, where console edits are most likely.

Cost and policy checks in the pull request

Because every change flows through a plan, the pull request is a natural place for guardrails. Alongside formatting and validation we add policy checks that fail the build on risky patterns: public storage buckets, security groups open to the world on management ports, unencrypted volumes, or databases without backups. Checkov or an equivalent policy engine runs on the plan output, and exceptions are explicit comments in code with a reason, so the exception is reviewable rather than invisible.

Cost estimation tools such as Infracost can show the monthly impact of a change in the pull request. That gives the reviewer a simple question to ask: is this price what we intended? It also feeds the cost optimisation reviews, because a spend increase can be traced to the exact commit that caused it.

Importing existing resources

Most estates have some resources created by hand. We bring them under management with import blocks, which let the import itself be reviewed in a plan rather than run as a one-off command:

import {
  to = aws_s3_bucket.assets
  id = "acme-assets-prod"
}

resource "aws_s3_bucket" "assets" {
  bucket = "acme-assets-prod"
}

After the import is applied, the plan must be empty. Any difference means the code does not yet describe reality, and we adjust the code, not the resource.

Upgrades

Upgrades are where maintenance earns its keep. Provider major versions can rename arguments or change defaults. Our process:

1. Pin versions, then read release notes for breaking changes.
2. Upgrade in a non-production state first.
3. Review the plan for unexpected replacements, especially on stateful resources.
4. Use lifecycle { prevent_destroy = true } on databases and similar critical resources as a seatbelt.
5. Apply to production in a low-risk window.

Decisions and trade-offs

Terragrunt or plain Terraform? Terragrunt reduces repetition across many environments but adds a layer to learn. Small estates do fine with plain modules and a clear directory layout.

Workspaces or directories? Workspaces share code but hide the environment inside state selection, which makes mistakes easy. We prefer separate directories or stacks per environment for production safety.

Modules: public, internal or none. Community modules save time but pull in their opinions and upgrade burden. Thin internal modules for repeated patterns are usually the best balance.

When not to buy this. If your estate is a handful of resources that rarely change, a documented repository and careful humans can be enough. Maintenance pays off as the number of environments and engineers grows.

Common failure modes

• Secrets in state. Sensitive values leak to anyone with bucket access. We restrict access and move secrets to a secrets manager.
• Stale locks. A crashed pipeline leaves a lock. We document safe unlocking.
• Version skew. Different Terraform binaries across laptops cause diffs. We pin through required_version and the pipeline image.
• Unreviewed imports. Importing without writing matching code leaves a plan that wants to change things.

Delivery and handover

The code stays in your repositories and the state in your cloud account. We document stack layout, upgrade cadence and break-glass steps in runbooks, and report drift and upgrade status monthly.

How to tell it is working

Plans are quick and boring, drift findings trend toward zero, nobody applies from a laptop, upgrades happen on schedule, and a new environment can be created from code without surprises. To review your current state layout, contact us.

References

• State, Terraform language documentation, HashiCorp
• Remote state, Terraform language documentation, HashiCorp
• Backend type s3, Terraform language documentation, HashiCorp
• State locking, Terraform language documentation, HashiCorp