Managing Secrets in Ansible with Vault and HashiCorp Vault

The credential in the playbook problem

Almost every Ansible repository we review has at least one of these: a database password in group_vars, an SSH private key committed “temporarily”, a cloud access key in a template, or a token echoed in a CI log. The playbook works, so nobody revisits it. Then someone adds a contractor to the repository, or the repository is mirrored, or an employee leaves with a clone, and the credential is effectively public.

Secrets management for Ansible has two parts: where the secret lives, and how it reaches the task that needs it. This page covers how we handle both within our Ansible configuration management service.

What good looks like

• No credential exists in plaintext in any repository, including history.
• Secrets are separated by environment, and a lower environment cannot read production.
• Access is least-privilege, and you can say who can read each secret.
• Secrets do not appear in job output, logs or error messages.
• A rotation is a routine procedure, not an emergency.

How it works in practice

Option 1: Ansible Vault

Ansible Vault encrypts data using AES-256 with a key derived from a password. You can encrypt entire files or single values. Single values are friendlier in code review because the variable name stays readable:

ansible-vault encrypt_string --vault-id prod@prompt 'S3cr3t-value' --name 'db_password'

The output is pasted into group_vars/production/vault.yml:

db_password: !vault |
  $ANSIBLE_VAULT;1.2;AES256;prod
  6338343262343733...

A useful convention is to keep an unencrypted file that references the encrypted one, so grep still finds variable names:

# group_vars/production/vars.yml
app_db_password: "{{ vault_app_db_password }}"

Run with the matching password source:

ansible-playbook site.yml --vault-id prod@~/.vault_pass_prod.sh

The password source can be a script that reads from your password manager, which keeps the password out of the repository and shell history. Vault IDs separate environments, so staging and production use different passwords.

Option 2: HashiCorp Vault

When you need audit logs, rotation or short-lived credentials, Ansible reads from HashiCorp Vault at run time using the community.hashi_vault collection:

- name: Read database credentials
  ansible.builtin.set_fact:
    db_creds: "{{ lookup('community.hashi_vault.hashi_vault',
                 'secret=kv/data/app/production/db',
                 url='https://vault.example.internal:8200',
                 auth_method='approle') }}"
  no_log: true

Authentication uses AppRole, a cloud identity or JWT rather than a long-lived root token. The role is bound to a policy that allows reading one path. With the database secrets engine, Vault can even mint a unique short-lived database account per run, so there is nothing long-lived to leak. Ansible lookups run on the controller, which affects network design: the controller must reach Vault, and the target hosts need not.

Option 3: Cloud secrets managers

If you are all-in on one cloud, AWS Secrets Manager or Azure Key Vault, read through the matching collection lookups, may be simpler than operating Vault yourself. You trade features like dynamic secrets for less to maintain.

Preventing leaks

Three habits matter more than the tool choice:

1. no_log: true on any task that touches a secret. Without it, a failure prints the full task arguments.
2. Pre-commit and CI scanning with a tool such as gitleaks, so a plaintext secret is rejected before merge. Our pipeline for playbook linting includes this check.
3. Never use secrets in command arguments, because they show up in process listings on the target.

Running in AWX

AWX has first-class credential types, including vault credentials and external secret lookup plugins for HashiCorp Vault and cloud managers. Credentials are stored encrypted, and users who can run a job template can use a credential without ever seeing its value. That is a major improvement over passwords on engineers’ laptops.

Decisions and trade-offs

When not to adopt HashiCorp Vault. If you have a handful of secrets and a team of three, running a highly available Vault cluster adds operational burden that outweighs the benefit. Ansible Vault with disciplined practice is perfectly respectable.

Common failure modes

• One password for everything. A single vault password makes every environment equally exposed. Use vault IDs.
• Vault password in the repository. Defeats the purpose. Hold it in a password manager, CI secret store or AWX credential.
• Secrets in facts. set_fact values can end up in fact caches or callback output. Use no_log and avoid caching them.
• Forgotten history. Removing a secret from the latest commit does not remove it from the past. We rotate first, then clean history where practical.
• Break-glass lockout. Only one person holds the vault password and is on holiday. We document escrow arrangements.

Delivery and handover

We deliver a secrets inventory (what exists, where, who uses it), the chosen vault configuration, CI checks for plaintext secrets, rotation runbooks and a short training session. Where hardening the hosts themselves is in scope, the server baseline role removes credentials left on disk. For wider secret handling in clusters, see our notes on sealed secrets.

What to do when a secret has already leaked

If a credential has been committed, assume it is compromised. Rotate it first, then remove it from the working tree, and only then decide whether rewriting history is worth the disruption. Rewriting does not help if the repository was ever cloned or mirrored, so rotation is the real fix. We write this order into the runbook so nobody wastes the first hour cleaning history while the credential is still valid.

How to tell it is working

• A repository scan finds no plaintext credentials, including in history.
• A staging engineer genuinely cannot decrypt production values.
• Rotating a credential takes minutes and touches one place.
• Job logs are clean even when a task fails.
• You can say who read a given secret last month, if you chose a runtime store.

To find out what is currently exposed in your repositories, contact us.

References

• Protecting sensitive data with Ansible vault, Ansible Community Documentation
• community.hashi_vault.hashi_vault lookup, Retrieve secrets from HashiCorp’s Vault, Ansible Community Documentation
• Vault product documentation, HashiCorp Developer