The secret sprawl problem
Secrets spread because pipelines need them and the fastest place to put one is the nearest text box. A database password in a variable group. A storage key in a YAML file “just for now”. A personal access token in a shell script. Each copy is a separate thing to rotate and a separate way to leak. When an engineer leaves, nobody knows what they could see.
Key Vault gives secrets a single home with access control, audit logs and versioning. This page describes how we integrate it into delivery as part of our Azure DevOps consulting. If your workloads run on Kubernetes, the Kubernetes side of secrets is also covered in our guide to ConfigMaps and Secrets.
What good looks like
- No credential exists in Git history, YAML or a plain-text variable.
- Each pipeline identity can read only the secrets its job needs.
- Secrets are fetched at the moment of use and are not stored as artifacts.
- The service connection to Azure has no long-lived client secret.
- Access to the vault is logged and unusual reads raise an alert.
- Rotation is routine, not an emergency.
How it works in practice
Vault design and access model
We usually use one vault per environment and sensitivity class, for example separate vaults for production and non-production. This keeps blast radius small and permissions simple. Soft delete is on by default in Azure, and we enable purge protection so a deleted secret can be recovered. Access uses Azure RBAC: the pipeline identity gets the Key Vault Secrets User role, which allows reading secret values, scoped to the vault or a specific secret. Humans who administer the vault get separate, audited roles.
Identity: no stored client secret
The service connection that talks to Azure should use workload identity federation, where Azure DevOps presents a short-lived token that Entra ID trusts through a federated credential. There is no client secret to expire, leak or rotate. Existing connections that use a secret can often be converted. The mechanics are explained in our page on service connections.
Reading secrets in a pipeline
The Azure Key Vault task fetches secrets for the run:
steps:
- task: AzureKeyVault@2
displayName: Fetch secrets
inputs:
azureSubscription: sc-prod-readonly
KeyVaultName: kv-shop-prod
SecretsFilter: "db-connection,payments-api-key"
RunAsPreJob: false
- script: ./deploy.sh
displayName: Deploy
env:
DB_CONNECTION: $(db-connection)
PAYMENTS_API_KEY: $(payments-api-key)
Two details matter. SecretsFilter lists the secrets needed rather than downloading everything. And secrets are passed to a script through env, because secret variables are not automatically exposed to scripts as environment variables. Alternatively, a variable group linked to the vault does the same without the task, which is convenient but gives less control over which secrets each pipeline sees.
Secrets inside Kubernetes
For applications on AKS, we avoid copying secrets through the pipeline at all. The Secrets Store CSI driver mounts Key Vault secrets into pods using a workload identity, so the cluster fetches them directly. This keeps secrets out of manifests and Helm values. Our AKS deployment integration page shows where that fits.
Network restrictions
For sensitive vaults we disable public network access and use private endpoints. Microsoft-hosted agents cannot reach such a vault, so builds that need it run on self-hosted agents in a network that can. We plan this together, because adding network restriction after the fact is a common cause of sudden pipeline failures.
Rotation and cleanup
Clean-up comes first. We search repositories and history with a scanner such as gitleaks, treat anything found as compromised, and rotate it. Then we set rotation rules: managed identities need none, database passwords rotate on a schedule, and third-party keys rotate by the provider’s mechanism. Key Vault can emit events when a secret nears expiry, which lets automation or alerts trigger rotation.
Decisions and trade-offs
Variable group link or Key Vault task? The linked group is simpler for teams. The task allows per-pipeline filtering and per-stage vaults. We choose the task for production.
One vault or many? Fewer vaults mean fewer things to manage, but permissions become broad. Splitting by environment is a sensible minimum.
Key Vault or an external secrets tool? If you are Azure-centric, Key Vault is natural. If you run across clouds, HashiCorp Vault or a cloud-neutral operator may suit better, at the cost of running it.
When not to bother. A prototype with no real data does not need a vault. The moment production credentials appear, it does.
Worked example: replacing a pasted database password
A typical starting point is a variable group holding a SQL connection string that three pipelines share, entered by an engineer two years ago. We first create a new credential in the database rather than reuse the exposed one, store it in the production vault, and grant the deployment identity the secrets reader role on that single secret. Each pipeline is then changed to fetch it with the Key Vault task, tested in a non-production stage first, and only afterwards is the old variable group entry deleted and the old database login disabled. The order matters: a new secret in place and proven working, then the old one revoked. Reversing it causes outages, and skipping the revocation leaves the original exposure open.
Common failure modes
- Secrets echoed by scripts. Enabling debug output prints variables.
- Over-broad access. Giving the pipeline access to the entire vault.
- Forgotten copies. Old variable group entries stay after migration.
- Expired client secrets. A service connection secret expires on a Friday and releases stop. Federation avoids this.
Delivery and handover
You receive configured vaults, role assignments as Terraform, pipeline examples, an inventory of removed and rotated secrets, and a one-page runbook for adding a secret. We test the failure case as well: a pipeline without access must be denied.
How to tell it is working
A repository scan comes back clean, you can answer “who can read production secrets” with a short list, rotation happens without outages, and a departing engineer does not trigger a scramble. To discuss your setup, contact us.