Why the last mile to the cluster goes wrong
Building an image is straightforward. Getting it safely into a Kubernetes cluster is where pipelines turn fragile. We regularly see a cluster-admin kubeconfig stored as a pipeline secret, kubectl apply run against whatever is in the current context, deployments reported as successful before pods are ready, and production changes made by whoever could edit the YAML.
This page describes how we connect Azure DevOps to AKS as part of our Azure DevOps consulting. For the cluster itself, see our Kubernetes consulting service.
What good looks like
- Images are built once, scanned, and tagged immutably with the commit or build identifier.
- Pipelines deploy with identities that can only touch the namespaces they own.
- A deployment is “done” only when the rollout is healthy, not when the command exits.
- Environments are promoted in order, with production behind approval.
- Rollback is a button or a command that has been practised.
How it works in practice
The image path
The build stage runs tests, builds the image, scans it, and pushes it to Azure Container Registry. Tags use the commit hash or build ID rather than latest, so a deployment is reproducible and a rollback is unambiguous. The registry is attached to the cluster so nodes can pull without stored passwords. More on registries is in our Azure Artifacts and ACR page.
Access: service connections and RBAC
Azure DevOps reaches the cluster through a Kubernetes service connection. Rather than a cluster-admin credential, we use a service account in the target namespace, bound with a Role limited to the resources the app needs. For production we use a separate connection with tighter scope and approval checks on it. Managed identity and workload identity options reduce stored secrets further, and are discussed in service connections.
Deploying with Helm
A deployment job targeting a Kubernetes environment runs Helm with environment-specific values:
- stage: DeployStaging
dependsOn: Build
jobs:
- deployment: Staging
environment: staging.shop
strategy:
runOnce:
deploy:
steps:
- checkout: self
- task: HelmDeploy@1
inputs:
connectionType: Kubernetes Service Connection
kubernetesServiceEndpoint: aks-staging
namespace: shop
command: upgrade
chartType: FilePath
chartPath: charts/shop
releaseName: shop
valueFile: charts/shop/values-staging.yaml
arguments: --atomic --timeout 5m --set image.tag=$(Build.BuildId)
The --atomic flag makes Helm roll back automatically if the release does not become ready within the timeout. The environment name uses the environment.resource form, so history is recorded against the namespace. For plain manifests, the KubernetesManifest@1 task supports the same pattern, including canary and blue-green strategies.
Verification and strategies
After a rollout, we check kubectl rollout status equivalents, run smoke tests against the new version, and fail the stage if either fails. Readiness probes must be meaningful, otherwise Kubernetes reports healthy pods that cannot serve traffic. For riskier services we use canary or blue-green deployment, with traffic shifted gradually through an ingress or service mesh.
Where GitOps fits
Instead of the pipeline pushing to the cluster, it can update a Git repository that Argo CD or Flux reconciles. This gives continuous drift correction and a full audit trail of cluster state. The trade-off is another system to run and a slightly longer path from merge to running pods. Our GitOps with Helm, Argo CD and Flux service covers it in depth, and a worked example is in the Argo CD tutorial.
Handling configuration and secrets
Applications need settings that differ per environment, and some are sensitive. We keep non-sensitive settings in Helm values files stored beside the chart, reviewed through pull requests like code. Sensitive values come from Azure Key Vault, surfaced to pods through the Secrets Store CSI driver or an operator, so they never appear in pipeline logs or Git. The pipeline passes references, not secrets.
Database migrations and ordering
Schema changes are the usual reason a clean rollout turns into an outage. We run migrations as a separate, explicit step, often a Kubernetes Job or a pipeline stage that completes before the new version receives traffic, and design migrations to be backward compatible so the previous version keeps working while the rollout proceeds. That makes a rollback safe: the old code still understands the new schema.
Pod disruption and capacity during rollouts
A rolling update needs spare capacity and sensible limits. We review replica counts, resource requests, pod disruption budgets and surge settings, so a deployment during busy hours does not starve the cluster or drop traffic. These are small settings that decide whether a release is invisible to users or not.
Decisions and trade-offs
Push or pull deployment? Push is quick to build and easy to understand. Pull gives stronger guarantees on drift and who can change the cluster. Small teams often start with push and add GitOps as environments multiply.
Helm, Kustomize or raw YAML? Helm for packaged apps with many variations, Kustomize for overlays on plain manifests, raw YAML for tiny services.
Shared or separate clusters per environment? Separate clusters isolate production but cost more. Namespaces on a shared non-production cluster are a common compromise.
When not to do this. If you run on App Service or Container Apps, adding AKS just for deployment sophistication is unnecessary. Choose the platform first.
Common failure modes
- Cluster-admin credentials in a library variable. We replace these with scoped identities.
latesttags. Nobody knows what is running. Immutable tags fix this.- Success without readiness. Pipelines go green while pods crash. Rollout checks fix it.
- Config baked into images. Environment settings belong in values, config maps and secrets from Key Vault.
Delivery and handover
You receive pipeline templates for AKS deployment, configured service connections and RBAC, environment definitions, a rollback runbook and a walkthrough for your engineers. We test the full path, including a deliberate failed release, before signing off.
How to tell it is working
Deployments are boring, failed rollouts revert themselves, anyone can say exactly which commit is running in each environment, and no pipeline holds more access than its job needs. To review your deployment path, book a consultation.