Azure DevOps Pipelines: The Complete Guide for Teams

Azure DevOps Pipelines has evolved significantly. YAML pipelines replaced Classic (GUI) pipelines as the recommended approach, offering code review, version history, and multi-stage deployments with environment approvals. Here is the complete guide.

Classic vs YAML pipelines

Migrate to YAML, Microsoft is not removing Classic but all new features go to YAML only.

Multi-stage pipeline anatomy

# azure-pipelines.yml
trigger:
  branches:
    include: [main]
  paths:
    exclude: [docs/**, '*.md']

pr:
  branches:
    include: [main, develop]

variables:
  - group: shared-variables  # variable group from Azure DevOps Library
  - name: imageRepository
    value: 'my-app'
  - name: containerRegistry
    value: 'myregistry.azurecr.io'

pool:
  vmImage: ubuntu-latest

stages:
  - stage: Build
    displayName: Build and Test
    jobs:
      - job: BuildJob
        steps:
          - task: NodeTool@0
            inputs:
              versionSpec: '20.x'
          - script: npm ci
          - script: npm run lint
          - script: npm test
            displayName: Run tests
          - task: PublishTestResults@2
            condition: always()
            inputs:
              testResultsFormat: JUnit
              testResultsFiles: '**/test-results.xml'
          - task: Docker@2
            displayName: Build and push image
            inputs:
              command: buildAndPush
              repository: $(imageRepository)
              dockerfile: Dockerfile
              containerRegistry: $(containerRegistryServiceConnection)
              tags: |
                $(Build.BuildId)
                latest

  - stage: DeployStaging
    displayName: Deploy to Staging
    dependsOn: Build
    condition: and(succeeded(), eq(variables['Build.SourceBranch'], 'refs/heads/main'))
    jobs:
      - deployment: DeployStaging
        environment: staging  # requires environment approval
        strategy:
          runOnce:
            deploy:
              steps:
                - task: KubernetesManifest@1
                  inputs:
                    action: deploy
                    namespace: staging
                    manifests: k8s/*.yaml
                    containers: $(containerRegistry)/$(imageRepository):$(Build.BuildId)

  - stage: DeployProduction
    displayName: Deploy to Production
    dependsOn: DeployStaging
    condition: succeeded()
    jobs:
      - deployment: DeployProduction
        environment: production  # manual approval gate
        strategy:
          runOnce:
            deploy:
              steps:
                - task: KubernetesManifest@1
                  inputs:
                    action: deploy
                    namespace: production
                    manifests: k8s/*.yaml
                    containers: $(containerRegistry)/$(imageRepository):$(Build.BuildId)

Environments and approvals

Environments are the Azure DevOps mechanism for deployment gates.

1. Go to Pipelines → Environments → New environment
2. Add Approvals and checks:
   • Approvals: require one or more named approvers before deployment
   • Business hours check: only deploy during working hours
   • Branch control: only deploy from main
   • Required template: enforce a specific pipeline template

# Environment usage in deployment job
- deployment: DeployProd
  environment:
    name: production
    resourceType: Kubernetes
    resource: prod-cluster  # specific k8s resource

Variable groups and Azure Key Vault

Variable groups store shared variables. Link them to Azure Key Vault to avoid storing secrets in Azure DevOps.

# Reference a variable group
variables:
  - group: production-secrets  # contains DB_PASSWORD, API_KEY etc.

# In a step
steps:
  - script: echo "Deploying to $(DEPLOY_ENV)"
    env:
      DB_PASS: $(DB_PASSWORD)  # from variable group / Key Vault

Linking Key Vault to a variable group (do this in the Library UI):

• Library → Variable groups → Link secrets from Azure Key Vault
• Select Key Vault, add specific secrets
• Pipeline service principal needs Key Vault Secrets User role

Service connections

Service connections are authenticated links to external services (Azure, Docker registries, Kubernetes, GitHub).

# Azure subscription service connection
- task: AzureCLI@2
  inputs:
    azureSubscription: 'my-azure-service-connection'
    scriptType: bash
    scriptLocation: inlineScript
    inlineScript: |
      az aks get-credentials --resource-group prod-rg --name prod-cluster

# Docker registry service connection
- task: Docker@2
  inputs:
    containerRegistry: 'my-acr-service-connection'
    repository: 'my-app'
    command: build
    tags: $(Build.BuildId)

Pipeline templates for reuse

Templates let you define jobs/stages/steps once and reuse them.

# templates/deploy-stage.yml
parameters:
  - name: environment
    type: string
  - name: namespace
    type: string
  - name: imageTag
    type: string

stages:
  - stage: Deploy_${{ parameters.environment }}
    jobs:
      - deployment: Deploy
        environment: ${{ parameters.environment }}
        strategy:
          runOnce:
            deploy:
              steps:
                - task: HelmDeploy@0
                  inputs:
                    namespace: ${{ parameters.namespace }}
                    releaseName: my-app
                    chartPath: helm/my-app
                    overrideValues: image.tag=${{ parameters.imageTag }}

Using the template:

# azure-pipelines.yml
stages:
  - stage: Build
    jobs:
      - job: Build
        steps: [...]

  - template: templates/deploy-stage.yml
    parameters:
      environment: staging
      namespace: staging
      imageTag: $(Build.BuildId)

  - template: templates/deploy-stage.yml
    parameters:
      environment: production
      namespace: production
      imageTag: $(Build.BuildId)

Self-hosted agents

When GitHub-hosted agents lack network access or specific tools:

# Download agent from Organization Settings > Agent pools > New agent
mkdir myagent && cd myagent
tar zxvf ~/Downloads/vsts-agent-linux-x64-3.x.x.tar.gz

# Configure
./config.sh \
  --url https://dev.azure.com/your-org \
  --auth PAT \
  --token YOUR_PAT \
  --pool default \
  --agent my-agent-name \
  --work _work \
  --run

# Install as service
sudo ./svc.sh install && sudo ./svc.sh start

Reference in pipeline:

pool:
  name: default  # your agent pool name
  demands:
    - Agent.OS -equals Linux

Pipeline caching

- task: Cache@2
  inputs:
    key: '"npm" | "$(Agent.OS)" | package-lock.json'
    restoreKeys: |
      "npm" | "$(Agent.OS)"
    path: $(npm_config_cache)
  displayName: Cache npm packages
- script: npm ci

Deployment strategies on AKS

# Blue-green with Kubernetes manifests
- task: KubernetesManifest@1
  inputs:
    action: deploy
    strategy: blue-green
    trafficSplitMethod: pod  # or 'smi' for Service Mesh Interface
    percentage: 20           # start with 20% on green
    namespace: production
    manifests: k8s/deployment.yaml
    containers: myregistry.azurecr.io/my-app:$(Build.BuildId)

# Promote green (after validation)
- task: KubernetesManifest@1
  inputs:
    action: promote
    strategy: blue-green
    namespace: production
    manifests: k8s/deployment.yaml

Key takeaways

• YAML pipelines are the only way forward, Classic pipelines get no new features.
• Use Environments with approval checks for all production deployments; skip this only for non-critical test environments.
• Link variable groups to Azure Key Vault, never store secrets as plain variable group values.
• Templates eliminate copy-paste between pipelines; maintain them in a central templates/ directory.
• Self-hosted agents are unavoidable for private network deployments to AKS; use the default pool or create a named pool per project.
• Pipeline caching for npm, pip, and Maven can cut CI time by 40-60%.