Why teams run their own agents
Microsoft-hosted agents are the right default: clean, patched, and free of operational work. Teams move to self-hosted pools for concrete reasons. A build must reach a database or artifact store that has no public endpoint. A job needs a licensed compiler, a GPU, or a large dependency cache that would take minutes to download on every run. Or the monthly minutes consumed have made owning capacity cheaper than renting it.
The risk is that a self-hosted pool becomes a pet: a few long-lived VMs nobody patches, with credentials baked in. This page describes how we build pools that stay disposable, as part of our Azure DevOps consulting work.
What good looks like
- Agents are created from a versioned image and replaced, never repaired by hand.
- Capacity follows demand: it grows when jobs queue and shrinks when idle.
- Agents authenticate with managed identities and short-lived tokens, not stored passwords.
- Network access is the minimum needed: private endpoints to the resources builds must reach, and no more.
- Queue time, job failures and image age are visible on a dashboard.
How it works in practice
Option 1: VM Scale Set agents
Azure DevOps has first-class support for scale set agent pools. You create a scale set, point a pool at it, and Azure DevOps manages scaling the instances up and down based on pending jobs. Instances can be configured to be used once and then deleted, which gives every job a clean machine.
The scale set runs from a custom image built with Packer, which installs your toolchain and the agent software. A pipeline rebuilds the image on a schedule so patches flow through. Provisioning with Terraform keeps the whole stack repeatable. A trimmed sketch of the scale set resource:
resource "azurerm_linux_virtual_machine_scale_set" "agents" {
name = "vmss-ado-agents"
resource_group_name = azurerm_resource_group.ci.name
location = azurerm_resource_group.ci.location
sku = "Standard_D4s_v5"
instances = 0
overprovision = false
upgrade_mode = "Manual"
source_image_id = var.agent_image_id
admin_username = "azagent"
identity {
type = "SystemAssigned"
}
# Azure DevOps manages instance count, so Terraform must not fight it.
lifecycle {
ignore_changes = [instances, tags]
}
}
Note the ignore_changes on instances. Azure DevOps owns the instance count, and if Terraform tries to reset it you get confusing scaling behaviour. Disabling overprovisioning and using manual upgrade mode are the settings Microsoft documents for scale set agent pools. Always check the current documentation for your region and SKU.
Option 2: Agents on AKS
If your team already runs Kubernetes, agents can run as pods. KEDA can scale the agent deployment based on the Azure DevOps queue length, and each pod can be a short-lived job runner. The attraction is shared cluster tooling, container-based toolchains and quick start times. The costs are Docker-in-Kubernetes complexity and a larger security surface, because build code now runs inside your cluster. We usually isolate agents on a dedicated node pool and namespace with network policy, and keep them away from production workloads. Our AKS deployment integration page covers the deploy side.
Identity and network
Agents need to talk to Azure DevOps outbound, and to whatever private resources your builds use. We place them in a dedicated subnet, restrict egress with a firewall or network security groups, and use private endpoints for registries, storage and Key Vault. A managed identity on the scale set or a workload identity on AKS removes stored cloud credentials. Details on credentials are in service connections.
Building the agent image
The image is the heart of a healthy pool. We define it in Packer, starting from a current operating system image and installing only what your pipelines use: language runtimes, the Azure CLI, Docker or a rootless alternative, and any private certificate authorities your network requires. A pipeline builds the image, runs a smoke test that executes a sample job against it, and publishes a versioned image to a Compute Gallery.
Keeping the toolchain small is a deliberate choice. Every extra package is more to patch and more to break. Where a tool is only needed by one team, we prefer running it in a container job, so the base image stays lean and the team controls its own versions.
Rolling out a new image
Because instances are disposable, rolling out is straightforward. Point the scale set at the new image version, let existing jobs finish, and new instances start from the updated image. If the new version misbehaves, rolling back is a change to one reference. We keep the previous version available for exactly this reason.
Cost control
Self-hosted capacity costs money whether or not it is used. Scale sets can scale to zero idle instances, with a small standby count if start-up time matters more than cost. We choose VM sizes by measuring real builds, not guessing, and use spot capacity only for work that tolerates interruption. Budget alerts on the resource group catch a runaway pipeline early. Our Terraform work makes all of this reviewable code rather than portal settings.
Decisions and trade-offs
Ephemeral or persistent workers? Ephemeral workers are cleaner and safer but lose local caches. We compensate with the pipeline Cache@2 task or a shared package feed, and only keep persistent workers when a measured build time justifies it.
One pool or several? Separate pools for production deployment and for untrusted pull request builds stop a malicious change from reaching powerful credentials.
Build your own or buy a larger hosted tier? Parallel job capacity on Microsoft-hosted agents may solve a queue problem faster than building infrastructure. We check this first.
When not to do this. If your only reason is “we want control”, the maintenance overhead often outweighs the benefit. Choose self-hosted when a specific requirement demands it.
Common failure modes
- Pets, not cattle. Hand-installed tools drift and builds become unreproducible.
- Over-privileged identities. An agent with subscription-wide Owner is an incident waiting to happen.
- No scale ceiling. A runaway pipeline can scale the pool to its maximum and run up cost. We set sensible limits and budget alerts.
- Stale images. Agents run for months with old runtimes. We alert on image age.
Delivery and handover
You receive Terraform for the pool, the image pipeline, a patching schedule, dashboards and a runbook. Because builds and deployments are only as good as their runners, this work pairs naturally with multi-stage pipeline design.
How to tell it is working
Jobs start within seconds or a minute rather than queueing, idle spend is low, no agent is older than your patch window, and nobody has logged into a build machine to fix it. To review your current setup, book a consultation.