Autoscaling Azure Pipelines agents with KEDA
May 27, 2021
With the addition of Azure Piplines support in KEDA, it is now possible to autoscale your Azure Pipelines agents based on the agent pool queue length.
Self-hosted Azure Pipelines agents are the perfect workload for this scaler. By autoscaling the agents you can create a scalable CI/CD environment.
💡 The number of concurrent pipelines you can run is limited by your parallel jobs.
KEDA will autoscale to the maximum defined in the ScaledObject and does not limit itself to the parallel jobs count defined for the Azure DevOps organization.
What are Azure Pipelines self-hosted agents?
Azure Pipelines jobs can run on different kinds of agents (docs). But if you want full control, you are going to have to use self-hosted agents. Agents are able to run on Linux, macOS or Windows machines and can be packaged in to a container.
When running self-hosted agents on Kubernetes, there is no out-of-the-box support for autoscaling. However, with KEDA v2.3 you can now autoscale your self-hosted agents on Kubernetes based on the amount of pending jobs in your agent pool.
You can run the agents as a
Deployment or a
Job in Kubernetes and scale them accordingly with a
ScaledObject or a
Deploying a self-hosted agent on Kubernetes as a Deployment
Create the container image
To create a basic Azure Pipelines agent image you can follow the instructions from the official docs.
Deploy on Kubernetes
You can easily deploy the agent as a Kubernetes deployment by using this Kubernetes manifest:
apiVersion: v1 kind: Secret metadata: name: azdevops data: AZP_TOKEN: <base64 encoded PAT> --- apiVersion: apps/v1 kind: Deployment metadata: name: azdevops-deployment labels: app: azdevops-agent spec: replicas: 1 selector: matchLabels: app: azdevops-agent template: metadata: labels: app: azdevops-agent spec: containers: - name: azdevops-agent image: <azdevops-image> env: - name: AZP_URL value: "https://dev.azure.com/<organization>" - name: AZP_POOL value: "<agent pool name>" - name: AZP_TOKEN valueFrom: secretKeyRef: name: azdevops key: AZP_TOKEN volumeMounts: - mountPath: /var/run/docker.sock name: docker-volume volumes: - name: docker-volume hostPath: path: /var/run/docker.sock
Autoscaling with KEDA
After the deployment is created you need to create the
ScaledObject in order for KEDA to start scaling the deployment.
To scale based on the queue length of an Azure Pipelines agent pool, you can use the
azure-pipelines trigger as of KEDA v2.3.
apiVersion: v1 kind: Secret metadata: name: pipeline-auth data: personalAccessToken: <base64 encoded PAT> --- apiVersion: keda.sh/v1alpha1 kind: TriggerAuthentication metadata: name: pipeline-trigger-auth spec: secretTargetRef: - parameter: personalAccessToken name: pipeline-auth key: personalAccessToken --- apiVersion: keda.sh/v1alpha1 kind: ScaledObject metadata: name: azure-pipelines-scaledobject spec: scaleTargetRef: name: azdevops-deployment minReplicaCount: 1 maxReplicaCount: 5 triggers: - type: azure-pipelines metadata: poolID: "1" organizationURLFromEnv: "AZP_URL" authenticationRef: name: pipeline-trigger-auth
1, so there will be one agent for each job.
⚠ The Azure Pipelines scaler supports scaling to zero but you need at least one agent registered in the agent pool in order for new jobs to be scheduled on the pool.
Running Azure Pipelines jobs
After deploying the agent and the KEDA
ScaledObject it is time to see the autoscaling in action.
First, check the current pods running in the deployment:
$ kubectl get pods NAME READY STATUS RESTARTS AGE azdevops-deployment-5854bbbf84-r86qv 1/1 Running 0 75s
Now let’s queue some builds.
As a result, you see that KEDA starts scaling out the pods to meet the pending jobs:
$ kubectl get pods NAME READY STATUS RESTARTS AGE azdevops-deployment-5854bbbf84-4gfbx 1/1 Running 0 36s azdevops-deployment-5854bbbf84-r86qv 1/1 Running 0 12m azdevops-deployment-5854bbbf84-tm47k 1/1 Running 0 36s
And they appear on Azure Pipelines as well:
Run the self-hosted agent as a Job
When running your agents as a deployment you have no control on which pod gets killed when scaling down. (see KEDA docs)
If you run your agents as a
Job, KEDA will start a Kubernetes job for each job that is in the agent pool queue. The agents will accept one job when they are started and terminate afterwards.
Since an agent is always created for every pipeline job, you can achieve fully isolated build environments by using Kubernetes jobs.
The following manifest is an example of a
ScaledJob combined with the Azure Pipelines agent.
You have to use a modified image for this where the agent terminates itself after running a job. (docs)
apiVersion: keda.sh/v1alpha1 kind: ScaledJob metadata: name: azdevops-scaledjob spec: jobTargetRef: template: spec: containers: - name: azdevops-agent-job image: <azdevops-image> imagePullPolicy: Always env: - name: AZP_URL value: "<organizationUrl>" - name: AZP_TOKEN value: "<token>" - name: AZP_POOL value: "<agentpool>" volumeMounts: - mountPath: /var/run/docker.sock name: docker-volume volumes: - name: docker-volume hostPath: path: /var/run/docker.sock pollingInterval: 30 successfulJobsHistoryLimit: 5 failedJobsHistoryLimit: 5 maxReplicaCount: 10 scalingStrategy: strategy: "default" triggers: - type: azure-pipelines metadata: poolID: "1" organizationURLFromEnv: "AZP_URL" personalAccessTokenFromEnv: "AZP_TOKEN"
You cannot queue an Azure Pipelines job on an empty agent pool because Azure Pipelines cannot validate if the pool matches the requirements for the job.
When you try to do this you will encounter the following error:
##[error]No agent found in pool keda-demo which satisfies the specified demands: Agent.Version -gtVersion 2.163.1
You can, however, use a workaround to register an agent as a placeholder which allows you to queue jobs on an agent pool that has no agents that are online.
Make sure you don’t execute any cleanup code in your container to unregister the agent when removing it to keep the placeholder agent registered in the agent pool.
ScaledJobs in action
To allow scaling to zero and create agents on-demand, a template agent was created as a placeholder to be able to queue jobs.
Now, let’s queue some pipelines:
KEDA will create a Kubernetes job for each pending job in the queue for the specified agent pool.
$ kubectl get pods NAME READY STATUS RESTARTS AGE azdevops-scaledjob-2hshf-mp5jl 1/1 Running 0 24s azdevops-scaledjob-5gzr5-p8625 1/1 Running 0 24s azdevops-scaledjob-mmlzc-rw5gm 1/1 Running 0 24s
Once that happens, you will see the agents are starting to process the pending Azure Pipelines jobs:
Once that happens, you will see the jobs in the Azure Pipelines’ agent pool:
ScaledJob is the preferred way to autoscale your Azure Pipelines agents if you have long running jobs.
The other option is using a
deployment and leveraging the container lifecycle. (docs)
Recent postsHelp shape the future of KEDA with our survey 📝
Announcing KEDA v2.9 🎉
HTTP add-on is looking for contributors by end of November
Announcing KEDA v2.8 🎉
How Zapier uses KEDA
Introducing PredictKube - an AI-based predictive autoscaler for KEDA made by Dysnix
How CAST AI uses KEDA for Kubernetes autoscaling
Announcing KEDA HTTP Add-on v0.1.0
Autoscaling Azure Pipelines agents with KEDA
Why Alibaba Cloud uses KEDA for application autoscaling
Migrating our container images to GitHub Container Registry
Announcing KEDA 2.0 - Taking app autoscaling to the next level
Give KEDA 2.0 (Beta) a test drive
Kubernetes Event-driven Autoscaling (KEDA) is now an official CNCF Sandbox project 🎉