Buildkite Pipeline Patterns#

Buildkite splits CI/CD into two parts: a hosted web service that manages pipelines, builds, and the UI, and self-hosted agents that execute the actual work. This architecture means your code, secrets, and build artifacts never touch Buildkite’s infrastructure. The agents run on your machines – EC2 instances, Kubernetes pods, bare metal, laptops.

Why Teams Choose Buildkite#

The question usually comes up against Jenkins and GitHub Actions.

Over Jenkins: Buildkite eliminates the Jenkins controller as a single point of failure. There is no plugin compatibility hell, no Groovy DSL, no Java memory tuning. Agents are stateless binaries that poll for work. Scaling is adding more agents. Jenkins requires careful capacity planning of the controller itself.

Over GitHub Actions: Buildkite agents run on your infrastructure, so you control the hardware, network, and security boundary. GitHub Actions hosted runners are shared VMs with fixed specs. Self-hosted GHA runners exist but lack Buildkite’s queue routing, agent targeting, and priority system. Buildkite also handles monorepos and dynamic pipelines more naturally than GHA’s static matrix strategy.

Over both: Buildkite’s dynamic pipeline model is genuinely unique. Pipelines can generate other pipelines at runtime. This enables patterns that are awkward or impossible in static YAML systems.

Pipeline YAML#

Pipelines live in .buildkite/pipeline.yml:

steps:
  - label: ":go: Build"
    command: "go build -o myapp ./cmd/myapp"
    agents:
      queue: "default"

  - wait

  - label: ":test_tube: Unit Tests"
    command: "go test ./... -v -count=1"
    artifact_paths:
      - "coverage.out"

  - label: ":test_tube: Integration Tests"
    command: "go test ./... -tags=integration -v"
    agents:
      queue: "large"

  - wait

  - label: ":docker: Build Image"
    command: "docker build -t myapp:${BUILDKITE_COMMIT} ."
    agents:
      queue: "docker"

Steps run top to bottom. wait steps create synchronization barriers – everything above must pass before anything below starts. Steps between barriers run in parallel by default.

Dynamic Pipelines#

This is Buildkite’s defining feature. A step can upload additional pipeline YAML at runtime:

steps:
  - label: ":pipeline: Generate pipeline"
    command: ".buildkite/generate-pipeline.sh | buildkite-agent pipeline upload"

The script can be anything that outputs valid pipeline YAML. A Go program that inspects changed files, a Python script that queries an API, a shell script with conditional logic:

#!/bin/bash
# .buildkite/generate-pipeline.sh

CHANGED_FILES=$(git diff --name-only HEAD~1)

cat <<YAML
steps:
YAML

if echo "$CHANGED_FILES" | grep -q "^api/"; then
  cat <<YAML
  - label: ":go: Test API"
    command: "cd api && go test ./..."
    agents:
      queue: "default"
YAML
fi

if echo "$CHANGED_FILES" | grep -q "^frontend/"; then
  cat <<YAML
  - label: ":react: Test Frontend"
    command: "cd frontend && npm test"
    agents:
      queue: "node"
YAML
fi

if echo "$CHANGED_FILES" | grep -q "^terraform/"; then
  cat <<YAML
  - label: ":terraform: Plan"
    command: "cd terraform && terraform plan -out=tfplan"
    agents:
      queue: "infra"
  - block: ":rocket: Apply Terraform?"
  - label: ":terraform: Apply"
    command: "cd terraform && terraform apply tfplan"
    agents:
      queue: "infra"
YAML
fi

This pattern is impossible in GitHub Actions or CircleCI without significant workarounds. Dynamic pipelines make Buildkite the natural choice for large monorepos where you want to run only the tests affected by the change.

You can also chain multiple pipeline upload calls. Each upload appends steps to the current build. This enables multi-stage dynamic generation where early steps produce information that later generators consume.

Agents and Queues#

Agents are lightweight Go binaries. Install them anywhere:

# Install on Linux
bash -c "$(curl -sL https://raw.githubusercontent.com/buildkite/agent/main/install.sh)"

# Start with a specific queue
buildkite-agent start --token $BUILDKITE_AGENT_TOKEN --tags "queue=docker,os=linux,arch=arm64"

Tags are key-value pairs that describe the agent’s capabilities. Steps target agents using the agents block:

steps:
  - label: "Build ARM64 image"
    command: "docker buildx build --platform linux/arm64 -t myapp:arm64 ."
    agents:
      queue: "docker"
      arch: "arm64"

  - label: "GPU training"
    command: "python train.py"
    agents:
      queue: "gpu"
      gpu: "a100"

Queue routing lets you maintain heterogeneous agent pools. Small builds go to spot instances. Docker builds go to machines with Docker installed. GPU jobs go to GPU nodes. Security-sensitive builds go to agents in isolated networks.

For Kubernetes-based agents, Buildkite provides the agent-stack-k8s controller that dynamically creates pods as agents:

# Helm values for agent-stack-k8s
agentStackSecret: buildkite-agent-token
org: myorg
image: buildkite/agent:3
podSpec:
  containers:
    - name: agent
      resources:
        requests:
          cpu: "2"
          memory: "4Gi"

Plugins#

Plugins extend step behavior without requiring agents to have specific tools installed. They are Git repositories that Buildkite clones at runtime:

steps:
  - label: ":docker: Build and Push"
    plugins:
      - docker-compose#v5.2.0:
          build: app
          push:
            - app:myregistry/myapp:${BUILDKITE_COMMIT}

  - label: ":junit: Tests"
    command: "go test ./... -v 2>&1 | go-junit-report > results.xml"
    plugins:
      - test-collector#v2.0.0:
          files: "results.xml"
          format: "junit"

  - label: ":ecr: Push to ECR"
    plugins:
      - ecr#v2.9.0:
          login: true
          account-ids: "123456789012"
      - docker#v5.11.0:
          image: "123456789012.dkr.ecr.us-east-1.amazonaws.com/myapp"
          build: "."
          push: "${BUILDKITE_COMMIT}"

Pin plugin versions with the #vX.Y.Z suffix. Plugins run as hooks in the agent lifecycle – pre-command, command, post-command, pre-artifact, post-artifact. The docker-compose and docker plugins are the most widely used, handling container builds without any Docker commands in your pipeline YAML.

Artifact Management#

Upload artifacts during a step and download them in later steps:

steps:
  - label: ":go: Build"
    command: |
      go build -o myapp ./cmd/myapp
      buildkite-agent artifact upload myapp
    artifact_paths:
      - "coverage.out"

  - wait

  - label: ":rocket: Deploy"
    command: |
      buildkite-agent artifact download myapp .
      chmod +x myapp
      ./deploy.sh myapp

artifact_paths automatically uploads matching files after the step completes. buildkite-agent artifact upload and download give you explicit control within scripts. Artifacts are stored in Buildkite’s managed storage or your own S3 bucket if configured.

For large artifacts, configure an S3 artifact backend:

# In the agent configuration
BUILDKITE_ARTIFACT_UPLOAD_DESTINATION=s3://my-buildkite-artifacts/${BUILDKITE_PIPELINE_SLUG}/${BUILDKITE_BUILD_NUMBER}

This keeps large build outputs off Buildkite’s infrastructure and gives you full control over retention and access policies.

Parallel Builds#

Buildkite’s parallelism key splits a single step into N parallel jobs:

steps:
  - label: ":test_tube: Tests"
    command: ".buildkite/run-tests.sh"
    parallelism: 8
    agents:
      queue: "large"

Inside the step, BUILDKITE_PARALLEL_JOB (0-indexed) and BUILDKITE_PARALLEL_JOB_COUNT identify the current shard:

#!/bin/bash
# .buildkite/run-tests.sh

ALL_TESTS=$(go list ./...)
SHARD=$(echo "$ALL_TESTS" | awk "NR % $BUILDKITE_PARALLEL_JOB_COUNT == $BUILDKITE_PARALLEL_JOB")
go test $SHARD -v

Combine parallelism with the Test Analytics plugin for timing-based splitting across runs, similar to CircleCI’s test splitting but using Buildkite’s Test Analytics dashboard for visualization.

Block Steps and Manual Gates#

Block steps pause the build until someone clicks “Unblock” in the UI:

steps:
  - label: ":test_tube: Tests"
    command: "go test ./..."

  - block: ":rocket: Deploy to production?"
    branches: "main"
    fields:
      - text: "Reason for deploy"
        key: "deploy-reason"
        required: true
      - select: "Target region"
        key: "region"
        options:
          - label: "US East"
            value: "us-east-1"
          - label: "EU West"
            value: "eu-west-1"

  - label: ":rocket: Deploy"
    command: "deploy.sh --region $(buildkite-agent meta-data get region)"
    branches: "main"

Block steps can include form fields. The values are accessible via buildkite-agent meta-data get. This lets you parameterize manual deployments – choosing a target region, entering a reason, or selecting a version – without separate tooling.

Environment and Secrets#

Environment variables can be set per-step, per-pipeline (in the Buildkite UI), or via agent environment hooks:

# /etc/buildkite-agent/hooks/environment
export AWS_DEFAULT_REGION="us-east-1"
export DOCKER_REGISTRY="123456789012.dkr.ecr.us-east-1.amazonaws.com"

# Source secrets from AWS Secrets Manager, Vault, etc.
eval $(aws secretsmanager get-secret-value --secret-id buildkite/prod --query SecretString --output text | jq -r 'to_entries | .[] | "export \(.key)=\(.value)"')

Agent hooks run on the agent machine before each job. This keeps secrets out of pipeline YAML entirely. The hook approach is more flexible than CircleCI contexts or GitHub Actions secrets because you control the retrieval mechanism – pull from Vault, AWS Secrets Manager, or any other source.

Common Mistakes#

Not using dynamic pipelines for monorepos. Running all tests on every commit wastes compute. Use a generator step that inspects changed files and only uploads relevant test steps.
Hardcoding agent queues. Use a default queue and only specify specialized queues when the step genuinely needs specific capabilities (Docker, GPU, architecture).
Ignoring agent lifecycle hooks. Pre-checkout and post-checkout hooks handle agent setup (Docker login, environment config) without cluttering pipeline YAML.
Not pinning plugin versions. Unversioned plugins pull the latest code, which can introduce breaking changes. Always use plugin-name#vX.Y.Z.
Using wait when depends_on would be better. wait blocks the entire pipeline. depends_on creates targeted dependencies between specific steps, allowing unrelated steps to proceed.