Delivery

It is 09:40 on on-sale day. A bug in inventory lets a seat hold expire two minutes early, and the fix is a one-line change. In a team without a delivery pipeline, the fix waits: someone must build it, someone must copy the jar, someone must restart servers one by one, and nobody wants to do that twenty minutes before the biggest traffic peak of the year. In a team with a good pipeline, the fix is merged, built, scanned, rolled out to 5 % of the traffic, checked automatically against the SLOs, and promoted — or rolled back — without a human touching a server.

This page builds that pipeline in four layers: the infrastructure is code, every change is integrated and tested continuously, the cluster pulls its desired state from Git, and each release is exposed gradually and judged by metrics.

flowchart LR
    pr["Pull request<br/>inventory fix"] --> ci["CI: build · test · scan<br/>SBOM · sign · push image"]
    ci --> cfg["Config repo:<br/>image digest bumped"]
    cfg --> argo["Argo CD:<br/>sync cluster to Git"]
    argo --> canary["Canary 5 % → 25 % → 50 %"]
    canary --> an{"Metrics analysis<br/>error rate, p99"}
    an -->|"healthy"| full["100 % — promoted"]
    an -->|"degraded"| rb["Automatic rollback"]

Infrastructure as code

The BoxOffice production environment has a VPC, subnets, a Kubernetes cluster, a managed PostgreSQL, a managed Redis, a Kafka cluster, buckets, DNS records and IAM roles — a few hundred resources. Created by clicking in a console, they cannot be reviewed, reproduced or rebuilt after a disaster, and staging silently drifts away from production. Infrastructure as code (IaC) describes those resources in versioned files that a tool applies.

Terraform (HashiCorp) is the most widely used IaC tool. In 2023 HashiCorp changed its license to the Business Source License; the community fork OpenTofu, hosted by the Linux Foundation, keeps the open-source MPL 2.0 license and the same language (HCL) and CLI workflow¹. The snippet below works in both:

variable "env"            { type = string }                 # "staging" | "prod"
variable "db_instance"    { type = string }                 # e.g. "db.r6g.large"
variable "private_subnets" { type = list(string) }

resource "aws_db_subnet_group" "boxoffice" {
  name       = "boxoffice-${var.env}"
  subnet_ids = var.private_subnets
}

resource "aws_db_instance" "inventory" {
  identifier                  = "boxoffice-inventory-${var.env}"
  engine                      = "postgres"
  engine_version              = "17"
  instance_class              = var.db_instance
  allocated_storage           = 100
  multi_az                    = var.env == "prod"           # standby in another AZ
  db_subnet_group_name        = aws_db_subnet_group.boxoffice.name
  username                    = "boxoffice"
  manage_master_user_password = true                        # password kept in Secrets Manager
  backup_retention_period     = 7
  deletion_protection         = var.env == "prod"
  tags = { project = "boxoffice", env = var.env, service = "inventory" }
}

resource "aws_s3_bucket" "tickets" {
  bucket = "boxoffice-tickets-${var.env}"
  tags   = { project = "boxoffice", env = var.env }
}

output "inventory_endpoint" { value = aws_db_instance.inventory.endpoint }

terraform {
  backend "s3" {                                  # shared, locked state
    bucket       = "boxoffice-tfstate"
    key          = "prod/terraform.tfstate"
    region       = "us-east-1"
    use_lockfile = true
  }
}

module "data" {
  source          = "../../modules/data"
  env             = "prod"
  db_instance     = "db.r6g.large"
  private_subnets = module.network.private_subnet_ids
}

tofu init          # download providers, configure the state backend
tofu plan          # show what would change — reviewed in the pull request
tofu apply         # apply exactly the reviewed plan

The tool keeps a state file mapping each resource in the code to a real resource ID. It must be remote and locked (two engineers applying at once corrupt it) and is sensitive (it may contain secrets). When someone changes a resource by hand, the next plan shows drift — a scheduled plan in CI that alerts on drift keeps the console honest.

Continuous integration

Continuous integration (CI) means every change is merged to the main branch frequently and verified automatically by building and testing it. Continuous delivery (CD) means every verified change is releasable at any time; continuous deployment goes one step further and releases every change automatically².

BoxOffice's CI runs on GitHub Actions. On a pull request it builds and tests; on main it also publishes a signed image and proposes the new version to the configuration repository:

name: inventory
on:
  pull_request: { paths: [ "inventory/**" ] }
  push:         { branches: [ main ], paths: [ "inventory/**" ] }

permissions:
  contents: read
  packages: write        # push to GitHub Container Registry
  id-token: write        # keyless signing with Sigstore
  attestations: write

jobs:
  build:
    runs-on: ubuntu-latest
    outputs:
      digest: ${{ steps.push.outputs.digest }}
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-java@v4
        with: { distribution: temurin, java-version: "21", cache: maven }
      - name: Unit and integration tests (Testcontainers)
        run: mvn -B -f inventory/pom.xml verify
      - uses: docker/setup-buildx-action@v3
      - uses: docker/login-action@v3
        with: { registry: ghcr.io, username: "${{ github.actor }}", password: "${{ secrets.GITHUB_TOKEN }}" }
      - name: Build and push image
        id: push
        uses: docker/build-push-action@v6
        with:
          context: inventory
          push: ${{ github.ref == 'refs/heads/main' }}
          tags: ghcr.io/boxoffice/inventory:${{ github.sha }}
          sbom: true                      # SBOM attached to the image
          provenance: mode=max            # SLSA provenance
      - name: Scan for critical vulnerabilities
        uses: aquasecurity/trivy-action@0.28.0
        with:
          image-ref: ghcr.io/boxoffice/inventory:${{ github.sha }}
          severity: CRITICAL
          exit-code: "1"
      - name: Sign image (keyless)
        if: github.ref == 'refs/heads/main'
        run: cosign sign --yes ghcr.io/boxoffice/inventory@${{ steps.push.outputs.digest }}

  propose-release:
    needs: build
    if: github.ref == 'refs/heads/main'
    runs-on: ubuntu-latest
    steps:
      - name: Bump the image digest in the config repo (staging overlay)
        run: |
          git clone https://x-access-token:${{ secrets.CONFIG_REPO_TOKEN }}@github.com/<your-org>/boxoffice-config.git
          cd boxoffice-config/apps/inventory/overlays/staging
          kustomize edit set image inventory=ghcr.io/boxoffice/inventory@${{ needs.build.outputs.digest }}
          git commit -am "inventory: ${{ github.sha }}" && git push

Notice what the pipeline does not do: it never runs kubectl apply against production and holds no cluster credentials. It only writes to Git. The cluster pulls.

image:
  name: image (${{ matrix.service }})
  needs: [test, performance]
  if: github.event_name == 'push' && github.ref == 'refs/heads/main'
    - name: Gate - no CRITICAL vulnerability that has a fix
      uses: aquasecurity/trivy-action@ed142fd0673e97e23eac54620cfb913e5ce36c25 # v0.36.0
      with:
        image-ref: boxoffice-${{ matrix.service }}:ci
        format: table
        severity: CRITICAL
        ignore-unfixed: true
        exit-code: "1"
    - name: Attest build provenance (who built it, from which commit, how)
      uses: actions/attest-build-provenance@4d101475d8b20a2381f78447822ac1eab6504dd8 # v4.2.2
      with:
        subject-name: ${{ env.IMAGE }}
        subject-digest: ${{ steps.push.outputs.digest }}
        push-to-registry: true

- name: Set the catalog image of the gitops overlay to this commit
  env:
    SHA: ${{ github.event.workflow_run.head_sha }}
  run: |
    OWNER="${GITHUB_REPOSITORY_OWNER,,}"
    cd k8s/overlays/gitops
    kustomize edit set image "boxoffice/catalog=ghcr.io/${OWNER}/boxoffice-catalog:sha-${SHA}"
    git diff --stat
- name: Commit and push (a commit made with GITHUB_TOKEN does not trigger CI again)
  env:
    SHA: ${{ github.event.workflow_run.head_sha }}
  run: |
    git config user.name "github-actions[bot]"
    git config user.email "41898282+github-actions[bot]@users.noreply.github.com"
    git add k8s/overlays/gitops/kustomization.yaml
    git diff --cached --quiet && { echo "already deployed"; exit 0; }
    git commit -m "deploy: catalog sha-${SHA::7}"
    git push

GitOps with Argo CD

GitOps applies the Kubernetes control loop to delivery itself. The OpenGitOps principles are: the desired state is declarative, versioned and immutable (in Git), pulled automatically by an agent in the cluster, and continuously reconciled⁴.

sequenceDiagram
    participant Dev as Engineer
    participant Git as Config repo (Git)
    participant Argo as Argo CD (in the cluster)
    participant K8s as Kubernetes API
    Dev->>Git: PR "inventory → sha256:ab12…" (reviewed, merged)
    Argo->>Git: poll / webhook: new commit
    Argo->>K8s: diff desired vs live → apply
    K8s-->>Argo: resources healthy
    Note over Argo,K8s: someone runs kubectl edit by hand…
    Argo->>K8s: drift detected → reverted to Git (self-heal)

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: inventory-prod
  namespace: argocd
spec:
  project: boxoffice
  source:
    repoURL: https://github.com/<your-org>/boxoffice-config.git
    targetRevision: main
    path: apps/inventory/overlays/prod
  destination:
    server: https://kubernetes.default.svc
    namespace: boxoffice
  syncPolicy:
    automated:
      prune: true        # delete resources removed from Git
      selfHeal: true     # revert manual changes
    syncOptions: [ CreateNamespace=true ]

Promotion between environments becomes a pull request that copies the digest from overlays/staging to overlays/prod — reviewed, auditable and trivially reversible.

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: boxoffice
  namespace: argocd
spec:
  project: default
  source:
    repoURL: https://github.com/your-user/boxoffice.git
    targetRevision: main
    path: k8s/overlays/gitops
  destination:
    server: https://kubernetes.default.svc
    namespace: boxoffice
  syncPolicy:
    automated:
      prune: true                    # delete what was removed from Git
      selfHeal: true                 # revert manual changes in the cluster
    syncOptions:
      - CreateNamespace=true

Deployment strategies

Deploying (putting new code on servers) and releasing (exposing it to users) are different steps. The strategies below differ in how they separate them:

Rolling updates are the Kubernetes default and are enough for most changes. For risky changes on the purchase path — inventory, order, payment — BoxOffice uses canaries, and for product features feature flags (with a vendor-neutral API such as OpenFeature), which decouple deploy from release entirely: the code ships on Tuesday, the feature turns on for 1 % of fans on Thursday.

Canary with automated metrics analysis

A canary is only as good as the decision to promote it. Looking at a dashboard and "feeling fine" is not a decision. Argo Rollouts replaces a Deployment with a Rollout that shifts traffic in steps and runs an analysis at each step, querying the metrics backend and aborting automatically when the canary is worse than its thresholds⁵.

apiVersion: argoproj.io/v1alpha1
kind: Rollout
metadata: { name: inventory }
spec:
  replicas: 10
  selector: { matchLabels: { app: inventory } }
  template:
    metadata: { labels: { app: inventory } }
    spec:
      containers:
        - name: inventory
          image: ghcr.io/boxoffice/inventory@sha256:ab12c3d4e5f6
  strategy:
    canary:
      canaryService: inventory-canary
      stableService: inventory-stable
      trafficRouting:
        plugins:
          argoproj-labs/gatewayAPI: { httpRoute: inventory, namespace: boxoffice }
      analysis:                                  # runs in the background from step 1
        templates: [ { templateName: inventory-slo } ]
        startingStep: 1
      steps:
        - setWeight: 5
        - pause: { duration: 10m }
        - setWeight: 25
        - pause: { duration: 10m }
        - setWeight: 50
        - pause: { duration: 10m }

apiVersion: argoproj.io/v1alpha1
kind: AnalysisTemplate
metadata: { name: inventory-slo }
spec:
  metrics:
    - name: error-rate
      interval: 1m
      failureLimit: 2                       # two bad minutes → abort and roll back
      successCondition: result[0] < 0.005   # < 0.5 % of requests fail
      provider:
        prometheus:
          address: http://prometheus.monitoring:9090
          query: |
            sum(rate(http_server_requests_seconds_count{service="inventory-canary",status=~"5.."}[2m]))
            /
            sum(rate(http_server_requests_seconds_count{service="inventory-canary"}[2m]))
    - name: p99-latency
      interval: 1m
      failureLimit: 2
      successCondition: result[0] < 0.300   # p99 below 300 ms (the hold SLO)
      provider:
        prometheus:
          address: http://prometheus.monitoring:9090
          query: |
            histogram_quantile(0.99, sum by (le) (
              rate(http_server_requests_seconds_bucket{service="inventory-canary"}[2m])))

strategy:
  canary:
    stableService: catalog-stable
    canaryService: catalog-canary
    analysis:                      # runs in the background from the first step
      templates:
        - templateName: catalog-slo
      startingStep: 1
    steps:
      - setWeight: 25
      - pause: { duration: 2m }
      - setWeight: 50
      - pause: { duration: 2m }
      # after the last step the canary becomes the new stable

metrics:
  - name: error-rate
    interval: 30s
    failureLimit: 1                # a second bad measurement aborts and rolls back
    successCondition: len(result) == 0 || isNaN(result[0]) || result[0] < 0.01
    provider:
      prometheus:
        address: http://prometheus.monitoring:9090
        query: |
          sum(rate(http_server_requests_seconds_count{service="catalog-canary", status=~"5.."}[1m]))
          /
          sum(rate(http_server_requests_seconds_count{service="catalog-canary"}[1m]))

How long must a canary run?

A canary at 5 % sees few requests, and rare errors need many requests to become visible. Before choosing pause durations, estimate how many canary requests you need to tell a real regression from noise. The table uses the standard normal-approximation sample size for comparing two proportions (one-sided α = 5 %, power 80 %), with an illustrative baseline error rate of 0.1 % and 400 requests per second reaching inventory:

Large regressions show up in a minute or two; a doubling of a small error rate needs a quarter of an hour at 5 %. That is why canary steps are measured in minutes, not seconds, why the analysis compares the canary with the stable baseline running at the same time (same traffic, same hour), and why low-traffic services are better served by longer pauses or synthetic traffic. Google's SRE workbook discusses these trade-offs in its chapter on canarying releases⁶.

Measuring delivery: the DORA metrics

The DORA research program (DevOps Research and Assessment) found four metrics that predict software-delivery performance and that move together in high-performing teams³:

The key finding — speed and stability are not a trade-off — is the argument for everything on this page: small, frequent, automatically verified changes are both faster and safer than large, rare, manual ones.

AI in the pipeline

Pipelines increasingly include AI steps: automated pull-request review, generated tests, agents that propose fixes for failing builds, AI-assisted canary analysis, release notes and incident summaries — and the security and governance questions they raise. These are covered in their own class: AI-Assisted Delivery. The foundations on this page — reproducible builds, GitOps, metric-gated rollouts — are what make those AI steps safe to adopt.

Trade-offs

How it connects

flowchart LR
    IAC["IaC: Terraform / OpenTofu<br/>plan, apply, remote state"] --> INFRA["VPC, cluster,<br/>managed data services"]
    INFRA --> DR["re-create after a region loss<br/>→ Disaster recovery"]
    CI["CI: build, test, scan,<br/>SBOM, sign"] --> IMG["image by digest"]
    IMG --> GIT["config repo:<br/>desired state in Git"]
    GIT --> ARGO["Argo CD pulls,<br/>reconciles drift"]
    ARGO --> STRAT["rolling, blue/green,<br/>canary, feature flags"]
    STRAT --> EXP["expand/contract:<br/>backward-compatible schema"]
    STRAT --> AN["canary analysis: error rate,<br/>p99 vs stable baseline"]
    AN --> SLO["SLIs and SLOs<br/>→ Reliability, Observability"]
    AN --> N["sample size:<br/>minutes, not seconds"]
    STRAT --> DORA["DORA metrics"]
    CI --> AI["AI steps in the pipeline<br/>→ AI-assisted delivery"]

Palette components: vpc (created by IaC) · sql · cache · kafka (managed, declared in Terraform) · ingress (Gateway API splits canary traffic) · metrics (queried by the canary analysis) · app (inventory, order, payment on canaries). Scenarios: — delivery happens outside the simulator; the metrics a canary is judged on are the ones of observability-stack. Continues in: Observability — SLOs, Reliability — disaster recovery, Performance — budgets in CI, AI-assisted delivery, Hands-on 13 — canary, Project milestones.