Agents in CI

The team upgrades Spring Boot across BoxOffice on a Monday morning. Fourteen services, fourteen red builds: a renamed configuration property here, a deprecated method there, a test that relied on an old default. Each fix is small and boring; together they take two engineers two days. An agent — a model that plans, uses tools, observes the result and tries again — can do most of this work while the engineers review. It can also, if badly designed, push to main, leak the deploy key, loop for an hour on a flaky test, or "fix" a red build by deleting the assertion. This page is about the design that gets the first outcome and makes the second impossible.

What agents do in a pipeline

The agent loop — the model chooses a tool, the harness runs it, the result goes back into the context, repeat — becomes, in CI, a loop over the repository and the build:

Every row ends in the same place: a proposal (a branch, a draft PR) that deterministic checks and a human approve. That is level 3 of the autonomy spectrum. Level 4 — applying changes without per-change approval — is reserved for changes that are both low-risk and fully verifiable, and rarely needs a model at all.

Anatomy of a fix-the-build agent

sequenceDiagram
    autonumber
    participant CI as CI run
    participant Orch as Workflow (deterministic)
    participant Agent as Agent loop (sandbox)
    participant Model as Model
    participant Repo as Git host
    CI-->>Orch: run failed (event)
    Orch->>Orch: dedupe by commit SHA, check branch is not ai-fix/*
    Orch->>Agent: failed log + checkout (no write token)
    loop at most N turns, M dollars
        Agent->>Model: context + tool results
        Model-->>Agent: tool call (read / edit / run build)
        Agent->>Agent: execute allow-listed tool, log it
    end
    Agent-->>Orch: working-tree diff + transcript
    Orch->>Orch: verify: paths allowed, no test removed, build green
    Orch->>Repo: separate job with write token: branch + draft PR
    Repo-->>Orch: human reviews, CI runs, merge or close

Read it as two programs. The agent is non-deterministic and untrusted: it may be wrong, confused or manipulated by text in the log. The orchestrator around it — triggers, dedupe, verification, the PR job — is ordinary deterministic code, reviewed like any other. All the safety lives in the orchestrator.

Running agents headless

Agents in CI run headless: no terminal, no human to answer "may I run this command?". Every decision a person would make interactively must be made in advance, as configuration.

gh run view "$RUN_ID" --log-failed | tail -n 300 | \
claude --bare -p "The CI build failed; the log is on stdin. Fix the root cause with the smallest change. Never delete or weaken a test." \
  --allowedTools "Read,Edit,Write,Bash(mvn -B *)" \
  --permission-mode dontAsk \
  --max-turns 25 --max-budget-usd 1.50 \
  --output-format stream-json --verbose > transcript.jsonl

name: Claude Code
on:
  issue_comment:
    types: [created]
  pull_request_review_comment:
    types: [created]
jobs:
  claude:
    if: contains(github.event.comment.body, '@claude')
    runs-on: ubuntu-latest
    permissions:
      contents: write
      pull-requests: write
      issues: write
      id-token: write
      actions: read
    steps:
      - uses: actions/checkout@v6
        with:
          fetch-depth: 1
      - uses: anthropics/claude-code-action@v1
        with:
          anthropic_api_key: ${{ secrets.ANTHROPIC_API_KEY }}

for step in range(MAX_STEPS):
    reply = client.chat.completions.create(model=MODEL, messages=messages, tools=TOOLS)
    msg = reply.choices[0].message
    messages.append(msg.model_dump(exclude_none=True))
    if not msg.tool_calls:
        break
    for call in msg.tool_calls:
        result = run_allow_listed(call.function.name, json.loads(call.function.arguments))
        audit(step, call, result)
        messages.append({"role": "tool", "tool_call_id": call.id, "content": result})

Sandboxing

The agent executes the build, and the build executes the repository's code: tests, Maven plugins, scripts. Assume that code — or text the agent reads — may be hostile, and contain it in layers:

No single layer is perfect. A tool allow-list of Bash(mvn *) still runs whatever the project's Maven plugins do; environment scrubbing does not protect the model API key held by the agent process itself. That is why the key has a spend limit, the job has no write token, and the output is verified outside the agent.

Permissions: the token is the blast radius

Whatever the model decides, it can never do more than its credentials allow. Design the credentials first:

• Least-privilege GITHUB_TOKEN: declare permissions: in every workflow; the job that runs the agent gets contents: read (and actions: read to fetch logs).
• Split jobs by privilege: the agent job produces a patch artifact; a small job with contents: write and pull-requests: write applies it and opens a draft PR. The job with write access runs no model and no untrusted code.
• No secrets in the agent's context: never paste secrets into prompts or tool results; keep deploy keys, cloud credentials and registry tokens out of the job entirely.
• Read-only by default: an agent that only diagnoses (explains the failure as a comment) needs no write access at all — start there.
• Branch protection and rulesets: main requires PRs, CI and a human approval; bots cannot bypass; changes to .github/workflows/ require a code owner.
• Human approval is structural, not a prompt instruction: the agent cannot merge because its token cannot.

jobs:
  fix:
    # Only failed runs, only branches of this repository, never our own fix branches.
    if: >-
      github.event.workflow_run.conclusion == 'failure' &&
      github.event.workflow_run.head_repository.full_name == github.repository &&
      !startsWith(github.event.workflow_run.head_branch, 'ai-fix/') &&
      !startsWith(github.event.workflow_run.head_branch, 'ai-tests/')
    runs-on: ubuntu-latest
    timeout-minutes: ${{ (vars.LLM_PROVIDER || 'ollama') == 'ollama' && 45 || 30 }}
    permissions:
      contents: read
      actions: read               # read the logs of the failed run
    outputs:
      fixed: ${{ steps.verify.outputs.fixed }}
  open-pr:                        # write access, but no model and no untrusted build here
    needs: fix
    if: needs.fix.outputs.fixed == 'true'
    runs-on: ubuntu-latest
    timeout-minutes: 5
    permissions:
      contents: write
      pull-requests: write

Budgets: turns, tokens, dollars, minutes

Each agent turn re-sends the whole context — instructions, tool definitions, the log, every file read, every tool result so far. Context grows with every step, so total input tokens grow roughly quadratically with the number of steps (AI systems — the numbers). Prompt caching reads the repeated prefix at a fraction of the price, which changes the slope dramatically. Illustrative assumptions: a 20,000-token starting context (instructions, tool definitions, log), 2,500 tokens added per step (tool result plus the model's output), 400 output tokens per step, list prices of a large model at the time of writing (US$ 4 input, US$ 20 output, US$ 0.20 cached read per million tokens):

Without caching, 40 turns cost more than eight times what 10 turns cost (about US$ 11 vs US$ 1.3); with caching, the curve is nearly linear and about eight times cheaper at 40 turns. Either way, the budget must be enforced by the harness, not requested in the prompt:

The ratio that matters is dollars per merged fix, not per run: an agent that costs US$ 0.80 per run but produces a mergeable fix one time in four costs US$ 3.20 per useful fix — compare that with the engineer-time it saves.

ROOT = Path.cwd().resolve()
MAX_STEPS = int(os.environ.get("AGENT_MAX_STEPS", "20"))
MAX_TOKENS = int(os.environ.get("AGENT_MAX_TOKENS", "400000"))   # input + output, whole run
MAX_USD = float(os.environ.get("AGENT_MAX_USD", "1.50"))
AUDIT = Path(os.environ.get("AGENT_AUDIT", "agent-audit.jsonl"))   # keep it outside the repo
def safe(path: str, write: bool = False) -> Path:
    p = (ROOT / path).resolve()
    if not p.is_relative_to(ROOT) or {".git", ".github"} & set(p.relative_to(ROOT).parts):
        raise PermissionError(f"{path}: outside the allowed area")
    if write and "src" not in p.relative_to(ROOT).parts:
        raise PermissionError(f"{path}: only files under services/<name>/src/ may be written")
    return p
        if done:
            break
        if usage.input_tokens + usage.output_tokens > MAX_TOKENS or usage.usd > MAX_USD:
            result = {"fixed": False, "explanation": f"stopped by budget after {step} steps"}
            break

Idempotency, retries and loops

Agents are triggered by events, and events repeat: a CI run is re-run, two pushes arrive in a minute, the agent's own branch fails CI. The messaging patterns of the course apply directly (Messaging — delivery semantics):

• Idempotency key: the failing commit SHA. One proposal per SHA (branch ai-fix/<sha8>, concurrency group on the SHA). Re-runs find the branch and stop.
• No self-triggering: never run the agent on its own branches (ai-fix/*), and remember that pushes with the GITHUB_TOKEN do not trigger new workflow runs on GitHub — a built-in loop breaker.
• Retries are for transport, not for the task: SDKs retry 429/5xx with backoff. Re-running a failed agent is a new attempt with a new cost — make it a human decision, or at most one automatic retry with a different strategy.
• Fail open for advisory agents: if the agent fails, the original red build is still there; nothing is worse than before.

Audit: who changed what, and why

An agent's change must be explainable months later. Keep:

This is the same discipline as for any production change, and it feeds the supply-chain provenance of the software: you can answer "which code in inventory was written by an agent, approved by whom?".

Dependency and migration agents

Dependency upgrades are the most common agentic workflow in practice. Bots such as Renovate or Dependabot open the upgrade PR deterministically; the agent's job starts when that PR is red: read the changelog and the failure, adapt the code, keep the change minimal. Group related updates (all Spring artifacts together), let patch-level updates with green CI auto-merge without AI, and send major versions to an agent that proposes and a human that approves.

Migrations — JUnit 4 → 5, javax.* → jakarta.*, Java 17 → 21 — mix a large mechanical part with a long tail of special cases. Use deterministic codemods for the mechanical part (OpenRewrite recipes for Java are the standard tool) and the agent for the leftovers the recipe cannot handle. Split the campaign into one PR per service, small enough to review, behind the normal CI and canary. Large-scale changes at Google followed the same principle long before LLMs: automate the change, shard it into independently reviewable pieces, and let the owners review each piece¹.

How it connects

flowchart LR
    TRIG["red build event<br/>trigger + dedupe"] --> HARN["harness = orchestrator"]
    HARN --> SB["sandbox, tool allow-list,<br/>no write token"]
    HARN --> BUD["budgets: turns, tokens,<br/>dollars, minutes"]
    HARN --> LOOP["agent loop<br/>→ AI systems agents"]
    LOOP --> VER["verification<br/>outside the agent"]
    VER --> PRIV["separate privileged step<br/>opens a draft PR"]
    PRIV --> HUM["human review"]
    HARN --> IDEM["idempotency, retries, loops<br/>→ Messaging semantics"]
    HARN --> AUD["audit trail:<br/>who changed what, and why"]

Palette components: orchestrator (the harness that enforces budgets and permissions) · toolreg (the allow-list of tools the agent may call) · llmgw (quotas and cost per run) · safety (no secrets or write tokens in the agent's reach) · logs (the audit trail) · metrics (merged fixes per proposal, cost per merged fix). Scenarios: ai-assistant with the spike profile (hundreds of agents at once compete with user traffic at the gateway). Continues in: AI systems — agents, Messaging — semantics (at-least-once runs, idempotent effects), Security & governance, Cloud-native — delivery, Hands-on 13 — fix agent.