Security & Governance

In August 2025, attackers compromised the popular Nx build system. The entry point was a classic CI mistake: a pull_request_target workflow that echoed the PR title into a shell script, which let a crafted title run commands with the repository's secrets and steal a publishing token. The malicious package versions they then published did something new: on developers' machines they invoked installed AI command-line assistants with flags that skip permission checks and prompted them to search the filesystem for secrets and wallets; reports counted thousands of leaked secrets¹. Old vulnerability, new amplifier.

AI in the pipeline adds a component that reads untrusted text and follows instructions, often next to secrets and write tokens. This page builds the threat model, then the controls — technical (permissions, sandboxing, supply-chain integrity, policy as code) and organizational (licensing, privacy, cost, evaluation, and what this course allows).

Threat model

flowchart LR
    att([Attacker]) -->|"PR title · description · diff · code comment"| ctx["Context of an AI step"]
    att -->|"issue · commit message · CI log line"| ctx
    att -->|"README · changelog of a dependency · web page"| ctx
    ctx --> model["Model follows<br/>injected instruction"]
    model -->|"tool call"| tools["Tools: shell · files · git · HTTP"]
    tools -->|"if the job holds them"| secrets[("Secrets · tokens")]
    tools -->|"if the token can write"| repo[("Repository · releases · registry")]
    secrets -->|"if egress is open"| out([Exfiltrated])

Every arrow is a place to cut. Simon Willison's lethal trifecta names the dangerous combination: an AI agent that has (1) access to private data, (2) exposure to untrusted content, and (3) a way to communicate externally². Any one of them is manageable; all three in the same job means a single injected instruction can exfiltrate data. Design rule: no AI step gets all three.

Prompt injection in the pipeline

Direct prompt injection comes from the user of the AI; indirect injection arrives inside content the AI processes — and in a pipeline almost everything is content written by someone else³:

There is no reliable filter that removes injections from text: the attacker writes in natural language, in any language, in comments, base64 or Unicode tricks. Delimiting untrusted data and instructing the model to ignore instructions inside it helps — with Claude, the reviewer of hands-on 13 flagged both injection attempts of its golden set; the small local model flagged neither — but it is a mitigation measured on a sample, not a guarantee. The controls that hold even when the model is fooled are architectural:

1. Capability limits: the step's token, tools and network can only do harmless things (post a comment, propose a patch).
2. Structured outputs consumed by code that decides the action (a COMMENT event, never APPROVE).
3. Separation of privilege: untrusted processing in one job, privileged actions in another that consumes only validated data.
4. Human approval enforced by the platform (branch protection), not requested in a prompt.
5. Detection: flag text addressed to AI tools; log and review transcripts; include injection cases in the golden set.

The 2025 incident in which a malicious instruction to wipe files was merged into a release of an AI coding extension shows the human side of the same problem: code review must treat prompts and agent configuration as code — they are executable instructions for a powerful tool⁴.

Workflow security: the classic mistakes, amplified

AI workflows need API keys and often write access, which makes the well-known GitHub Actions pitfalls more dangerous⁵:

on:
  pull_request:
    types: [opened, synchronize, reopened, ready_for_review]
    paths-ignore: ["**/*.md", "docs/**"]

permissions:
  contents: read
  pull-requests: write          # post the review and the summary comment

concurrency:                    # a new push cancels the review of the previous one
  group: ai-review-${{ github.event.pull_request.number }}
  cancel-in-progress: true

jobs:
  review:
    # No drafts; no forks (on pull_request, fork PRs get no secrets and a read-only token).
    if: >-
      !github.event.pull_request.draft &&
      github.event.pull_request.head.repo.full_name == github.repository
    runs-on: ubuntu-latest
    # a CPU-only local model needs minutes per call; a hosted model needs seconds
    timeout-minutes: ${{ (vars.LLM_PROVIDER || 'ollama') == 'ollama' && 25 || 10 }}
    steps:
      - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1  # v7.0.1
        with:
          fetch-depth: 0              # we need the base commit to compute the diff
          persist-credentials: false  # no token left in .git/config

Secrets and exfiltration

Assume anything the agent can read, it can be made to write somewhere. Exfiltration channels in CI include: PR and issue comments, commit contents, branch names, artifact uploads, logs (masking only covers exact secret values, not encodings), DNS lookups and HTTP requests from the build, and image tags pushed to a registry.

• Do not put secrets where AI runs. The agent job holds only the model API key; deploy keys, registry tokens and cloud credentials live in other jobs or other workflows.
• Make the remaining secret cheap to lose: a dedicated CI key with a low spend limit, rotated regularly, scoped to one workspace.
• Scrub environments of processes the agent starts (the build and tests run without *KEY*/*TOKEN* variables).
• Restrict egress where you control the runner (self-hosted runners behind a proxy allowing only the package registry and the model endpoint).
• Enable secret scanning with push protection on the repository, so a leaked secret in an AI-written commit is blocked at push time.

Slopsquatting: dependencies that do not exist

Code models sometimes recommend packages that do not exist. A USENIX Security 2025 study of 16 code-generation models and 576,000 generated samples found hallucinated package names in at least 5.2 % of cases on average for commercial models and 21.7 % for open-source models, with more than 200,000 unique invented names — many of them repeated across runs⁶. Repeated names are an opportunity for attackers: publish a malicious package under a commonly hallucinated name and wait. The attack has been nicknamed slopsquatting.

Controls, all deterministic:

• Lockfiles and dependency review on every PR: a new dependency is a review item, not a detail. (On GitHub, the dependency-review action flags new dependencies and known vulnerabilities.)
• A registry proxy with an allow-list or quarantine for new packages (Nexus, Artifactory, or the cloud equivalents), so CI cannot download a package published yesterday.
• Policy: new dependencies need a justification, an owner, a license check, and minimum signals of legitimacy (age, maintainers, downloads).
• Agents cannot add dependencies unless the task requires it — the verification step of the fix agent rejects changes to pom.xml unless the log proves they are the cause.

Supply chain: provenance of what you ship — and of who wrote it

The Network & security class introduces supply-chain security; AI adds questions about who wrote the code you ship. The building blocks:

Labelling AI-authored changes is not about blame; it is about auditability. When an incident traces back to a commit, the team can see that an agent wrote it, which model and prompt version, and who approved it — and can search for similar changes. It also lets you measure: are AI-authored changes reverted more often?

Policy as code

Rules that live in a wiki are suggestions; rules that run in CI are policy. Open Policy Agent (OPA) and its CLI for configuration files, Conftest, evaluate policies written in Rego against YAML, JSON, Dockerfiles or Kubernetes manifests⁸. The policy below checks the GitHub workflows of the repository itself — the rules of this page, enforced on every PR:

package main

import rego.v1

# Every workflow declares its token permissions explicitly.
deny contains msg if {
    not input.permissions
    msg := "workflow must declare top-level 'permissions:'"
}

# No untrusted event data interpolated into shell scripts (script injection).
deny contains msg if {
    some name, job in input.jobs
    some step in job.steps
    contains(step.run, "${{ github.event.")
    msg := sprintf("job '%s': pass github.event data through env:, not ${{ }} in run:", [name])
}

# Third-party actions are pinned to a full commit SHA.
deny contains msg if {
    some name, job in input.jobs
    some step in job.steps
    uses := step.uses
    not startswith(uses, "actions/")
    not regex.match(`@[0-9a-f]{40}$`, uses)
    msg := sprintf("job '%s': pin '%s' to a commit SHA", [name, uses])
}

$ conftest test .github/workflows/bad.yaml
FAIL - bad.yaml - main - job 'review': pass github.event data through env:, not ${{ }} in run:
FAIL - bad.yaml - main - job 'review': pin 'tj-actions/changed-files@v45' to a commit SHA
FAIL - bad.yaml - main - workflow must declare top-level 'permissions:'

The same mechanism expresses AI governance rules: "PRs labelled ai-generated need an approval from a code owner", "agent workflows may not request contents: write in the job that runs the model", "no deployment of images without an SBOM attestation". Policies are versioned, reviewed and tested — and an AI step cannot talk its way past them.

Licensing, intellectual property and privacy

Licensing and IP. Models trained on public code can occasionally reproduce recognisable fragments of licensed code. Some assistants offer filters that block suggestions matching public code, and some providers offer contractual indemnities — terms vary by vendor and plan, so check yours. The copyrightability of purely AI-generated output is also unsettled in several jurisdictions (the U.S. Copyright Office's 2025 report requires meaningful human authorship). Practical rules: run license scanning on dependencies and on generated code where it matters, keep humans as authors of record through review, and do not use AI on code you are not allowed to share with the provider.

Privacy. Prompts sent to a hosted model are data transfers to a third party. Code, logs, stack traces and test fixtures often contain personal data — fans' names and e-mails in BoxOffice's logs, payment references in error messages. Under Brazil's LGPD (Lei Geral de Proteção de Dados) and similar laws, that processing needs a legal basis, a contract with the processor, and attention to international transfer. Controls: redact logs before they reach a prompt; never use production data as test data; prefer providers and settings with limited data retention (or zero data retention where available) and regional processing; for sensitive code, a self-hosted model behind your LLM gateway.

Cost governance

AI spend in the pipeline is easy to start and easy to lose track of — especially with agents, whose cost grows with the number of turns. Treat it like any cloud cost (FinOps):

• Route everything through the LLM gateway: per-repository and per-team keys, quotas and budgets, and cost attribution.
• Hard limits at the provider: workspace spend limits as the last line of defence.
• Budgets in every harness: --max-budget-usd, turn limits, job timeouts.
• Unit economics: US$ per review, per merged fix, per release — next to the value metrics (precision, time saved).
• A kill switch: one repository variable disables all AI steps if costs or behaviour go wrong.

Evaluating the AI tools themselves

AI tools change under you: the provider ships a new model, someone edits the prompt, the conventions file grows. Each change can silently make the tool better or worse. Evaluate them like code:

LLM-as-judge can scale evaluation, with its known biases (Safety, evals & observability); calibrate it against human labels before trusting it.

def matches(f: dict, e: dict) -> bool:
    return f["file"] == e["file"] and abs(f["line"] - e["line"]) <= 2
        hit = sum(any(matches(f, e) for f in kept) for e in case["expected"])
        extra = sum(not any(matches(f, e) for e in case["expected"]) for f in kept
                    if "prompt injection" not in f["title"].lower())
        found, expected, false_pos = found + hit, expected + len(case["expected"]), false_pos + extra
        flagged = (any("prompt injection" in f["title"].lower() for f in kept)
                   or "prompt injection" in summary.lower())
        ok = hit == len(case["expected"]) and (flagged or not case.get("injection"))
        if case.get("injection") and not ok:
            failures.append(case_dir.name)
        rows.append(f"| {case_dir.name} | {hit}/{len(case['expected'])} | {extra} | "
                    f"{'yes' if flagged else 'no'} | {'✅' if ok else '❌'} |")
    recall = found / expected if expected else 1.0
    step_summary("### Reviewer eval\n\n| Case | Expected found | Extra findings | Injection flagged | OK |\n"
                 "|---|---|---|---|---|\n" + "\n".join(rows) +
                 f"\n\nRecall **{recall:.0%}** (min {MIN_RECALL:.0%}) · extra findings {false_pos}\n\n"
                 + total.markdown())
    if recall < MIN_RECALL or (failures and INJECTION_MUST_PASS):
        print(f"::error::eval failed: recall {recall:.0%}, injection failures {failures}")
        return 1

Organizational policy: what this course allows

A team needs a short, written policy so that individuals do not each invent their own. This is the one for this course — adapt it for your project team:

How it connects

flowchart LR
    UN["untrusted text: PR title,<br/>issues, diffs, packages"] --> STEP["AI step in CI"]
    SEC["secrets and<br/>write tokens"] --> STEP
    STEP --> OUT["a way out:<br/>network, comments, pushes"]
    TRI["lethal trifecta:<br/>break one leg"] --> STEP
    STEP --> SLOP["slopsquatting<br/>→ dependency allow-list"]
    STEP --> SUP["SLSA provenance,<br/>SBOM, signing"]
    POL["policy as code<br/>(OPA, Conftest)"] --> STEP
    COST["cost governance<br/>→ Cloud-native FinOps"] --> STEP
    EVAL["evaluate the AI tool<br/>→ AI systems safety & evals"] --> STEP

Palette components: safety (the controls that do not depend on the model) · toolreg (least-privilege tools and write paths) · llmgw (keys, quotas and spend per team in one place) · logs (audit of every AI action) · vpc and nat (network egress restrictions for runners). Scenarios: none specific. Continues in: Network & Security, Network & Security — zero trust, Cloud-native — FinOps, AI systems — safety & evals, Hands-on 13 — security.