Code Review

A pull request to order changes the checkout so that it confirms the seat hold with one SQL statement instead of an HTTP call to inventory — "it is faster". The code compiles, the tests pass, and the reviewer, at 18:40 on a Friday, approves. Three weeks later, during an on-sale, inventory changes its hold table and checkout breaks. The rule that was violated — one schema per service — was written in a wiki page nobody opened. A reviewer that had read that page, every time, on every PR, would have caught it.

That is the promise of AI code review: a reviewer that reads every diff in minutes, never gets tired, and can be told the team's rules. This page is about turning the promise into a tool people actually trust.

Why code review, and what it is for

Modern code review — a lightweight, tool-based review of each change before merging — is near-universal in industry. Studies at Microsoft and Google found that its main outcomes are not only finding defects but knowledge transfer, shared code ownership and consistency; at Google the median review latency was under a day, with most changes small¹². Two consequences for AI review:

• An AI reviewer can take over part of the defect-finding and consistency work, but not the knowledge transfer: a human must still read the code to learn it.
• Review latency matters. The first human look often waits hours; an AI review in two minutes gives the author something to fix before the human arrives, so the human review is shorter.

What AI review is good and bad at

Two systematic weaknesses deserve names. Gullibility: the model reads text written by the author — PR description, code comments — and can be steered by it ("this was approved by security"); see prompt injection. Plausible noise: a model asked to review will find something; without explicit permission to return nothing, it pads the review with style remarks and restatements.

Anatomy of an AI reviewer

sequenceDiagram
    autonumber
    participant GH as Git host
    participant R as Reviewer job
    participant Ctx as Context builder
    participant GW as LLM gateway
    participant Post as Poster
    GH->>R: PR opened / new push
    R->>Ctx: diff + PR metadata
    Ctx->>Ctx: number lines, filter files, budget tokens
    Ctx->>Ctx: add conventions, ADRs, owners, related code
    Ctx->>GW: system (stable, cached) + user (untrusted data)
    GW-->>R: JSON findings (schema-validated)
    R->>R: filter: changed lines, severity, confidence, dedup, cap
    R->>Post: findings to post
    Post->>GH: one review with inline comments
    Post->>GH: metrics: tokens, US$, kept vs dropped

The model is one box of eight. Most of the engineering — and most of the quality — is in the context builder and the filter.

Context engineering

A model can only review against what it is given. Context engineering is deciding what goes into the limited, priced context window, and in which order.

Three practical rules:

1. Order for caching. Put stable content first (instructions, conventions, ADRs) and per-run content last (diff). Prompt caching is a prefix match: if the stable part is byte-identical, repeated reviews read it at a fraction of the price (LLM gateway).
2. Budget, do not truncate. A 5,000-line PR does not fit — and should not be reviewed as one blob. Keep whole files by priority (production code before tests before config), list the skipped files in the review, or split: one call per file or per service (map), then one call to merge and deduplicate (reduce). Cutting a diff in the middle produces confident nonsense about code the model never saw.
3. Write conventions for humans first. If a rule is unclear to a new engineer it is unclear to the model. A rule that needs a paragraph of nuance is usually better as a deterministic check (below).

Structured output, posted as review comments

Free text is for humans; the pipeline needs data. Ask for JSON that validates against a schema — with Claude, structured outputs constrain the response to the schema; with other providers, JSON mode plus validation and one repair attempt:

{
  "summary": "Adds direct SQL access from order to the inventory schema.",
  "findings": [
    {
      "file": "services/order/src/main/java/boxoffice/order/OrderService.java",
      "line": 21,
      "severity": "blocker",
      "category": "architecture",
      "confidence": "high",
      "title": "order writes to the inventory schema",
      "explanation": "CONVENTIONS.md: a service reads and writes only its own schema. This UPDATE couples order to inventory's table layout and bypasses inventory's locking, so a hold can be confirmed after it expired.",
      "suggestion": "Call InventoryClient.confirmHold(holdId) and handle 409 Conflict."
    }
  ]
}

On GitHub, all findings of one run go into one review (POST /repos/{owner}/{repo}/pulls/{number}/reviews with a comments array of path, line, side), so the author gets one notification, not twelve³. A review event of COMMENT keeps the bot advisory; the code — not the model — chooses that event.

Noise control

Every comment spends a developer's attention. A reviewer is judged by its worst comments: after a few useless ones, people stop reading — including the useful ones. The controls, in the order they act:

The thresholds trade precision (the fraction of posted comments that are useful) against recall (the fraction of real problems found). The table uses an illustrative labelled sample of 100 PRs — the model emitted 60 high-, 90 medium- and 150 low-confidence findings, of which 48, 45 and 30 were real problems, out of 150 real problems in total:

Posting everything finds the most problems and is ignored by everybody (3 comments per PR, 4 out of 10 useful). Posting only high confidence is trusted and misses two thirds of the problems. Most teams start at high + medium for severities ≥ minor, measure, and move per category: secrets and SQL injection can stay at medium; style never gets posted.

def review(diff: str, title: str, body: str, root: Path = Path(".")):
    """Review a diff: (summary, kept findings, dropped [(finding, why)], usage, skipped files)."""
    files = diffutil.split_files(diff)
    files, skipped = diffutil.budget(files, MAX_DIFF_LINES)
    if not files:
        return "No reviewable changes.", [], [], Usage(model()), skipped
    numbered = "\n\n".join(diffutil.numbered(p, lines) for p, lines in files.items())
    user = (f"<untrusted_pr_title>\n{title}\n</untrusted_pr_title>\n"
            f"<untrusted_pr_description>\n{body[:4000]}\n</untrusted_pr_description>\n"
            f"<untrusted_diff>\n{numbered}\n</untrusted_diff>")
    data, usage = complete_json(load_context(root), user, SCHEMA, effort="medium")

    kept, dropped = [], []
    added = {p: diffutil.added_lines(lines) for p, lines in files.items()}
    for f in data["findings"]:
        if f["line"] not in added.get(f["file"], set()):
            dropped.append((f, "not an added line"))
        elif SEVERITIES.index(f["severity"]) < SEVERITIES.index(MIN_SEVERITY):
            dropped.append((f, f"below {MIN_SEVERITY}"))
        elif f["confidence"] == "low":
            dropped.append((f, "low confidence"))
        else:
            f["fp"] = fingerprint(f)
            kept.append(f)
    kept.sort(key=lambda f: (-SEVERITIES.index(f["severity"]), f["confidence"] != "high"))
    for f in kept[MAX_COMMENTS:]:
        dropped.append((f, f"over the limit of {MAX_COMMENTS} comments"))
    return data["summary"], kept[:MAX_COMMENTS], dropped, usage, skipped
    try:
        summary, kept, dropped, usage, skipped = review(diff, title, body)
    except LLMError as err:
        step_summary(f"### AI review skipped\n\n{err}")
        return 0                      # fail open: an AI outage must not block merges

Advisory or gate?

A safe rollout: shadow (findings logged, not posted; compare with what humans found) → advisory for volunteer teams → gate per category once precision is proven. Even as a gate, an AI check is a required status check — never an approval. Approval stays human, enforced by branch protection.

Measuring the reviewer

You cannot tune what you do not measure, and "the team likes it" is not a measurement.

An LLM can also judge comments ("was this comment correct?"), which is cheaper than human labels — but LLM-as-judge has biases (it tends to agree with confident text, and with itself). Use it to triage, and calibrate it against a human-labelled sample.

Architecture conformance

The opening story — order writing to the inventory schema — is an architecture rule. Such rules are the most valuable thing an automated reviewer can check, because humans forget them and violations are expensive later. But many of them do not need a model at all.

@AnalyzeClasses(packages = "boxoffice.order")
class ArchitectureTest {

    @ArchTest
    static final ArchRule no_direct_kafka_outside_outbox =
        noClasses().that().resideOutsideOfPackage("..outbox..")
            .should().dependOnClassesThat().haveSimpleName("KafkaTemplate")
            .because("ADR-0004: events are published through the transactional outbox");
}

The best design combines both: deterministic rules block, because they are exact; the AI reviewer explains and catches what the rules cannot express, and its recurring findings become candidates for new deterministic rules. Defense in depth again — and the database user of order without grants on inventory.* is the last layer that does not depend on any reviewer.

How it connects

flowchart LR
    PR["pull request event"] --> CTX["context: diff, conventions,<br/>ADRs, symbol search"]
    CTX --> RET["retrieval<br/>→ AI systems RAG & Memory"]
    CTX --> LLM["model call<br/>→ LLM gateway, prefix cache"]
    LLM --> OUT["structured findings<br/>(JSON schema)"]
    OUT --> NOISE["noise control: changed lines,<br/>confidence, dedup, cap"]
    NOISE --> GATE{"advisory<br/>or gate?"}
    GATE --> PREC["precision + golden set<br/>→ AI systems safety & evals"]
    ARCH["architecture rules:<br/>deterministic checks first"] --> OUT
    PR --> INJ["PR text is untrusted<br/>→ Security & governance"]

Palette components: orchestrator (the reviewer job around the model) · llmgw (routing, quotas, cached prefix of conventions and ADRs) · memory (the context assembled for each diff) · safety (untrusted PR text, validated output) · metrics (precision and comments per PR). Scenarios: none specific. Continues in: AI systems — RAG & Memory, AI systems — LLM Gateway, AI systems — safety & evals, Security & governance, Hands-on 13 — the reviewer.