Testing

BoxOffice's test suite has grown to about 4,000 tests. A full CI run takes 25 minutes, roughly one run in thirty fails for reasons nobody understands and passes on retry, and the inventory service — the one where a bug means double-booked seats — has the fewest tests, because its logic is the hardest to test. AI can write tests, triage failures and choose which tests to run. Each of those is useful, and each has a trap: a generated test can protect a bug, a triage bot can hide a real failure behind the word "flaky", and a test selector can skip the one test that mattered. This page is about getting the benefit without falling into the traps.

Where AI helps in testing

Generated unit tests and the oracle problem

A test has two parts: the inputs (arrange, act) and the oracle — the expected result that decides pass or fail. Models are excellent at the first part: they enumerate edge cases (empty list, null, boundary dates, the 61st minute) faster than people. The second part is the problem. To know that a seat hold must last 10 minutes, the model needs the requirement; given only the code, it reads plusMinutes(8) and writes isEqualTo(now.plusMinutes(8)). The test passes and now protects the bug. This is the test oracle problem, a long-standing topic of software-testing research¹, and generated tests make it acute.

What generated tests are reliably good for:

• Characterization tests before a refactoring: pin down current behaviour, refactor, confirm nothing changed. Here "asserts current behaviour" is exactly the goal.
• Regression tests for a bug fix, when the prompt includes the bug report and the expected behaviour.
• Edge cases a human then checks against the specification.

What they cannot do is decide what is correct. Requirements, examples in the issue, or an ADR must be in the context — and a human must read the assertions.

Coverage lies; mutation testing does not (as much)

Line coverage measures which lines ran during the tests. A test that calls every method and asserts nothing has full coverage and catches no bug. Mutation testing measures whether the tests notice a change: the tool creates mutants — small modifications such as < → <=, + → -, a removed method call, a constant return — and runs the tests against each one. A mutant is killed if some test fails; the mutation score is killed ÷ total. Surviving mutants point at behaviour no test checks. The idea is old (DeMillo, Lipton and Sayward, 1978); tools like PIT make it practical on the JVM².

flowchart LR
    cls["Changed class<br/>HoldService"] --> base["PIT, existing tests<br/>baseline 41 %"]
    base --> gen["Model writes tests<br/>(class + requirements + neighbours)"]
    gen --> run{"compile, pass twice?"}
    run -->|"no: errors"| gen
    run -->|"yes"| pit["PIT, existing + new tests"]
    pit --> better{"score ≥ baseline + 5 pp?"}
    better -->|"no: surviving mutants"| gen
    better -->|"yes"| review["Draft PR → human reads assertions"]

The loop above feeds the surviving mutants back to the model ("line 42: changed conditional boundary in isExpired() survived") — the approach Meta reported for generating tests that target specific classes of faults³. It turns a vague request ("write more tests") into a precise one ("write a test that fails if this < becomes <=").

MAX_ATTEMPTS = int(os.environ.get("AI_TESTS_MAX_ATTEMPTS", "3"))
MIN_GAIN = float(os.environ.get("AI_TESTS_MIN_GAIN", "0.05"))   # +5 percentage points
FORBIDDEN = ["Thread.sleep", "@SpringBootTest", "@Disabled", "System.exit", "new Socket(",
             "HttpClient.new", "RestTemplate("]
        cmd = ["mvn", "-B", "-q", "-f", f"{module}/pom.xml", "test", f"-Dtest={name}",
               "-Dsurefire.failIfNoSpecifiedTests=false"]
        ok, out = run(cmd)
        if ok:
            ok, out = run(cmd)                      # second run: catch flaky tests
        if not ok:
            errors = [l.replace(f"{Path.cwd()}/", "") for l in out.splitlines()
                      if "ERROR" in l or "FAIL" in l][:2] or out.strip().splitlines()[-2:]
            history.append(f"{attempt}. does not compile or fails: " + " / ".join(errors)[:300])
            feedback = f"<previous_attempt_failed>\n{out}\n</previous_attempt_failed>"
            continue
        new_score, killed, total, survivors = mutation_score(module, fqcn, f"{package}.{cls}*Test")
        history.append(f"{attempt}. passes twice; mutation score {new_score:.0%} "
                       f"(needs {base_score + MIN_GAIN:.0%})")
        if new_score >= base_score + MIN_GAIN:
            accepted = True
            break
        feedback = ("<previous_attempt_too_weak>The tests pass but these mutants survive; add "
                    "assertions that would fail for them:\n" + "\n".join(survivors[:25])
                    + "\n</previous_attempt_too_weak>")

    if not accepted and test_file.exists():
        test_file.unlink()                          # never leave an unproven test behind

Contract tests

In a microservice system the most expensive test failures happen between services: order expects inventory to return 409 Conflict for an expired hold; inventory changes it to 410 Gone. Consumer-driven contract tests (Pact, Spring Cloud Contract) capture what each consumer actually uses, and the provider's build verifies it still satisfies every contract.

AI helps draft them: given the consumer's client code and the provider's OpenAPI document, a model can write the interactions (POST /inventory/holds/{id}/confirm → 200 or 409) much faster than a person. The trap: if the model derives the contract from the provider's implementation, the contract describes what the provider does today — not what the consumer needs — and verifies nothing. Generate from the consumer side, and let the provider verification (deterministic) be the judge.

Property-based tests

Example-based tests check a few inputs; property-based tests state an invariant and let a generator try hundreds of inputs, shrinking any failure to a minimal example. They are a natural fit for the parts of BoxOffice where bugs are expensive:

class SeatMapProperties {

    @Property
    void seatsAreNeverDoubleBooked(@ForAll("operations") List<SeatOperation> ops) {
        SeatMap map = new SeatMap(Venue.small());           // 100 seats
        ops.forEach(op -> op.applyTo(map));                  // holds, confirms, expiries, releases
        assertThat(map.sold() + map.held() + map.available()).isEqualTo(100);
        assertThat(map.soldSeats()).doesNotHaveDuplicates();
    }

    @Provide
    Arbitrary<List<SeatOperation>> operations() {
        return SeatOperation.arbitrary().list().ofMaxSize(200);
    }
}

The example uses jqwik for JUnit 5. Models are good at proposing invariants from a domain description ("sold + held + available = capacity", "a confirmed seat is never released by expiry") and at writing generators. Two checks keep them honest: plant a bug and confirm the property fails (a property that never fails may be vacuous), and read every invariant against the domain rules.

Load-test scripts from OpenAPI

Every BoxOffice service publishes an OpenAPI document. A model can turn it — plus a sample of real access logs — into a k6 script with realistic endpoints, payloads and checks in seconds. What it cannot know is the workload model: the mix of requests (95 % reads of the catalog during an on-sale), the arrival process (open model with a ramp, not a fixed number of looping users), think time, and data distribution (everyone wants the same 500 front-row seats). Those decisions come from the performance class and from your measurements, and a generated script that silently uses a closed model with no think time measures the wrong system.

A safe workflow: generate the script, review the scenario block (executor, rates, thresholds) by hand, run a smoke test, and keep the script in load/ like any other code.

Flaky-test triage

A flaky test passes and fails on the same code. Flakiness is a large, well-documented problem in industry CI: Google reported that a significant fraction of its tests had shown flaky behaviour, and that flakiness was a major cost of its CI system⁴. When a build goes red, someone must decide: real regression, broken test, infrastructure problem, or flake?

A model reading the failure log plus history can classify quickly — but the deterministic signals should come first, because they are cheap and reliable:

The LLM's job is the part that is hard to do with rules: reading a 2,000-line log, finding the first meaningful error, matching it to the diff, and writing a short explanation with a proposed category. Its output is a label with evidence, not an action:

{
  "category": "test-bug",
  "confidence": "medium",
  "evidence": "HoldExpiryTest.expiresAfterTenMinutes uses Instant.now() twice; fails when the clock crosses a second boundary (3 of 412 runs on main in 30 days, no code change).",
  "suggested_action": "Inject a fixed Clock; until then, quarantine with an owner and a ticket."
}

Test selection

Running all 4,000 tests on every push costs 25 minutes; most of them cannot be affected by a one-line change in notification. Test impact analysis selects the tests that can be affected:

• Static: the build graph (changed module → modules that depend on it → their tests). Exact for module boundaries; coarse inside a module.
• Coverage-based: tests whose recorded coverage includes the changed lines. Precise, but needs coverage maps kept up to date.
• Predictive: a model trained on history predicts which tests are likely to fail for this change; Meta reported catching the vast majority of failures while running a fraction of the tests⁵. LLMs can contribute features (what the change is about), but the decision is statistical.

The safety net is non-negotiable: the full suite still runs — on main after merge, nightly, or before release — so a missed test is caught hours later, not in production. The numbers make the trade-off concrete (illustrative assumptions in the code):

At 95 % recall the team saves about twelve runner-hours of CI per day and lets roughly one breaking change every three weeks slip to the nightly run — where it is caught and attributed to a commit. Whether that is acceptable depends on how expensive a red main is for the team; for inventory, many teams always run the service's full suite.

Synthetic test data

Tests need data: venues with odd seat maps, events across time zones, names with accents, e-mail addresses at the length limit. Models generate rich, realistic fake data quickly. Three rules:

1. Never send production data (customer names, e-mails, payment details) to a model to "make it realistic". Under Brazil's LGPD and similar laws, that is processing personal data by a third party, with legal obligations — and it is unnecessary.
2. Freeze the output. Generate once, review, commit the fixture (or a seeded generator). Tests that call a model at run time are slow, cost money and are non-deterministic.
3. Ask for the edge cases explicitly: an event at 23:30 on the day daylight-saving time changes, a venue with 0 seats in a section, a name with 300 characters. Realistic data is less valuable than adversarial data.

Trade-offs

How it connects

flowchart LR
    GEN["AI generates inputs:<br/>unit, property, contract tests"] --> ORA{"the oracle:<br/>who decides correct?"}
    ORA --> REQ["requirements<br/>and humans"]
    GEN --> MUT["mutation testing<br/>judges the tests"]
    GEN --> K6["k6 scripts from OpenAPI<br/>→ Performance"]
    FL["flaky-test triage:<br/>deterministic signals first"] --> QU["quarantine is<br/>a policy decision"]
    SEL["test selection:<br/>recall vs minutes"] --> NET["full-suite<br/>safety net"]
    DATA["synthetic test data"] --> LGPD["no production<br/>personal data"]

Palette components: llmgw (every generation and triage call) · app (the services under test, with inventory as the riskiest) · sql (the seat-hold rules that properties and contract tests protect) · logs and metrics (failure history and flaky rates that feed the triage). Scenarios: flash-sale (the load the generated k6 scripts must reproduce). Continues in: Performance (load tests and budgets), Performance — load testing, Databases — transactions (what "no double booking" means), Code review, Hands-on 13 — tests.