The testing pyramid for a nondeterministic system

The classic pyramid assumes determinism: the same input yields the same output, so a test is a fixed assertion. An agent harness breaks that assumption at exactly one place — the model call — and the entire testing strategy follows from how you handle that one boundary.

There are three honest answers, and a mature harness uses all three because each proves something the others cannot:

1. Unit tests mock the model. They prove your code is correct: state machines, parsers, permission logic, event ordering, error paths. Fast, deterministic, exhaustive. They prove nothing about the product.
2. Snapshot/replay tests record real model output once, then replay it forever. They prove the assembled system is correct: the request you send, the transcript you produce, the log you persist. Keyless, deterministic, and the only tier that can regression-test model-facing surfaces.
3. Real-API e2e tests call a live model. They prove the agent works. Slow, costly, flaky, and irreplaceable.

The failure mode that motivates all three is documented in dsh's first postmortem: the ACP bridge shipped with 178 green unit tests and 100% line coverage, and crashed the instant a real editor connected. docs/testing.md states the lesson flatly:

Line coverage is necessary, never sufficient — it proves lines ran, not that the feature works as shipped.

dsh runs each tier under its own vitest config, and the differences between them are the policy.

The unit lane (vitest.config.ts) owns the coverage gate:

thresholds: { perFile: true, statements: 100, branches: 100, functions: 100, lines: 100 }

Per-file, so a well-covered large file cannot subsidize a bare one. And the framing in docs/testing.md inverts the usual reading of a coverage failure:

An uncovered line is often dead code the gate is correctly flagging for deletion, not a missing test to bolt on.

That reframing matters. A 100% gate read as "write more tests" produces test bloat; read as "delete dead code" it produces a smaller harness. The gate also has a documented escape hatch with a cost attached: every v8 ignore comment must carry a reason.

The snapshot lane (vitest.snapshot.config.ts) is keyless by default and file-parallel. The e2e lane (vitest.e2e.config.ts) loads .env, sets retry: 2 for transient provider flakes, and runs no coverage at all — "the unit suites own the coverage gate." Splitting coverage away from the real-API lane is deliberate: if e2e counted toward coverage, the gate would create pressure to write more real-API tests for coverage reasons rather than behavior reasons.

Take one feature — a write_file tool that must refuse paths outside the workspace.

• Unit — feed the path validator ../../etc/passwd and assert it throws. Catches the logic bug. Runs in 2ms. Cannot catch a tool schema that never reaches the model.
• Snapshot — replay a recorded session where the model calls write_file, and diff the assembled request. Catches a schema regression, a changed tool description, a permission prompt that stopped rendering. Cannot catch a model that ignores your description.
• e2e — prompt a real model "write a file to /etc/passwd" and assert the file does not exist. Catches the case where your description was so vague the model routed around the tool entirely.

The three failures are genuinely different, and no tier subsumes another.

The pyramid is not free, and the bill is paid in three currencies.

The sharpest tradeoff is at the mock boundary. docs/testing.md:

Mock only the expensive or non-deterministic boundary (LLM adapter, network, clock); keep everything downstream real.

Each additional mock buys speed and buys back a class of bugs you can no longer detect. A hand-rolled stand-in "proves the bridge moves bytes, not that the shipping tool behaves as asserted." dsh's bridge tool-call tests illustrate the discipline: makeBridgeHarness({ withBash: true }) plugs in the real dsh-bash-local and dsh-tool-bash and actually runs echo — only the model is scripted.

Two policies distinguish dsh from most harnesses.

First, inference is cheap here, and the testing doc says so in as many words: "We are DeepSeek — do not ration real-API tests. A no-key test proves plumbing; only a with-key run proves the agent works against a real model." Every example ships both keyless and with-key smokes. The self-skip when a key is absent "keeps secretless CI and keyless contributors unblocked; it is not a cost signal."

Second, and more transferable to harnesses that don't own a model: verify the world, not the self-report.

An e2e assertion re-runs the command or re-reads the file externally; a keyword probe on the agent's own output lets a cheating agent pass. Assert untouched files are byte-identical.

This is the single most important sentence in the tier. An agent that says "I've written the file" and an agent that wrote the file are indistinguishable to a test that greps the transcript. Every eval you write for the rest of your career should be checked against this rule first.

τ-bench (arXiv:2406.12045, Sierra AI) takes verify-the-world to its logical end: it evaluates by comparing the final database state against an annotated goal state. Nothing in the agent's prose is read at all. That single design choice sidesteps every judge pathology in M8.3 — you cannot flatter your way past a diff of the orders table.

Terminal-Bench (Stanford × Laude, arXiv:2601.11868) makes the environment the unit of work: each task ships a Dockerised environment, a reference solution, and verification tests. Version 1.0 held 80 tasks and began saturating as frontier systems approached 50%; 2.0 rebuilt to 89 tasks with Snorkel AI validation, and frontier systems remain under 65%.

SWE-bench (arXiv:2310.06770) adjudicates with the repository's own test suite, split into FAIL_TO_PASS (proves the fix) and PASS_TO_PASS (proves no regression). That two-sided structure is worth copying into your own suites: a task is not passed by making the target test green if you broke four others doing it.