Snapshot/replay: making model output regression-testable

You cannot assert on model output. You can assert that, given the model output you recorded last Tuesday, your harness still assembles the same request, produces the same transcript, and persists the same log. That is the whole trick, and it converts an untestable surface into a diffable one.

The recorded model response becomes a fixture. The pipeline around it stays completely real — real subprocesses, real tool executors, real persistence. You are not testing the model. You are testing everything the model touches.

dsh drives this with a single environment variable, DSH_SNAPSHOT, whose three modes are documented in vitest.snapshot.config.ts:

Replay is the keyless default: boot real subprocess paths from recorded model responses and diff assembled requests, normalized protocol or transcript output, and persisted-log expected outputs. record calls the real API and updates fixtures and expected outputs; refresh replays committed scripts and updates current expected outputs. Replay/refresh never load .env; only record reads a key.

The mode split maps exactly onto three different things that can change:

refresh is the mode people skip and then regret skipping. If you renamed a field in your transcript renderer, you do not need a new model call — you need the same recorded conversation re-rendered. Burning real API quota to update a formatting change is a smell, and it also silently changes your test's input, which means a genuine behavior regression can slide in under cover of a formatting diff.

There is a concurrency contract attached, and it is a good example of why parallelism is a per-mode decision rather than a global switch. File parallelism is enabled only in replay:

Record and refresh stay serial: record spends real API quota per scenario, and refresh write-back harvests volatile values from fixtures already on disk, so concurrent writers would corrupt goldens.

A naive fixture stores the raw HTTP exchange verbatim, and then every test fails on the first run because the timestamp moved. dsh's rule:

Session fixtures keep headers and payloads but omit body sequence/time envelopes. Replay synthesizes them; runtime persistence is unchanged.

So the fixture holds the semantic content and the replay harness regenerates the volatile envelope. The persisted log written during replay is then identical to the one written at runtime, which is the property that makes the assertion meaningful.

The second normalization decision is subtler, and it's the one most teams get wrong. Your system prompt and tool schemas are enormous and change constantly. If every fixture pins their full text, a one-word prompt edit churns thousands of lines across dozens of goldens, reviewers stop reading the diffs, and the suite stops working as a review instrument. But if no fixture pins them, you lose the ability to catch a prompt regression at all.

dsh resolves this asymmetrically: one ACP scenario (text-turn) pins the full system-prompt and tool-schema content; every other fixture tokenizes it "so an edit churns one line." Exactly one witness for the content, one line of churn everywhere else.

Fixtures rot. Recorded output ages against provider behavior. A replay suite can stay green for months while the real model has drifted into a different response shape — which is precisely why the e2e tier is not optional.

A golden is only a guard if it fails. docs/testing.md: "A guard only guards if the regression actually fails it… introduce the regression, watch red, revert." This is not paranoia. Postmortem 0001's Loader smoke stayed green through the exact regression it existed to catch, because a plugin without inject loads fine either way. The fix was an explicit expect('default' in mod).toBe(false) — plus proving it failed before trusting it.

Review discipline is the real cost. "Review every JSONL and expected-output diff." A snapshot suite that gets rubber-stamped --update is worse than no suite: it consumes CI time and provides false assurance.

Snapshot suites are owned by the surface they cover, not centralized: examples/<name>/tests/snapshots/ for ACP automation scenarios over the dsh-acp-snapshot factory, apps/cli/tests/snapshots/ for interactive-terminal journeys, apps/web/tests/snapshots/ for browser-rendered GUI journeys, examples/jsonrpc-agent/tests/snapshots/ for the TypeScript SDK client, and scripts/snapshots/python-sdk-single-exe/ for the Python one. The two SDKs project the agent loop and SessionEventMap independently, so a change to either updates both.

The web lane makes a further operational choice: CI forces read-only DSH_SNAPSHOT=replay and never writes expected outputs. Record and refresh stay local. A CI job that can rewrite its own goldens cannot fail.

Fixture format changes are handled by migration rather than re-recording: scripts/migrate-packed-session-fixtures.ts rewrites old layouts into canonical packed rows.

Most agent harnesses record at the HTTP layer with a VCR-style library, which is cheaper to build and strictly weaker: it pins the transport exchange, not the assembled request or the persisted log, so a bug between your request builder and your log writer passes through invisibly. dsh's fixtures sit at the session layer and the assertions cover assembled request, normalized protocol output, and re-persisted log — three surfaces, one recording.

Benchmark harnesses take the opposite approach entirely: Terminal-Bench and SWE-bench re-execute against real environments every run and accept the nondeterminism, because their unit of measurement is task success rather than harness behavior. That is the right call for a leaderboard and the wrong call for a regression suite, and knowing which one you are building is the whole decision.

---