From postmortem to durable invariant

The final skill, and the one that compounds. A postmortem that ends in "we fixed the bug and added a test" has converted an incident into a point fix. A postmortem that ends in a durable invariant has converted it into a class of bugs that can no longer happen.

docs/postmortem/README.md sets the bar for when a write-up is warranted — the bug must be subtle ("the mechanism is non-obvious and a careful engineer would re-derive it the hard way"), systemic ("the reason it escaped is a gap in tests/tooling/conventions, not a one-off typo"), and costly to rediscover. And it names the actual subject: "the interesting part is why our process let it through, not just the one-line fix."

Every postmortem opens with an Executive summary a busy reader absorbs in thirty seconds — what broke, the root cause in plain terms, why it escaped, and the durable lesson — before Summary / Timeline / Root cause / Guardrails.

Reading a postmortem for invariants means climbing from the specific to the general, and stopping at the highest rung you can actually enforce.

1. The fix — what changed. Not an invariant.
2. The guard — a test that fails if this exact bug returns. Better; still one bug.
3. The rule — a convention that makes the class of bug visible in review.
4. The invariant — a machine-checked property that makes the class of bug impossible or loud.

Postmortem 0001 climbs all four rungs. Fix: delete export default apply. Guard: a no-key session/new e2e over real stdio through the real Loader — verified to fail when the regression is restored. Rule: docs/testing.md's "test the real entry path." And the general lesson, at rung four:

A test that constructs a plugin by hand cannot validate how the plugin loads. At least one test must drive the real Loader/export path end-to-end. When the headline operation does not call the model, that test needs no API key — so it belongs in CI, not behind a key gate.

That last clause is what makes it durable. A guard behind a key gate does not run in CI, and a guard that does not run is documentation.

Postmortem 0004 shows the same ladder in a different domain. A Landlock partial-enforcement notice shared a landlock-run: prefix with genuine launcher failures, so ripgrep's ordinary exit-1-for-no-matches surfaced as SANDBOX_UNAVAILABLE. The root cause was representational: "The public sandbox result type could express only a bag of substrings. It could not state that Landlock failure requires exit 125, that evidence must occur within one fatal line, or that one exact line under the same prefix is informational." The guardrail changed the type — RunnerFailureRule now carries allowed exit codes, per-line fatal signatures, and exact informational-line exclusions — so the ambiguity is no longer expressible. Its lessons generalize cleanly:

Process attribution requires a conjunction of independent evidence; a shared prefix is not a protocol.

An adapter must preserve structured failures owned by the seam below it instead of replacing them with its own nearest generic category.

Note also what 0004 says about its own test matrix: "The test matrix mirrored that representation." The fakes could emit a fatal prefix or no prefix, but never a benign prefix followed by a child-controlled nonzero exit. When your test doubles are built from the same wrong model as your production types, coverage is 100% over the wrong state space.

dsh gives rung four a home. ctx.invariants (packages/runtime-diagnostics/invariants/) lets a package register runtime checks:

ctx.invariants.register(PACKAGE_NAME, install)

Registration returns an effect-scoped disposer and reserves the npm package name even when filters disable the checks, so ownership is unambiguous regardless of configuration. Violations raise InvariantError with code: 'INVARIANT', the owning packageName, and a message prefixed invariant violated by "<package>": …. The failure is attributed before anyone starts debugging, which is the operational payoff of rung four over rung two: a failed test tells you a test failed; a violated invariant tells you which package's contract broke.

The scope rule is deliberately narrow: a check may assert over authoritative event streams or mutable data, never service or method presence. Asserting that a service exists tests your wiring, not your semantics, and wiring is already the Loader's job.

The part most worth stealing is the enforcement of the absence of an invariant. Every package ships a ./invariant companion, and an empty one must carry a leading comment starting No runtime invariant: explaining package-specifically why nothing is checkable. pnpm run verify-package-invariants mechanically rejects generated markers and unexplained empties. The effect: "we didn't think about invariants here" is not a reachable state — you either have one, or you have written down why you cannot.

packages/llm/token-meter/src/invariant.ts shows what a good empty one looks like. It explains that token estimates are per-call outputs, that the private session cache is invalidated at its event mutation boundary, that the usage folds replace same-step samples so totals need not be monotone, and that the composition fold "makes its message figure equal measure().surfaceTokens by construction rather than by a relation worth observing at runtime." That last phrase is the whole art: an invariant worth checking is one that could be violated. A property true by construction is not an invariant, it is a tautology, and checking it burns runtime proving that your own code calls itself.

Bug #2: AgentLoop.resume() read this.ctx.sessionPersistence — an optional service deliberately absent from its static inject, because injecting it would make non-persistent demos pend forever. The property proxy resolves by an ancestor-only fiber walk, which fails when reached through a foreign shadow, because the service lives on a sibling branch. Tests missed it because they called ctx.agents.resume(...) from top-level test code, where ctx.fiber.runtime is null and the proxy takes an early bypass straight to the global store.

Climbing the ladder:

1. Fix — read it as ctx.get('sessionPersistence').
2. Guard — an e2e that drives session/load through the real Loader.
3. Rule — the postmortem's lesson: "For a service a plugin reads opportunistically but does NOT declare in static inject, use ctx.get(name), never ctx.<name>."
4. Invariant — the mechanical form: a check that flags a direct ctx.<service> property read for any service absent from the reading plugin's inject. Rung three depends on a reviewer noticing; rung four does not.

The general shape, and the one to carry into your own postmortems: the durable invariant is usually a statement about what the test harness must exercise, not about what the code must do. Both of 0001's bugs shared one root process gap — no test exercised the plugin through its real load path or its real call topology. The code fixes were one line each. The process fix is what prevents the next one.

Hands-on lab

The Module 1 spine, under test. Four parts, each building on the last. Everything here is method — implement it with whatever runner and provider you use; nothing depends on a specific tool's flags.

Goal. Three test lanes with three different contracts over the spine.

Create separate configurations for unit, snapshot/replay, and real-API e2e. The unit lane mocks only the model adapter and keeps every downstream component real, and owns a coverage gate. The snapshot lane is keyless by default and never calls a provider. The e2e lane loads credentials, self-skips without them, allows retries for transient flake, and runs no coverage.

Write one test of each tier for a single tool in your spine: a unit test for its argument validation, a snapshot test that the tool's schema appears correctly in the assembled request, and an e2e test that prompts a real model to use it and then verifies the world — re-read the file or re-run the command externally. Never grep the agent's own output.

Success criteria. The unit lane runs with no network. The snapshot lane passes with no API key. The e2e lane self-skips cleanly without a key and passes with one. Deleting the tool from your registry turns all three red for three different reasons.

Stretch. Add a per-file coverage threshold and resolve every failure by deleting dead code rather than adding tests. Record how many lines you removed.

Goal. Model output that can be regression-tested.

Implement record, replay, and refresh for your spine. record calls the real API and writes fixtures plus expected outputs. replay boots the real pipeline from fixtures and diffs against expected outputs, keyless. refresh re-renders from committed fixtures and updates expected outputs without any model call.

Record one multi-turn session with at least one tool call. Then apply the normalization asymmetry: pin the full system prompt and tool schemas in exactly one scenario and tokenize them in every other. Ensure fixtures omit volatile envelopes (timestamps, request IDs) and that replay synthesizes them.

Success criteria. Replay passes offline and is byte-stable across runs. Editing one word of the system prompt churns one line in every scenario except the pinned one. Changing your transcript renderer is fixable with refresh alone. Record and refresh run serially; replay may parallelize.

Stretch. Prove your guard guards: introduce a real regression (drop a field from the assembled request), watch red, revert. If it stayed green, the golden was not covering what you assumed.

Goal. Ten tasks, a layered rubric, and a judge you have measured.

Build a suite of ten tasks with execution-based verification wherever possible. Give each a FAIL_TO_PASS-style assertion (proves the work was done) and a PASS_TO_PASS-style assertion (proves nothing else broke, including "untouched files are byte-identical").

Run each task n = 5 times and report both pass@k and pass^k. For each task, decide and record whether its actions are reversible, and state which metric is honest for it.

Then add an LLM judge for one task that resists execution-based grading, and stress-test it before trusting it:

• Position bias — run every pairwise comparison in both orders and report consistency. Below roughly 65% and your judge's verdicts are noise.
• Verbosity bias — pad a correct answer with redundant restatement that adds no information. If the judge prefers it, it is grading length.
• Reference-guided grading — grade the same set with and without a reference answer and compare error rates.

Success criteria. Every task reports pass^k alongside pass@k, with reversibility recorded. Your judge report includes a position-consistency number. You can name at least one task where the judge disagreed with execution-based grading, and say which was right.

Stretch. Deliberately introduce solution leakage into one task (put the answer in the prompt) and measure the score inflation. Compare your figure to SWE-Bench+'s ~3.1× and consider how you would have detected it without knowing.

Goal. A failure diagnosable from logs alone, ending in a durable invariant.

Instrument the spine with two channels: a ledger channel mirroring session events with a stable (session.id, event.seq) identity, and an ops channel for signals that deliberately omits it. Emission must be non-blocking. Add token accounting per turn and per session, and distinguish provider-reported usage from your own estimate in the type, not in a comment.

Now break something deliberately — truncate tool results at the wrong boundary, or drop one message from the assembled request — and hand the logs to someone else without telling them what you changed. They must diagnose it from logs alone.

Then write the postmortem, following the repository's shape: Executive summary, Summary, Timeline, Root cause, Guardrails, Lessons. Climb the ladder explicitly and label the rungs: fix, guard, rule, invariant. Implement the invariant as a machine-checked property whose violation names the owning component.

Success criteria. Your debugger identifies the bug without re-running the agent. The postmortem's Executive summary is absorbable in thirty seconds. The invariant fires on the original bug and does not fire in normal operation. You can state why it is a real invariant rather than a property true by construction.

Stretch. Adopt the strongest rule in this module: require every component to declare either an invariant or a written explanation of why nothing is checkable, and add a check that mechanically rejects empty declarations without an explanation.

Common pitfalls

1. Treating a benchmark number as a model property. The failure: you read "System X scores 62% on SWE-bench Verified" and conclude something about the model. You have actually read a joint measurement of model, scaffold, test suite, and prompt ordering — Epoch AI measured scaffold alone moving results up to 11 points for GPT-5 and 15 for Kimi K2 Thinking. Fix: never cite a score without its scaffold and subset. Compare systems only under an identical harness, and never compare a Verified number against an original SWE-bench number — different populations, since 68.3% of the original was filtered out.

2. Reporting pass@k for irreversible actions. The failure: your agent issues refunds and you report pass@8 because it looks better. pass@k measures "at least one of 8 attempts succeeded," which for a system that mutates the world means eight refunds were issued. τ-bench names this a category error, and the gap is not cosmetic — GPT-4o's 61.2% pass^1 on retail falls to roughly 25% at pass^8. Fix: classify every task by reversibility first. Irreversible actions get pass^k. And read pass^k as a consistency measurement, not a harder version of the same number.

3. Trusting an LLM judge you have not stress-tested. The failure: you build an eval on judge scores and ship on a 3-point improvement, without knowing your judge's position consistency is 46%. On near-identical answers, measured consistency ranged from 23.8% (Claude-v1) to 65.0% (GPT-4); reordering alone flipped 82.5% of comparisons in one study. Fix: measure position consistency and verbosity susceptibility before trusting any verdict, run every comparison in both orders, and use reference-guided grading wherever a reference exists — it cut a math-grading error rate from 14/20 to 3/20 where chain-of-thought alone only reached 6/20.

4. Testing the self-report instead of the world. The failure: your e2e greps the agent's output for "successfully created" and passes. An agent that hallucinates success and an agent that succeeds are identical to that assertion — and this is the exact class postmortem 0001 describes, where 178 green tests and 100% coverage shipped a bridge that crashed on connect. Fix: re-run the command, re-read the file, diff the database. Assert untouched files are byte-identical. Grade on the world's state, as τ-bench does.

5. Rubber-stamping snapshot updates. The failure: a snapshot fails, you re-record, CI goes green, and you have just approved an unreviewed behavior change. Worse in CI: a job that can write its own goldens cannot fail. Fix: review every fixture and expected-output diff. Force read-only replay in CI. Use refresh (no model call) when only rendering changed, so the diff you review is the diff you caused. And prove each guard fails by introducing its regression on purpose.

6. Merging the audit record and the alerting signal. The failure: one telemetry stream serving both debugging and alerting. You sample it for volume and lose the record; you keep it complete and drown the signal; you sum a best-effort stream and get confidently wrong numbers. Fix: two channels. The ledger carries a stable (session.id, event.seq) identity so receivers can dedupe; ops records deliberately omit it because they are signals to alert on, not entries to sum. And keep emit non-blocking — it runs on the event hot path.

7. Debugging by re-prompting. The failure: the agent misbehaves, you tweak the prompt and re-run, ten times, spending real money to sample a stochastic system. You end up with a prompt that worked once and no understanding. Fix: read the log first. Find the divergence turn, read the assembled request the model actually received, and classify the failure as harness, tool, or model behavior — only the third is a prompting problem. Trust the trace, not the theory: postmortem 0001's elegant explanation was a real mechanism and the wrong bug.

8. Stopping a postmortem at the fix. The failure: "we deleted the bad line and added a test." The bug class remains, and the process gap that let it through is untouched. Fix: climb to the highest enforceable rung — fix, guard, rule, invariant — and prefer invariants that change the type so the ambiguity becomes inexpressible, as postmortem 0004's RunnerFailureRule did. Then check that the guard actually runs in CI: postmortem 0001's most reusable lesson is that a key-gated test is not a guard.

Key terms

• pass@k — probability that at least one of k attempts succeeds: 1 - E_task[C(n-c,k)/C(n,k)]. Honest only when failed attempts are free and reversible.
• pass^k — probability that all k i.i.d. trials succeed, averaged across tasks: E_task[C(c,k)/C(n,k)]. The consistency measure, and the honest one for irreversible actions.
• FAIL_TO_PASS / PASS_TO_PASS — SWE-bench's two-sided adjudication: tests that must newly pass (the fix worked) and tests that must keep passing (nothing else broke).
• Snapshot/replay — record real model output once, then replay it through the real pipeline forever, so model-facing surfaces become diffable.
• Golden / expected output — the committed artifact a replay run diffs against. A guard only guards if the regression actually fails it.
• Position bias — a judge's verdict depending on the order in which candidates are presented. Measure by swapping and reporting consistency.
• Verbosity bias — preferring longer answers that add no information. Note that identical-answer ties are graded correctly, so naive tests miss it.
• Self-preference — a judge favoring its own output; established causally in arXiv:2404.13076 via self-recognition, not by the MT-Bench measurement, which explicitly declines to conclude.
• Reference-guided grading — giving the judge a known-good answer to grade against. The most effective single mitigation for math and reasoning grading.
• Construct validity — whether a benchmark measures what it claims. SWE-bench's 39% trivial tasks and ~50% Django concentration are construct-validity problems, not difficulty problems.
• Solution leakage — the answer being present in the task input. 32.67% of SWE-bench instances, contributing to roughly 3.1× score inflation.
• Ledger channel — telemetry mirroring the session log, carrying (session.id, event.seq) so receivers dedupe. The record.
• Ops channel — telemetry signals deliberately without that identity: things to alert on, not entries to sum.
• Token surface — the ordered set of nodes currently priced into a request. Surface order is authoritative; log order can disagree after compaction.
• Measurement baseline — whether a token figure came from provider-reported usage or a local estimate, encoded in the type (kind: 'usage' | 'estimated' | 'none').
• logRevision — the number of durable events a measurement consumed, making two measurements comparable.
• Durable invariant — a machine-checked property that makes a bug class impossible or loud, attributed to an owning component when violated.
• Extraction ladder — fix → guard → rule → invariant. A postmortem's value is the highest rung it reaches.

Summary

What should have changed in your mental model:

A benchmark score is a joint measurement, not a model property. Model, harness, tests, and prompt ordering all move it, and scaffold alone has been measured moving SWE-bench Verified results by up to 11 points for one system and 15 for another. You are one of those four terms. Read every leaderboard row as a claim about a whole configuration, and refuse to compare across subsets — Verified is a correction of a two-thirds-defective dataset, not a difficulty tier.

Single-number scores are the least informative artifact an evaluation produces. The informative artifacts are distributions and failure taxonomies: pass^k alongside pass@k, per-task reversibility, judge consistency, and a named breakdown of how the failures failed. A team that reports one percentage has thrown away everything it paid to learn.

Nondeterminism is confined to one boundary, and the entire test strategy follows from that. Mock the model for unit tests, record it for snapshot tests, call it for e2e — three tiers proving three genuinely different things, none subsuming another. And 100% line coverage proves lines ran, not that the feature ships: 178 green tests and full coverage shipped a bridge that crashed the moment a real editor connected.

Grade the world, not the self-report. Re-read the file, re-run the command, diff the database, assert untouched files are byte-identical. Every question you can answer by execution is a question a biased judge cannot get wrong — and the judges are measurably biased, by position (down to 23.8% consistency), by verbosity (91.3% fooled by padding), and by self-preference (causally, per arXiv:2404.13076). When you must use one, give it a reference answer.

Telemetry has two audiences, and one stream serves neither. Split the faithful record from the alertable signal, give the record a stable identity so it can be deduped, and deliberately deny the signal one so nobody sums it. Keep emission non-blocking, and type your token figures by provenance so nobody mistakes an estimate for a measurement during an incident.

Debug from the log, not by re-prompting. Find the divergence turn, read the request the model actually received, and classify the failure as harness, tool, or model before forming any hypothesis. Trust the trace, not the theory — the elegant explanation is sometimes a real mechanism attached to the wrong bug.

A postmortem's value is the highest rung it reaches. Fix, guard, rule, invariant. The durable invariant is usually a statement about what the test harness must exercise rather than what the code must do — and it must be one that could actually be violated, running where it will actually run.

That is the course. Eight modules, one spine, and one habit worth more than any single mechanism in it: when someone hands you a number, ask what else was in the measurement.

Further reading

In the dsh repository — read in this order

1. docs/testing.md — the three-tier policy, the with-key stance, "verify the world, not the self-report," and "test the real entry path."
2. vitest.config.ts, vitest.snapshot.config.ts, vitest.e2e.config.ts — the three lanes, and the contract differences between them.
3. docs/subsystems/session-telemetry.md — two channels, the emit() boundary axiom, redaction as a waterfall over the exported copy only.
4. docs/subsystems/token-meter.md — provenance-typed baselines, positional surface pricing, logRevision.
5. docs/subsystems/session-projection.md — pure synchronous folds over committed events, and why an async unit would tear the consistency cut.
6. docs/subsystems/invariants.md — ctx.invariants.register, InvariantError attribution, and the enforced-explanation rule for empty companions.
7. docs/postmortem/README.md plus postmortems 0001 and 0004 — when a postmortem is warranted, and two worked climbs of the extraction ladder.

Per objective

• M8.1 — docs/testing.md tiers section; vitest.config.ts coverage thresholds; packages/core/agent-loop/tests/contract-regressions.spec.ts.
• M8.2 — vitest.snapshot.config.ts mode documentation; scripts/migrate-packed-session-fixtures.ts; docs/testing.md "When a snapshot test is required."
• M8.3 — arXiv:2406.12045 (τ-bench, pass^k); arXiv:2306.05685 (MT-Bench judge biases); arXiv:2404.13076 (self-preference, causal).
• M8.4 — packages/session/session-telemetry/src/index.ts; packages/llm/token-meter/src/types.ts; packages/session/session-stats/.
• M8.5 — docs/postmortem/0001-acp-default-export-drops-inject.md; docs/subsystems/session-projection.md.
• M8.6 — packages/llm/token-meter/src/estimate.ts; packages/llm/llm-deepseek/, packages/llm/llm-pi-ai/; packages/session/session-title-llm/.
• M8.7 — packages/llm/llm-retry/src/index.ts (durable-before-wait, provider-owned policy); vitest.e2e.config.ts retry rationale.
• M8.8 — docs/postmortem/0004-landlock-partial-notice-misclassified-child-failures.md; packages/llm/token-meter/src/invariant.ts as a model empty companion.

External

• Jimenez et al., SWE-bench: Can Language Models Resolve Real-World GitHub Issues? — arXiv:2310.06770 (ICLR 2024). 2,294 instances, 12 repositories; the best system at publication resolved 1.96%.
• SWE-Bench+: Enhanced Coding Benchmark for LLMs — arXiv:2410.06992. Solution leakage and weak tests; roughly 3.1× inflation.
• Wang, Pradel, Liu — arXiv:2503.15223 (ICSE 2026). Differential testing on Verified: 29.6% of plausible patches diverge from ground truth; 6.2 percentage points of inflation.
• OpenAI, Introducing SWE-bench Verified (August 2024). 93 developers, 1,699 samples, 500 retained.
• Epoch AI, SWE-bench Verified analysis — scaffold dependence and construct validity.
• Yao et al., τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains — arXiv:2406.12045 (Sierra AI). pass^k, database-state grading.
• τ²-bench — arXiv:2506.07982. Dual control as a Dec-POMDP; communication dominates reasoning.
• Terminal-Bench — arXiv:2601.11868 (Stanford × Laude). Per-task Dockerised environments; the environment is the unit of work.
• Zheng et al., Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena — arXiv:2306.05685 (NeurIPS 2023 D&B). Position, verbosity, and limited-capability biases; 85% GPT-4/human agreement.
• Panickssery, Bowman, Feng, LLM Evaluators Recognize and Favor Their Own Generations — arXiv:2404.13076 (NeurIPS 2024). Self-preference established causally.
• Wang et al., Large Language Models are not Fair Evaluators — arXiv:2305.17926. Reordering flips 82.5% of comparisons; calibration mitigations.
• A Survey on LLM-as-a-Judge — arXiv:2411.15594. Task-agnostic versus judgment-specific bias taxonomy.
• OpenTelemetry GenAI semantic conventions — standard span and attribute naming for model calls.