Module 8 — Evaluation and operations

Seven modules have built a spine and then bent it into new shapes: a loop, tools, context, permissions, subagents, extension seams. Every one of those modules asked you to change the harness. This one asks a different question, and it is the one that decides whether any of that work was real: how do you know it works, and what do you do at 3am?

The uncomfortable answer is that the field's default instrument — a benchmark percentage — mostly does not tell you. A number like "62% on SWE-bench Verified" is not a property of a model. It is a joint measurement of the model and the harness that scaffolds it and the tests that adjudicate it and the order in which the prompt happened to arrive, and any of those four can move it by double digits without a single weight changing. That is not a cynical framing; it is the published finding of the people who maintain those benchmarks. Epoch AI, re-running SWE-bench Verified across scaffolds, reports that "the choice of scaffold has the single biggest impact on the overall performance" — up to 11 points for GPT-5 and up to 15 points for Kimi K2 Thinking on the same model weights.

You are the harness. You are one of the four terms in that measurement, and after this module you should never again read a leaderboard row as though you weren't.

So this module is about building the instruments that do tell you something: a testing pyramid that survives nondeterminism, snapshot/replay so model output can be regression-tested at all, eval suites with rubrics whose judge you have actually stress-tested, telemetry dense enough that a failure is debuggable from logs alone, cost and latency controls you can reason about, and the operational discipline — rollout, failover, incident response — that turns a postmortem into a durable invariant instead of a memory.

The through-line: a single-number score is the least informative artifact these evaluations produce. Everything below is about producing the more informative ones.

---

Objectives

• M8.1 Build the testing pyramid for a nondeterministic system: unit, snapshot/replay, real-API e2e.
• M8.2 Design snapshot/replay testing so model output can be regression-tested at all.
• M8.3 Write agent evals: task suites, rubrics, LLM-as-judge and its failure modes.
• M8.4 Instrument telemetry — traces, spans, token and cost accounting per turn and per session.
• M8.5 Debug a misbehaving agent methodically from session logs rather than by re-prompting.
• M8.6 Manage cost and latency: model routing, effort tiers, caching, and batching.
• M8.7 Operate in production: rollout, provider outages and failover, versioning, incident response.
• M8.8 Read a postmortem and extract a durable invariant.

Prerequisites and time

Modules 1–7. You need the Module 1 spine running, the tool loop from Module 3, the session event log from Module 4, the permission layer from Module 5, and at least one capability seam from Module 7 — the lab puts all of them under test at once. Budget 5 hours: roughly 2 hours reading, 3 hours on the lab.

---