Cost and latency: routing, effort, caching, batching

Four levers, ordered by how much they change while you pull them:

• Model routing — a cheap model for easy work, an expensive one for hard work.
• Effort tiers — same model, more or less reasoning per request.
• Caching — do not pay twice for the same prefix.
• Batching — trade latency for throughput on work nobody is waiting for.

The unifying insight: not every model call in an agent loop is the same kind of call. Summarizing a tool result, deciding which tool to call next, and writing the final patch have wildly different difficulty and wildly different consequences for getting them wrong. Paying the same price for all three is a choice, and usually not the right one.

Routing needs a classifier, and the classifier is the problem. If it is a model call, you have added latency and cost to every request in order to save on some. If it is a heuristic, it will misroute. If it is per-call-site — "titling uses the cheap model, the main loop uses the strong one" — it is static, boring, and works, which is why dsh's structure points that way: packages/session/session-title-llm/ and its siblings exist as separate plugins precisely so session titling is configured independently of the agent's model.

Effort tiers are the highest-leverage lever with the least code. The same request at low effort may cost a fraction of high effort, and for the many mechanical steps in an agent loop, high effort buys nothing. The trap is evaluating a tier change on pass^1 — a lower tier often preserves average quality while degrading consistency, which is exactly the property pass^1 cannot see and pass^k can.

Caching is where agent loops have unusual structure in your favor: every turn resends the entire conversation prefix, and that prefix is stable and append-only, so prompt caching pays disproportionately. The constraint is that a cache hit requires a byte-identical prefix, which puts a real cost on anything that mutates history — a timestamp in the system prompt, a reordered tool schema, or a compaction that rewrites the middle invalidates everything downstream. Compaction from Module 4 is genuinely in tension with caching, and there is no clever resolution: you are trading a context saving against a cache saving and must measure which is larger for your workload.

Batching trades latency for throughput and is unavailable for interactive turns by definition. Where it applies is offline: eval suites, bulk classification, backfills. Running your eval suite through a batch API rather than an interactive one is often the difference between running it nightly and running it monthly.

You route session titling to a cheaper model. To know whether it worked:

1. Measure the baseline per call site, not per session — logRevision makes measurements citable and comparable.
2. Check baseline.kind. If the cheap provider does not return usage, your comparison is estimate-versus-measurement and the delta is partly an artifact.
3. Evaluate quality with pass^k, not pass^1. Titling that is right 95% of the time and catastrophically wrong 5% may be worse than the expensive model at 99%, depending on what a bad title costs you.
4. Watch latency at the tail. A cheaper model with a worse p99 can make the whole session feel slower even at a better mean.

Every lever adds a failure mode. Routing means two providers to keep working and two response shapes to handle. Caching creates a hidden coupling between prompt construction and cost that will confuse whoever next edits the system prompt. Batching means a second code path with different error semantics.

The measurement problem is genuine: these levers interact. Routing changes which prefix gets cached; effort tiers change token counts, which change compaction timing, which changes cache hit rates. Changing two at once and attributing the delta to either is how teams end up with folklore instead of data.

Metering and pricing policy are separated. packages/llm/token-meter/ measures; it does not know your rate card. estimateMessage(message: Message): number gives a heuristic count, and the union-typed baseline keeps heuristic and provider-reported figures distinguishable at the type level. Splitting measurement from pricing means a rate-card change touches no measurement code — and, more importantly, means the numbers you compare across providers were computed the same way.

Provider selection is plugin composition rather than a config switch: packages/llm/llm-deepseek/, packages/llm/llm-pi-ai/. Routing is a wiring decision in cordis.yml, not a branch in the loop — Module 7's argument applied to cost.

Many harnesses expose a single model setting and leave routing to the user. Some frameworks build automatic routers with a classifier model in front — powerful, and it adds a model call plus a new failure mode to the critical path of every request. The static per-call-site approach gives up adaptivity and gains predictability, which for a cost lever is usually the better trade.

---