Statelessness: why every turn resends everything

The provider endpoint you call is stateless with respect to your conversation. It does not store your history between requests. Consequently every request carries the entire conversation, from the system prompt to the message you just typed. Turn 20 resends everything sent in turns 1 through 19, plus everything that has accumulated since.

This one fact generates most of the engineering in this course.

Let a conversation have turns t = 1..n, where turn t adds a_t tokens (your message plus the model's reply plus any tool output). The prompt sent at turn t is the sum of everything before it. Total input tokens billed across the session is therefore:

Σ(t=1..n) Σ(i=1..t) a_i      —  quadratic in conversation length

Ten turns of 2,000 tokens each is not 20,000 input tokens; it is about 110,000. Latency follows the same curve, because time-to-first-token grows with prompt length. This is why an agent session feels progressively slower and why the bill for a long debugging session is dominated by input, not output — a fact that surprises people who reason about cost from output length.

Three mitigations exist, and every serious harness uses all three:

1. Prompt caching. Providers will hash a stable prefix and charge a reduced rate for re-reading it. This makes the quadratic term much cheaper, but only if the prefix is byte-identical across requests. Anything that mutates early context — a timestamp in the system prompt, reordered tool schemas, a re-sorted file list — invalidates the cache from that byte onward. Cache discipline is therefore an architectural constraint, not an optimisation: the front of your prompt must be stable.
2. Compaction / summarisation. Replace a prefix of history with a summary once it grows past a threshold. Module 4 covers this in depth.
3. Externalising context. Keep bulk material out of the conversation and let the model fetch it with a tool. Paying 300 tokens for a read_file call beats paying 20,000 tokens every turn for a file the model needed once.

dsh treats the sampling and routing parameters of a call as request-header state precisely because of caching. From packages/llm/llm/src/call-config.ts:

"Provider routing, model, reasoning effort, and sampling values are request-header state that can affect cache reuse; request waterfalls replace them and the loop logs changed snapshots instead of allowing silent per-call drift."

The failure this prevents is subtle and expensive: if any component could tweak temperature or swap the model on an individual call, the prefix would change without anyone noticing, the cache would silently stop hitting, and the only symptom would be a bill. Making the config a logged, compared snapshot — callConfigEquals() in the same file decides whether a proposed config is a real change worth logging or the held one restated — turns an invisible cost regression into an explicit, auditable event.

Aggressive compaction saves money and buys headroom at the cost of fidelity: the model can no longer see the exact error text from turn 3, and it will sometimes re-derive a wrong conclusion you had already corrected. Aggressive caching saves money at the cost of flexibility: you can no longer cheaply personalise the front of the prompt per request. There is no configuration that is simultaneously cheap, complete, and adaptive; pick the two your product needs.

The token accounting convention is that TokenUsage counts are disjoint (packages/llm/llm/src/types.ts), which means billed input is inputTokens + cacheReadTokens + cacheWriteTokens rather than inputTokens alone. This is a deliberate normalisation, and the DeepSeek adapter does the arithmetic to honour it — from packages/llm/llm-deepseek/src/translate.ts:

"DeepSeek's prompt_tokens INCLUDES cache hits (

prompt_tokens = prompt_cache_hit_tokens + prompt_cache_miss_tokens

); the harness TokenUsage convention is DISJOINT counts, so cache reads are subtracted out of inputTokens."

The failure prevented is a cost dashboard that double-counts cached tokens and tells you caching made things worse.

Some provider APIs offer server-side conversation state, letting you send only the delta. That removes the quadratic resend but hands the provider custody of your history, which forecloses offline replay, cross-provider portability, and local redaction. Coding harnesses have broadly declined that trade; they keep the transcript local and pay the resend.