M4.8

Exploit prompt caching

2,856 words · about 18 min

The concept

Concept

Providers cache the prefix of your request. If request N+1 begins with the same tokens as request N, the provider can skip recomputing that prefix, and it charges much less for the cached portion. Given that a harness resends the entire conversation on every step, the potential saving is enormous — most of a long conversation's prompt is byte-identical to the previous step's.

The rule is one sentence: caching is prefix-based, and invalidation starts at the first changed token and runs to the end.

Every practical consequence follows from that sentence.

The ordering rule

Stable content first, volatile content last.

Order your request so that everything unchanging comes first: harness identity, persona, tool schemas, then conversation history oldest-first. Anything that changes between steps goes at the very end. Then normal conversation growth is pure append, and each step's cached prefix is the previous step's entire request.

The naive mistake that destroys everything

Here is the canonical failure. You want the model to know the current time, so you put it in the system prompt:

You are a helpful coding assistant.
Current time: 2026-08-27T14:23:11Z
Working directory: /home/user/project

This looks harmless. It is catastrophic.

The system prompt is the first thing in the request. The timestamp changes on every single step. So the first changed token is near the very beginning of the prompt, and invalidation runs from there to the end — meaning the entire request is invalidated on every step. Your cache hit rate is zero. Not degraded: zero. You are paying full uncached price for a 100,000-token prefix that is 99.99% identical to the last one, forever, because of one timestamp.

The fix is not to remove the timestamp — the model may genuinely need it. The fix is to move it. Put it at the end, after the history, where invalidating from that point forward costs you only the tokens that follow it — which is nothing, because it is last.

This is exactly what dsh's runtime-context mechanism does, and it is why PromptContext exists as a separate registration kind from PromptSection. RuntimeContextProjection in packages/core/agent-loop/src/runtime-context.ts emits volatile context as a trailing user message, and only when it actually changed:

project(current: string, sections: readonly ContextSnapshotSection[]): UserMessage | undefined {
  if (this.retained === undefined && current.length === 0) return
  const snapshot = current.length === 0 ? CLEARED : current
  if (this.retained?.text === snapshot) return
  return createUserMessage({ ... })
}

Two optimisations in four lines. Volatile content is positioned last, so its change invalidates nothing before it. And it is emitted only on change, so an unchanged snapshot appends nothing at all and the request is a pure prefix extension of the previous one. When contexts disappear entirely, a marker states so explicitly rather than silently vanishing:

Current runtime context: none. Earlier runtime-context snapshots no longer apply.

Removing the old message would rewrite history and invalidate the cache; appending a superseding one does not.

Worked example: the arithmetic

Worked example

Assume a 100,000-token conversation, a provider charging cached input at 10% of uncached, and 50 steps.

Timestamp in the system prompt. Every step invalidates everything: 50 × 100,000 = 5,000,000 tokens at full price.

Timestamp as a trailing message. Step 1 is a full miss at 100,000 tokens. Each subsequent step reuses the prior prefix and pays full price only on the delta — say 500 tokens of new content. That is 100,000 + 49 × 500 = 124,500 uncached tokens, plus roughly 49 × 100,000 = 4,900,000 cached tokens at 10%, which costs the equivalent of 490,000 uncached tokens.

Effective cost: about 614,500 versus 5,000,000. Roughly an 8× reduction in prompt-side cost, from a positioning decision.

What else invalidates

dsh requires every package README to declare a KV Cache effect — the requirement is specified in docs/cookbook/adding-a-package.md and enforced by a verification script. The declarations are precise about the conditions:

packages/core/system-prompt/README.md: "Prefix-stable while identity, persona, variables, section text, and order render identically. Any change may invalidate reuse from the first changed system-prompt token."
The same README on tools: "Prefix-stable while the visible schema set, rendering, and order are unchanged. Registration, restriction, or reordering may invalidate reuse from the first changed schema token."
packages/core/agent-loop/README.md: "Ordinary history growth is append-only and preserves reusable entries. A surface replacement or compaction invalidates reuse from the first shadowed history token."

That last one is the cost nobody budgets for: compaction destroys your cache. It replaces a span of early history, so everything from that point forward is invalidated. The step after a compaction pays full price for the entire remaining prompt. That belongs in the ladder's cost accounting — it is another reason pruning-first is right, and it means a harness that compacts aggressively can spend more on cache misses than it saves on tokens.

Two further points from the same source set. Tool ordering matters: the system-prompt package canonicalises tool order via toolOrder rather than letting registration order decide, because a non-deterministic schema order would produce a different prefix on every process start. And dsh exploits the rule in an unexpected place — buildSummarizationInput in region.ts replays the header's system and tools plus the region's messages, with the JSDoc explaining: the summarisation call is "a genuine prefix of the conversation and reuses the provider's KV cache." Even the call that causes invalidation is itself constructed to be a cache hit.

Tradeoffs and honest limits

Tradeoffs

Caching has a TTL, and it is provider-specific and typically short — minutes. An agent that pauses for human approval past the TTL loses the cache regardless of how well-ordered the request is. Some providers offer longer retention at a price; dsh models this in adapter capability flags (supportsLongCacheRetention, cacheControlFormat in packages/llm/llm-pi-ai/src/catalog.ts) rather than assuming a uniform model, because the capabilities genuinely differ.

Provider mechanics differ enough to matter. Some cache automatically on prefix match; some require explicit cache-control markers. Some charge for cache writes; DeepSeek reports no cacheWriteTokens at all through the dsh adapter, while pi-ai reports both reads and writes. A harness must not assume a uniform model, and cache behaviour and eviction remain outside any package's contract — as the repo's own documentation standard says, "provider cache availability and eviction remain outside the package contract."

The honest limit: cache hit rate is a hope, not a guarantee. You can order your request perfectly and still miss because the provider evicted you, because you crossed a TTL, or because a load balancer routed you to a different machine. Design for the saving, but never make correctness depend on it.

Hands-on lab

Extend the Module 1 spine with context management. Work in order — each step depends on the last.

Step 1 — Measure. Write estimateTokens(messages) using the fixed-density heuristic: ceil(text.length / 4) + 4 per block, +4 per message. Report system, tools, and history separately, as three numbers.

Success: your harness prints a three-way breakdown before every request, and the three sum to a plausible total.

Step 2 — Budget. Add a contextWindow config value, a thresholdRatio (default 0.8), and a retainRatio (default 0.16). Compute thresholdTokens and retainTokens. Reject at startup if retainTokens >= thresholdTokens.

Success: a config with retainRatio: 0.9 fails at load with a message naming both values — not at the first compaction.

Step 3 — Prune. Before each request, walk history for tool results over 8,192 characters and replace them with head 4,096 + marker + tail 1,024. Slice by code point (Array.from(text)), not by index. Log tokens reclaimed.

Success: a session that reads three large files drops measurably in token count with no change to conversational structure, and a result containing an emoji survives pruning without producing invalid UTF-8.

Step 4 — The ladder. Implement the pressure check in order: gate on threshold → prune → remeasure → gate on threshold again → only then compact.

Success: a session pushed over threshold by one large tool result is brought back under by pruning alone, and your logs show zero summarisation calls.

Step 5 — Compact. Walk history backwards accumulating tokens until you reach retainTokens, then walk further backwards until the boundary has no unanswered tool call crossing it. Summarise everything before it with a model call whose instruction mandates named sections — the task statement first. Verify the summary is smaller than what it replaced; throw if not.

Success: compaction reduces history below threshold; the retained tail is verbatim; no tool call is ever separated from its result; and a deliberately oversized summary is rejected rather than committed.

Step 6 — Cache ordering. Add a current-timestamp context. Implement it twice: once in the system prompt, once as a trailing message emitted only when changed. Run a ten-step session both ways and record cacheReadTokens from the provider.

Success: you can state the measured hit-rate difference in tokens, and explain why the number is what it is.

Stretch — Spill. Add a maxInlineBytes policy. Results above it write to disk with open(path, 'wx', 0o600) and leave a notice containing the locator and retrieval instructions. Make spill failure non-fatal: a spill error must never turn a successful tool call into an error result.

Common pitfalls

1. No headroom for the next tool result. Budgeting only for the response leaves an agent at 98% occupancy free to call a tool that returns 40KB, blowing the window on the following step with a tool call outstanding. Fix: size headroom as maximum response plus largest plausible tool result. Verify by asking what happens if your biggest tool fires at the highest occupancy your threshold allows.

2. Volatile content in the system prompt. A timestamp, session id, or token counter in the prompt invalidates the cache from near byte zero on every request, taking hit rate to exactly zero. Fix: move volatile content to a trailing message, and emit it only when it changed. Audit by diffing two consecutive request bodies — everything before the first difference is what you are caching, and if that is near zero you have found the bug.

3. Splitting a tool call from its result. A compaction boundary placed by token count alone can put an assistant tool-call message inside the summarised region and its result outside, producing a malformed request. Fix: snap the boundary to a pairing-balanced cut, walking backwards only. Test with a session ending mid-tool-call.

4. Compacting before pruning. Summarising first spends a model call and loses reasoning to reclaim tokens that a string operation would have reclaimed for free. Fix: prune, remeasure, re-gate, and only then compact. Assert in tests that a prune-satisfiable session performs zero summarisation calls.

5. Releasing the compaction lock before the commit. Appending compaction/end before the replacement lands means a crash in between leaves a record claiming compaction finished when it did not. Fix: append the start event synchronously before any await, and the end event last. Test by killing the process mid-summarisation and confirming the session recovers by detecting an orphaned lock.

6. Trusting the estimate as exact. Four characters per token is an average. Base64, minified JSON, and CJK text price very differently, and a harness that treats the estimate as authoritative will overflow while reporting 70% occupancy. Fix: anchor to provider-reported usage where available, keep the heuristic for the unsent frontier, and size headroom to absorb the error.

7. Letting cross-session memory grow unbounded. Instruction files accumulate because nobody deletes from them; eventually they consume the window they exist to use well. Fix: enforce a byte budget across all discovered files, state explicitly when content was truncated, and review the files as you would any other code.

8. Context management that fails the turn. A compactor that throws and aborts the turn has converted a possibly-survivable situation into a definite failure. Fix: log and continue; let the reactive overflow path handle it; let the provider's error reach the user if all else fails. Failing honestly is the fourth rung of the ladder, and it beats failing for the wrong reason.

Key terms

Context window — the maximum tokens in one request, re-spent from zero on every step.
Headroom — reserved budget covering the model's response plus the next tool result.
Threshold ratio — the occupancy fraction at which pressure handling triggers (dsh default 0.8).
Retain ratio / retain tokens — the recent history never summarised (dsh default 0.16 of the window). Must be strictly less than the threshold or compaction cannot converge.
Escalation ladder — prune → compact → spill → fail, climbed only as far as necessary.
Pruning — replacing the middle of an oversized tool result with a marker, keeping head and tail.
Compaction — replacing a span of history with a model-generated summary.
Spill — storing an artefact outside the context window and leaving a retrievable handle.
Surface — the projection of the session log that the model actually sees; compaction and pruning mutate the surface, never the log.
Shadowing — hiding events from the surface while retaining them in the log.
Tool-pairing balance — the property that no unanswered tool call crosses a given cut.
Locator — an opaque branded handle to spilled content, paired with a retrieval hint.
KV cache / prompt cache — provider-side reuse of a request prefix; invalidated from the first changed token to the end.
Prefix stability — the property that consecutive requests share a byte-identical opening span.
Runtime context snapshot — volatile facts emitted as a trailing message rather than in the system prompt.
Memory tiers — in-turn context, session persistence, cross-session memory files.

Summary

The Module 1 spine treated context as unlimited, and its only failure mode was total. You now have four graduated responses instead of one cliff.

Five things should have changed in your mental model.

The window is a recurring budget with four categories, one of which grows. You can price all four before sending. Headroom is not slack — it is a reservation covering the response and the next tool result, and omitting it produces failures at the worst moment.

The ladder is ordered by cost and by what gets destroyed. Prune before compact, always, because pruning is a free string operation that discards mechanical tool output while compaction costs a model call and discards reasoning. dsh encodes this as prune → remeasure → re-gate → compact, and the second gate means a well-pruned session may never summarise at all.

Compaction is a transaction, not a transformation. The lock is appended synchronously before any await and released last, so a crash leaves a detectable orphaned lock rather than a false claim of success. The summary rides on an ordinary message with a surface replacement, so the original stays in the log — model-visible content shrinks while the record does not.

Fidelity verification is genuinely unsolved. What dsh checks is convergence — that the summary is smaller — not fidelity. The real protections are structural: prescribed sections with the task statement first, mandated verbatim identifiers, an explicit rule against summarising a prior summary, and the log that keeps the original for a human. Anyone claiming to verify summary fidelity automatically should be asked how.

Cache ordering is free money you can throw away with one line. Stable first, volatile last. A timestamp in the system prompt takes your hit rate to zero and costs roughly 8× on prompt-side tokens. And compaction invalidates the cache from the first shadowed token — a cost that belongs in the ladder's accounting alongside the summarisation call itself.

Module 5 takes up the question this module kept deferring: several mechanisms here — spill writing files at content-derived paths, instruction files arriving from a possibly-untrusted repository — are attack surfaces, and the precedence rule saying workspace instructions never override system instructions is the first appearance of a boundary that module makes explicit.