Classifying and recovering from failures

The model request fails constantly in production: 429s, 500s, connection resets, provider-side timeouts, context overflows, and empty completions. If your loop treats all of these as "error, end the turn," it is unusable. If it retries all of them, it will retry a context-overflow forever — the request that overflowed will overflow again identically.

Recovery requires classification. Classification requires that failures arrive as structured data, not as an exception whose message you regex.

dsh normalizes every provider failure to one type (docs/subsystems/llm-streaming.md):

interface LlmFailure {
  message: string
  code: string
  status?: number
  providerRetryAfterMs?: number
  requestId?: string
}

There are exactly two sanctioned error paths, both carrying this one type: an adapter may throw from stream(), or it may deliver a terminal finish chunk with kind: 'error' or kind: 'aborted' and a failure. Two paths rather than one because a failure before the stream opens and a failure mid-stream are genuinely different situations; one type rather than two because the classifier should not care which happened.

Three canonicalizations matter:

• Context overflow has exactly one code. Every adapter maps its provider's wording to CONTEXT_WINDOW_EXCEEDED via isContextWindowExceededError(). Provider drift — OpenAI, Anthropic, and DeepSeek all phrase this differently, and each changes wording over time — is absorbed in the adapter. A compaction plugin listens for the one canonical code and never sees a provider string. docs/agent-lifecycle.md notes that dsh-compaction-basic uses agent/request-error only for canonical context overflow.
• An empty completion is a retryable error, not a silent success. It maps to EMPTY_RESPONSE and is retried by default. The failure this prevents: a harness that accepts an empty completion as a valid answer produces a turn that ended with the agent saying nothing, which is indistinguishable to the user from a bug.
• One adapter call is one provider attempt. Adapters disable their SDK's built-in retries. Otherwise retry policy is split between a library you do not control and a policy you do, and the observable retry count is a product of two numbers.

Transport stalls are handled below the classifier: streamIdleTimeoutMs (a positive finite value, five-minute default) arms a watchdog only while the iterator's next() is outstanding, maps its own expiry to TIMEOUT, and preserves an earlier caller abort as ABORTED. Arming only during an outstanding next() is the detail — a watchdog armed across the whole stream would fire on a slow consumer rather than a stalled provider.

Malformed tool calls are a separate class, handled in packages/core/agent-loop/src/tool-calls.ts:

function parseArguments(raw: string): unknown {
  try { return raw ? JSON.parse(raw) : {} } catch { return raw }
}

Three behaviors in three lines: empty input becomes {} (models routinely emit '' for a no-argument tool), valid JSON parses, and invalid JSON is preserved as the raw text rather than throwing. The tool's own schema validation then rejects it and returns a normal error result the model can read and correct. The failure prevented: throwing on unparseable arguments would kill the turn over a model typo, when the model is entirely capable of fixing it if told.

packages/llm/llm-retry/src/index.ts registers on agent/request-error. Its module doc states the invariant: each scheduled retry is durable before its cancellable wait. The mechanism:

1. Recover the current retry count by scanning the session log for the last llm/retry event matching (turn, step, provider, policyKey). Retry state lives in the log, not in memory. A process that dies during a backoff and resumes does not restart the retry count from zero.
2. Compute the delay: exponential, initialDelayMs * 2 ** min(retry - 1, 1024) capped at maxDelayMs, multiplied by jitter 1 - jitterRatio + 2 * jitterRatio * random(). Jitter exists to prevent a fleet of agents that all got 429'd from retrying in lockstep and re-creating the overload.
3. Honor failure.providerRetryAfterMs when finite, positive, and within maxDelayMs. If the provider asks for longer than the cap, a normal-mode policy gives up (delegating downstream) while an always-mode policy falls back to its local delay.
4. Append llm/retry before the cancellable wait, then llm/retry-started after it, then return { kind: 'retry' }.

Step 4's ordering is the durability point. Logging before the wait means a crash during a four-minute backoff leaves evidence that a retry was scheduled; logging after would lose it. And the wait is cancellable — cancellableDelay resolves false on abort — so Ctrl-C during a backoff is immediate rather than blocking for four minutes.

Two policy modes exist: normal (finite maxRetries, default five, with an explicit retryableCodes allowlist) and always (unbounded, same backoff shape, for deployments that would rather wait than fail). The plugin itself has empty config — providers own their retryPolicy — and passing retryPolicy to the plugin throws with a message telling you where it belongs. That is fail-loud misconfiguration again: a silently ignored retryPolicy would look configured and behave unconfigured.

In the loop, the recovery is a waterfall. On finish.kind === 'error' | 'aborted', step() runs agent/request-error and continues the inner retry loop only when the returned action is { kind: 'retry' }; otherwise it throws LlmError. Recovery is delegated to policy; the loop only knows "retry" or "give up."

Log-durable retry state costs log volume and couples retry to the session — a retry outside a session has nowhere to record itself. In-memory counters are simpler and lose everything on restart. Which is right depends on whether resumption matters; for a harness whose central promise is --resume, it does.

The always mode is genuinely dangerous and the repo does not pretend otherwise: an unbounded retry against a permanently failing provider is an infinite loop with backoff. It exists for unattended deployments where failing is worse than waiting, and it should be a deliberate choice.

Most harnesses retry in memory with a fixed count and jittered exponential backoff — the same math, none of the durability. Some retry inside the provider SDK, which dsh explicitly forbids so that the retry count is observable. The distinctive dsh choices are (a) durable retry events and (b) unifying request recovery and compaction behind the same agent/request-error extension point, so "the context overflowed, compact and retry" and "the provider 429'd, wait and retry" are the same waterfall with different listeners. The cost is that ordering between those listeners becomes a configuration concern.