Operating in production

Everything above assumes a stable target. Production does not supply one: providers deprecate models, have outages, silently change behavior, and rate-limit you at the worst possible moment. Four disciplines follow.

Rollout. A prompt change is a code change with a much larger blast radius, because it changes behavior across every task simultaneously with no compiler to catch it. It deserves the same staged treatment: canary, monitor, expand.

Failover. Providers go down. The question is what your harness does mid-turn, and there is a real distinction between retrying a request (safe — the model is stateless) and retrying a turn that has already executed tools (not safe — the tools already ran, and re-running them is the irreversibility problem from M8.3 arriving in production).

Versioning. "The agent" is a composition: model version, prompt version, tool schemas, harness version. Any of them can change independently, and a bug report without all four is not actionable.

Incident response. At 3am you need to know what changed, what the blast radius is, and how to stop the bleeding. Sequence: stop the bleeding, then diagnose. Rolling back a prompt takes seconds; understanding why it broke takes hours; doing them in the wrong order costs you the difference.

packages/llm/llm-retry/ is instructive because of one property stated in its own module doc: "Each scheduled retry is durable before its cancellable wait."

Unpack that. Retry state is written to the session log before the backoff wait begins. So a process that dies mid-backoff can resume knowing a retry was pending, rather than either losing the turn or double-issuing it. Retry is treated as a state machine over durable state, not as a for loop with a sleep — and every operational concern about crash safety and cancellation follows from that framing.

The package also declines a responsibility, which is the more transferable lesson. Its Config is empty, and passing retryPolicy to it is a hard error:

if (key === 'retryPolicy') {
  throw new Error('llm-retry: retryPolicy belongs under each provider configuration')
}

Retry policy belongs to the provider, because rate limits, backoff windows, and which errors are retryable are provider facts. A single global retry policy is wrong for every provider except the one it was tuned against. Note also that the validator rejects unknown keys rather than ignoring them — a typo in a retry setting fails at startup instead of silently disabling itself.

The e2e lane's retry: 2 is the pragmatic counterpart, with an honest comment: real model calls hit shared-key concurrency quotas, and a retry there is for transient flake, not for correctness. Retries in tests hide flakiness; retries in production hide outages. Both are load-bearing, and both must be visible in telemetry — otherwise a provider degrades for a week while your dashboards stay green.

1. Version the prompt and put its identifier in every ledger record's attributes. Without this you cannot answer "which sessions ran the new prompt."
2. Run the eval suite offline first. Reference-guided where a judge is involved; report pass^k and the failure taxonomy, not one number.
3. Canary a fraction of traffic. Watch error rate, tokens per turn, tool-call distribution, and completion rate — not just error rate. A prompt regression frequently shows up as a different tool mix long before it shows up as errors.
4. Keep the rollback one config change, and rehearse it. A rollback path that has never been executed is a hypothesis.

Failover to a second provider means your agent behaves differently during an incident: different model, different tool-calling quirks, different failure modes. You have traded a hard outage for a soft, harder-to-detect degradation. Sometimes right, never free, and it must be visible — an unannounced silent failover is an incident that lasts until someone notices the quality drop.

Canarying agents is harder than canarying services because the signal is noisier. A 5% canary on a stochastic system may take a long time to distinguish a real regression from variance, and a change that is unambiguously bad across 5 offline eval tasks can be statistically invisible on 5% of live traffic for a day.

Version pinning fights model deprecation. Pin, and you eventually get a forced migration on the provider's schedule; float, and you get silent behavior changes on the provider's schedule. Both are bad; pinning at least makes the change an event you can plan around.

Configuration is fail-loud at startup. The invariants service validates its own config and rejects contradictions rather than resolving them silently, with a documented precedence — blocklist overrides allowlist. Cordis config errors surface at load, not at first use, which is the difference between a deployment that refuses to start and one that runs for six hours before hitting the broken path.

Snapshot goldens double as a provider-change detector: when a provider's behavior shifts, record mode produces a diff you must review — a mandatory reading of what changed, rather than an alert you may or may not notice.

Managed agent platforms handle failover and version pinning for you, and take the model-choice decision with them. Self-hosted harnesses own both and must build the retry, failover, and versioning machinery themselves — which is why llm-retry's durable-before-the-wait property is worth copying rather than re-deriving during an incident.