M3.3
Defensive validation and errors that teach
904 words · about 6 min
Concept
Arguments arrive from a probabilistic text generator. They will be wrong: missing
required keys, wrong types, string "3" for a number, extra invented keys, and
occasionally not JSON at all. The harness's job is to reject them precisely and
instructively, because the error message is the only training signal the model
receives inside a session.
An error message has one job: make the corrected retry obvious. "Invalid arguments" fails that test. "invalid arguments: /file_path: expected string, got number" passes it.
Mechanism
dsh validates before the body runs. Inside defineTool, the wrapped execute
validates first and throws a typed error:
// packages/core/tools/src/schema.ts (defineTool)
async execute(args: unknown, exec: ToolRunContext): Promise<JsonValue> {
const violations = validate(args)
if (violations.length > 0) throw new ToolArgsError(violations)
return userExecute(args as InferArgs<S>, exec) as Promise<JsonValue>
}ToolArgsError extends HarnessError and carries structure, not just prose:
// packages/core/tools/src/schema.ts
export class ToolArgsError extends HarnessError {
readonly violations: string[]
constructor(violations: string[]) {
super(`invalid arguments: ${violations.join('; ')}`, 'INVALID_ARGS')
this.name = 'ToolArgsError'
this.violations = violations
}
}Three properties make this work. Violations are path-qualified — validateArgs
returns "path-qualified violations" in schema-walk order, so the model is told
which field is wrong, not merely that something is. All violations are
reported at once, joined with ; , so a model with two wrong fields fixes both
on one retry instead of ping-ponging. And the error carries a stable machine code
INVALID_ARGS alongside the human message, so policy and telemetry route on the
code while the model reads the prose.
That last property generalizes. Every structured dsh tool failure surfaces as
isError: true with error.info = { name, code }. Module 2 showed
{ name: 'AbortError', code: TOOL_ABORTED_BEFORE_DISPATCH } and
{ name: 'ToolTimeoutError', code: TOOL_TIMEOUT }. An unknown tool maps to
UNKNOWN_TOOL; an invalid canonical return value throws ToolOutputError
(INVALID_TOOL_OUTPUT). Consumers route on the code; models read the message.
A harness that only produces strings forces every downstream policy to
string-match, and every message improvement then becomes a breaking change.
Worked example — an error that teaches vs one that doesn't
Worked exampledsh's filesystem package has a purpose-built module for exactly this, and its
module docstring states the principle outright: the provider's FS_STALE_VERSION
and FS_NOT_OBSERVED messages "state the condition but not the only correct
recovery."
// packages/fs/tool-fs/src/error.ts
const REMEDIES: Partial<Record<FsErrorCode, string>> = {
FS_STALE_VERSION: 're-read the file, then retry',
FS_NOT_OBSERVED: 'read the file, then retry',
}
export function remediateFsError(error: unknown): unknown {
if (!(error instanceof FsError)) return error
const remedy = REMEDIES[error.code]
if (!remedy) return error
return new FsError(`${error.message} — ${remedy}`, error.code, { cause: error })
}Study the layering, because it is the correct general pattern. The provider's
message stays "machine-oriented and unchanged." The remedy is appended at the
model boundary, in the tool package. The FsError code is preserved so retry,
permission, and UI layers keep routing on it, and the original chains as cause.
Anything unrecognized passes through untouched.
So the model receives "the file changed since this session last observed it —
re-read the file, then retry" instead of FS_STALE_VERSION. The first produces a
correct next action; the second produces a guess.
Tradeoffs
Verbose errors cost tokens, and they cost them on the failure path where the model
is already burning a turn. Extremely detailed validation output — dumping the full
schema on every mismatch — can exceed the size of a successful result. The
calibration: name the path, the expectation, and what was received; do not paste
the schema. Also resist over-coercion. Silently converting "3" to 3 makes the
current call succeed and teaches the model nothing, so it keeps emitting strings;
worse, coercion rules that differ between tools produce inconsistent behavior the
model cannot learn. Coerce only where the ambiguity is genuinely harmless.
How `dsh` does it
How dsh does itValidation is generated from the same declaration as the type and the wire schema,
so it cannot drift. defineTool validates types, required keys, literal
constraints, exact-one unions, and nested values before execute runs. What it
deliberately does not do is check semantics the DSL cannot express — non-empty
strings, positive numbers, cross-field rules — and the authoring reference says so
explicitly: "You still hand-check constraints the DSL does not express." Knowing
where the generated guarantee stops is part of the contract.
There is one subtle and instructive asymmetry. The same validator is used for display presenters, but with the opposite failure mode:
// packages/core/tools/src/schema.ts
// Presentation is display-only and may run on REPLAY of arbitrary logged args
// (possibly from an older schema), so it must never throw: validate softly and
// fall back to `undefined` (a generic UI presentation) on any mismatch, rather
// than the hard `ToolArgsError` the execute path raises.Execution is strict because a wrong call must not run. Display is soft because a schema change must not crash a replay of last month's session. Same validator, two policies, chosen per consequence.
How others differ
Claude Code's Edit tool enforces a validation rule that is a semantic
precondition rather than a type check: the target string must be unique in the
file, and the call fails if it is not. It also enforces a read-before-write
precondition. dsh implements the same class of guarantee through fs/* event
gates (@deepseek-ai/dsh-fs-observation-policy) rather than inside the tool,
which is why the remediation lives in a separate boundary module. Both harnesses
concluded that the expensive filesystem failures are preconditions, not types.
Check yourself
3 questions on M3.3