Certified Agent Harness Engineer
Course sections

M3.3

Defensive validation and errors that teach

904 words · about 6 min

Concept

Arguments arrive from a probabilistic text generator. They will be wrong: missing required keys, wrong types, string "3" for a number, extra invented keys, and occasionally not JSON at all. The harness's job is to reject them precisely and instructively, because the error message is the only training signal the model receives inside a session.

An error message has one job: make the corrected retry obvious. "Invalid arguments" fails that test. "invalid arguments: /file_path: expected string, got number" passes it.

Mechanism

dsh validates before the body runs. Inside defineTool, the wrapped execute validates first and throws a typed error:

// packages/core/tools/src/schema.ts (defineTool)
async execute(args: unknown, exec: ToolRunContext): Promise<JsonValue> {
  const violations = validate(args)
  if (violations.length > 0) throw new ToolArgsError(violations)
  return userExecute(args as InferArgs<S>, exec) as Promise<JsonValue>
}

ToolArgsError extends HarnessError and carries structure, not just prose:

// packages/core/tools/src/schema.ts
export class ToolArgsError extends HarnessError {
  readonly violations: string[]
  constructor(violations: string[]) {
    super(`invalid arguments: ${violations.join('; ')}`, 'INVALID_ARGS')
    this.name = 'ToolArgsError'
    this.violations = violations
  }
}

Three properties make this work. Violations are path-qualified — validateArgs returns "path-qualified violations" in schema-walk order, so the model is told which field is wrong, not merely that something is. All violations are reported at once, joined with ; , so a model with two wrong fields fixes both on one retry instead of ping-ponging. And the error carries a stable machine code INVALID_ARGS alongside the human message, so policy and telemetry route on the code while the model reads the prose.

That last property generalizes. Every structured dsh tool failure surfaces as isError: true with error.info = { name, code }. Module 2 showed { name: 'AbortError', code: TOOL_ABORTED_BEFORE_DISPATCH } and { name: 'ToolTimeoutError', code: TOOL_TIMEOUT }. An unknown tool maps to UNKNOWN_TOOL; an invalid canonical return value throws ToolOutputError (INVALID_TOOL_OUTPUT). Consumers route on the code; models read the message. A harness that only produces strings forces every downstream policy to string-match, and every message improvement then becomes a breaking change.

Worked example — an error that teaches vs one that doesn't

Worked example

dsh's filesystem package has a purpose-built module for exactly this, and its module docstring states the principle outright: the provider's FS_STALE_VERSION and FS_NOT_OBSERVED messages "state the condition but not the only correct recovery."

// packages/fs/tool-fs/src/error.ts
const REMEDIES: Partial<Record<FsErrorCode, string>> = {
  FS_STALE_VERSION: 're-read the file, then retry',
  FS_NOT_OBSERVED: 'read the file, then retry',
}

export function remediateFsError(error: unknown): unknown {
  if (!(error instanceof FsError)) return error
  const remedy = REMEDIES[error.code]
  if (!remedy) return error
  return new FsError(`${error.message} — ${remedy}`, error.code, { cause: error })
}

Study the layering, because it is the correct general pattern. The provider's message stays "machine-oriented and unchanged." The remedy is appended at the model boundary, in the tool package. The FsError code is preserved so retry, permission, and UI layers keep routing on it, and the original chains as cause. Anything unrecognized passes through untouched.

So the model receives "the file changed since this session last observed it — re-read the file, then retry" instead of FS_STALE_VERSION. The first produces a correct next action; the second produces a guess.

Tradeoffs

Verbose errors cost tokens, and they cost them on the failure path where the model is already burning a turn. Extremely detailed validation output — dumping the full schema on every mismatch — can exceed the size of a successful result. The calibration: name the path, the expectation, and what was received; do not paste the schema. Also resist over-coercion. Silently converting "3" to 3 makes the current call succeed and teaches the model nothing, so it keeps emitting strings; worse, coercion rules that differ between tools produce inconsistent behavior the model cannot learn. Coerce only where the ambiguity is genuinely harmless.

How `dsh` does it

How dsh does it

Validation is generated from the same declaration as the type and the wire schema, so it cannot drift. defineTool validates types, required keys, literal constraints, exact-one unions, and nested values before execute runs. What it deliberately does not do is check semantics the DSL cannot express — non-empty strings, positive numbers, cross-field rules — and the authoring reference says so explicitly: "You still hand-check constraints the DSL does not express." Knowing where the generated guarantee stops is part of the contract.

There is one subtle and instructive asymmetry. The same validator is used for display presenters, but with the opposite failure mode:

// packages/core/tools/src/schema.ts
// Presentation is display-only and may run on REPLAY of arbitrary logged args
// (possibly from an older schema), so it must never throw: validate softly and
// fall back to `undefined` (a generic UI presentation) on any mismatch, rather
// than the hard `ToolArgsError` the execute path raises.

Execution is strict because a wrong call must not run. Display is soft because a schema change must not crash a replay of last month's session. Same validator, two policies, chosen per consequence.

How others differ

Claude Code's Edit tool enforces a validation rule that is a semantic precondition rather than a type check: the target string must be unique in the file, and the call fails if it is not. It also enforces a read-before-write precondition. dsh implements the same class of guarantee through fs/* event gates (@deepseek-ai/dsh-fs-observation-policy) rather than inside the tool, which is why the remediation lives in a separate boundary module. Both harnesses concluded that the expensive filesystem failures are preconditions, not types.


Check yourself

3 questions on M3.3

Start drill