M3.9

Testing tools

2,305 words · about 14 min

Concept

Tools are the part of a harness that touches the real world, so they are where mocks lie most. A test suite that mocks the filesystem, mocks the subprocess, and asserts on the agent's own claim of success will pass while the product is broken.

Mechanism — the tiers

Mechanism

dsh (docs/testing.md) runs five:

Tier	Command	Purpose
Unit	`pnpm run test`	Vitest over `tests/**`; every registry gets an HMR-safety test
Coverage gate	`pnpm run test:coverage`	Per-file 100% on `packages///src`
Real-API e2e	`pnpm run test:e2e`	Self-skips without provider keys
Snapshot	`pnpm run test:snapshot`	Recorded transcripts; `:record` / `:refresh`
Web browser snapshot	`pnpm run test:web`	Required Linux PR gate, `DSH_SNAPSHOT=replay`

The coverage gate is per-file and absolute, with a rationale worth quoting: "an uncovered line is often dead code the gate is correctly flagging for deletion." And the necessary caveat: "line coverage is necessary, never sufficient."

Worked example — three testing doctrines

Worked example

"Prefer the real implementation over a mock." A bash-tool test does not stub the shell; the harness helper plugs in the real local shell plugin plus the real bash tool and runs an actual echo. This catches the entire class of bugs where the mock and the real backend disagree about argument handling.

"Verify the world, not the self-report." An end-to-end assertion re-reads the target file from disk with an independent call and checks its contents. It does not grep the agent's output for the word "done." A keyword probe on the agent's own narration lets a cheating — or merely optimistic — agent pass. This is the single most important idea in the module for anyone who has ever shipped a green suite over a broken feature.

"Test the real entry path." Exercise the tool through the registry, not by calling execute directly. Calling execute bypasses argument validation, pre-execute policy, guards, the timeout wrapper, output-schema validation, and finalizeContent — which is to say, it bypasses nearly everything M3.4 described. A test that calls the body directly can be fully green while the tool is unreachable, unauthorized, or producing invalid output in production.

The suite in packages/core/tools/tests/ maps onto the objectives directly: schema.spec.ts and json-schema.spec.ts (M3.2/M3.3), execution-mode.spec.ts (M3.6), execution-signal-types.spec.ts (cancellation), scoped.spec.ts (agent scoping), invariant.spec.ts (identity protection), and gen-tool-catalog.spec.ts — which regenerates docs/tool-catalog.md by booting each plugin and harvesting schemas(), making the published catalog a test artifact rather than documentation someone has to remember to update.

Tradeoffs

Real-implementation tests are slower and platform-sensitive; a test that really spawns bash will not run unmodified on Windows. Real-API tests cost money and are non-deterministic, which is why they self-skip without keys — and why dsh states a with-key policy of "inference is cheap here": the failure mode of skipping a test you meant to run is worse than the token cost. Snapshot tests are the most efficient way to cover a rendering surface and the easiest to refresh into meaninglessness; a refreshed snapshot that nobody reviewed asserts nothing.

How others differ

Aider's per-model edit-format selection is effectively a benchmark harness treated as a test tier — the format choice is empirical, so the evaluation must be continuous. Module 8 develops this into evaluation operations. For now the point is narrower: for tools, the correctness question ("did the file change?") is mechanically checkable, and you should check it mechanically rather than asking the model.

Hands-on lab — from if/else dispatch to a real registry

The Module 1 spine dispatched tools like this:

// Module 1 spine — naive dispatch
async function runTool(name, args) {
  if (name === 'read')  return await fs.readFile(args.path, 'utf8')
  if (name === 'write') return await fs.writeFile(args.path, args.content)
  if (name === 'bash')  return (await exec(args.command)).stdout
  throw new Error('unknown tool: ' + name)
}

Five defects, one per lab step: no schema validation, no structured errors, no result bounds, no permission seam, and no separation between the canonical value and what the model reads.

Lab 1 — Build the registry and validate arguments. Replace the if/else with a Map from tool name to definition, where each definition carries { name, description, parameters, execute }. Write defineTool(def) that wraps execute so it validates args against parameters first and throws a typed ToolArgsError carrying an array of path-qualified violations. Report all violations at once, not the first. Add a schemas() method that projects only { name, description, parameters } — assert in a test that execute never appears in the projection. Success: calling read with {} returns a model-visible error naming file_path, and the loop continues rather than crashing.

Lab 2 — Structured errors. Give every failure a stable machine code alongside its human message: INVALID_ARGS, UNKNOWN_TOOL, TOOL_TIMEOUT. Normalize anything the body throws into a failure carrying { message, info: { name, code } } under an isError: true discriminant. Then add a remediation layer at the model boundary, in the shape of remediateFsError: a code-to-remedy map that appends the correct recovery sentence and preserves the code. Success: an unknown tool name produces UNKNOWN_TOOL and the loop continues; a stale-write error tells the model to re-read and retry.

Lab 3 — Result shaping. Give read offset and limit parameters defaulting to 1 and 2000, prefix every line N: , and append the three distinguishable footers from formatReadOutput — capped, continuing, and end-of-file. Add a per-line character cap and a total byte cap. Truncate bash output to its tail and say so. Success: reading a 10,000-line file returns a bounded result whose footer names the exact offset for the next call, and the model successfully paginates without being told how.

Lab 4 — The permission seam. Insert a preExecute(execution) hook between resolution and execution, returning an allow, deny (with a reason), or ask decision. Ship a default implementation that returns allow for everything, and make ask degrade to deny when no approval handler is mounted. Then add a separate guard list whose members return string | undefined — a denial reason or abstention, with no allow result — and run guards after the hook. Prove the monotonicity property with a test: register a permissive hook and a denying guard in either order, and the call is denied both times. Success: the seam exists, denials become model-visible errors rather than exceptions, and Module 5 can fill in real policy without touching the registry.

Lab 5 — Value/content split and concurrency classification. Have execute return a canonical JSON value, and add a pure render(args, value) that projects it to model-visible text. Add isConcurrencySafe?(args) to the definition, default missing predicates to exclusive, and group consecutive parallel calls with a bound of 10 while treating exclusive calls as ordering barriers. Preserve model order in the appended results regardless of completion order. Success: three read calls overlap; a write between two reads forces a barrier; and the transcript is byte-identical across runs.

Common pitfalls

Serializing the whole tool object into the request. Symptom: mysterious token growth, provider errors about unknown schema fields, internal timeouts and concurrency hints leaking to the model. Fix: project the model-facing schema through an explicit allowlist of { name, description, parameters } and test the projection.
Throwing on unparseable arguments. Symptom: one malformed emission kills the session. Fix: parse defensively — fall back to the raw string, let schema validation produce a model-readable error, and keep the loop alive. Every malformed input becomes a message, never a crash.
Errors that state the condition but not the recovery. Symptom: the model retries the identical failing call two or three times, then gives up or invents a workaround. Fix: append the correct next action at the model boundary while preserving the machine code for policy and telemetry.
Truncating silently. Symptom: the agent confidently reports a clean build from the visible tail of a log whose error scrolled off. Fix: every capped result says it was capped, says how much is missing, and names the exact call that retrieves more.
Assuming read-only implies parallel-safe. Symptom: intermittent subprocess exhaustion or non-reproducible flakiness under load. Fix: default to exclusive; opt in per tool and per arguments; remember that spawning a process is interference even when nothing is mutated.
Permission checks that a later hook can reverse. Symptom: a plugin reordering silently disables a security control. Fix: make the final authorization stage monotonic — give guards no way to express "allow," so denial cannot be undone by ordering.
A mode parameter instead of separate tools. Symptom: the model invents mode strings, and a payload object whose valid shape the schema cannot describe. Fix: if a parameter's validity depends on another parameter's value, split the tool. If the set is genuinely closed and shallow, use an enum.
Testing execute directly. Symptom: a fully green suite over a tool that is unregistered, unauthorized, or emitting output that fails schema validation in production. Fix: test through the registry entry path, and assert on the world rather than the agent's self-report.

Key terms

Tool schema — the model-facing projection of a tool: { name, description, parameters }, where parameters is a JSON Schema object. Built by an explicit allowlist so host-only fields never reach the provider.
Tool definition — the host-side object: schema plus execute, output, finalizeContent, timeoutMs, isConcurrencySafe, and presenters.
Canonical value — the typed JSON a tool's body returns; execution-local, validated against output.schema, and deliberately omitted from durable events.
Content — the rendered blocks the model actually reads, produced by a pure render(args, value).
Waterfall — a reorderable listener chain where each listener may transform, short-circuit, or delegate via next(). dsh runs three around tool execution: tools/pre-execute, tools/execute, tools/post-execute.
Guard — a post-hook authorization stage returning a denial reason or undefined. Has no allow result, which makes the security boundary monotonic.
Monotonic authority — the property that a later stage can only narrow permission, never widen it; the reason listener order cannot defeat a denial.
Structured error — a failure carrying both a human message and a stable machine code (INVALID_ARGS, UNKNOWN_TOOL, TOOL_TIMEOUT, INVALID_TOOL_OUTPUT). Consumers route on the code; models read the message.
Remediation — appending the correct recovery action to a provider error at the model boundary while preserving its code and chaining the original as cause.
Line-number/offset idiom — returning N: text lines plus a footer naming the exact offset for the next page.
Spill — saving a capped result's full data and reporting its path, converting a hard cap into a soft one.
Execution mode — parallel or exclusive; exclusive is the fail-closed default and acts as an ordering barrier in the scheduler.
MCP / ACP / A2A — agent-to-tools, agent-to-editor, and agent-to-agent protocols respectively.
Deferred tool loading — advertising a tool's name without its full schema and fetching the schema on demand, to keep the cached prompt prefix stable.

Summary — what changed in your mental model

You began this module thinking of a tool as a function the model calls. You should end it thinking of a tool as a small, permanent piece of documentation with an executable attachment. The schema is re-read on every request forever; the error message is the only in-session training signal; the result is a durable occupant of the context window that speaks in the user's voice.

Four ideas should have moved:

Schema design is the product. Naming, description economy, flat parameters, enums, and honest defaults determine first-call accuracy more than any loop improvement. Every clause of a description should be traceable to a specific failure it prevents, and every clause costs tokens forever — that tension is the craft.

Errors are an interface, not an exception path. A structured failure carries a stable code for machines and a remedy sentence for the model. The remedy belongs at the model boundary, not in the provider.

Bounds are correctness, not optimization. Truncation caps, spill paths, continuation footers, and pagination coordinates are what let an agent work on a repository larger than its context window. A cap that does not announce itself is a correctness bug.

Authority must be monotonic and fail closed. No predicate means exclusive. No approval service means deny. No allow result on a guard. Every one of those defaults costs latency or convenience and buys the property that a reordering, a missing plugin, or a wrong assumption cannot silently widen what an agent is permitted to do.

Module 4 takes the context window this module has been carefully filling and asks how to manage it deliberately. Module 5 fills in the permission seam Lab 4 left open.