Certified Agent Harness Engineer
Course sections

M1.4

The four standing responsibilities

828 words · about 5 min

Concept

Regardless of language, vendor, or topology, a harness is permanently accountable for four things. Naming them separately is useful because each fails in its own way and each has its own test strategy.

  1. Composing context — deciding what the model sees this step.
  2. Executing effects — turning tool calls into real actions.
  3. Enforcing policy — deciding what is allowed, and asking when unsure.
  4. Surviving failure — behaving correctly when the model, the network, the tool, or the user misbehaves.

Mechanism

Composing context answers: which system prompt sections, which history, which tool schemas, what got truncated? The failure mode is the silent one — nobody notices that a system prompt section stopped being included, and the agent just gets subtly worse. The defence is making assembly explicit and inspectable rather than string-concatenated at the call site.

Executing effects answers: how do arguments become an action, and how does the outcome become an observation? The failure modes are unbounded output (40,000 lines of test log evicting the entire conversation), unbounded time (a tool that never returns), and lost correlation — a result that cannot be matched to its call. Every tool result must cite the id of the call it answers; without it, a parallel step produces a transcript the model cannot interpret and the provider will often reject.

Enforcing policy answers: is this call permitted, and who decides? The failure mode is obvious and catastrophic. The subtle version is policy drift — a check that exists in one code path and not another. The defence is a single choke point that every call passes through.

Surviving failure answers: what happens on a 429, a malformed tool call, a context-length error, a mid-tool interrupt? The failure mode is a loop that spins — retrying a permanently-failing call forever, burning money — or one that dies and loses work. Two invariants matter most: cancellation must be cooperative (propagated, not merely ignored), and the transcript must remain valid after a failure, because a conversation containing a tool call with no matching result is often unrecoverable.

Worked example

dsh shows all four in one code path. packages/core/agent-loop/src/tool-calls.ts schedules a step's tool calls, and the module docstring is a compressed statement of responsibilities 2 and 4:

"Abort records synthetic error results for skipped calls so replay stays valid. A terminal scheduler failure preserves already-recorded tool/call events without fabricating results."

Concretely, when a turn is cancelled mid-flight:

function appendSkippedToolCall(session: Session, turn: number, step: number, block: ToolCallBlock): void {
  const callSeq = appendToolCall(session, turn, step, block)
  appendToolResult(session, turn, step, block, {
    content: [{ type: 'text', text: 'Error: tool call aborted before dispatch' }],
    isError: true,
    error: { message: 'tool call aborted before dispatch',
             info: { name: 'AbortError', code: TOOL_ABORTED_BEFORE_DISPATCH } },
  }, callSeq)
}

The harness manufactures a result for a call that never ran. That looks like lying to the model until you see the failure it prevents: a resumed session whose history contains a tool call with no result is a malformed conversation. The synthetic result keeps the transcript replayable, and it tells the truth about what happened. Note also that appendToolResult links the result to its call via sourceEventSeqs: [callSeq] — correlation is structural, not inferred.

Argument parsing shows the same defensive instinct:

function parseArguments(raw: string): unknown {
  try {
    return raw ? JSON.parse(raw) : {}
  } catch {
    return raw
  }
}

Malformed JSON from the model is not a crash. It is preserved as text and passed down to the tool layer, which will reject it and report a usable error the model can correct on the next step. A JSON.parse that threw into the loop would take the whole turn down over one bad token.

Tradeoffs

Centralising all four responsibilities in the loop makes them auditable and makes the loop large. Distributing them into plugins keeps the loop small at the cost of a harder question — "what actually ran?" — which is why any harness that distributes them needs a strong event log to answer it.

How `dsh` does it

How dsh does it

Two conventions from AGENTS.md carry the weight. "Plugins, not loop changes" keeps the loop from accreting features. "Model-visible ⟺ logged" is the invariant tying context composition to persistence: anything the model can see must exist as an event in the session log, and model history is derived from the log rather than stored alongside it. The failure prevented is divergence — a UI, a transcript, and a model request that disagree about what happened.

How others differ

Claude Code exposes hooks (PreToolUse, PostToolUse) so external code can participate in responsibilities 2 and 3 without forking; policy lives in settings files with allow/deny rules. Codex CLI leans harder on sandboxing — enforcing policy through OS-level confinement rather than per-call adjudication. Both choices are defensible: adjudication is precise but prompt-fatiguing; confinement is coarse but unbypassable. Module 5 examines the trade in detail.


Check yourself

4 questions on M1.4

Start drill