Certified Agent Harness Engineer

Concept

Security decays under feature pressure. Each capability is added by someone focused on making it work, and the boundary everything above depends on is exactly what is easy to route around by accident. The countermeasure is a checklist applied to every new capability, so safety review is a process step rather than an act of individual vigilance.

Mechanism: the checklist

Mechanism

Trust boundaries. Where does untrusted input enter, and is it validated there? AGENTS.md enumerates them: "parser/config, queued, model/tool JSON, durable/file, worker, process, and wire." Model/tool JSON is a boundary — the model's arguments are untrusted input, always.

Fail-closed. For every error path — provider missing, timeout, malformed response, abort — is the outcome deny? Write out the failure matrix. An unavailable outcome a caller treats as success is the bug.

Path handling. Canonicalized before containment? Both root and target canonicalized the same way? Symlinks handled explicitly with a no-follow check at deletion? Platform behaviour tested (junctions, case-insensitivity, /private/tmp)?

Process spawning. Scrubbed environment? Secrets out of argv? Temp files private, random-named, exclusively opened? Confined if the deployment expects confinement, and failing closed if it cannot be?

Model visibility. Does anything reaching the model carry secrets, host paths that leak layout, or internal state? Conversely, per "Model-visible ⟺ logged": is everything the model sees on the durable record? An unlogged model-visible fact is invisible to incident response.

Loud misconfiguration. "Misconfiguration fails loud." Does a wrong config throw at load, or silently disable enforcement? Silent disablement is the worst failure mode in this module — the system reports healthy and enforces nothing.

Grant scope. Does any new grant widen along an axis the user did not choose? Is it revocable and enumerable?

Invariant stability. "Protocol constants, external specs, and security invariants stay fixed." A test failing because it asserts a security property is evidence of a regression, never an invitation to update the assertion.

Worked example — reviewing a new capability

Worked example

Suppose you add an HTTP-request tool:

Trust boundary: the URL is model-controlled. Validate the scheme (reject file:, gopher:), reject internal ranges and the cloud metadata endpoint (169.254.169.254) — and re-check after DNS resolution and after every redirect, because rebinding and redirect chains each defeat a check done only on the original URL. Note the shape: canonicalize-then-contain again, in a different namespace.
Fail-closed: unresolvable host, timeout, TLS failure → error, not empty success.
Model visibility: the response body is untrusted content — mark provenance (M5.6); keep request headers out of errors (M5.7).
Secrets: does it forward any ambient credential or cookie jar? If so it is an exfiltration primitive — the model chooses the destination, the harness supplies the authorization.
Grant scope: per-host approval, or all-or-nothing? All-or-nothing pushes users to "all."
Logging: URL and status logged; body under the size and redaction policy.

Ten minutes, and it catches the SSRF, the header leak, and the exfiltration primitive.

Tradeoffs

Checklists become ritual. A team ticking eight boxes without thinking has a compliance artifact, not a control. The mitigation is keeping each item tied to the failure it prevents — which is why docs/defensive-patterns.md is written as six named bug classes with the incident behind each rather than abstract rules. Rules get skipped; scar tissue does not.

How `dsh` does it

How dsh does it

dsh makes several items structural rather than advisory: the tool pipeline is generated from code, so the ordering of permission and sandbox steps is documented from reality; the pipeline enforces stage order with invariants that fail loudly ("tools/execute must follow tools/pre-execute"); presets throw at load when composed over a non-confining executor; confine() throws rather than passing through; duplicate service registration throws instead of silently replacing.

How others differ

Most projects encode this as a SECURITY.md plus review culture. dsh's choice — a short numbered list of bug classes with their incidents, plus hard invariants in AGENTS.md — trades comprehensiveness for the property that every rule is memorable and traceable to a real failure. It will not cover a novel class until that class has bitten someone, which is the honest cost.

Hands-on lab — securing the Module 1 spine

Work in order; each step depends on the last.

1. Write the threat model (20 min). One page. Enumerate every channel by which non-user bytes reach context in your spine. For each: who controls it, what an attacker gains, what bounds the damage today. You will find a channel you had not considered.

2. Add the permission seam (45 min). Insert a decision point between tool-call parsing and execution returning a closed union of allow/deny/ask. Requirements: no rule matched → deny; approver missing or throwing → deny; an exhaustiveness assertion the type-checker enforces. Log the decision before executing.

3. Add approval with correct scope (30 min). Implement ask so a positive outcome authorizes exactly one call. Deliberately do not implement "always allow" yet.

4. Feel the fatigue (15 min). Run a task touching ten-plus files. Count prompts. This step is the point of the lab — the number is the argument.

5. Fix it three ways (45 min). (a) Presets: one posture decision replacing per-call ones. (b) Pattern grants: offer "allow this pattern" showing the literal pattern text, and make the offered pattern the narrowest that removes the repetition. (c) Structural: make workspace reads need no prompt, and justify why in one sentence. Re-run step 4 and compare.

6. Filesystem fence (40 min). Canonicalize target and root; contain; re-canonicalize immediately before the syscall. Verify against ../ traversal, an absolute path outside the root, a symlink inside the workspace pointing out, and a path whose parent is a symlink. Reads pass; writes are contained. Write down the residual TOCTOU risk in the style of packages/fs/fs-sandbox/README.md.

7. Environment scrub (20 min). Scrub credential-shaped names from child environments, merging explicit caller entries after the scrub. Verify by running env through your shell tool and grepping the result for your key.

8. Untrusted-content marker (30 min). Wrap file-read and fetch results in a provenance fence naming the source and denying authority. Escape the closing delimiter in the content, and write a test using a payload that tries to close the fence early. Then write two sentences on why this is mitigation, not a solution.

9. Re-exploit (25 min). Re-run the M5.1 chain with a repo file containing an instruction addressed to the agent. Document which control stopped it — or, more instructively, how far it got.

Success criteria. No path from tool call to execution bypasses the decision point. Every error path denies. All four filesystem attacks are blocked and the residual risk is written down. A key in the parent environment does not appear in env output. The fence-breakout test passes. Prompt count on the step-4 task drops materially with no grant widening beyond what the user was shown. The step-9 injection either fails or produces a written account of the missing control.

Common pitfalls

1. Fail-open on the error path. The check is right, and catch (e) { return ALLOW } sits three lines below because the author was thinking about robustness rather than security. Fix: a closed union with no default-allow, an exhaustiveness assertion, and a unit test asserting deny for every failure mode. dsh normalizes a throwing or rogue answerer to 'unavailable', which callers treat as deny.

2. Lexical path containment. child.startsWith(root) before resolution — defeated by .., by symlinks, and by the near-miss /workspace-evil matching a /workspace prefix. Fix: canonicalize both sides, then compare with a segment-aware containment helper, never a raw string prefix.

3. Scope creep in a grant. The user approves npm test; the implementation records "allow npm," and npm publish now runs silently. Fix: the recorded pattern must be exactly what was displayed — add a test asserting the stored grant equals the shown one.

4. Security state as an in-memory flag. Approval policy in a variable cannot be audited afterward, survives restarts inconsistently, and disagrees with the log. Fix: derive it from the event log. dsh's effectiveApprovalPolicy(events) is a pure reverse fold — replaying the log is the state.

5. Silent unconfined fallback. The sandbox binary is missing, so the code logs a warning and runs the command anyway. The system reports healthy and enforces nothing — the worst outcome here, because everything above assumed confinement holds. Fix: throw. "Silent unconfined passthrough is never legal."

6. Two copies of one boundary. The fs fence computes writable roots one way, the shell sandbox another. They agree today and diverge at the next change, and nobody is watching the divergence. Fix: one shared function — dsh derives both from a single writableRoots.

7. Trusting a version check instead of probing. "Kernel ≥ 5.13, therefore Landlock works" — but a container or hardened kernel can expose the syscalls and refuse enforcement. Fix: probe functionally, report actual enforcement, and collapse "missing binary" and "unenforcing kernel" into one degradation path so only one path needs testing.

8. Re-rendering arguments in the approval prompt. The prompt formats arguments a second time, the rendering drifts from what executes, and the user approves something that is not what runs. Fix: reference the already-displayed call by id. dsh's ApprovalRequest deliberately carries no arguments.

9. Treating a filter as a boundary. Scanning tool output for "ignore previous instructions" and calling injection handled. The attacker rephrases. Fix: filters are telemetry; least authority is the boundary.

Key terms

Allow / deny / ask — the three-outcome vocabulary of a permission decision. Approval fatigue — degradation of decision quality from excessive prompting; converts a security control into a rubber stamp. Canonicalize-then-contain — resolve a path fully through the filesystem, then test containment against a likewise-canonicalized root. Confused deputy — a privileged component induced to misuse its authority on behalf of a less-privileged party; the model's structural role in every harness. Enforcement completeness — a reported fact (full/partial) about how much of a requested policy the platform actually applied. Fail-closed — an undetermined or errored decision resolves to deny. Landlock — unprivileged Linux LSM providing path-based, allow-list filesystem confinement, inherited across execve. Least authority — grant the minimum capability sufficient for the task; the only defence that holds when the model's judgment fails. Monotonic guard — a check that can only deny or abstain, so no ordering of guards can undo a denial. No-follow check — metadata inspection (lstat) that does not traverse a symlink, used to reject link-shaped paths at a trust boundary. One-shot grant — authorization for exactly one call, not recorded as standing policy. Preset — a named bundle of independent enforcement knobs, chosen once instead of per-action. Prompt injection — instructions embedded in content that reaches the model as data and are acted on as if authorized. Read/write asymmetry — permitting broad reads while narrowly confining writes, justified by reversibility and blast radius. Scrubbed environment — a child environment with credential-shaped names removed before explicit entries merge in. Seatbelt — macOS sandbox policy engine invoked via the deprecated sandbox-exec CLI; allow-default with targeted denies. TOCTOU — time-of-check-to-time-of-use; the window between validating a path and using it. Trust boundary — a point where data crosses from a less-trusted domain into a more-trusted one and must be validated.

Summary

The Module 1 spine has no decision point between "the model said this" and "the OS did this," and this module reconstructs one. Start from the threat surface (M5.1): injection through any content channel, over-broad file access, irreversible commands, exfiltration — with the model as confused deputy rather than adversary. The permission model (M5.2) is a total function to allow/deny/ask that must fail closed and resolve conflicts by a stated order, deny > ask > allow, where abstention is never allowance. Approval flows (M5.3) must fight fatigue, because a prompt firing constantly is worse than none — mitigated by presets, correctly narrow pattern scoping, and above all structural elimination, since a workspace write under an enforced sandbox needs no prompt. Filesystem policy (M5.4) is canonicalize-then-contain with a real symlink and TOCTOU story, applied asymmetrically: reads broad, writes narrow. Sandboxes (M5.5) enforce below the confined code, and each has a named escape surface — Landlock's filesystem-only scope and partial ABIs, bubblewrap's shared kernel and /proc magic links, Seatbelt's allow-default posture and deprecated CLI, Windows' retained Everyone SID and hard links, containers' shared kernel, remote sandboxes' control plane — and all must fail closed. Untrusted content (M5.6) is the unsolved one: tool output is data, never instructions, but a single token stream leaves no parser to enforce it, so design for injection succeeding and bound the damage. Credentials (M5.7) travel as references, never values, with each of six leak paths closed deliberately. M5.8 makes all of it a checklist applied to every new capability, because a boundary nobody re-checks is a boundary someone will route around.

Defensive patterns as a review checklist