Certified Agent Harness Engineer
Course sections

M6.7

Durable and long-running work

848 words · about 5 min

Concept

Four categories, distinguished by exactly one question: what survives a process restart?

  • In-turn work. Lives inside a single agent turn. Survives nothing. The parent blocks; a crash loses it.
  • Background jobs. Outlive the turn that started them, within the process. Survive turn boundaries, not process death.
  • Workflows. Scripted orchestration with their own budget and lifecycle. In dsh, a workflow blocks the parent turn and does not survive restart.
  • Scheduled work. Fires on a time rule. Whether it survives restart depends entirely on the delivery model, and the honest answer is usually "less than you think."

Getting this wrong produces the most expensive class of agent bug: work the user believes is happening and that is not. A background job whose process was restarted is not "delayed"; it is gone, and unless something reports that, the user waits forever.

Mechanism: ownership, concurrency, and delivery guarantees

Mechanism

Jobs (packages/jobs/jobs/src/index.ts, local implementation in packages/jobs/jobs-local) model this with an explicit registry. JobKindMap today has bash and subagent — a background subagent is a job, unified with background shell work under one lifecycle. JobStatus is running | stopping | completed | killed | failed; JobStart carries { kind, label, outputLimitBytes?, owner?, run(): JobHooks }; JobHooks is { cancel(reason?), done: Promise<JobOutcome>, readOutput?() }.

Two properties are worth internalizing. Concurrency is per-owner — maxConcurrentJobsPerOwner defaults to 10, so one runaway session cannot starve every other session. And the registry "refuses work while no attached job controller serves the spec's owner," which means orphan jobs are impossible by construction rather than by cleanup. Settlement is first-wins: a job that is both cancelled and completing races once and records one outcome.

Scheduling (docs/subsystems/schedule.md) is where the honesty is sharpest. ScheduleRecord is After | At | Every, with every_seconds floored at 300. The delivery mode is ScheduleDeliveryMode = 'session-local' — and the docs state the consequences plainly: no cold-session scheduler and no external notification channel. If the session is not live, the schedule does not fire. Delivery is described as "best-effort at-least-once rather than exactly-once."

The firing discipline is worth copying: due work waits for full idle, claims the maintenance phase, and queues exactly one followup(). It never calls steer(). A timer is not permitted to interrupt the user's work — it waits its turn.

Worked example — the overnight job that wasn't

Worked example

A user asks for a large migration and the agent starts it as a background subagent job, replying "running in the background, I'll report when done." The user closes the terminal.

What survives: nothing. The job registry is process-local; the controller detaches; the job is gone. There is no notification channel to tell the user.

The correct designs, in order of increasing cost: (a) tell the user truthfully that the work lives only while the session does; (b) make the child write progress into the workspace, so a later session can read where it got to; (c) push the work to a real durable queue outside the harness. Note that (b) is the pattern the Ralph loop formalizes in M6.8 — the workspace is the durable store, because the filesystem survives what the process does not.

Tradeoffs

Durability costs storage, resume complexity, and a genuinely harder correctness argument (a resumed job must not repeat side effects). Session-local delivery is vastly simpler and gives up unattended operation. Per-owner concurrency caps trade throughput for fairness. And the deepest tradeoff: every durability guarantee you make becomes a promise the user relies on, so an over-claimed guarantee is worse than an honestly-absent one.

How `dsh` does it

How dsh does it

Alongside jobs and the session-local scheduler, dsh handles one subtle interaction between forking and durable state. Because a fork folds only events at or after SessionHeader.seedLength, the child retains the parent's history without adopting the parent's active reminders. A fork does not inherit its parent's timers. Two independently reasonable features would, composed naively, have duplicated every scheduled item on every fork; the seed boundary prevents it.

The plan subsystem (docs/subsystems/plan.md) contributes the opposite lesson. ctx.planMode exposes plan/mode as a log-only whole-value-replace event with a pure foldPlanMode(events, end?) reducer, and set() returns one of 'committed' | 'queued' | 'cancelled' | 'noop'. But the docs are blunt: "Plan mode is soft guidance." It shapes the prompt; it does not enforce. Compare packages/todo/tool-todo, whose README is equally blunt: "The list belongs to the ONE agent session that called the tool. There is no subagent/shared/swarm scope." A todo list is not a coordination mechanism between agents, and every attempt to use it as one is a bug.

How others differ

Most harnesses treat background work as a shell-level concern and provide no durable job model at all. The strongest contrast is ACP's inversion: under the Agent Client Protocol the agent is a subprocess of the editor, so the editor owns the lifetime and the agent's durability question becomes the editor's. Worth distinguishing three protocols students routinely conflate: MCP connects an agent to tools; ACP connects an editor to an agent; A2A connects agents to each other as peers. Different layers, different trust models, not substitutes.


Check yourself

1 question on M6.7

Start drill