Telemetry: traces, spans, tokens, and cost

Testing tells you the harness was correct yesterday, on the cases you thought of. Telemetry tells you what is happening right now, on the cases you didn't. For an agent the natural trace shape follows the loop: a session span containing turn spans containing step spans, each step holding a model request and zero or more tool calls.

Two things distinguish agent telemetry from ordinary service telemetry, and both come from the same fact — the model is expensive and the model is opaque.

Token and cost accounting is not an optimization concern, it is a primary operational metric. A web service is measured in latency and error rate; an agent is measured in latency, error rate, and dollars per completed task, and the third moves by an order of magnitude between a good and a bad prompt.

Telemetry has two audiences with incompatible requirements. One wants a faithful record of what the agent did (auditing, debugging, replay). The other wants aggregate signals to alert on (error rates, latency percentiles). Merging them serves neither: the record gets sampled away and the signal gets buried.

dsh's dsh-session-telemetry splits exactly on that distinction. Every record carries a channel:

interface SessionTelemetryRecord {
  channel: 'ledger' | 'ops'
  time: number
  severity: SessionTelemetrySeverity
  attributes: Record<string, string | number>
  body: unknown
}

Ledger records mirror the session event log — the faithful record. They carry session.id, event.type, and event.seq, and receivers dedupe on (session.id, event.seq). Because they carry a stable identity, a duplicate is harmless and a retry is safe.

Ops records are signals: telemetry.op, session.id, and for agent-error also agent.id, turn, step, error.name. The subsystem doc is explicit about why they deliberately omit the ledger's identity: they "are signals to alert on, not entries to sum." If you could sum them you would be tempted to, and a best-effort at-least-once channel summed as though it were exactly-once produces confidently wrong numbers.

Both channels are best-effort, and the design says so rather than pretending otherwise. The volume-control mechanism is worth studying: only the first assistant/chunk of each (turn, step) ships, because shipping every streamed token would multiply telemetry volume by the length of every response for no additional signal. The consequence is stated up front — "seq gaps are routine on the wire and never a loss signal." An alerting rule that fired on seq gaps would page you constantly and correctly detect nothing.

The seam's boundary axiom is the transferable part:

the harness's aspect ends at emit(); batching, retry, queueing, and loss policy belong to the reporting SDK.

That is a capability-seam decision straight out of Module 7. dsh defines the vocabulary and the emission point; dsh-session-telemetry-otel provides an OpenTelemetry-backed sink. A harness that builds its own batching and retry has built a worse version of something the OTel SDK already ships, and has coupled its hot path to a delivery concern.

Which brings up the hard constraint: emit MUST be non-blocking. It is called synchronously on the session/event hot path. A sink that awaits a network call has just made every agent turn wait for your observability vendor.

measure(session, requestHeader?) returns a TokenMeasurement:

interface TokenMeasurement {
  readonly logRevision: number
  readonly baseline: TokenMeasurementBaseline
  readonly surfaceDeltaTokens: number
  readonly totalTokens: number
  readonly surfaceTokens: number
  readonly nodes: readonly TokenSurfaceNode[]
}

Three design points generalize well beyond dsh.

The baseline is typed by provenance. TokenMeasurementBaseline is a union of { kind: 'none' }, { kind: 'estimated', tokens }, and { kind: 'usage', tokens, usage }. Provider-reported usage and your own heuristic estimate are different kinds of number, and the type system says so. Any consumer that must not confuse a measurement with a guess checks baseline.kind === 'usage'. Harnesses that flatten both into one tokens: number produce dashboards where nobody can tell which bars are real.

Position determines price. nodes is ordered head-to-tail, and each TokenSurfaceNode carries a durable seq and its token count. The doc warns that "surface order is authoritative; replacement nodes can have higher durable seqs than later positional nodes" — after compaction, log order and surface order genuinely disagree, and pricing by log order gives the wrong answer.

logRevision makes a measurement citable. It is "the number of durable events consumed; equal to the next unread event seq." A measurement is a snapshot at a revision, so two measurements are comparable only when you know both revisions. The cost is stated rather than hidden: "Every call clones those positional nodes, so measurement is O(surface)" — do not call it in a per-token loop.

An alert fires: cost per session tripled overnight. With aggregate metrics you know only that. With ledger records plus token measurements you can ask:

1. Did tokens per turn rise, or did turns per session rise? Two completely different bugs — a context-management regression versus a loop that stopped terminating.
2. If tokens per turn rose, nodes locates it positionally: a system prompt that grew, a tool result that stopped being truncated, a compaction that stopped firing.
3. If baseline.kind flipped from 'usage' to 'estimated', your provider stopped returning usage and the "cost increase" may be an estimator artifact rather than real spend.

That third branch is why provenance in the type pays for itself. Without it, the incident is a day spent chasing a cost rise that never happened.

The two-channel split costs a decision at every emission site and a schema harder to explain than one flat stream. It buys the ability to sample ops aggressively while keeping the ledger complete.

Best-effort delivery means the ledger is not an audit log. If you need one, the canonical session event log — not the telemetry export — is the source of truth.

Redaction is where the honesty is sharpest. The redaction waterfall "ships NO rules of its own… exported data is precisely as clean as the rules a deployment mounts." A default-empty redaction policy exports everything: a defensible framework choice and an indefensible deployment. The complementary rule is one to internalize — redaction applies to the exported copy only; the canonical session log is never rewritten. Rewriting history to satisfy an export would corrupt replay, projections, and every audit that depends on the log.

The redaction hook is a waterfall (Module 7's dispatch modes):

'session-telemetry/record'(record: SessionTelemetryRecord, next: () => SessionTelemetryRecord): SessionTelemetryRecord

Waterfall, so each handler transforms and passes on, and multiple redaction plugins compose instead of racing. Source: packages/session/session-telemetry/src/index.ts.

Sharing is a tri-state, SessionTelemetrySharingStatus = 'full' | 'feedback-only' | 'disabled' — richer than a boolean because "the user will share feedback but not transcripts" is a real and common consent state that a boolean forces you to round in one direction or the other.

Sinks implement emit, optional flush, and an async shutdown. The asymmetry is the point: emission is synchronous and non-blocking, teardown is allowed to await.

For derived per-session state, packages/session/session-stats/ contributes a projection over the session-projection seam rather than maintaining a parallel counter. That contract requires a pure synchronous fold over committed events, because "an async unit would tear the carriers' consistency cut" — every client sees the same state as of the same event.

Most harnesses emit one undifferentiated stream and let the backend sort it out, which works until you need to sample. OpenTelemetry's GenAI semantic conventions are converging on standard span and attribute names for model calls — worth adopting for attribute naming even when the record shape is your own, since it is the difference between your traces being queryable by a standard dashboard and not.

Benchmark harnesses instrument almost nothing: a task either passed or it didn't. That is correct for a leaderboard, and it is exactly why leaderboard scores cannot tell you why a system failed — which is the entire content of this module's through-line.