Skip to content
Night Owls.dev
Jump to a page
Concept · observability

Can you see what every agent did, when, and at what cost? Yes — down to the individual model call, opt-in.

Every generation a native engine runs is measured at the source: one swarm.usage event per model call — which agent, which model, the token breakdown, the priced cost, and a stable generationId — and one swarm.turn_usage per segment that sums them. You wire nothing and pay nothing for it. The moment you attach an observer, a table, the ready-made sink, or a telemetry exporter, that stream becomes an on-demand audit trail of the whole run. This page is the coverage, the four recording paths, and the honest gaps.

swarm.usageswarm.turn_usageopt-in

What the framework measures, at the source

Metering is not bolted on after the fact — it is emitted by the engine as the loop runs. Every model call produces a swarm.usage event attributed to the agent that made it (the orchestrator, or a specific delegate), and every segment seals exactly one swarm.turn_usage that sums the segment's spend with a per-agent bySlug split. Both are plain domain types, so nothing about consuming them couples you to the engine.

the usage events (@nightowlsdev/core)
// Two usage events carry the whole audit trail. Both are plain domain types — no @mastra in the shape.

// 1) ONE swarm.usage per model generation, attributed to the agent that made the call.
{
  type: "swarm.usage",
  data: {
    slug: "researcher",            // WHICH agent generated (the orchestrator, or a delegate)
    modelId: "claude-sonnet-4-5",  // WHAT model actually ran (after tier routing)
    breakdown: { inputTokens, outputTokens, /* cache/reasoning classes… */ },
    cost: { usd: 0.0123 },         // the METERED (token-priced) cost — see "metered vs billed" below
    generationId: "run_abc:0:2",   // STABLE key: <runId>:<segmentIndex>:<within-segment #> — retry-safe, unique
  },
}

// 2) ONE swarm.turn_usage per segment (the billing unit for the turn), summing every generation in it —
//    the orchestrator AND every delegate — with a per-agent bySlug split.
{
  type: "swarm.turn_usage",
  data: {
    breakdown, cost,               // the segment total
    bySlug: [{ slug: "researcher", breakdown, cost }, /* … */ ],
  },
}

// 3) swarm.external_usage — non-LLM external spend (e.g. a paid tool/provider), flushed BEFORE the
//    turn_usage that itemizes it, so a segment's ledger is complete and correctly ordered.

Why generationId is the point. It is <runId>:<segmentIndex>:<within-segment #> — stable across retries and unique per model call. That is what makes the audit trail exact: recording keyed on it is idempotent (no double-count when a delegate's usage bubbles up, no double-insert on a realtime replay or a durable resume), and it is the join key to reconcile the framework's metered cost against a provider's post-hoc billed amount later.

Four opt-in ways to record it — pick by where the data should land

PathWireUse when
1 · Synchronous observerdefineSwarm({ onEvent })Any custom destination; the lowest-level hook. Sees every event as it happens.
2 · Ready-made ledgersupabaseUsageSink({ pool, table })A Supabase/Postgres usage table, idempotent on generationId, no code to write.
3 · Durable event logstorage.events (a StorageAdapter)Replay/query the WHOLE run (not just usage) later, or stream it live over Realtime.
4 · Traces + metric seriesdefineSwarm({ telemetry }) · @nightowlsdev/metricsAn APM/OTel backend, or a separate bucketed metric-series plane with compare().

They compose — a durable event log and the sink and telemetry is a normal setup. And they are all genuinely opt-in: a swarm that wires none of them is byte-identical and carries no metering overhead beyond emitting events no one is listening to.

Path 1 — the onEvent observer (and the opt-in switch)

onEvent is the synchronous seam: a function that receives every SwarmEvent with its SwarmContext. Filter for swarm.usage and you have the per-call record the instant a generation completes. This is where "let the user see what was done and when" is switched on.

lib/swarm.ts
import { defineSwarm } from "@nightowlsdev/core";

// PATH 1 — the synchronous observer. onEvent sees EVERY event, including every swarm.usage the instant a
// generation completes. This is the opt-in switch: no onEvent ⇒ nothing recorded, zero overhead. A throwing
// observer is isolated (the engine swallows it so a sink hiccup never breaks a run) — so log your own failures.
const swarm = defineSwarm({
  /* … model, agents, tools … */
  onEvent: (ev, ctx) => {
    if (ev.type === "swarm.usage") {
      audit.record({
        at: new Date(), runId: ctx.runId, tenant: ctx.tenantId,
        agent: ev.data.slug, model: ev.data.modelId,
        tokens: ev.data.breakdown, usd: ev.data.cost.usd,
        generationId: ev.data.generationId,   // idempotency key for exactly-once
      });
    }
  },
});

Path 2 — the ready-made ledger (supabaseUsageSink)

If the destination is a Postgres table, you don't write the observer at all. supabaseUsageSink is an onEvent handler that upserts each swarm.usage into your table, keyed idempotently by generationId — so a retry or a realtime replay inserts at most once, and delegation never double-counts.

lib/swarm.ts
import { supabaseUsageSink } from "@nightowlsdev/storage-supabase";

// PATH 2 — the ready-made ledger. supabaseUsageSink IS an onEvent handler that upserts every swarm.usage into
// a host table, keyed idempotently by generationId ("on conflict (generation_id) do nothing"), so a realtime
// replay or a task retry inserts at most once. No OTel pipeline, no double-count under delegation.
const sink = supabaseUsageSink({ pool: storage.ctx.pool, table: "llm_usage_logs" });

const swarm = defineSwarm({ /* … */, onEvent: sink });
// Default row: generation_id, run_id, org_id, agent_slug, model_id, input_tokens, output_tokens, cost_usd.
// Pass `map` to reshape it to your own columns; return null to skip an event.

Compose it, don't replace your own. The sink only handles swarm.usage and ignores everything else, so if you already have an onEvent observer, run both — the engine composes host observers and isolates faults in both directions, so a throwing sink can't take down your own handler or the run.

Path 3 — the durable event log (replay the whole run)

onEvent is live-only; a StorageAdapter with an events table is the durable record. It persists every SwarmEvent under a monotonic seq, so long after a run finishes you can list it back — per run or per conversation container — and reconstruct not just the spend but the full sequence of tool calls, delegations, questions, and answers. It is the same feed the timeline UI subscribes to.

reading the audit log back
// PATH 3 — the durable event log. A StorageAdapter with an events table persists EVERY SwarmEvent (not just
// usage) under a monotonic `seq`, so the whole run is replayable and auditable long after it finished.
import { createSupabaseStorage } from "@nightowlsdev/storage-supabase";
const storage = createSupabaseStorage({ dbUrl: process.env.DATABASE_URL! });

// Read it back on demand — per run, or per conversation container (root thread + every lane sub-thread):
const events = await storage.events.list(tenantId, runId, /* sinceSeq */ 0);
const usage  = events.filter((e) => e.type === "swarm.usage");     // every model call this run made
const spend  = events.filter((e) => e.type === "swarm.turn_usage"); // the per-segment billing records

// Or stream live (the same feed the UI subscribes to over Realtime):
for await (const ev of storage.events.subscribe(runId)) { /* tee to your audit UI */ }

Path 4 — OpenTelemetry traces and the metric-series store

For an APM or trace backend, a telemetry exporter emits the same measurements shaped as OTel-style spans: one root run span, one generation span per model call (carrying its costUsd), and one span per tool_call → tool_result pair. Separately, @nightowlsdev/metrics is an opt-in observation plane for rolled-up series with period-over-period compare().

lib/swarm.ts
import { defineSwarm } from "@nightowlsdev/core";

// PATH 4 — OpenTelemetry-shaped spans, for an APM/trace backend. A telemetry exporter emits one root `run`
// span, one `generation` span per model call (carrying the per-call `costUsd`), and one span per
// tool_call → tool_result pair. Same measurements as the events, shaped as a trace.
const swarm = defineSwarm({
  /* … */
  telemetry: { export: async (spans) => { await myOtelPipeline.send(spans); } },
});
// Also: @nightowlsdev/metrics is a SEPARATE opt-in observation plane — (orgId, orgScope, metric, at, dims)
// → value with calendar bucketing and a fully-pinned period-over-period compare(). Its own package, its own
// pool, its own migration; wire none and a swarm is byte-identical.

The honest gaps — what this does NOT claim

"Every LLM call is measurable" is true for the loops the framework runs. Three boundaries are worth stating plainly so the claim stays honest.

BoundaryWhat it means for the audit trail
Adapter engines (remote loops)An engine-a2a / engine-trigger-chat / engine-eve loop runs on a remote product, so the framework records what the remote reports: a per-segment swarm.turn_usage from all three, plus a coarse per-call swarm.usage from the tier-2 adapters (engine-eve / engine-trigger-chat) when the remote attaches usage — not a per-call itemization it can't see (the tier-1 engine-a2a is opaque and carries no token data at all). Native's exact per-call swarm.usage — and swarm.external_usage, which is native-engine-only — do not cross the remote boundary.
Metered vs. billed costcost.usd is the framework's token-priced estimate, not the provider's post-hoc invoiced amount. Reconcile via generationId if your provider exposes actuals.
Unpriced models & embeddingsPricing comes from a host-owned price table (model id → rate). An unpriced model still emits tokens and a generationId but a best-effort cost; embedding calls are priced by the same table when configured.