@nightowlsdev/runner-background
RunnerThe durable runner for Night Owls, runs a swarm turn in Trigger.dev v4 or Vercel Workflow so an ask-suspended run can park for days and resume across process death.
What it does
`createBackgroundRunner({ engine, storage, backend })` returns the core `Runner` whose durable task body loops `engine.run`/`engine.resume`, opening a parkable waitpoint (keyed by `followupId`) on each `ask` suspend so a human-in-the-loop turn can park (default `askTimeout` 7d) and resume across process death; `run()` streams live progress over Supabase Realtime rather than an HTTP task stream. The whole vendor surface reduces to a 3-verb `DurableBackend` seam (enqueueRun / createWaitpoint / completeWaitpoint), with three implementations: `createTriggerBackend` (Trigger.dev v4 waitpoints, with `runDurableTask` + `registerBackgroundSwarm`/`getRegisteredSwarm` for the deployed task), `createVercelBackend` (Vercel Workflow hooks, host-injected callables), and an in-memory backend for hermetic tests. `@trigger.dev/sdk` and `workflow` are optional peers (wire exactly one). FR-061: the Vercel Workflow durable body now ships in-package, so instead of hand-writing the workflow you import `swarmWorkflow` from the `@nightowlsdev/runner-background/vercel-workflow` subpath, and `createVercelBackend` gains a `cancelRun` when you inject a `cancelWorkflow` callable; long segments still need Trigger, because a single `@workflow/next` step tops out around 800s. FR-064: `reapStuckRuns` returns the count it actually REAPED (not the rows it selected) and takes an `onReapError({ runId, error })` callback, so a host learns which runs are still stuck instead of losing them in an aggregate. It asserts the backend is the single retry owner. It never imports `@mastra/*` and re-exports no Mastra types (the engine wall).
Install
pnpm add @nightowlsdev/runner-backgroundKey exports
- createBackgroundRunner
- registerBackgroundSwarm
- getRegisteredSwarm
- buildTaskBody
- createTriggerBackend
- runDurableTask
- createVercelBackend
- swarmWorkflow (FR-061: the in-package Vercel Workflow body, @nightowlsdev/runner-background/vercel-workflow subpath)
- reapStuckRuns (FR-064: returns the count actually reaped + onReapError)
- createInMemoryBackend
- InMemoryDurableBackend
- verifyDurableLifecycle
- nightOwlsPlugin
- DurableBackend
- Waitpoint
- DurableTaskBody
- BackgroundRunnerOpts
- BackgroundSwarm
- DurableVerifyResult
Usage
import { createBackgroundRunner, createTriggerBackend } from "@nightowlsdev/runner-background";
const backend = createTriggerBackend(/* Trigger.dev v4 wiring */);
const runner = createBackgroundRunner({ engine, storage, backend });
// Suspended HITL turns park on a waitpoint and resume across process death.
await runner.run(ctx, { message: "Draft the launch post." });
// FR-064: reapStuckRuns returns the count it actually REAPED (not the rows it selected), and takes
// onReapError({ runId, error }) so you learn which runs are still stuck:
// const reaped = await reapStuckRuns({ /* backend + storage */ onReapError: ({ runId, error }) => log(runId, error) });
// FR-061: to park on Vercel Workflow instead of Trigger, the durable body now ships in-package as swarmWorkflow:
// import { swarmWorkflow } from "@nightowlsdev/runner-background/vercel-workflow";
// const backend = createVercelBackend({ cancelWorkflow }); // gains cancelRun; long turns still need Trigger (~800s/step)What it provides
runner-background is the DURABLE runner for a core swarm: it runs a turn inside durable background infrastructure (Trigger.dev v4 or Vercel Workflow) so an `ask`-suspended, human-in-the-loop run can park for hours or days and resume across process death. createBackgroundRunner({ engine, storage, backend }) returns the same core Runner interface, but instead of streaming the engine's iterable to the caller, the turn executes in the vendor's infra and the HTTP caller streams live progress over Realtime by subscribing to the persisted swarm_events. The whole vendor surface reduces to a three-verb DurableBackend seam, and it ships two production backends, an in-memory one for hermetic tests, a stuck-run reaper, and a timer-less recurrence plane.
When to use it
- A HITL approval must survive process death or a serverless cold start — the run parks on a waitpoint (default askTimeout 7d) and the compute goes to zero until a human answers.
- A swarm turn is long-running or background work that must outlive the HTTP request that started it.
- You already run Trigger.dev v4 (or Vercel Workflow) and want durable agent runs on that runtime.
- You need scheduled / recurring agent runs (the FR-057 recurrence plane) or a watchdog that fails runs stranded by a crash (the reaper).
When not to
- The browser stays connected for the whole turn and you just want to stream tokens — @nightowlsdev/runner-nextjs streams the engine's own iterable straight to the client with no vendor infra.
- There is no HITL, no long turn, and no durability requirement — plain core in-process is enough and this adds a backend to wire.
- You will not stand up a durable backend — the in-memory backend is dev/test only; a process restart drops its in-flight continuations.
Alternatives
- @nightowlsdev/runner-nextjs (interactive)The turn completes within one connected request and you want SSE streaming without Trigger/Vercel. The two compose: pass this durable runner as runner-nextjs's `background` option to keep the same routes but enqueue instead of stream.
- Plain core in-processA run that never suspends and finishes inside the process — you keep engine.run()'s iterable and skip the durable snapshot entirely, losing park-across-process-death.
- @nightowlsdev/engine-trigger-chatYou want to render a deployed Trigger chat.agent Session as an agent. That is an experimental, reduced-governance adapter over a DIFFERENT Trigger surface — runner-background is the GA durable layer (Trigger v4 durable tasks + wait tokens).
Strengths
- Durable HITL: a parked turn consumes zero compute while it waits (Trigger checkpoints across process death), and N sequential approval turns run inside one durable run because the body re-parks on each new question.
- Vendor-agnostic three-verb seam (enqueueRun / createWaitpoint / completeWaitpoint), so the in-memory backend runs the REAL engine end-to-end with no Docker and no cloud — the default test suite is fully hermetic.
- Single retry owner is asserted at construction (retries.owner must be "backend"), so a second retry owner can never double-execute a tool's side effects.
- Resume is cross-tenant safe and idempotent: the tenant comes from ctx, never the request body, and a duplicate resume is a no-op.
- Engine-wall clean — it never imports @mastra/* and re-exports no Mastra types.
- Post-drive hygiene (delta compaction, message-search indexing) is awaited at the drive's finally in its own contained try/catch, so a scheduled run's messages still get folded and indexed even though no request ever existed — and a throwing chore cannot change the run's outcome.
Limits & trade-offs
- You must wire exactly one durable backend for real durability; the in-memory backend drops in-flight continuations on restart (completeWaitpoint returns false, forcing recovery from the Postgres snapshot).
- The Vercel Workflow driver is best-effort + gated smoke, and a single @workflow/next step tops out around 800s — long HITL segments still need Trigger.
- Nothing in-package holds a timer: the reaper and the recurrence plane need the host to drive ticks (a cron route or a long-lived interval).
- The deployed task bundles a live engine, which can't travel over the wire — you register an engine+storage factory in a shared config module that both the task file and the app import.
- resume streams the continuation over Realtime, not an HTTP task stream, so storage.events.subscribe (e.g. Supabase Realtime) must be wired, and the task must use the DIRECT Postgres port (5432, not the 6543 pooler) so it finds the saved snapshot.
- Options that must reach the DEPLOYED task body (askTimeout, compactDeltas, messageSearch, bugCapture, release, maxReparks) have to be set on the REGISTERED SWARM as well: the Trigger backend discards the task body it is handed and the deployed task rebuilds it from getRegisteredSwarm(), so an option passed only to createBackgroundRunner is configured and inert in production.
- Hosting a durable body inside one of your OWN existing tasks is supported, but it is a shape with four constraints nothing enforces for you — a hosting task must not reattempt, the swarm registry is an unkeyed singleton, cancellation tags are enqueue-side only, and the store must implement the waitpoint trio. See the section below before adopting it.
How it works
createBackgroundRunner returns a Runner whose durable task body is a loop. engine.run(input, ctx) streams events; on an `ask` tool suspend the engine saves the Mastra snapshot to Postgres, sets the run suspended, and emits swarm.question. The body only READS that event to detect the suspend (it never re-appends — the engine is the sole event writer), opens backend.createWaitpoint keyed by the followupId (plus an FR-065 per-suspension salt so a re-suspended followup can't be woken with a prior suspension's completed token), persists the waitpoint id, and parks. When the host calls resume, the waitpoint resolves, the body calls engine.resume and loops; it terminates only on a terminal status. run() enqueues then yields storage.events.subscribe(runId) for live Realtime progress; resume() authorizes against storage.runs.findSuspended(ctx.tenantId, followupId), completes the waitpoint, and streams the durable continuation.
Hosting a durable body in an existing task
The wiring above builds ONE dedicated task. That is not the only supported shape. A durable body can be embedded in an existing, NON-scaffolded Trigger task by calling buildTaskBody({ engine, storage, backend }) directly inside that task's own run(), with the swarm registry bypassed entirely. This is supported: a host bolting durable agents onto a fleet of pre-existing background jobs does not have to restructure them into thin callers that enqueue() into one dedicated task. Nothing about the task handle is special — park() is a wait.forToken with no task affinity, maxDuration is never read by any runtime code, and the §12 single-retry-owner assert validates the host-process option object rather than the Trigger task's own retry config.
CONSTRAINT 1 — retry.maxAttempts must be 1 on any task hosting a durable body, and this is safety-critical rather than a scaffold preference. Trigger counts the first execution as an attempt, so maxAttempts: 1 is precisely what DISABLES reattempts. A task-level retry re-drives engine.run from scratch with no idempotency guard on runId, and storage.runs.create no-ops on conflict WITHOUT throwing, so the retry proceeds silently into a full fresh generation. The runner's ownership cell protects less than its name suggests: it gates runner-owned orchestration writes and waitpoint attachment only. The re-driven ENGINE independently writes its own events and its own status transitions (suspended / done / failed) after that losing create, without consulting the cell at all — so a retry can overwrite or duplicate persisted state, not merely emit a harmless duplicate event. Worse, the waitpoint's identity is `${runId}:${toolCallId}` and toolCallId is minted fresh by a re-drive, so a retry produces a new followupId, a new createWaitpoint key and a new token, and ORPHANS the original park's attached token — one run left advertising two answerable asks, one of which nothing is driving. The package's own test suite proves this split rather than asserting it in prose.
CONSTRAINT 2 — getRegisteredSwarm() is an unkeyed, last-write-wins singleton scoped to one loaded module instance. Its factory and cache are unkeyed module-scope globals, and registerBackgroundSwarm replaces the factory and clears the cache, so whichever registration runs last within that runtime/module instance wins (do not assume every vendor task bundle necessarily shares one). A host embedding many tasks should register the swarm ONCE and call buildTaskBody directly in every other task, bypassing the registry — never call registerBackgroundSwarm more than once per deployed bundle or module instance, because the second call silently redefines what every other task resolves.
CONSTRAINT 3 — cancelRun's native-kill tag is enqueue-side only, and a manually-triggered task CAN stamp it itself. createTriggerBackend().enqueueRun stamps ngorun-<runId> on the vendor run and cancelRun finds and kills the run by that tag; a task you trigger yourself is not tagged by that path, so stamp the same tag from inside your own run() with tags.add(). That is the fix, not merely a caveat. Without it native cancellation finds nothing tagged, the task receives NO cancellation signal at all and keeps running — a correctness and side-effect hazard, not just wasted compute. Its tool side effects continue, and it can subsequently overwrite the already-written failed / run_cancelled row with suspended, done, or another terminal while emitting later events; the earlier cancel write is not protected against that overwrite. Runner.cancel would then have recorded that it tried to cancel while nothing actually stopped. The tradeoff this trades TO, not just away from: stamping the tag makes the ENTIRE hosting Trigger run hard-cancellable, not just the agent's parked step — Runner.cancel ends in a real vendor kill of whatever run carries the tag. This integration wires no graceful handling by default, though Trigger v4 does give a cancelled task an aborted signal and an onCancel hook that may await cleanup for up to 30s before termination, which a host needing that can wire independently. On a host task that also does its own pre-existing work, an agent-level STOP can now terminate that job mid pre-work, mid agent execution, or after the agent returns but before your own post-work commits. There is no way to make a parked agent cancellable without this; keep the hosting task's pre/post work idempotent or compensable, and stamp the tag as late as reasonably possible — immediately before buildTaskBody's body(), not at the top of run() — to narrow (not eliminate) the window.
CONSTRAINT 4 — this shape effectively requires a waitpoint-capable storage implementation, not an arbitrary StorageAdapter. attachWaitpoint / getWaitpoint / clearWaitpoint are OPTIONAL on the public storage type, so an adapter without them satisfies the contract and still breaks this path: token attachment silently optional-chains to a no-op; resume optional-chains the read back and falls back to the followupId as the token id — valid for the Vercel backend, where the hook key IS the followupId, but NOT Trigger's own generated waitpoint_* token id, so a resume can complete the WRONG token and be falsely acknowledged as successful; and without clearWaitpoint a consumed token stays advertised as live. recordSuspend is required as well, or the followup is never resumable at all. Plainly: pattern (B) requires the conformant @nightowlsdev/storage-supabase implementation, which carries the followup/waitpoint migration — not any StorageAdapter implementation.
The one thing bypassing the registry costs you is that resolveTaskBodyArgs' field list stops carrying options on your behalf — pass bugCapture, release, maxReparks, compactDeltas, messageSearch and askTimeout to buildTaskBody yourself.
Indexing a durable turn for message search
messageSearch: { plane, indexOnDriveEnd? } indexes the conversation's new messages after a durable drive completes. The plane is @nightowlsdev/storage-supabase's createMessageSearch({ pool, embedder, embedderTag, dimensions }); this package mirrors only the one method it calls (indexContainer — it never reads), so it takes on no dependency on pgvector or an embedder. indexOnDriveEnd defaults to TRUE: unlike compactDeltas, indexing only ever inserts into its own tables, so it deletes nothing a host might be treating as an audit trail.
This is the ONLY place a durable turn's messages get indexed. @nightowlsdev/runner-nextjs indexes when an INTERACTIVE turn ends, and it cannot cover the durable path: its runEnd fires at ENQUEUE, before the run has written a single message. A run driven here has no request behind it at all, so the drive's end is its only terminal. A deployment running both wires both, and the overlap is safe — the plane owns its own budget, cursor and per-(org, container, model) lock, so concurrent calls on one conversation serialize inside it rather than racing.
It is CONTAINED by construction. The call is awaited (nothing is waiting on this worker's latency, and a detached promise in a task body can be cut short the instant the task resolves) inside its own try/catch at the drive's finally, and every failure — a wrong-dimension schema, an embedder outage, an unwired table, a contended lock — is swallowed with a log. A throwing indexer can never fail the run, the resume, or the drive: the message is the product.
⚠ On the DEPLOYED Trigger path, set it on the registered swarm too. createTriggerBackend().enqueueRun ignores the task body it is handed — it only triggers the deployed task with { input, ctx }, and that task rebuilds its body from getRegisteredSwarm() inside Trigger's infra. A messageSearch passed only to createBackgroundRunner therefore reaches the in-process path and nothing on the deployed one: configured, documented, and inert. BackgroundSwarm carries the same field for exactly this reason, as bugCapture, release, maxReparks and compactDeltas each did before it — and as askTimeout does after it (FR-075, the sixth field to ride this hand-maintained seam and the fifth wave to do so — and the one that genuinely shipped inert, in 2.13.0). The Vercel Workflow path runs it as a 'use step'.
Examples
Durable HITL on Trigger.dev v4
A shared config module registers the engine+storage factory; the app builds the runner. The deployed task file (elided) calls runDurableTask(swarmRun, payload).
// swarm.config.ts — imported by BOTH the deployed task file and the app
import { registerBackgroundSwarm } from "@nightowlsdev/runner-background";
import { SwarmEngine } from "@nightowlsdev/core";
import { createSupabaseStorage } from "@nightowlsdev/storage-supabase";
registerBackgroundSwarm(() => ({
engine: new SwarmEngine(swarm),
// Direct Postgres (5432), NOT the 6543 pooler — resume must find the saved snapshot.
storage: createSupabaseStorage({ dbUrl: process.env.DATABASE_URL_5432! }),
}));
// lib/swarm.ts — the app side
import { createBackgroundRunner, createTriggerBackend, getRegisteredSwarm } from "@nightowlsdev/runner-background";
import { swarmRun } from "../trigger/swarm"; // the deployed task() that calls runDurableTask
const { engine, storage } = getRegisteredSwarm();
export const runner = createBackgroundRunner({
engine,
storage,
backend: createTriggerBackend(swarmRun),
retries: { owner: "backend" }, // must be "backend" — a second retry owner double-runs side effects
});Host a durable body inside your OWN existing task (FR-074)
Pattern (B): the task keeps its own id and shape, the registry is bypassed, and the four constraints are stated at the lines that satisfy them. Storage must be @nightowlsdev/storage-supabase.
// trigger/process-invoice.ts — one of your OWN pre-existing tasks
import { task, tags } from "@trigger.dev/sdk";
import type { RunInput, SwarmContext } from "@nightowlsdev/core";
import { buildTaskBody, createTriggerBackend } from "@nightowlsdev/runner-background";
import { engine, storage } from "../lib/swarm"; // storage-supabase — constraint 4
export const processInvoice = task({
id: "billing.process-invoice",
// ⚠ CONSTRAINT 1 — Trigger counts the first execution as an attempt, so this DISABLES
// reattempts. A reattempt re-drives engine.run, orphans the parked waitpoint, and
// overwrites persisted state (the engine writes status + events past the ownership cell).
retry: { maxAttempts: 1 },
run: async (payload: { input: RunInput; ctx: SwarmContext }) => {
// …your existing pre-work stays exactly as it is — kept OUTSIDE the cancellable window by
// stamping the tag as late as possible, below…
// ⚠ CONSTRAINT 3 — self-triggered runs carry no ngorun- tag, so runner.cancel() would find
// nothing to kill. Stamp it ourselves, right before the agent runs — not at the top of
// run() — so only "the agent is executing" becomes cancellable, not your own pre-work.
await tags.add(`ngorun-${payload.ctx.runId}`);
// ⚠ CONSTRAINT 2 — buildTaskBody DIRECTLY; no registerBackgroundSwarm in this file. The
// registry is an unkeyed singleton and a second registration redefines it for every task.
const body = buildTaskBody({
engine,
storage,
backend: createTriggerBackend(processInvoice),
askTimeout: "36h", // FR-075 — reaches this body because it is passed here directly
});
await body(payload.input, payload.ctx);
},
});Index a durable turn's messages for search
Set it in BOTH places on the Trigger path: the backend discards the task body, and the deployed task rebuilds it from the registered swarm.
import {
registerBackgroundSwarm,
createBackgroundRunner,
createTriggerBackend,
type MessageSearchConfig,
} from "@nightowlsdev/runner-background";
import { createMessageSearch } from "@nightowlsdev/storage-supabase";
const messageSearch: MessageSearchConfig = {
plane: createMessageSearch({
pool: storage.ctx.pool,
embedder, // (texts) => Promise<number[][]>
embedderTag: "openai/text-embedding-3-small",
dimensions: 1536,
}),
// indexOnDriveEnd defaults to true — indexing only inserts into its own tables.
};
// The DEPLOYED task body reads this one.
registerBackgroundSwarm(() => ({ engine, storage, messageSearch }));
// The in-process / in-memory path reads this one.
export const runner = createBackgroundRunner({
engine,
storage,
backend: createTriggerBackend(swarmRun),
retries: { owner: "backend" },
messageSearch,
});Reap stalled runs from a serverless cron (FR-064)
reapStuckRuns returns the count it actually REAPED; onReapError names the ones it could not fail, and onStalled is the host's billing seam.
import { reapStuckRuns } from "@nightowlsdev/runner-background";
// A cron route (the in-process startReaper interval can't run on serverless).
const reaped = await reapStuckRuns(
{ storage, pool },
{
maxAgeMs: 15 * 60_000, // no new event for 15m ⇒ stalled (default)
onStalled: async ({ runId, tenantId }) => {
if (tenantId) await credits.refund(tenantId, runId); // never charge for non-delivery
},
// FR-064 — reaped !== selected; onReapError names the runs still stuck.
onReapError: ({ runId, error }) => log.error("still stuck", runId, error),
},
);A weekly scheduled agent run (FR-057 recurrence plane)
No in-package timer — the host drives plane.tick() from a cron route (createScheduleHandler in runner-nextjs).
import { createSchedulePlane, createPgOccurrenceLedger, createRunOutcome } from "@nightowlsdev/runner-background";
const plane = createSchedulePlane({
ledger: createPgOccurrenceLedger({ pool }), // storage-supabase migration 0031
enqueueAgentRun: (input, ctx) => runner.enqueue(input, ctx),
runOutcome: createRunOutcome(storage), // suspended -> "running" is pinned
schedules: [
{ id: "weekly-rank-check", cron: "0 9 * * 1", timezone: "Europe/London",
agent: { bindings: async () => loadBindings() } },
],
});
await plane.tick(); // at-most-once per occurrence key, on any backendDoing the parts it doesn't support
- Streaming a turn to a still-connected browserThat is @nightowlsdev/runner-nextjs's interactive SSE. To keep those routes but run durably, pass this runner as runner-nextjs's `background` option — chatRoute then enqueues and returns { runId }, and the react client subscribes over Realtime.
- Parking on Vercel Workflow instead of TriggerImport swarmWorkflow from the @nightowlsdev/runner-background/vercel-workflow subpath and wire createVercelBackend with the host-injected startWorkflow / createHook / resume callables (inject cancelWorkflow to gain cancelRun). A single @workflow/next step tops out near 800s, so long HITL segments still need Trigger.
- Running the reaper or the schedule tick on a timerThere is no in-package timer. In a long-lived process call startReaper (an unref'd, globalThis-guarded interval); on serverless point a cron at runner-nextjs's createReaperHandler / createScheduleHandler, or call reapStuckRuns / plane.tick() directly.
- Searching the history this indexesThis package only WRITES — its plane mirror carries indexContainer and nothing else. The read side is @nightowlsdev/runner-nextjs's searchRoute(), which runs the participation gate the store deliberately does not; point both at the same createMessageSearch plane.
- Recovering a continuation after an in-memory restartThe in-memory backend backs each waitpoint with a held-open Promise, so a restart loses it: completeWaitpoint returns false when no waiter exists, signalling the caller to recover from the durable Postgres snapshot. Real durability across process death requires the Trigger or Vercel backend.
Related
- core — The engine + Runner contract this wraps durably; its suspend/resume snapshot is what parks and resumes.
- runner-nextjs — The interactive transport — compose it via its `background` option, and its cron handlers drive this package's reaper + schedule ticks.
- storage-supabase — Implements attachWaitpoint / getWaitpoint and ships the followups + occurrence-ledger migrations; the durable snapshot and swarm_events live there, as does the createMessageSearch plane this indexes into.
- recurrence — The FR-057 schedule-as-intake walkthrough for the recurrence plane's semantics (at-most-once, catch-up, staleClaimAfter).
- react — The client hooks that subscribe over Realtime to render a durable run's persisted event stream.