Skip to content
Night Owls.dev
Jump to a page
Concept · Stable

Client tools. Let an agent act in the browser.

A client tool is a tool the model calls but your browser runs: frame the camera on an entity, read the current selection, trigger an upload, apply a change to the live scene. The agent decides, your code performs the side effect, and the suspended run waits for the result. The seam shipped earlier (FR-002); as of @nightowlsdev/react@2.8.0 you register these tools with one prop on <SwarmChat>, no raw-hook rebuild.

An ergonomics unlock, not a new capability. The client-tool mechanism (defineClientTool, useClientTools, the swarm.client_action suspend and resume flow) is stable and has shipped since FR-002. FR-035 exposes it on the batteries-included <SwarmChat>, so a component-based host adopts client tools without rebuilding the timeline and composer on the raw hook.

The round-trip, in five steps

  1. 1. The model calls the tool by name, with typed input.
  2. 2. The engine has no server execute to run, so it suspends the run and emits a swarm.client_action with a followupId and the typed input.
  3. 3. The browser matches the tool name to a handler and runs it — the actual side effect, in the user's session.
  4. 4. The handler's return value (or a thrown error) is posted back as the tool's output.
  5. 5. The engine resumes the run with that output and the model continues.

Declare a client tool (server)

A client tool has no server execute. Declare its name and schemas with defineClientTool and attach the returned tool to an agent like any other. When the model calls it, the engine emits a swarm.client_action event and suspends the run until the browser answers.

tools.ts
import { z } from "zod";
import { defineClientTool } from "@nightowlsdev/core";

// A client tool has no server execute. When the model calls it, the engine emits
// swarm.client_action and suspends the run; the browser runs the handler and posts the result back.
export const frameObject = defineClientTool({
  name: "frameObject",
  description: "Frame the camera on a scene entity",
  inputSchema: z.object({ entityId: z.string() }),
  outputSchema: z.object({ ok: z.boolean() }),
  needsApproval: false, // set true to force a confirm before the browser runs it
});

Run it on the component, in one prop

If you already mount <SwarmChat>, pass a handler map to clientTools. The component runs useClientTools internally against its own run: each handler receives the model's typed input, returns a value that becomes the tool's output, and the run resumes. Throw to fail the tool.

chat.tsx
import { SwarmChat } from "@nightowlsdev/react";

// Pass a handler map to the component: it runs useClientTools internally against its own run.
// The handler gets the model's typed input and returns the tool's output; throw to fail it.
<SwarmChat
  agentSlug={agentSlug}
  threadId={threadId}
  clientTools={{ frameObject: ({ entityId }) => { camera.frame(entityId); return { ok: true }; } }}
  confirmClientAction={(action) => window.confirm(`Run ${action.tool}?`)}  // optional
/>;

Keep a human in the loop

Mark a tool needsApproval: true on the server to force a confirm before the browser runs it, then pass confirmClientAction to gate it: return false to decline and the tool fails with { error: "declined by user" }. confirmClientAction is opt-in. Omit it and an approval-gated tool runs without a prompt, so wire it whenever a tool has a real side effect.

When to reach for one — and the alternatives

A client tool is the right tool only when the work must happen in the browser: it touches the DOM, a canvas, a WebGL scene, the clipboard, a file picker, or client-only state the server never sees. If the effect can run on the server, a plain server tool is simpler and safer — the run never has to suspend and round-trip.

Reach forWhenRuns where
defineTool (server)the effect can run server-side — a DB write, an API call, a computationyour server, inline (no suspend)
defineClientToolthe effect needs the browser — camera, canvas, selection, upload, live scenethe browser (run suspends, then resumes)
an MCP toolthe capability lives in an external MCP server you connectthat server, over the MCP transport
a HITL askyou need a human decision or input, not a browser actionthe human answers; nothing executes on their behalf

The distinction from a swarm.question is worth stating: an ask collects a typed answer from a person; a client action asks the browser to do something and report the result. Both suspend the run through the same durable machinery, so both need a store that can persist the pause (see below).

Engine support, and what a suspend means for durability

Two constraints are honest to state up front. First, engine support: the client-tool path is implemented on the default core / Mastra engine. The alternative engine-openai-agents declares clientTools: false and has no client-action path at all, and the computed danger level on an action (what the confirm step surfaces) is resolved by the Mastra engine only. If you plan to run client tools, stay on an engine that advertises them.

Second, a client action is a genuine run suspension. The engine writes a follow-up record and a snapshot before it emits the event, then waits. With an in-memory or :memory: store that pause lives only in the current process — close the tab or restart and there is nothing to resume. For a client tool whose answer may come back after a reload (or in a background runner), back the swarm with a durable snapshot store so the suspended run can be rehydrated and resumed with the browser's result.

The server-side gate is deny-only. Enforce-level rules and the policy floor can block a client action before the browser is ever asked to run it (a deny fails the tool), and a baseline ask is surfaced to the browser's confirm step — but the server never runs a second approval on your behalf. The only place a client action is approved by a human is the browser's confirm, which is exactly why confirmClientAction matters for anything with a real side effect.

Two rules worth knowing

  • Register on the run-owner lane. The swarm.client_action fires on the run and is root-attributed; a delegate lane never receives it, so mount clientTools on the run-owner <SwarmChat> (one per run or thread).
  • One tool, one handler path. useClientTools dedups per instance, so do not register the same tool in both clientTools and a host-side useClientTools (both would answer the one followupId). Partition instead: some tools via clientTools, the rest via the DIY hook.

The DIY escape hatch (advanced)

For advanced wiring, bubble the matched pair with onClientTools and mount useClientTools yourself. You cannot call the hook inside the callback (hooks rule), so latch the bubbled { clientAction, respondClientAction } into state and call the hook at the top level. The common case is clientTools; reach for this only when you need to own the loop.

host.tsx
import { useState } from "react";
import { SwarmChat, useClientTools, type UiClientAction } from "@nightowlsdev/react";

// Advanced: own the wiring. You cannot call useClientTools inside the callback (hooks rule),
// so latch the bubbled pair into state and mount the hook at the top level.
function Host() {
  const [run, setRun] = useState<{
    clientAction: UiClientAction | null;
    respondClientAction: (r: { output?: unknown; error?: string }) => Promise<void>;
  } | null>(null);

  useClientTools({
    clientAction: run?.clientAction ?? null,
    respond: run?.respondClientAction ?? (async () => {}),
    handlers: { frameObject: ({ entityId }) => { camera.frame(entityId); return { ok: true }; } },
  });

  return <SwarmChat agentSlug={agentSlug} threadId={threadId} onClientTools={(_slug, io) => setRun(io)} />;
}