Skip to content
Night Owls.dev
Jump to a page
Guide · FR-048

Let an agent see the file.

Paste a screenshot and say “fix this”. Drop in the PDF instead of describing it. Two seams make that work and both are yours: onUpload puts the bytes in your bucket, and resolveAttachment turns an id back into something a model can read. Everything between them — the wire, the admission chain, the capability honesty — is the framework's.

The bytes never cross the framework's wire. What travels is an AttachmentRef: an opaque id plus display hints. There is no url / path / key field to fill in, and a ref carrying one is rejected at every entry — because a signed URL on an event is a bearer token in a permanent log that is also broadcast to every subscriber of the thread.

Take the upload — your bucket, your auth

The framework has no storage adapter for blobs and does not want one. Your route accepts the file, writes it wherever you already write files, records the row that maps an id to an object, and returns the ref.

app/api/attachments/upload/route.ts
// app/api/attachments/upload/route.ts — YOUR bucket, YOUR auth. The framework never sees the bytes.
import type { AttachmentRef } from "@nightowlsdev/core";
import { authenticate } from "@/lib/auth";
import { bucket, recordAttachment } from "@/lib/bucket";

export async function POST(req: Request): Promise<Response> {
  const authCtx = await authenticate(req);
  if (!authCtx) return Response.json({ error: "unauthorized" }, { status: 401 });

  const form = await req.formData();
  const file = form.get("file");
  const container = form.get("container");
  if (!(file instanceof File) || typeof container !== "string") {
    return Response.json({ error: "file and container required" }, { status: 400 });
  }

  // ⚠ KEEP THE REAL EXTENSION IN THE KEY. OpenRouter gates image URLs on a file-extension regex
  // (/^https?:\/\/.+\.(jpg|jpeg|png|gif|webp)(?:[?#].*)?$/i), so a signed URL over a bare UUID key
  // does not match — admission then refuses the URL and falls back to bytes, or drops the
  // attachment with `url-refused`. OpenRouter's PDF parsing keys on the filename too.
  const id = crypto.randomUUID();
  const dot = file.name.lastIndexOf(".");
  const key = `${authCtx.tenantId}/${container}/${id}${dot > 0 ? file.name.slice(dot) : ""}`;

  await bucket.put(key, file);
  await recordAttachment({
    id, key, container, tenantId: authCtx.tenantId,
    mediaType: file.type, filename: file.name, byteLength: file.size,
  });

  // The ref is the WHOLE wire payload — there is no url/path/key/bucket field to leak. A ref
  // carrying one is rejected at every entry, so this shape is the only thing that reaches a run.
  const ref: AttachmentRef = { id, mediaType: file.type, filename: file.name, bytes: file.size };
  return Response.json(ref);
}

Keep the real file extension in the object key. It reads like housekeeping and it is a transport rule. OpenRouter gates image URLs on a file-extension regex, so a signed URL over a bare UUID key does not match the live model's supportedUrls — the framework then refuses to send that URL, asks your resolver again for bytes, and drops the attachment with url-refused if it gets a second URL. The same provider defaults an unrecognised file to application/pdf and keys its parsing on the filename. Six provider packages carry a canary suite that pins these cells against the installed adapters, so a provider upgrade that changes one fails a test rather than a customer conversation.

Resolve — the only code that may produce a URL

One function, two purposes. purpose: "model" is the engine asking for something to put in front of a model; purpose: "ingest" is a tool asking for bytes or text. The ingest arm has no URL variant by type, so a tool cannot mint one, cannot supply its own scope, and cannot reach outside its run's container.

lib/attachments.ts
// lib/attachments.ts — the ONLY code that turns a ref into something a model can read.
import type { AttachmentCanonical, ResolveAttachmentFn } from "@nightowlsdev/core";
import { attachmentRow, bucket } from "@/lib/bucket";

export const resolveAttachment: ResolveAttachmentFn = async (ref, scope, want) => {
  // YOUR OBLIGATION. `scope` is trusted and framework-built (never read off the request body);
  // `ref` is untrusted hints. A ref belonging to another container must resolve exactly like an
  // absent one — "not-found", never "forbidden" — or the reason string becomes an existence oracle
  // for ids in someone else's thread.
  const row = await attachmentRow(ref.id);
  if (!row || row.container !== scope.container || row.tenantId !== scope.tenantId) {
    return { kind: "unsupported", reason: "not-found" };
  }

  // `canonical` is the TRUSTED record. Admission and part construction read it exclusively, and the
  // per-file budget is enforced against canonical.byteLength — never against the ref's `bytes` hint.
  const canonical: AttachmentCanonical = {
    mediaType: row.mediaType,
    filename: row.filename,
    byteLength: row.byteLength,
  };
  const text = async () => new TextDecoder().decode(await bucket.bytes(row.key));

  // purpose "ingest" — a TOOL asked, through ctx.resolveAttachment. Bytes or text only: this arm
  // has no URL variant, by type.
  if (want.purpose === "ingest") {
    return row.mediaType.startsWith("text/")
      ? { kind: "text", canonical, text: await text() }
      : { kind: "bytes", canonical, data: await bucket.bytes(row.key) };
  }

  // purpose "model". `want.transportHint` is CONSERVATIVE and computed before anything is resolved:
  // "url-ok" only means the profile COULD accept a URL for this media class. Admission still tests
  // the final URL and canonical mediaType against the LIVE model's supportedUrls, and asks you once
  // more with "bytes-only" if it refuses. Answering with bytes is always legal.
  if (row.mediaType.startsWith("image/")) {
    return want.transportHint === "url-ok"
      ? { kind: "image", data: await bucket.signedUrl(row.key, 900), canonical }
      : { kind: "image", data: await bucket.bytes(row.key), canonical };
  }
  if (row.mediaType === "application/pdf") {
    return { kind: "pdf", data: await bucket.bytes(row.key), canonical };
  }
  if (row.mediaType.startsWith("text/")) {
    return { kind: "text", canonical, text: await text() };
  }

  // Anything no provider reads (docx, xlsx, video) is YOUR extraction problem. Saying so puts a
  // `resolution-failed` notice on the operator's screen instead of a silently missing document.
  return { kind: "unsupported", reason: `no transport for ${row.mediaType} — extract it host-side first` };
};

Admission runs on what you returned, not on a table. Before any URL-carrying part exists the engine checks, in order: kind/MIME consistency (an image result whose canonical type is text/plain is dropped); scheme and host policy (https only unless you opt in, plus a private-address guard that core owns because the fetcher underneath has none); then the live model instance's supportedUrls regexes against the final URL string and the canonical media type. No match means the URL is not sent — you are asked once more with bytes-only, and a second URL drops the attachment with a warning. Two providers (Groq, Ollama) publish an empty supportedUrls, so they never receive a URL by construction.

transportHint is a hint about the profile, not a promise about the model: it is computed before anything is resolved, from the ref's untrusted display type, so it is deliberately conservative. Returning bytes when the hint said url-ok is always correct. One engine — engine-openai-agents — always hints bytes-only and rejects a URL result outright, because its bridge flattens a URL image's media type regardless of what the provider would have accepted.

lib/swarm.ts
// lib/swarm.ts — the two fields FR-048 adds to the config you already have.
import { defineSwarm } from "@nightowlsdev/core";
import { resolveAttachment } from "@/lib/attachments";
import { agents, cost, modelFactory, models, storage } from "@/lib/swarm-config";

export const { engine } = defineSwarm({
  storage, agents, models, modelFactory, cost,

  resolveAttachment,
  // Post-resolution ceiling, checked against canonical.byteLength. Default 20 MiB.
  // `allowInsecureUrls` (default false) is the only way an http: URL is ever admitted: https plus a
  // private-address guard is the standing policy, and a data: URL is refused outright.
  attachments: { maxBytesPerFile: 10 * 1024 * 1024 },
});

Wire the composer

One prop. The paperclip renders only when onUpload is wired and the engine declares at least one class it can carry, and the file picker's accept filter is derived from those same classes — so an engine that carries images but not PDFs never offers a PDF. Upload state per chip is pending / done / error; there is no numeric progress seam in v1.

app/chat.tsx
"use client";
import { SwarmChat, SwarmProvider, previewContainerOf, type AttachmentRef } from "@nightowlsdev/react";

async function uploadToHost(file: File, container: string): Promise<AttachmentRef> {
  const form = new FormData();
  form.set("file", file);
  form.set("container", container);
  const res = await fetch("/api/attachments/upload", { method: "POST", body: form });
  if (!res.ok) throw new Error(`upload failed (${res.status})`);
  // Throwing marks THAT chip as failed. It never blocks the send: the text still goes.
  return (await res.json()) as AttachmentRef;
}

export function Chat({ threadId }: { threadId: string }) {
  return (
    <SwarmProvider
      theme="ink"
      // Both or neither: either endpoint alone is inert and chips render as plain labels.
      endpoints={{
        attachmentToken: "/api/swarm/attachments/token",
        attachmentPreview: "/api/swarm/attachments",
      }}
    >
      <SwarmChat
        agentSlug="coordinator"
        threadId={threadId}
        // The paperclip appears only when onUpload is wired AND the engine declares a class.
        onUpload={(file) => uploadToHost(file, previewContainerOf(threadId))}
      />
    </SwarmProvider>
  );
}

SwarmFab does not plumb onUpload. The floating chat mounts its own SwarmChat and does not forward the prop, so a FAB-only host cannot enable the paperclip without composing the lane itself. Known limitation, not a configuration mistake.

Building your own composer instead? The same seam is on the hook. Note the shape: send(text, "slug") still works, but only the options object can carry attachments.

components/own-composer.tsx
"use client";
import { useState } from "react";
import { AttachmentChip, useSwarmChat, type AttachmentRef } from "@nightowlsdev/react";

export function OwnComposer({ agentSlug, threadId }: { agentSlug: string; threadId: string }) {
  const chat = useSwarmChat({ agentSlug, threadId });
  const [text, setText] = useState("");
  const [refs, setRefs] = useState<AttachmentRef[]>([]);

  return (
    <form
      onSubmit={(e) => {
        e.preventDefault();
        // An ATTACHMENT-ONLY send is legal — "" plus one ref is a valid turn, as long as the engine
        // declares a class it can carry. `send(text, "slug")` still works; only the options object
        // can carry attachments.
        if (!text && refs.length === 0) return;
        void chat.send(text, { attachments: refs });
        setText("");
        setRefs([]);
      }}
    >
      {refs.map((ref) => (
        <AttachmentChip
          key={ref.id}
          attachment={ref}
          threadId={threadId}
          state="done"
          onRemove={() => setRefs((prev) => prev.filter((r) => r.id !== ref.id))}
        />
      ))}
      <input value={text} onChange={(e) => setText(e.target.value)} />

      {/* Per-ref outcomes of the LAST request: refs the server rejected or dropped over the cap.
          Nothing is silently discardedsurface these or the operator sends into a void. */}
      {chat.attachmentDispositions
        ?.filter((d) => d.disposition !== "accepted")
        .map((d) => <p key={d.id ?? d.reason}>{d.id ?? "attachment"}: {d.disposition} — {d.reason}</p>)}
    </form>
  );
}

Previews: mint, then redeem

A chip renders a thumbnail by minting a short-lived token for one attachment and redeeming it. Both hops are authenticated, both re-run the participation gate, and the token is bound to the identity that minted it — a leaked token is useless to anyone else. Wire nothing and chips still render as labels; no request is made.

lib/attachment-preview.ts
// lib/attachment-preview.ts
import type { AttachmentPreviewResult, AttachmentPreviewScope } from "@nightowlsdev/runner-nextjs";
import { attachmentRow, bucket } from "@/lib/bucket";
import { attachmentPreviewSecret } from "@/lib/env";
import { runner } from "@/lib/runner";

// The route has already authenticated the caller, re-run the participation gate, and compared the
// caller's identity to the token's claims. What it CANNOT know is your bucket layout, so ownership
// is still yours to check: a foreign ref returns null and the route answers 404 — the same answer
// an unknown id gets, so nothing is confirmed to a prober.
async function resolvePreview(
  ref: { id: string },
  scope: AttachmentPreviewScope,
): Promise<AttachmentPreviewResult> {
  const row = await attachmentRow(ref.id);
  if (!row || row.container !== scope.container || row.tenantId !== scope.tenantId) return null;

  // { stream } proxies the bytes through your own origin — always safe to read.
  return { stream: await bucket.stream(row.key), mediaType: row.mediaType };

  // { redirect: await bucket.signedUrl(row.key, 900) } is also legal and answers 302 — but the
  // browser READS the redeem response to build a blob: URL, so the target must be CORS-readable.
  // If your CDN is not, return { stream }.
}

// Built ONCE and re-exported by the two route files below, so both handlers share one secret, one
// TTL and one resolver. Constructing it per route file also works, but then two files own two
// copies of that configuration.
export const preview = runner.attachmentPreviewRoutes({
  resolvePreview,
  secret: attachmentPreviewSecret,
  ttlMs: 15 * 60_000, // default; a 410 triggers exactly one transparent re-mint client-side
});

Two route files re-export it — the mint is a plain POST, the redeem sits under a dynamic segment carrying the attachment id.

app/api/swarm/attachments/token/route.ts
// app/api/swarm/attachments/token/route.ts — MINT
import { preview } from "@/lib/attachment-preview";

export const POST = preview.POST;
app/api/swarm/attachments/[id]/route.ts
// app/api/swarm/attachments/[id]/route.ts — REDEEM (the dynamic segment carries the attachment id)
import { preview } from "@/lib/attachment-preview";

export const GET = preview.GET;

The statuses are distinguishable on purpose. 400 malformed body or token · 401 unauthenticated (checked before the token is read — the token is a carrier, not a credential) · 403 gate or identity mismatch · 404 unknown or foreign ref · 410 expired. Only the 410 is retried, exactly once, because it is the only failure the client can fix by itself.

A { redirect } target must be CORS-readable. The browser reads the redeem response itself and hands the DOM a managed blob: URL — which is what makes previews work for bearer-token hosts, where an <img> would send no Authorization header at all. If your CDN is not CORS-readable, return { stream } and let the route proxy the bytes.

What each engine can actually carry

EngineCapabilities.multimodal is a declaration of native file-part transport, and it is optional: absent means unsupported, so no engine acquires multimodal support by omission. These are not policy choices — each row states what that engine's transport can carry.

EngineimagepdftextHITL inside an attachment turn
core / mastrayesyesyesrefused
engine-ai-sdkyesyesyesfull attach → suspend → resume cycle
engine-openai-agentsyesnonorefused
a2a · eve · trigger-chatundeclared → unsupportedn/a

pdf: false on engine-openai-agents is a transport fact, not a backlog item: its bridge throws on the file item those classes would need. A PDF or text-class ref sent to it produces an unsupported-engine notice and the run continues without it.

The v1 trades, stated up front

Each of these is a consequence of something below the framework, and each one announces itself on the warning channel rather than degrading quietly.

Approval and attachments are mutually exclusive within one turn — on two of the three engines. On core/mastra and engine-openai-agents a turn carrying attachments cannot be parked: the vendor snapshot embeds the content and there is no safe rewrite seam, so a tool needing a human decision fails with attachments-suspend-unsupported instead of corrupting or leaking state. The tool does not run, no follow-up or snapshot row is written, the run never becomes suspended, and the model sees the denial. Send the request again without the attachment when a human must decide. engine-ai-sdk owns its snapshot format and supports the full cycle: parts are stripped, their positions recorded beside the refs, and re-resolved through the whole admission chain on resume.

On core/mastra, an attachment turn runs memory-stateless. Mastra persists input messages before the model executes, with no seam to send rich input while storing text-only — so that turn skips Mastra's memory wiring and the engine does the work itself: it prepends recalled history and afterwards writes a sanitized, text-only user message plus the assistant message, attribution intact. Recent-history recall is reproduced. Semantic recall, working memory and observational memory are skipped for that turn, read and update alike, and each enabled mode emits its own memory-degraded notice. The next turn without attachments is back to full semantics; turns without attachments never leave the existing path.

A strict workflow runs, without the attachments. Workflow agent steps take a string, and strict-workflow selection happens inside the engine after the agent row loads — the runner cannot know in advance. So the engine emits one workflow-unsupported notice and runs the workflow with no attachments; no ref reaches a model call. That turn no longer carries attachments, so the workflow's own human-approval steps still suspend normally.

Delegates receive text, never a part. Sub-agents are agents-as-tools whose argument schema is a string, generated below the framework. The orchestrator sees the image; a delegate gets whatever text you put in front of it. Scope it deliberately — “the editor can see the screenshot but the media specialist cannot” is a documented consequence, and it arrives as a bug report if nobody said so first.

One sighting per turn, and one hydration path. File parts bind on the first model pass only: a continue-nudge, a cost-cap re-invocation and the next turn in the thread never re-send or re-bill them. Attachments ride the persisted swarm.message event, so a reload through the thread-events timeline renders them — while the older history route, which returns messages rather than events, renders attachments-less. Hydrate from thread events if chips matter after a refresh.

When it refuses, it says so

Nothing about an attachment fails silently, and the division is simple: what the runner rejects comes back in the HTTP response — per-ref dispositions on the acknowledgement, or a 400 when every ref was invalid and there was no text — because the run never started. What the engine hits emits a swarm.attachment_warning event, persisted like every other event and rendered by the bubble it belongs to.

codewhat happened
dropped-invalidThe ref failed shape validation — an unknown key, a nested object, a non-string id, or a url/data-shaped field at any depth.
over-capMore than 10 refs after dedupe, or the serialized array exceeded 8 KB. Refs drop from the tail; each drop is reported.
unsupported-mediaThe resolved kind and canonical mediaType disagree (an image-kind result labelled text/plain). Dropped before any part is built.
unsupported-engineThis engine declares no transport for that class — or no transport profile could be resolved for the provider at all.
resolution-failedNo resolveAttachment is wired, or yours returned unsupported. The reason string is the one you returned.
url-refusedThe resolved URL failed admission (scheme, private host, or the live supportedUrls regex) and the one bytes-only retry produced another URL.
lost-on-resumeengine-ai-sdk re-resolved a ref on resume and could not get it back; the position carries a placeholder instead.
memory-degradedOne enabled advanced memory mode was skipped for this turn (see the trades below). One warning per mode.
attachments-suspend-unsupportedA tool needed a human decision inside an attachment turn on an engine that cannot park one. The tool did NOT run.
workflow-unsupportedA strict workflow was selected for a run carrying attachments. The workflow runs; the attachments do not travel with it.

The first one many deployments will meet is unsupported-engine. An attachment's transport profile is resolved from the model provider, and an agent version published before the provider registry landed carries no provider to resolve from — so every attachment on it refuses, with a message that names the fix: publish the agent version with a provider, or declare attachmentTransportProfile on a custom adapter. Nothing is broken and nothing was sent; re-publish the agent and attachments start working. An adapter may also declare the profile "unsupported" deliberately — unknown means unsupported here, never a silent best-effort degrade.

Two shapes are refused at the door rather than accepted and dropped. A stock ai-sdk FileUIPart in the chat body is rejected by name — it carries a raw URL a server would fetch and persist — with a message naming the custom { type: "attachment", ref } part to use instead; previously it vanished silently. And an attachment-only send against an engine that declares no multimodal support answers 400 rather than launching an empty billed turn.

Reading an attachment from a tool

Formats no provider reads — docx, xlsx, a video transcript — have to become text somewhere, and that somewhere is yours. The tool context carries an ingest-only resolver bound to the run for exactly this.

lib/tools/read-attachment.ts
// A tool that reads an attachment. FR-054's ingestion pipeline uses this same handle.
import { defineTool } from "@nightowlsdev/core";
import { z } from "zod";

export const readAttachment = defineTool({
  name: "read_attachment",
  description: "Read the text of a document the user attached to this conversation.",
  inputSchema: z.object({ attachmentId: z.string() }),
  outputSchema: z.object({ text: z.string() }),
  execute: async ({ attachmentId }, ctx) => {
    // The fourth engine-populated handle. It is ingest-shaped BY TYPE — no URL arm exists to hand a
    // model — it closes over THIS run's trusted scope, and it is `undefined` (never a no-op faking
    // an `unsupported` result) when the host wired no resolver.
    if (!ctx.resolveAttachment) return { text: "no attachment resolver is configured" };

    const resolved = await ctx.resolveAttachment({ id: attachmentId });
    if (resolved.kind === "text") return { text: resolved.text };
    if (resolved.kind === "bytes") return { text: new TextDecoder().decode(resolved.data) };
    return { text: `unavailable: ${resolved.reason}` };
  },
});

Extracted text is untrusted input. A PDF is written by whoever wrote the PDF, and it is about to be read by an agent that may hold publishing or deletion tools. Frame it as advisory reference — the same treatment page context and fetched pages already get — rather than dropping raw document text into a prompt. A dedicated composition helper with its own budget is specified and has not shipped; until it does, use the advisory framing you already have and cap what you inject.