Get a 40-page document into the index. Batched, resumable, never half-searchable.

Storing a document is the step in the middle nobody specified: you can decide whether content should be kept, enumerate what is already stored, and carry a file to the agent, but taking a long transcript and actually getting it into the knowledge base was a host-private script. The old path embedded every chunk in one provider call and upserted them in one transaction, so a 500-chunk document could never land while any single batch was flaky, and the failure looked like a timeout rather than "your document was too big". This pipeline chunks, admits, and embeds a whole document in batches that commit as they go, behind an atomic finalize, so a partial ingest is resumable and never retrievable until it is whole. Pass no new fields and an existing ingest call behaves exactly as today.

IngestArgs.text is a plain string, so every adopter used to invent the mediaType → text step privately: no shared failure taxonomy, no structure preservation, and no way for the framework to refuse an empty extraction. The DocumentExtractor contract lives in @nightowlsdev/core; a host registers one on SwarmConfig.extractors, and it wins over the built-in identity extractor (which handles text/* and application/json and fetches nothing).

import type { DocumentExtractor, ExtractedDocument, ExtractorInput } from "@nightowlsdev/core";

// The framework ships an identity extractor for text/* and application/json only. Anything else is
// host-side: register your own on SwarmConfig.extractors and it wins over the built-in.
export const pdfExtractor: DocumentExtractor = {
  supports: (mediaType) => mediaType === "application/pdf",
  async extract(input: ExtractorInput): Promise<ExtractedDocument> {
    if (input.kind !== "bytes") throw new Error("pdf extraction needs bytes");
    const parsed = await parsePdf(input.bytes);
    return {
      text: parsed.text,               // whitespace-only => the wrapper rejects it (kind:"empty"), never ingested
      mediaType: input.mediaType,
      pages: parsed.pageCount,
      structure: parsed.headings,      // heading offsets INTO text — chunking snaps boundaries to them
      truncated: parsed.truncated,     // a host cap fired — surfaced onto the report, never silent
      warnings: parsed.imageOnlyPages > 0
        ? [String(parsed.imageOnlyPages) + " pages were images with no text layer"]
        : undefined,
    };
  },
};

// Your PDF/Office library (pdf-parse, unpdf, mammoth, officeparser). None ship with the framework.
declare function parsePdf(bytes: Uint8Array): Promise<{
  text: string;
  headings: Array<{ heading: string; charOffset: number }>;
  pageCount: number;
  truncated: boolean;
  imageOnlyPages: number;
}>;

The pipeline is chunk → classify every chunk → embed only what is admitted → upsert. The ordering is a cost decision, not a stylistic one: admission before embedding means you never pay a provider to embed a chunk you are about to quarantine. The consequence, stated plainly, is that releasing a quarantined chunk later requires embedding it then — a small deferred cost, not a free flag flip.

Chunking is deterministic, so re-ingesting an unchanged sourceVersion produces the same chunkIdxes — which is the idempotent upsert key. Defaults are 1200 characters with 150 of overlap. When the extractor supplied structure, a boundary within ±20% of the target size snaps to the nearest heading offset rather than the raw whitespace break, because a chunk that straddles two sections retrieves worse than either. Tune size up for reference tables, down for chatty transcripts; the admission gate is per chunk, so smaller chunks classify more finely.

Batching moved into the store, so behaviour no longer depends on whether the host's injected embedder happens to batch. batchSize (default 64, validated 1..512) is the chunks-per-embed-call, and each batch commits on its own. The 5-tuple already makes every chunk row idempotent, so a per-batch commit plus resume: true turns a timeout into "continue from chunk 384" instead of a rollback of the whole document.

import { createKnowledgeStore } from "@nightowlsdev/knowledge";

const kb = createKnowledgeStore({
  dbUrl: process.env.KB_DATABASE_URL!,
  embedder: myEmbedder,
  // Store-level chunking defaults, overridable per ingest. 1200 chars is a few paragraphs; 150 overlap
  // keeps a fact that straddles a boundary retrievable from either chunk.
  chunk: { size: 1200, overlap: 150 },
});

// A long document, ingested from a queue worker. Batched INSIDE the store and resumable, so a mid-document
// embed timeout becomes "continue from chunk 384" instead of "start over, forever".
export async function ingestGuide(tenantId: string, sourceVersion: string, text: string) {
  const report = await kb.ingest({
    tenantId,
    sourceType: "guide",
    sourceId: "onboarding.md",
    sourceVersion,          // the re-ingest unit: an edited document bumps this and re-chunks WHOLESALE
    text,
    batchSize: 64,          // chunks per embed call, committed per batch (not one document-long transaction)
    resume: true,           // skip chunk_idx values already committed for this exact 5-tuple, then finalize
    onProgress: (p) => console.log("embedded " + p.embedded + "/" + p.total + " (batch " + p.batch + "/" + p.batches + ")"),
  });
  if (!report.complete) {
    // complete:false is a first-class OUTCOME, not a throw. Re-run with resume:true to finish the version.
  }
  return report;
}

declare const myEmbedder: import("@nightowlsdev/knowledge").Embedder;

Skipping already-present chunk_idxes is only safe if the resume runs the same plan. A changed admission policy, a re-extraction that moved a heading, an edited paragraph, a different target-visibility rule — any of these would splice two corpora into one document, silently. So each version carries a plan_fingerprint on its manifest, and a resume is a true continuation only when it matches.

// Resume is gated on the chunk-plan FINGERPRINT, not on chunk_idx presence alone. The fingerprint
// (computeChunkPlanFingerprint) hashes, in a length-prefixed canonical frame:
//   chunker version, size, overlap, structure offsets, the BUILT-IN admission version (always),
//   the host admission policy, the target-visibility rule + its host-owned version, the org scope,
//   and the full text.

// Same 5-tuple, IDENTICAL plan, incomplete  ->  resume:false REJECTS (IngestIncomplete); pass resume:true
// Same sourceVersion, DIFFERENT plan         ->  IngestPlanMismatch — bump sourceVersion, never mix corpora
// Same identity, DIFFERENT orgScope          ->  IngestScopeMismatch — use setScope to move it, don't re-label

The built-in admission version is hashed unconditionally, policy or no policy, which closes the policy-less hole: a corpus ingested under the old built-in detectors is invalidated by a detector change exactly as a policy-bearing one is. The visibility rule is expressed as a host-owned visibilityForVersion string, never a hash of the callback's source — a function-source hash fails open, because two closures with different captured state stringify identically. And the manifest is the enforcement point that survives an all-rejected version, where no chunk row exists to expose a scope or plan collision at all.

ingest resolves to a KnowledgeIngestReport — never throws for a partial document. Its numbers are a partition, and they close: the classification counts are invariant across every invocation of one fingerprint, while embedded and skipped are per-invocation progress.

// KnowledgeIngestReport — the store.ingest return (and the retention receipt's report).
{
  chunks: 512,        // every chunk the plan CLASSIFIED, rejected ones included — the progress denominator
  embedded: 384,      // admitted chunks embedded AND committed by THIS invocation
  skipped: 96,        // admitted chunks already committed by a prior run (resume found them present)
  quarantined: 12,    // verdict "quarantine" — stored, UNEMBEDDED, never retrievable
  rejected: 8,        // verdict "reject" — never embedded, never stored at all
  complete: false,    // pending = admitted - embedded - skipped > 0 — re-run with resume:true
}

// The equations that hold on EVERY return (with admitted = chunks - quarantined - rejected):
//   chunks   = admitted + quarantined + rejected
//   admitted = embedded + skipped + pending
//   complete === (pending === 0)

complete: false is not an error state; it is "pending > 0, call again with resume". An identical re-ingest of a finalized version is a true no-op that reports embedded: 0, skipped: admitted, so a retry loop is cheap and honest. truncated and warnings ride through from extraction, and deleted is an advisory tally of surgical chunk removals, deliberately outside the equations.

The whole pipeline exists to answer a query. Once a version is finalized, its admitted chunks are exactly what search_knowledge returns — embedded with the same injected embedder, filtered by tenant, by the fail-closed visibility allow-list, and by the caller's scope, then fenced before the model sees them.

import { searchKnowledgeTool } from "@nightowlsdev/knowledge";

// Grant this to an agent as a skill. It embeds the query with the SAME injected embedder, filters by
// tenant + the fail-closed visibility allow-list + the caller's scope (Rule A), and FENCES the snippets.
const searchTool = searchKnowledgeTool(kb, { egress: false });

// A finalized document's admitted chunks are visible. Its quarantined chunks carry the reserved
// "quarantine" visibility and its still-ingesting rows carry "ingesting" — both OUTSIDE the allow-list,
// so a partial or quarantined chunk is unreachable by retrieval until finalize flips it to its target.

The two reserved visibilities are the seam between ingest and retrieval: ingesting hides a document while its batches are still landing, and quarantine hides a chunk admission judged sensitive. Neither can appear in a configured allow-list — they are reachable only through an explicit per-call filter.visibility, which is how an operator review path reads what admission set aside, at the call site, on a service-role store.

When an operator drops a document in chat, the agent needs something to call. That something is the retention flow: this pipeline is the first-party knowledge executor behind it. Registering the destination composes the pieces — resolve the attachment, extract, chunk, admit per chunk, embed only what is admitted, upsert — rather than duplicating them.

// FR-051 wires the agent-callable path; FR-054's executor is what runs behind it.
import { knowledgeRetentionExecutor } from "@nightowlsdev/knowledge";

// Register the knowledge destination. The engine then exposes two tools to the agent:
//   propose_retention  — a SUSPENDING intrinsic: it parks for operator approval; no row exists yet
//   ingest_document    — the per-destination executor that runs ONLY on the approved resume
retention: {
  executors: [knowledgeRetentionExecutor({ store: kb })],
}

// ingest_document is a durable, org-visible write (needsApproval, danger >= 2 under FR-044), and it is
// in RESERVED_STORED_SKILL_NAMES so an imported skill cannot shadow the ingest path. The org scope comes
// from ctx.orgScope, NEVER a tool argument — core's never-a-tool-argument test pins that repo-wide.

propose_retention suspends for operator approval and no durable row exists before that decision; ingest_document runs only on the approved resume, derives the document identity from the approved snapshot rather than execution-time arguments, and earns a verified: true receipt only when the version finalized with at least one admitted chunk. An all-rejected or still-incomplete document yields no claimedRef, so the receipt honestly reads verified: false.

• ingest(args): Promise<KnowledgeIngestReport> on createKnowledgeStore. New fields: batchSize, resume, onProgress, visibilityFor / visibilityForVersion, and admission. All additive.
• DocumentExtractor, ExtractedDocument, ExtractorInput, ExtractionError, identityExtractor, extractDocument — the extraction contract, all from @nightowlsdev/core.
• chunkText / ChunkOptions / CHUNKER_VERSION: deterministic, structure-aware chunking.
• computeChunkPlanFingerprint / ChunkPlanInputs: the resume-safety fingerprint, exposed for host revalidation.
• IngestPlanMismatch, IngestIncomplete, IngestScopeMismatch, KnowledgeReleaseNotFinalized: the typed rejects.
• knowledgeRetentionExecutor: the first-party knowledge retention executor behind ingest_document.
• knowledgeMigrations({ dimensions }) now includes knowledge_0005 (nullable embedding, a partial ANN index, the unembedded-is-quarantined CHECK, target_visibility, and the kb_ingest_state manifest). Eject the array and apply in order; a migration you add takes knowledge_0006.