Skip to content
Night Owls.dev
GitHubGet started
Jump to a page

@nightowlsdev/knowledge

Adapter/Storage

An optional pgvector document knowledge base, RAG over arbitrary documents (guides, brand voice, reference docs), plus a ready tenant-scoped `search_knowledge` tool.

What it does

A document knowledge base distinct from conversation memory (which is thread-scoped): agents retrieve reference material by meaning. `knowledgeMigration({ dimensions })` returns a migration for `nightowls.kb_documents`, a `vector(N)` column (sized to your embedder) with an HNSW cosine index, RLS, first-class scoping columns (`org_id`/`source_type`/`source_id`/`source_version`/`visibility`), and a `(org_id, source_id, source_version, chunk_idx)` idempotency key. `createKnowledgeStore({ pool | dbUrl, embedder })` ingests documents (deterministic chunk → embed → idempotent upsert), searches (embed the query → an ALWAYS org-scoped, optionally source_type/visibility-filtered pgvector cosine search), and prunes superseded versions. The embedder is injected (the swarm's text provider may not embed, e.g. Ollama Cloud has no embeddings endpoint). `searchKnowledgeTool(store)` is a ready read-only `search_knowledge` SwarmTool that takes the tenant from the run context (never a tool arg, so the model can't cross tenants) and fences retrieved snippets as untrusted reference material. ⚠ NOT for metrics: this plane holds text for semantic retrieval, so a number embedded in a document comes back through a similarity search with no notion of ordering, of "the value on the 3rd", or of summing a month. The neighbouring temptation is worse, `@nightowlsdev/graph`'s `corroborate` never updates an edge's properties, so a graph "fact" recording a number silently keeps the FIRST value it ever saw while its confidence rises with every repetition. Both failures are silent. Numbers that change over time belong in `@nightowlsdev/metrics`.

Install

pnpm add @nightowlsdev/knowledge

Key exports

  • knowledgeMigration
  • createKnowledgeStore (+ listSources / getSource / deleteSource / stats)
  • createKnowledgeHandlers (FR-042: framework-agnostic Request/Response)
  • searchKnowledgeTool
  • chunkText
  • Embedder

Usage

knowledge.ts
import { knowledgeMigration, createKnowledgeStore, searchKnowledgeTool } from "@nightowlsdev/knowledge";
import { defineAgent } from "@nightowlsdev/core";

// 1. Eject the migration (dimensions MUST match your embedder, e.g. 1536 for text-embedding-3-small).
export const MIGRATIONS = [/* …engine migrations… */ knowledgeMigration({ dimensions: 1536 })];

// 2. Build the store. The embedder is INJECTED (batch text -> vectors of length `dimensions`).
const kb = createKnowledgeStore({ pool, embedder });

// 3. Ingest documents (chunk -> embed -> idempotent upsert), then grant the ready tool to an agent.
await kb.ingest({ tenantId, sourceType: "guide", sourceId: "brand-voice", text: brandVoiceMd });
const editor = defineAgent({ slug: "editor", skills: [searchKnowledgeTool(kb)], /* … */ });
// The tool takes the tenant from the run context (never a tool arg) and fences retrieved snippets.

What it provides

knowledge is an optional pgvector document knowledge base: RAG over arbitrary documents — brand voice, guides, reference docs — distinct from conversation memory, which is hard-pinned to the current thread. createKnowledgeStore({ pool, embedder }) ingests documents (deterministic chunk, embed, idempotent upsert) and searches them by meaning, always bound to org_id in code before ranking. searchKnowledgeTool(store) is a ready read-only search_knowledge tool that takes the tenant from the run context (never a tool argument) and fences retrieved snippets as untrusted reference material.

When to use it

  • Agents need to retrieve reference material by meaning — a handbook, brand voice, product docs, past reports.
  • You want citation-grade verbatim chunks in the model's context, not distilled entities (that is @nightowlsdev/graph).
  • Multi-tenant retrieval where the tool must never let the model cross tenants — the tenant comes from ctx.
  • Your swarm's text provider cannot embed (e.g. Ollama Cloud has no embeddings endpoint) — the embedder is injected, so you pick any.

When not to

  • You are storing a NUMBER that changes over time — impressions, revenue, latency, a score. A similarity search has no notion of ordering, of 'the value on the 3rd', or of summing a month. Use @nightowlsdev/metrics.
  • You need relationships, provenance, or 'when did it stop being true' — that is @nightowlsdev/graph.
  • You want thread-scoped conversation recall — that is core's memory.semanticRecall, hard-pinned to the current thread.

Alternatives

  • @nightowlsdev/graphYou need resolved entities and relations, structural provenance, and bi-temporal invalidation — reasoning and navigation rather than verbatim text. Ingest a document into both: knowledge quotes it, graph reasons over it.
  • @nightowlsdev/metricsThe data is a numeric time series — an (org, scope, metric, at, dims) observation plane with calendar bucketing and a period-over-period compare().
  • core memory.semanticRecallYou want recall of what was said in THIS thread. Knowledge is cross-document and not per-thread; semantic recall is the thread-scoped sibling.

Strengths

  • A ready-made read-only tool: tenant-from-ctx, output fenced untrusted, needsApproval:false.
  • The embedder is injected — decoupled from the swarm's text model; any dimensions, as long as the migration matches.
  • Idempotent, versioned ingest: re-ingesting a version upserts chunks in place; bump the version to replace, prune the superseded chunks.
  • Batched, resumable, never-partially-searchable ingest (FR-054): a crash leaves either the old version or the new one live, never a half-written mix.
  • Visibility fails closed (default ['org']) and FR-055 sub-org scope (Rule A on reads, exact-scope on writes) is inert until you use it.
  • The FR-044 egress declaration on the tool lets the approval gate treat a hosted embedder honestly instead of guessing.

Limits & trade-offs

  • You own the pg Pool and must apply ALL migrations in order — knowledgeMigrations(), not the 0001-only knowledgeMigration(); the vector dimension must match your embedder or ingest fails outright.
  • No job runner — a large document must be handed to a background worker (onProgress + resume), never ingested inside a request handler.
  • Not for numbers or relationships — a similarity search over an embedded number is a silent wrong answer the README calls out explicitly.
  • egress is undeclared by default, which the danger matrix reads as unsafe (danger 2), so the tool asks under 'auto' until you declare it.
  • deleteByScope offboarding requires an onOffboard audit sink wired first; runs / threads / events have no purge path today.

How it works

knowledgeMigrations({ dimensions }) creates nightowls.kb_documents — a vector(N) column with an HNSW cosine index, RLS, first-class scoping columns (org_id / source_type / source_id / source_version / visibility / org_scope), and an idempotency key on (org_id, source_type, source_id, source_version, chunk_idx). createKnowledgeStore({ pool, embedder }) chunks text deterministically, embeds admitted chunks in batches under a reserved 'ingesting' visibility, then runs one atomic finalize that flips them live and prunes the prior version — so the corpus is never half-written. search embeds the query and runs a two-stage ANN-then-filter pgvector cosine search, always bound to org_id before ranking. searchKnowledgeTool(store, { egress }) wraps that as a search_knowledge SwarmTool whose tenant comes from ctx and whose snippets are fenced as untrusted reference material.

Examples

Migration, store, ingest, grant the tool

Eject ALL migrations (dimensions must match your embedder), build the store with an injected embedder, ingest, then grant the read-only tool.

knowledge-example-1.ts
import { knowledgeMigrations, createKnowledgeStore, searchKnowledgeTool } from "@nightowlsdev/knowledge";
import { defineAgent } from "@nightowlsdev/core";

// 1. Eject EVERY migration, in order — 1536 = openai text-embedding-3-small.
export const MIGRATIONS = [...engineMigrations, ...knowledgeMigrations({ dimensions: 1536 })];

// 2. The embedder is INJECTED (batch text into vectors of the configured length).
const kb = createKnowledgeStore({ pool, embedder });

// 3. Ingest — chunk, embed, idempotent upsert by source_type + source_id + version + chunk_idx.
await kb.ingest({ tenantId: orgId, sourceType: "guide", sourceId: "brand-voice", text: brandVoiceMd });

// 4. Grant the read-only tool. egress:true = a hosted embedder (the query text leaves); false = local.
const editor = defineAgent({ slug: "editor", skills: [searchKnowledgeTool(kb, { egress: true })] });

A direct, host-side search

search is ALWAYS org-scoped in code and can be narrowed by source_type or visibility.

knowledge-example-2.ts
const hits = await kb.search({
  tenantId: orgId,
  query: "what is our refund policy?",
  topK: 5,
  filter: { sourceType: "guide" },
});

Batched, resumable ingest for a large document

A big document belongs in a background worker — this package ships no job runner. Persist complete:false and drive resume until it returns complete:true.

knowledge-example-3.ts
let report = await kb.ingest({
  tenantId: orgId, sourceType: "guide", sourceId: "handbook", sourceVersion: "2026-06",
  text, batchSize: 64,
  onProgress: (p) => log(p.embedded + "/" + p.total),
});
// report: { chunks, embedded, skipped, quarantined, rejected, complete, ... }
if (!report.complete) {
  // same identity + same text, resume the version an earlier call left incomplete.
  report = await kb.ingest({ tenantId: orgId, sourceType: "guide", sourceId: "handbook", sourceVersion: "2026-06", text, resume: true });
}

Doing the parts it doesn't support

  • Storing numbers that change over timeDo not embed a metric into a document — a similarity search cannot order or sum it. Use @nightowlsdev/metrics, an (org, scope, metric, at, dims) observation plane with bucketing and compare().
  • Relationships and provenanceFor 'who told us, when did it stop being true, what else connects to it', use @nightowlsdev/graph. Ingest a document into both — knowledge keeps verbatim chunks, graph keeps distilled entities.
  • Async ingest of a large documentThis package ships no job runner. onProgress + resume + complete:false are exactly a queued worker's primitives — hand a big document to a background worker and drive resume:true until complete:true.
  • An HTTP surface over the storecreateKnowledgeHandlers({ store, auth }) exposes framework-agnostic (Request) => Promise<Response> handlers for search / sources / stats / delete. auth is mandatory (a null or a throw is a 401); there is no unauthenticated path.

Related

  • graph — The neighboring plane for entities, relations, and provenance — ingest a document into both.
  • metrics — Where numbers that change over time belong; a knowledge search cannot order or sum them.
  • document-ingestion — The pipeline guide: chunk, admit, and embed a whole document, resume-safe and never partially searchable.
  • knowledge-and-tools — Enumerate and delete knowledge sources, and manage the surrounding tool grants.
  • org-scope — The FR-055 sub-org partition (Rule A reads, exact-scope writes) this store implements.