Skip to content
Night Owls.dev
GitHubGet started
Jump to a page

@nightowlsdev/gepa

Quality

A reflective, metric-gated prompt optimizer, reflect on eval feedback to propose better persona fields, keep the best.

What it does

@nightowlsdev/gepa is a reflective prompt optimizer for Night Owls agents: it reflects on an eval's *textual* feedback to propose better persona fields, scores each candidate against a metric (from @nightowlsdev/eval), and keeps the best, a minimal, in-house, metric-gated hill-climb (not the full GEPA algorithm; no DSPy/Ax dependency). It runs OFFLINE (script / CI / job, never inside a user request) and publishes improved versions through the normal versioning path under a service actor. A bounded, inspectable optimizer, NOT autonomous self-improvement: it needs a metric you supply and a human to promote the winner.

Install

pnpm add @nightowlsdev/gepa

Key exports

  • optimizePrompt
  • worstCases
  • types: PromptFields, OptimizeOpts, OptimizeResult

Usage

gepa.ts
import * as pkg from "@nightowlsdev/gepa";

What it provides

gepa is a reflective prompt optimizer for Night Owls agents: optimizePrompt reflects on an eval's TEXTUAL feedback to propose better persona fields, scores each candidate against a metric (an EvalReport from @nightowlsdev/eval), and keeps the best — a minimal, in-house, metric-gated hill-climb. It captures GEPA's core idea (reflection-on-feedback plus metric-gated selection) WITHOUT the full pareto-frontier algorithm and with no DSPy/Ax dependency. Both the evaluation and the reflection LLM are injected seams, so it stays engine-vendor-free and hermetically testable.

When to use it

  • You have a trustworthy eval metric and want to improve an agent's personality / capabilities text against it automatically.
  • You want an inspectable, bounded optimizer you can read end to end — not a black-box tuner.
  • You run optimization OFFLINE (a script, CI, or a job) and a human promotes the winning version.
  • You already use @nightowlsdev/eval and want its per-case textual feedback to drive reflection.

When not to

  • You have no metric you can verify — a weak or gameable eval makes the optimizer produce a prompt that 'scores' better while being worse.
  • You want to tune structural wiring (which skills, delegates, or model an agent has) — gepa optimizes prompt TEXT only, never the wiring.
  • You want inline / online self-improvement inside a user request — gepa is offline and human-gated by design.
  • You expected the full GEPA algorithm (pareto frontier over instances) — this is a best-aggregate hill-climb, deliberately smaller.

Alternatives

  • Manual prompt iterationYou have a handful of cases and a clear intuition — hand-editing the personality and re-running eval is faster than wiring a reflect seam.
  • DSPy / Ax (the full GEPA)You need the complete pareto-frontier-over-instances algorithm and can take on the external dependency. gepa deliberately omits that depth to stay small and dependency-free.

Strengths

  • Minimal and inspectable — a metric-gated hill-climb you can read in one file, not an opaque optimizer.
  • Strictly-beats gating: a candidate is adopted only if it beats the current best, and the base is always evaluated first as the floor — so the result never regresses below where you started.
  • Reflection-on-feedback: worstCases pulls the k lowest cases, each with its worst scorer's TEXTUAL feedback, so the reflector learns WHAT to fix, not just that it scored low.
  • Both seams are injected (evaluate + reflect), so the package carries no Mastra, no AI SDK, no LLM client, no DB, and is testable with plain stubs.
  • Deploy-what-you-measure: if the reflector omits capabilities, the candidate inherits them from the current best, so the scored prompt is exactly the one a deploy step ships.
  • Candidates in a round run concurrently with per-candidate failure isolation — a transient timeout drops that candidate rather than aborting the round.

Limits & trade-offs

  • Not the full GEPA: no pareto-frontier-over-instances — it optimizes for the best aggregate, which can overfit a small dataset.
  • It optimizes only PromptFields (personality + capabilities free-form text); skillNames / delegateSlugs / modelId are structural wiring and are never touched.
  • It is only as good as the metric you supply — garbage-in yields a confidently-worse prompt.
  • Offline only — never runs inside a user request — and it needs a human to promote the winner through the normal versioning path (not autonomous).
  • You must wire a reflection LLM (the reflect seam); the package provides none.

How it works

optimizePrompt(opts) evaluates the base PromptFields first — that score is the floor. Then, for each round, worstCases(bestReport, reflectTopK) selects the lowest-scoring cases (each carrying its worst scorer's feedback) and hands them to the injected reflect({ fields, failures }) to propose candidate fields; each candidate is scored by the injected evaluate(fields) (typically wrapping @nightowlsdev/eval's runEval), and a candidate is adopted only if it STRICTLY beats the current best. It stops early when there are no failures left or the aggregate hits 1 (a perfect score can only tie). Candidates within a round run concurrently and a throwing candidate resolves to null and is dropped. The OptimizeResult carries best, baseScore, bestScore, improved, and the per-round history.

Examples

Optimize an agent's persona against an eval metric

evaluate wraps eval's runEval (the fitness); reflect is your frontier-tier reflection LLM.

gepa-example-1.ts
import { optimizePrompt } from "@nightowlsdev/gepa";
import type { PromptFields } from "@nightowlsdev/gepa";
import { runEval } from "@nightowlsdev/eval";

const base: PromptFields = {
  personality: "Terse support triager.",
  capabilities: ["triage tickets"],
};

const result = await optimizePrompt({
  base,
  rounds: 3,
  candidatesPerRound: 2,
  reflectTopK: 3,
  // Fitness: score these fields with the eval suite; the report's aggregate is the signal.
  evaluate: (fields) =>
    runEval({ suite: "support", agentSlug: "triager", dataset, runAgent: runWith(fields), scorers }),
  // Reflection LLM: read the worst cases' feedback, propose better persona fields.
  reflect: ({ fields, failures }) => myReflector(fields, failures),
});

if (result.improved) {
  // Offline only: a HUMAN promotes result.best through the normal versioning path.
  console.log(result.bestScore, result.best.personality);
}

Inspect the worst cases feeding reflection

worstCases is exported directly, so you can preview what the reflector will see.

gepa-example-2.ts
import { worstCases } from "@nightowlsdev/gepa";
import { runEval } from "@nightowlsdev/eval";

const report = await runEval({ suite: "support", agentSlug: "triager", dataset, runAgent, scorers });
const failures = worstCases(report, 3); // [{ input, output, feedback }] — lowest-scoring first

Doing the parts it doesn't support

  • Tuning tools, delegates, or the modelgepa optimizes prompt TEXT only (personality + capabilities). Structural wiring is out of scope — change skillNames / delegateSlugs / modelId through the agent-config surface and re-run the eval to measure the effect.
  • Publishing the winnergepa returns result.best but never publishes. Promote it through the normal versioning path under a service actor (offline), with a human approving the promotion — the optimizer is bounded, not autonomous.
  • The full pareto-frontier GEPAThis is a best-aggregate hill-climb. For pareto-over-instances, reach for DSPy/Ax; gepa deliberately trades that depth for a dependency-free, inspectable core.

Related

  • eval — Supplies the EvalReport fitness function and the per-case textual feedback gepa reflects on.
  • core — composeSystemPrompt renders exactly the PromptFields gepa optimizes; agent versions live here.
  • agent-builder — Where a human promotes the optimized version through the versioning path.