@nightowlsdev/eval

Quality

The scoring plane, run an agent over a fixture dataset, score each run's event trajectory, aggregate into a per-version EvalReport.

What it does

@nightowlsdev/eval is the scoring plane for Night Owls: `runEval` executes an agent over a fixture dataset (via an injected RunAgent seam), scores each run's `SwarmEvent` trajectory with a set of named scorers, and aggregates the verdicts into a per-version `EvalReport`, a CI regression signal that doubles as a prompt-optimizer's fitness function. Ships deterministic scorers (no-failure, completion, tool-success, expected-tool) plus an optional LLM-judge scorer whose model call is INJECTED, so the package stays @mastra-free and hermetically testable. This is the 'plug in a metric you can verify → self-correct against it' seam, the honest, bounded half of self-improvement.

Install

pnpm add @nightowlsdev/eval

Key exports

runEval
defaultDeterministicScorers
noFailureScorer / completionScorer / toolSuccessScorer / expectedToolScorer / llmJudgeScorer
types: Scorer, EvalCase, EvalReport

Usage

eval.ts

import * as pkg from "@nightowlsdev/eval";

What it provides

eval is the scoring plane for Night Owls: runEval executes an agent over a fixture dataset (via an injected RunAgent seam), scores each run's SwarmEvent trajectory with a set of named scorers, and aggregates the verdicts into a per-version EvalReport — a CI regression signal that doubles as a prompt-optimizer's fitness function. It ships deterministic scorers (no-failure, completion, tool-success, expected-tool) plus an optional LLM-judge scorer whose model call is injected, so the package stays engine-vendor-free and hermetically testable. This is the 'plug in a metric you can verify, then self-correct against it' seam.

When to use it

You want a repeatable regression signal for an agent version in CI — score the same fixtures on every change and fail on a drop.
You are optimizing prompts with gepa and need a fitness function whose per-case textual feedback drives the reflection.
You want cheap, trustworthy scoring that reads the event log (no model call) before layering in an advisory LLM judge.
You want scoring that stays deterministic and unit-testable — both the agent run and the judge are injected, so you can test with plain fakes.

When not to

You just want to eyeball a single run — the harness is overkill for a one-off.
You have no metric you actually trust — an LLM judge you cannot verify is not a gate, and a bad metric trains an optimizer on noise.
You want a full experiment-tracking / dataset-management platform — this stores nothing; the dataset is an in-memory EvalCase[].

Alternatives

An ad-hoc assertion scriptYou have one or two checks and no need for per-scorer aggregation, textual feedback, or a report shape a reflector can consume. You lose the failure-isolation and the gepa fitness contract.
A hosted eval platformYou need dataset versioning, run history, and dashboards out of the box. eval is deliberately a library, not a service — it produces the EvalReport and hands it back.

Strengths

Engine-wall clean: only a type-only dependency on core (for SwarmEvent). The agent run and the judge are injected, so scoring stays deterministic and testable with fakes.
Deterministic scorers need no model call — cheap, flake-free signal you can trust in CI (tool-success reads normalizeToolResult, not a raw ok flag, so an envelope-returning tool that failed is scored as failed).
Feedback is TEXTUAL, not just a scalar — it is what a human reads and what gepa reflects on to improve a prompt.
Per-case failure isolation: a runAgent crash or a throwing scorer is recorded as score 0 with the error as feedback, so one bad case never aborts the suite (a partial report beats none).
The EvalReport is attributable to an agentVersion, so scores tie to a specific prompt version.

Limits & trade-offs

You write the RunAgent drain yourself — the harness never runs agents; the host owns how a run executes and gets drained to { events, output }.
The built-in deterministic scorers only observe the trajectory + final text — there is no golden-output diff scorer, and EvalCase.expect is a reserved bag the built-ins do not read.
The LLM-judge scorer is advisory and injected: you supply the judge model call, and its scores are non-deterministic trend signals, not hard gates.
No dataset management or storage — the dataset is an in-memory array you assemble each run.

How it works

runEval(opts) loops over the dataset. For each EvalCase it calls the injected runAgent(testCase), which drains a real agent run into { events, output } — the SwarmEvent trajectory plus the final assistant text. It then applies every Scorer to that ScoredRun, each returning a 0..1 score with textual feedback, and aggregates per-case (unweighted mean) and per-scorer (mean across cases) into an EvalReport. Deterministic scorers read the event log: noFailureScorer (no terminal swarm.run_failed), completionScorer (non-empty output), toolSuccessScorer (fraction of tool_results that did not fail), and expectedToolScorer(name) (a specific tool was called). llmJudgeScorer defers to an injected JudgeFn. A thrown runAgent or scorer is isolated to that case as score 0.

Examples

Run a suite with deterministic scorers + an expected tool

Drain a real agent run to { events, output }; runEval scores and aggregates it.

eval-example-1.ts

import { runEval, defaultDeterministicScorers, expectedToolScorer, type EvalCase, type RunAgent } from "@nightowlsdev/eval";
import type { SwarmEvent } from "@nightowlsdev/core";

const dataset: EvalCase[] = [
  { id: "ticket", input: "log a support ticket for a broken export" },
];

// The engine seam: drive a run and DRAIN it to its trajectory + final text.
const runAgent: RunAgent = async (c) => {
  const events: SwarmEvent[] = [];
  let output = "";
  for await (const ev of engine.run({ message: c.input }, ctx)) {
    events.push(ev);
    if (ev.type === "swarm.message") output += ev.data.delta ?? ev.data.text ?? "";
  }
  return { events, output };
};

const report = await runEval({
  suite: "support", agentSlug: "triager", agentVersion: 3,
  dataset, runAgent,
  scorers: [...defaultDeterministicScorers, expectedToolScorer("log_ticket")],
});
// report.aggregate is the regression signal; report.cases carry the per-case textual feedback.

Add an advisory LLM-judge scorer

The judge model call is injected, so the package stays engine-vendor-free and testable.

eval-example-2.ts

import { llmJudgeScorer, type JudgeFn } from "@nightowlsdev/eval";

const judge: JudgeFn = async ({ rubric, question, answer }) => {
  const verdict = await gradeWithMyModel(rubric, question, answer); // your model call
  return { score: verdict.score, feedback: verdict.notes };
};

const helpfulness = llmJudgeScorer({
  name: "helpfulness",
  rubric: "Did the answer resolve the user's request clearly?",
  judge,
});
// Add `helpfulness` to runEval's scorers. Judge scores are advisory trend signals, not hard gates.

Doing the parts it doesn't support

Actually executing the agent runseval never runs an engine. Implement RunAgent to drive engine.run(...) (or a runner) and drain the async event stream into { events, output } — the host owns how runs execute, mirroring its real extractOutput.
Golden-output or per-case expected-tool assertionsThe built-in scorers observe the trajectory + final text. For a per-case assertion, write your own Scorer that reads EvalCase.expect (the reserved bag) off the run, or compose expectedToolScorer(name) per suite.

gepa — Consumes the EvalReport as its fitness function and the per-case feedback as its reflection signal.
core — The SwarmEvent trajectory eval scores comes from core's run loop.
agent-builder — Where an optimized/scored agent version is published through the normal versioning path.