@nightowlsdev/eval
QualityThe scoring plane, run an agent over a fixture dataset, score each run's event trajectory, aggregate into a per-version EvalReport.
What it does
@nightowlsdev/eval is the scoring plane for Night Owls: `runEval` executes an agent over a fixture dataset (via an injected RunAgent seam), scores each run's `SwarmEvent` trajectory with a set of named scorers, and aggregates the verdicts into a per-version `EvalReport`, a CI regression signal that doubles as a prompt-optimizer's fitness function. Ships deterministic scorers (no-failure, completion, tool-success, expected-tool) plus an optional LLM-judge scorer whose model call is INJECTED, so the package stays @mastra-free and hermetically testable. This is the 'plug in a metric you can verify → self-correct against it' seam, the honest, bounded half of self-improvement.
Install
pnpm add @nightowlsdev/evalKey exports
- runEval
- defaultDeterministicScorers
- noFailureScorer / completionScorer / toolSuccessScorer / expectedToolScorer / llmJudgeScorer
- types: Scorer, EvalCase, EvalReport
Usage
import * as pkg from "@nightowlsdev/eval";