Skip to content
Night Owls.dev
Jump to a page

@nightowlsdev/eval

Quality

The scoring plane, run an agent over a fixture dataset, score each run's event trajectory, aggregate into a per-version EvalReport.

What it does

@nightowlsdev/eval is the scoring plane for Night Owls: `runEval` executes an agent over a fixture dataset (via an injected RunAgent seam), scores each run's `SwarmEvent` trajectory with a set of named scorers, and aggregates the verdicts into a per-version `EvalReport`, a CI regression signal that doubles as a prompt-optimizer's fitness function. Ships deterministic scorers (no-failure, completion, tool-success, expected-tool) plus an optional LLM-judge scorer whose model call is INJECTED, so the package stays @mastra-free and hermetically testable. This is the 'plug in a metric you can verify → self-correct against it' seam, the honest, bounded half of self-improvement.

Install

pnpm add @nightowlsdev/eval

Key exports

  • runEval
  • defaultDeterministicScorers
  • noFailureScorer / completionScorer / toolSuccessScorer / expectedToolScorer / llmJudgeScorer
  • types: Scorer, EvalCase, EvalReport

Usage

eval.ts
import * as pkg from "@nightowlsdev/eval";