> ## Documentation Index > Fetch the complete documentation index at: https://overmind-b84ae13c-codex-workshop-queue-capacity.mintlify.site/llms.txt > Use this file to discover all available pages before exploring further. # Eval > Evaluator kinds and their configuration, claims and behaviour bindings, eval sets, live trace scoring, and eval runs — with the endpoints behind them. Eval is where you define what "good" means for a capability and measure it — in batch runs that compare models and prompts side by side, and continuously on production traces as they arrive. The same evaluators serve both. The **Evaluations** area of the Console has three surfaces: **Runs**, **Eval sets**, and the **Eval library**. ## Evaluators Generated evaluations preserve provider finish reasons. A response stopped by its output token limit remains inspectable, but the sample is marked degraded and its judgments are skipped rather than assigned quality zeros. Degraded samples are excluded from trusted score aggregates and counted in run health. Check this coverage before comparing model quality: a high average over the few complete samples is not a successful overall run. Training-linked evaluations with degraded evidence report failure and no headline comparison score. Rubric judges default to generative grading. Qualified bounded checklist, claim-support and per-turn decisions can opt into [Jev with generative fallback](/agent-testing/decision-engine). Authoring and holistic reasoning remain generative. Evaluator policies and score provenance identify which path ran. An evaluator is one reusable, versioned scorer — editing one writes a new `version` row, so past runs keep meaning what they meant. Five kinds exist: A model grades the response against a rubric you write in plain language. The rubric is compiled into a checklist of atomic, weighted yes/no questions; on a boolean evaluator an item can be a **gate** that fails the verdict regardless of the rest. Judges return reasoning and per-item sub-scores. The managed Correctness and Faithfulness judges use a **proportional** mode instead: they enumerate the output's claims and score the share that holds. Judge responses are cached content-addressed, so re-scoring identical inputs is free. A fixed rule, no model involved. Checks: `exact_match`, `contains`, `regex`, `json_schema_valid`, `schema_field_conformance`, `field_present`, `reference_field_compare`, `canonical_fields`, `tool_selection` (precision/recall/F1 over called tools), `tool_args_match`, `goal_state_match`, `card_constraints`, `latency`, `cost`, and `turn_count`. Judges the path the agent took, not just the output. Two modes: **match** compares the tool-call sequence against a reference with `strict`, `unordered`, `subset`, `superset`, or `subsequence` semantics (with per-tool argument-match overrides), or **judge** hands the trajectory to an LLM. An LLM judge for long, multi-step runs; on live traces the summarising cascade keeps the judge under its context budget. Scores a whole dataset at once rather than one example at a time: `accuracy`, `precision`, `recall`, `f1` (macro/micro/weighted), `calibration` (expected calibration error with a bootstrap interval), `exact_match`, `bleu`, `rouge_l`, `embedding_cosine`, `chrf`, and `duplicate_rate`, with `output_normalize` set to `strip` (default), `first_word`, `last_word`, `first_line`, `last_line`, or a `regex:` extractor. When the references are labels, the run also reports per-class precision, recall, F1 and a confusion matrix. Every evaluator declares a **scope** (`final_output`, `turn`, `step`, `trajectory`, `sample`, or `dataset`), a **score type** (`numeric`, `categorical`, or `boolean`) with a `pass_threshold`, an **evidence requirement** (`model_output`, `harness_artifact`, `trajectory`, or `reference`), and a **surface** — `model`, `harness`, or `any` — that says where it may run: a `model` grader never runs on live trace scoring, a `harness` grader never runs on a generate variant. If a run cannot supply the required evidence, the evaluator abstains or is marked not-applicable instead of producing a number. ### Claims Every evaluator also carries a **claim**: what it asserts and at which grain. The claim is what [trace scoring](/agent-testing/trace-scoring#verdicts-and-the-execution-score) uses to compose one execution score out of many verdicts. | Claim type | Asserts | Composes as | | - | - | - | | `conformance` | The output has the required shape (a schema, a field, a format) | A scoring leaf whose own `passed` flips on failure | | `verification` | A fact in the output checks out against evidence | A scoring leaf whose own `passed` flips on failure | | `progress` | A step of the behaviour was taken | A weighted leaf | | `quality` | The delivered result is good | A weighted leaf | | `grounding` | The deliverable rests on the evidence gathered | A weighted leaf | | `safety` | Nothing harmful or forbidden happened | A cap on the score | The **grain** is the slice a claim is judged on: `unit` (one turn or step), `trajectory` (the path through the behaviour), `terminal` (the deliverable), or `session` (the whole conversation). Grains land in three weighted phases — steps, trajectory, output — and no single evaluator vetoes the composite. ### Behaviour bindings An evaluator can be bound to one of the capability's [behaviours](/core/capabilities#the-scan) with a role: an **outcome** evaluator judges the behaviour's deliverable, a **step** evaluator judges one anchor segment of its route. The Eval library groups evaluators by behaviour (the Console labels it "Task"), shows each behaviour's coverage (outcome plus every step), and lists the rest under **Unassigned**. `GET /api/behaviours/coverage/?capability={id}` returns the same rollup. Judges are authored behaviour-scoped from the behaviour's group in the Eval library or through the `upsert_evaluator` MCP tool: `POST /api/evaluators/author/` takes `behaviour`, `behaviour_role` (`outcome` or `step`), and — for a step judge — the `anchor_segment`, validated against the behaviour's contract. `GET /api/behaviours/{id}/authoring-context/` grounds the authoring with the behaviour's latest contract: anchor sequence, steps, tool set, and terminal. A judge whose `applicable_roles` include `trace_scoring` must be bound to a behaviour, and `eval_set` plus `eval_set_role` attach the new judge to an eval set in the same call. `PUT /api/evaluators/{id}/author/` re-authors an existing judge. ### Authoring Author from the Eval library by hand, or describe what you want to test and let Overmind generate the evaluator (`POST /api/evaluators/generate-prompt/`). The generated prompt template references the agent's output as `{{output.}}` and dataset fields as `{{reference.}}`, and everything stays editable before saving. `overmind sync` refreshes the Default eval set of every current capability, in tiers: Tier 0, rule-based checks compiled from the capability's card (its output contract, fields, and constraints — no model call); Tier 1, generative LLM judges grounded in the card, the prompt, and the behaviour map; and the behaviour outcome and step judges for trace scoring. Tier 1 authors once per capability; when it authors nothing for the generative role, one floor judge fills the role until a later push covers it. Authoring reads the capability's **evidence profile** — what its telemetry carries (declared behaviours, explicit units, real tool spans, provenance tags, a declared deliverable) — and refuses a spec whose evidence the traces cannot supply. Eval library grouped by behaviour Evaluators are platform-managed templates, project-wide, or scoped to a single capability. ## Eval sets An eval set is a named group of evaluators, optionally mapped to a capability. On **Evaluations → Eval sets**, choose **New eval set**, enter a name, and search the whole evaluator library by name, type, or mapped capability. Select multiple evaluators; selected rows show their name, type, and capability and can be removed before creation. Leave capability set to **None** for a reusable project set. `POST /api/eval-sets/` accepts `evaluator_ids` and an optional nullable `capability`; the set and members are created together. MCP provides `create_eval_set` and the read-only `overmind://eval-sets/{eval_set}` resource. Neither creation path starts an evaluation or activates the set. Each member receives its applicable **roles**: * `generative` members score outputs produced during eval runs and optimisation runs. * `trace_scoring` members run on every production trace, per task execution, writing verdicts and the execution score into [Observability](/core/observability#scores-on-arrival). The mechanics are on [Trace scoring](/agent-testing/trace-scoring). The same evaluator can hold both roles. One set per capability is **active** (`POST /api/eval-sets/{id}/activate/`); it is what trace scoring, optimisation runs and training runs default to. `POST /api/eval-sets/{id}/members/` adds members. A set without a capability can be selected for project evaluation and training, but cannot be activated for live trace scoring until a capability is assigned. ## Eval runs ### Context warnings Training setup shows advisory context estimates for the model being evaluated and its generative judges. These include prompt, tools and reserved output tokens. Judge preflight uses reference-answer size as a proxy for the answer that has not been generated yet. Unknown model limits are shown as unverified. These are estimates, not exact tokenizer counts; warnings never prevent launch. The **Judge model** selector below **Eval set** starts with the set's saved judge or **Mixed models** when its evaluators use different judges. A selection applies to this training run's generative judges and fallbacks, not the saved set or its Jev policy. The dropdown marks estimated context fit and shows the budget change across all judges for one dataset pass per evaluated model. Undersized options are greyed out but remain selectable. Estimated limit overflows turn the affected judge or benchmark selector orange. **Start** also turns orange with a warning icon; clicking it opens a short risk summary with **Start anyway**. Unverified, unavailable and pending checks stay neutral and do not open this warning. Changing the model is optional. Before launch, context warnings suggest alternatives with verified context and output capacity. Judge suggestions preserve configured model order, exclude the model family under test and require structured-output support. Generation suggestions require tool support when the input declares tools. These checks do not predict model quality. No suggestion changes the selection automatically. A judge change should be applied to both sides of a comparison. Evaluation cost comparisons use published token rates, all checked rows and the full reserved output for one pass per model or judge. They are not a total-run quote: retries, extra tool turns, caching, hosting and training are excluded. Missing prices remain unavailable. In training setup, training alternatives use the existing dataset ranking and show training-cost and duration estimates for their recommended configurations separately from judge evaluation costs. Benchmark deployment costs are not inferred from the base model's token prices. API clients can preview a selected version with `POST /api/eval-runs/context-check/` using `project`, `dataset`, optional `cell`, `eval_set`, `capability`, `variants` and a run-specific `judge_model`. The response includes `judge_models` with fit and budget comparisons. Training launch accepts the selected `eval_judge_model`; omitting it preserves the eval set's per-evaluator models. Each preview variant can supply `output_tokens` for a planned absolute output reservation; otherwise the registered model parameters or provider-call default apply. The request reads data without launching a run or calling a model. MCP `check_evaluation_readiness` accepts proposed `variants` and `judge_model` and returns advisory `context_checks` and `judge_models`; these do not change `ready`. MCP `start_finetune` accepts `eval_judge_model`. Standalone REST creation and MCP `run_evaluation` accept `judge_model`, matching the preview. Omit it or pass blank to preserve each evaluator's saved judge. An explicit choice applies only to that run's generative judges and fallbacks, including replay judges attached after launch; it does not edit the library, deterministic checks or Jev policy. Judge selection is frozen once created: start a new run to use another judge. Run detail shows the snapshot judges; the MCP run resource exposes `judge_model`, `run_evaluators` and the context checks. Runtime checks use the actual model messages and assembled judge prompt. The judge funnel preserves that prompt instead of clipping it to a fixed character limit. If a judge exhausts its output budget, it can retry once with a larger budget only when published context and output limits permit it. It does not switch judges or shorten the evidence. Permanent context rejection and exhausted grading attempts are technical errors, not quality zeros. Score `_judge` metadata records attempt budgets, finish reasons, usage, request IDs and costs where the provider supplies them. ### Configuration An eval run is one experiment: **one data source × N variants × M evaluators.** * **Data source** — `dataset` (the version is used and frozen, so the run stays reproducible as the dataset evolves) or `trace_filter`, drawing directly from captured traffic. `max_items` caps how many rows are drawn. * **Variants** — the columns of the experiment. A variant's `mode` is `generate` (re-run a model over each row) or `existing` (grade the recorded output as-is). Each pins a model (a catalogue model, a registered endpoint, or one of your trained models) and optionally a prompt; one variant is the baseline (`is_baseline`) every other is measured against. * **Evaluators** — attached as frozen snapshots at creation, so later library edits cannot change what a completed run measured. Creation refuses a run whose eval set has no live generative LLM judge, and a run over a closed-form labelled dataset whose eval set carries no comparator against the gold labels. In the Console, **Evaluations → New evaluation** selects an eval dataset, eval set, model to test, maximum samples and optional run-only judge. Model selectors show the provider logo and model name, matching training setup. Judge options also show estimated context fit and cost comparisons; context warnings remain advisory. Estimates cover the full dataset, while maximum samples limits the run. Eval runs can also be created through the API and the `run_evaluation` MCP tool. An eval dataset's cell also offers **Run the optimiser** and **Use in a training job**. ```bash theme={null} # Create (which also launches), follow, and compare a run curl -X POST -H "X-Api-Key: $OVERMIND_API_KEY" -H "Content-Type: application/json" \ -d '{ "project": "", "name": "Triage — prompt v3", "data_source": "dataset", "dataset": "", "variants_input": [ {"label": "production", "mode": "existing", "is_baseline": true}, {"label": "prompt v3", "mode": "generate", "model_name": ""} ], "eval_set": "" }' \ "https://api.overmindlab.ai/api/eval-runs/" curl -H "X-Api-Key: $OVERMIND_API_KEY" "https://api.overmindlab.ai/api/eval-runs/{id}/comparison/" curl -X POST -H "X-Api-Key: $OVERMIND_API_KEY" "https://api.overmindlab.ai/api/eval-runs/{id}/run/" # re-launch: deletes samples and scores first curl -X POST -H "X-Api-Key: $OVERMIND_API_KEY" "https://api.overmindlab.ai/api/eval-runs/{id}/cancel/" # revokes in-flight tasks curl -H "X-Api-Key: $OVERMIND_API_KEY" "https://api.overmindlab.ai/api/eval-runs/datasets/" # datasets that have runs ``` Execution fans out server-side: sample preparation (generation), then one scoring task per sample × evaluator, then aggregation into the run summary. A run's `status` moves `pending` → `running` → `completed`, or ends `failed` or `cancelled`. For recorded conversations, a generated variant can set `params.generation_strategy` to `per_assistant_turn`. The model produces one decision for each recorded assistant turn, using the conversation recorded before that turn as context. Tool calls are evaluated as decisions without executing their tools. Each subsequent turn uses the recorded history, keeping its context independent of the candidate's previous decisions. Such a run attaches a per-turn judge automatically, and `params.replay_dimensions` names what it grades. This also applies to conversations containing a single recorded assistant turn; a tool-only decision does not require a final answer. A sample that exceeds the generation worker's time limit records an error and is excluded from scoring. Scoring starts after sample preparation finishes, so a run can be generating while its score count is still zero. ### Reading the report Eval run report with results, distributions and operations The report shows per-variant rollups (mean score, pass rate) per evaluator, the score distribution per evaluator and variant, cost and latency per model — generation cost and eval cost separated — and a per-datapoint table. Opening a datapoint shows each evaluator's score with its reasoning; the **Compare runs** tab lays one run's results next to another's. Each dataset row has its own entry in the breakdown, with responses from different model variants grouped under that row. Search or page through the entries, then open one to inspect its input and each available model output. A model with no sample for that row does not display another model's response. The datapoint popup shows **Input** beside **Output and evaluation**. Select a model to switch both panels together. Generated output is the model's response, not a label extracted by a statistical evaluator; the **Reference answer** is shown separately as grading data. Expand **Generated messages** to inspect recorded assistant and tool messages. Generated samples capture the initial model-runner input, including the system prompt and tool definitions. This is not the provider's rendered chat template or every subsequent replay request. A sample without a captured request shows **Dataset input** from the run's pinned version. A missing pinned source is shown as unavailable, never reconstructed from a mixed input/output transcript. The MCP `overmind://eval-runs/{eval_run}` resource exposes up to five bounded sample inspections with separate input, output, reference, and evaluator reasoning. `input_source` distinguishes captured requests, pinned dataset rows, and unavailable input; `samples_truncated` indicates additional samples. Per-datapoint evaluator scores with reasoning * **Outcomes are typed.** Every score's `outcome` is `scored`, `abstained`, `not_applicable`, `skipped`, or `error` — and only `scored` results feed averages and pass rates. * **Samples can be degraded.** Each trace is normalised into a canonical transcript before scoring; if it cannot be reconstructed, the sample is marked `degraded` with a `degraded_reason` and excluded from aggregates. Failure scores also carry a `failure_role` — `root_cause` or `propagated` — so one upstream failure does not read as five independent ones. Scores accumulate into a per-capability history (`GET /api/evaluators/score-history/`) that powers the trend chart on the capability page. ## Calibrating judges Samples accept **annotations** — human labels with an optional note, written through the `annotate_evaluation_sample` MCP tool — giving you ground truth to compare judge behaviour against. When a judge and your annotations disagree consistently, tighten the rubric; the compiled checklist shows which criterion is drifting. This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.