# Evals

Score a prompt version against a dataset with judges, assertions, and a verdict.

An **eval** measures how well a prompt version performs against a [dataset](/docs/datasets). It is a single YAML definition attached to one prompt **version**: it pins a dataset, describes how to call the candidate prompt, optionally adds LLM **judges**, declares **assertions** over each output, and sets a passing threshold. Running it executes the candidate over every case and reports a pass-rate and a verdict — usable as a CI gate.

There is exactly one eval per version. You edit it while the version is a **draft**; publishing the version freezes the eval along with it.

## How a run works

When you run an eval, for each case in the dataset:

1. **Resolve inputs** — each candidate `inputs` entry is a CEL expression evaluated over the `case`.
2. **Call the candidate** prompt with the resolved inputs.
3. **Run judges** that the assertions reference (lazily — a judge only runs when an assertion needs it).
4. **Evaluate every assertion** against the output, the case, and the judges.
5. **Record pass/fail** for the case.

At the end, the pass-rate (cases passed ÷ total cases) is compared to your `passingThreshold` to produce the verdict. A run is an **immutable snapshot** of the eval config — to change what runs, edit the YAML and push again.

## The eval YAML

The backend is the source of truth for the schema. Run `sufleur eval get @workspace/name@version` to print the current definition — a complete, editable skeleton when none exists yet — rather than writing one from scratch. The full shape:

```yaml
description: extraction quality
dataset:
  ref: "@acme/extraction-cases@2.0.0"   # raw version — no "v" prefix
prompt:
  inputMapping:                          # provider/model/params come from the version's model config, not the eval
    files:                               # each file declares its own input schema,
      - file: systemPrompt               # so each file carries its own inputs
        role: system
        inputs:
          taxonomy: allowed_types        # CEL over the dataset case
      - file: userPrompt
        role: user
        inputs:
          text: case.text
judges:
  - alias: quality
    prompt: "@acme/answer-grader@1.0.0"   # the judge's provider/model/params come from ITS version's model config
    inputMapping:
      files:
        - file: userPrompt
          role: user
          inputs: { answer: output.answer }
assertions:
  - kind: schema                         # output conforms to the version's output schema
  - kind: expression
    label: quality_high
    expression: "judge.quality.score > 0.7"
verdict:
  passingThreshold: 0.8                   # 0–1; omit for no gate
```

### Top-level fields

| Name | Description |
| --- | --- |
| `description` | Free-text label for the eval. |
| `dataset.ref` | The dataset version to run against, as `@workspace/name@version`. Required before a run. |
| `prompt` | The candidate prompt version being evaluated: its `inputMapping`. Provider, model, and parameters come from the version model config, not the eval. |
| `judges` | Optional LLM judges that score or label the output; referenced from assertions. |
| `assertions` | The checks applied to every output. A case passes only if all of its assertions pass. |
| `verdict.passingThreshold` | Pass-rate (0–1) the run must reach to PASS. Omit for no gate — the run still reports a score but never fails. |

> [!WARNING] Model config lives on the version, and providers must be configured
>
> Provider, model, and parameters now live on each prompt **version's** model config — set them in
> the web app or with `sufleur version set-model-config` — not in the eval YAML. Before an eval can
> run, the candidate version **and every judge version** must have a model config. An eval can only
> run against providers your workspace has configured — list them with `sufleur workspace providers
>   @workspace` (add `--models` to see each provider's models).

## Input mapping

Both the candidate prompt and each judge take an `inputMapping` with a `files` list. Each prompt file declares its **own** input schema, so each entry carries:

- `file` — which of the prompt's files to send, and the message `role` for it (`user` or `system`).
- `inputs` — a map of **that file's** template variables to CEL expressions. For the candidate they are evaluated over the `case`; for a judge they are evaluated over the candidate's `output` (plus `case` and `input`).

Because inputs are scoped per file, two files can declare the same variable name without colliding.

```yaml
inputMapping:
  files:
    - file: systemPrompt
      role: system
      inputs:
        taxonomy: allowed_types
    - file: userPrompt
      role: user
      inputs:
        text: case.text
```

## Judges

A **judge** is itself a prompt that scores or labels the candidate's output — useful when "correct" is not a simple equality check. Each judge has:

- a unique `alias` (letters, digits, and underscores; must start with a letter or underscore),
- its own `prompt` version reference (its provider, model, and parameters come from that version's model config),
- an `inputMapping` whose per-file `inputs` are CEL over the candidate's `output` (plus `case` and `input`).

A judge's result is read back in assertions as `judge.<alias>.<field>` — for example `judge.quality.score`. If the judge prompt declares an output schema with a `score` field, that is what `judge.quality.score` resolves to.

## Assertions

Each assertion is one check applied to every case. There are two kinds:

| Name | Description |
| --- | --- |
| `kind: schema` | The output must validate against the prompt version's output schema. No expression needed. |
| `kind: expression` | A CEL boolean in the `expression` field; the case passes the assertion when it evaluates to true. |

A `schema` assertion requires the version to have an [output schema](/docs/concepts#output-schemas). Every assertion also takes an optional `label` that names it in the run report.

## CEL expressions

Assertions and input mappings are written in [CEL](https://github.com/google/cel-spec) (Common Expression Language) — a small, safe expression language. The bindings available depend on where the expression appears:

| Name | Description |
| --- | --- |
| `case` | The current dataset case, typed by the dataset schema. In scope everywhere. |
| `output` | The candidate's output — the parsed object when the version has an output schema, otherwise the raw string. In judge inputs and assertions. |
| `input.<file>` | The candidate's resolved inputs for this case, namespaced per prompt file — e.g. `input.userPrompt.text`. In judge inputs and assertions. |
| `judge.<alias>` | A judge result, e.g. `judge.quality.score`. In assertions only. |

Scope by location:

- **Candidate file `inputs`** — only `case` is in scope; the expression produces a value.
- **Judge file `inputs`** — `output`, `case`, and `input` are in scope; produces a value.
- **`expression` assertions** — `output`, `case`, `input`, and `judge` are in scope; the expression must produce a **boolean**. Reference a candidate input as `input.<file>.<var>`.

### Operators and functions

- Comparison: `==`, `!=`, `<`, `<=`, `>`, `>=`
- Boolean logic: `&&`, `||`, `!`
- Membership: `in`
- String methods: `.contains(s)`, `.startsWith(s)`, `.endsWith(s)`, `.matches(regex)`
- Size of a string, list, or map: `.size()`

### Examples

| Name | Description |
| --- | --- |
| `output.answer == case.expected` | Output field equals the case's expected value. |
| `output == case.expected` | The whole raw output equals the expected string. |
| `judge.quality.score > 0.7` | A judge scored the output above 0.7. |
| `output.answer.contains("done")` | The answer contains a substring. |
| `output.summary.size() > 0` | The summary is non-empty. |
| `case.expected in output.choices` | The expected value is one of the output's choices. |
| `output.ok && judge.grader.score >= 8` | Combine an output flag with a judge score. |

> [!NOTE] CEL is sandboxed
>
> Expressions are bounded: at most 256 nodes, 32 levels of nesting, and 250 ms per evaluation.
> Keep assertions to direct checks and push anything heavier into a judge.

## Authoring an eval with the CLI

Evals live on a draft version. The local loop mirrors prompt editing — get, edit, validate, push:

```bash
sufleur eval get @acme/extraction@draft --file ./eval.yaml       # write the current YAML (or skeleton) to a file
# …edit ./eval.yaml…
sufleur eval validate @acme/extraction@draft --file ./eval.yaml  # parse + type-check, save nothing
sufleur eval push @acme/extraction@draft --file ./eval.yaml      # validate, then save
sufleur eval delete @acme/extraction@draft                       # remove the eval
```

`sufleur version dump @acme/extraction@draft --to ./working` also writes `./working/eval.yaml`, so a dumped directory is a complete working copy you can edit and push from.

**Diagnostics.** Both `validate` and `push` report three severities:

- **error** — a blocking problem (bad syntax, an unresolved ref, an invalid value). `push` refuses and changes nothing; `validate` exits non-zero.
- **note** — non-blocking (a failed type-check, an unavailable model). `push` still saves the eval, but it will not run cleanly until you resolve the note.
- **warning** — advisory only.

Run `validate` and clear any errors before `push`.

## Running and inspecting

```bash
sufleur eval run @acme/extraction@draft            # enqueue a run; prints the run id
sufleur eval run @acme/extraction@draft --watch    # …and stream progress to completion
sufleur eval runs @acme/extraction@draft           # list recent runs (newest first)
sufleur eval show <run-id>                          # one run's status, verdict, score, timing
sufleur eval watch <run-id>                         # follow an in-flight run to completion
```

A run needs (1) a pinned `dataset.ref` and (2) the candidate and judge providers configured in the workspace; `eval run` errors clearly if either is missing.

> [!TIP] Use it as a CI gate
>
> `eval run --watch` and `eval watch` exit `0` when the run passes (or has no threshold) and
> non-zero when the verdict fails, the run errors, or the watch times out. Drop either into a
> pipeline step to block a release on a failing eval. Without `--watch`, `eval run` returns as soon
> as the run is enqueued.

For the full command surface and flags, see the [CLI reference](/docs/cli#evals).