Evals

An eval measures how well a prompt version performs against a dataset. It is a single YAML definition attached to one prompt version: it pins a dataset, describes how to call the candidate prompt, optionally adds LLM judges, declares assertions over each output, and sets a passing threshold. Running it executes the candidate over every case and reports a pass-rate and a verdict — usable as a CI gate.

There is exactly one eval per version. You edit it while the version is a draft; publishing the version freezes the eval along with it.

How a run works

When you run an eval, for each case in the dataset:

1. Resolve inputs — each candidate inputs entry is a CEL expression evaluated over the case.
2. Call the candidate prompt with the resolved inputs.
3. Run judges that the assertions reference (lazily — a judge only runs when an assertion needs it).
4. Evaluate every assertion against the output, the case, and the judges.
5. Record pass/fail for the case.

At the end, the pass-rate (cases passed ÷ total cases) is compared to your passingThreshold to produce the verdict. A run is an immutable snapshot of the eval config — to change what runs, edit the YAML and push again.

The eval YAML

The backend is the source of truth for the schema. Run sufleur eval get @workspace/name@version to print the current definition — a complete, editable skeleton when none exists yet — rather than writing one from scratch. The full shape:

description: extraction quality
dataset:
  ref: "@acme/extraction-cases@2.0.0"   # raw version — no "v" prefix
prompt:
  inputMapping:                          # provider/model/params come from the version's model config, not the eval
    files:                               # each file declares its own input schema,
      - file: systemPrompt               # so each file carries its own inputs
        role: system
        inputs:
          taxonomy: allowed_types        # CEL over the dataset case
      - file: userPrompt
        role: user
        inputs:
          text: case.text
judges:
  - alias: quality
    prompt: "@acme/answer-grader@1.0.0"   # the judge's provider/model/params come from ITS version's model config
    inputMapping:
      files:
        - file: userPrompt
          role: user
          inputs: { answer: output.answer }
assertions:
  - kind: schema                         # output conforms to the version's output schema
  - kind: expression
    label: quality_high
    expression: "judge.quality.score > 0.7"
verdict:
  passingThreshold: 0.8                   # 0–1; omit for no gate

Top-level fields

Input mapping

Both the candidate prompt and each judge take an inputMapping with a files list. Each prompt file declares its own input schema, so each entry carries:

• file — which of the prompt's files to send, and the message role for it (user or system).
• inputs — a map of that file's template variables to CEL expressions. For the candidate they are evaluated over the case; for a judge they are evaluated over the candidate's output (plus case and input).

Because inputs are scoped per file, two files can declare the same variable name without colliding.

inputMapping:
  files:
    - file: systemPrompt
      role: system
      inputs:
        taxonomy: allowed_types
    - file: userPrompt
      role: user
      inputs:
        text: case.text

Judges

A judge is itself a prompt that scores or labels the candidate's output — useful when "correct" is not a simple equality check. Each judge has:

• a unique alias (letters, digits, and underscores; must start with a letter or underscore),
• its own prompt version reference (its provider, model, and parameters come from that version's model config),
• an inputMapping whose per-file inputs are CEL over the candidate's output (plus case and input).

A judge's result is read back in assertions as judge.<alias>.<field> — for example judge.quality.score. If the judge prompt declares an output schema with a score field, that is what judge.quality.score resolves to.

Assertions

Each assertion is one check applied to every case. There are two kinds:

A schema assertion requires the version to have an output schema. Every assertion also takes an optional label that names it in the run report.

CEL expressions

Assertions and input mappings are written in CEL (Common Expression Language) — a small, safe expression language. The bindings available depend on where the expression appears:

Scope by location:

• Candidate file inputs — only case is in scope; the expression produces a value.
• Judge file inputs — output, case, and input are in scope; produces a value.
• expression assertions — output, case, input, and judge are in scope; the expression must produce a boolean. Reference a candidate input as input.<file>.<var>.

Operators and functions

• Comparison: ==, !=, <, <=, >, >=
• Boolean logic: &&, ||, !
• Membership: in
• String methods: .contains(s), .startsWith(s), .endsWith(s), .matches(regex)
• Size of a string, list, or map: .size()

Examples

Authoring an eval with the CLI

Evals live on a draft version. The local loop mirrors prompt editing — get, edit, validate, push:

sufleur eval get @acme/extraction@draft --file ./eval.yaml       # write the current YAML (or skeleton) to a file
# …edit ./eval.yaml…
sufleur eval validate @acme/extraction@draft --file ./eval.yaml  # parse + type-check, save nothing
sufleur eval push @acme/extraction@draft --file ./eval.yaml      # validate, then save
sufleur eval delete @acme/extraction@draft                       # remove the eval

sufleur version dump @acme/extraction@draft --to ./working also writes ./working/eval.yaml, so a dumped directory is a complete working copy you can edit and push from.

Diagnostics. Both validate and push report three severities:

• error — a blocking problem (bad syntax, an unresolved ref, an invalid value). push refuses and changes nothing; validate exits non-zero.
• note — non-blocking (a failed type-check, an unavailable model). push still saves the eval, but it will not run cleanly until you resolve the note.
• warning — advisory only.

Run validate and clear any errors before push.

Running and inspecting

sufleur eval run @acme/extraction@draft            # enqueue a run; prints the run id
sufleur eval run @acme/extraction@draft --watch    # …and stream progress to completion
sufleur eval runs @acme/extraction@draft           # list recent runs (newest first)
sufleur eval show <run-id>                          # one run's status, verdict, score, timing
sufleur eval watch <run-id>                         # follow an in-flight run to completion

A run needs (1) a pinned dataset.ref and (2) the candidate and judge providers configured in the workspace; eval run errors clearly if either is missing.

For the full command surface and flags, see the CLI reference.