What a scenario holds
A scenario has an identity — name, domain, description, tags — and its content lives in versions, so you can change a case without losing what it used to say. Each version carries:Versions move through review
draft → validating → awaiting review → published → archived
The first three are reversible — a version can go back for more work. Published and archived are
one-way, so a case other people’s results depend on cannot be quietly rewritten underneath them.
Suites group scenarios against an agent
A suite is a named group scoped to the agent it tests. Membership carries two things beyond the scenario itself: whether it is enabled, and its weight — so a suite can hold a case temporarily switched off, and can count the cases that matter more for more.Reading a result
Each run records the scenario, the version, per-phase results, a score, and a failure kind. Two outcomes look similar in a list and mean different things:Ungraded — the case ran to completion but was never judged. The work happened; the verdict did
not. Most often the judge could not be dispatched.Failed — the case never completed at all: it failed, was blocked, or was cancelled. It was never
gradeable, so there is no score to be missing.
Next steps
Evaluations
Grading the traffic that actually arrived.
Model Evaluation
Comparing a candidate against production on the same input.
Continuous improvement
Turning verdicts into a proposed change.
Runs & evidence
A scenario run is a run — open it and read it.