> ## Documentation Index
> Fetch the complete documentation index at: https://docs.komodor.com/llms.txt
> Use this file to discover all available pages before exploring further.
# Evaluations
> Grade agent runs with LLM judges against explicit rubrics, and read exactly why a score is what it is.
A run that succeeded can still be wrong. **Evaluations** is how the Komodor Agentic Operation Platform (KAOP)
grades what agents actually produced — an LLM judge reads a run and returns a score per rubric with
its reasoning attached. This page explains how grading is set up, what a verdict contains, and how to
read quality as a trend rather than a collection of opinions.
## The objects, once
| Term | What it is |
| ------------- | --------------------------------------------------------------------------------------------------------------------------------------- |
| **eval rule** | The configuration that decides which agents' runs get graded, how often, and by whom. |
| **judge** | The agent that does the grading. A judge is an ordinary agent that advertises the grading capability, listed on the **Evaluators** tab. |
| **rubric** | The named dimension a score is reported against — `quality`, `precision`, `recall`, and whatever else a judge measures. |
| **verdict** | One judge's score on one rubric for one run, with its reasoning. |
| **eval** | Everything recorded about one graded run, across every judge that graded it. |
**Grading is itself agent work.** A judge is an agent, and its grading pass is a real run against
your control plane — with an input, tool calls, and an output. That is why a verdict can carry
reasoning rather than a bare number: the reasoning is something an agent produced and the platform
stored, not a label the platform inferred.
## Setting up grading
Grading starts when you create an eval rule. Rules are managed on the **Evaluators** tab, alongside
the judges themselves.
**Single run** grades each matching run on its own — the default. **Run pair (shadow vs.
production)** grades a completed comparison pair together and belongs to
[Model Evaluation](/manage-your-agents/optimize/model-evaluation).
A rule names the agents whose runs it grades. Nothing outside that list is touched.
**10%**, **25%**, **50%**, or **100%**, defaulting to 10%. The rate is per rule: the whole panel
of judges evaluates the same fraction of matching runs, so their verdicts are comparable.
One or several. Each selected judge produces its own independent verdict.
Free text telling the judges what "good" means for these agents specifically — for example,
that a weekly report must lead with the total, or that a table missing a share-of-total column
is a defect. Optional, and the most under-used field on the form.
The rule starts matching new runs immediately, samples them at the rule's rate, and dispatches
one grading task per judge. Edits apply to new matching runs from then on.
A rule can be **Active** or **Paused**, and pausing it stops grading without losing anything already
recorded.
**What each judge measures lives in the judge, not in the rule.** A rule says whose runs to grade
and how often; the judge decides which rubrics it reports and how it computes them, and the rule's
instructions give it your nuance. That is why two judges on the same rule can return different
rubrics — and why adding a judge that measures something new needs no change to the rule.
By default a rule grades runs that completed. Grading is also **arm-aware**: when an agent is
running a candidate arm, the candidate's runs advance their own sampling counter and their verdicts
are excluded from the production agent's average — so a live experiment cannot quietly move the
numbers it is being compared against.
## Reading a verdict
An LLM verdict is a score from **0 to 1**, shown to two decimals with its rubric name attached —
`safety 0.92`. Colour carries the at-a-glance signal (strong, middling, weak) and no word is put on
it, because a three-word band is a claim the score does not support on its own.
Open an eval and each judge's review expands into three parts:
| Section | What it holds |
| ------------------------ | ----------------------------------------------------------------------------------------------------------------------------------------- |
| **How it was scored** | The judge's own method, in one line — for example, precision and recall at 50 points each, equal weight. |
| **What was checked** | The individual criteria, each with an outcome of **pass**, **partial**, **fail**, or **info**, its reasoning, and the evidence behind it. |
| **What moved the score** | The adjustments that took the number off full marks — each with its signed points, why it applied, and where. |
Below those sits the judge's analysis prose, and the eval links straight to the graded run so you
can read what the agent actually did.
Adjustment points are in **the judge's own scale**, not normalized to 0–1. A judge working out of 50
reports points out of 50, labelled with the method that produced them. KAOP never re-scales or sums
them across judges, because that would present a number nobody computed.
### Two rubrics stay two numbers
If a judge measures `precision` and `recall`, you see two scores — never their mean. A blend cannot
distinguish a reviewer that invents findings from one that misses them, which is precisely the
distinction two rubrics exist to preserve. Where a single headline is genuinely needed, KAOP shows
the **lowest** rubric with its name attached, which is the honest summary of "is this run's
evaluation good".
### Human grades are kept separate
A person can grade a run directly, on a 1-to-5 star scale, and those grades are shown in their own
amber styling — never band-coloured, never blended into a judge's average, and never moved onto an
agent's scorecard. A human grade is one person's spot check; an LLM verdict at a known sample rate
is a measurement. Reporting them together would let machine grading quietly inflate or mask what
your team actually thinks.
## The four views
| Tab | What it answers |
| ----------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------- |
| **Overview** | How each rubric is trending, which agents moved, and who did the evaluating. |
| **Runs** | Every evaluated run, one record each, with every judge's verdict — the surface you work in day to day. |
| **Evaluators** | The eval rules, and the judges doing the grading. |
| **Agent Score Results** | Every agent running a candidate beside production, and how the two compare. See [Model Evaluation](/manage-your-agents/optimize/model-evaluation). |
### Overview, in detail
Pick a window of **7d**, **30d**, or **90d** — 30 by default — and optionally narrow to one rubric,
one agent, or one judge. Every filter narrows the whole page together, so the tiles always describe
the same slice the chart is drawing.
* **Tiles** — evaluations in the window, agents evaluated, active judges, and the **weakest rubric**
named alongside its average. Never a cross-rubric mean.
* **Score over time** — a daily series per rubric, with volume. A day with no verdict is omitted
rather than plotted as zero: a zero would draw a cliff on every quiet weekend that looks
identical to a quality collapse.
* **Movers** — the agents whose rubric average moved most against the preceding window, split into
**Dropped** and **Improved**. A mover needs a baseline on both sides, so an agent first evaluated
this window is not reported as the period's biggest improvement.
* **Evaluators** — each judge with its volume, its average, and when it first evaluated anything, so
a newly added judge is visible as new and its early verdicts can be sanity-checked against the
incumbent's.
Counts on Overview are of LLM verdicts only. That is deliberate — the tab exists to trend a
measurement, and a human spot check is not one.
### Per-agent quality
An agent that has been evaluated carries its own score-over-time series per rubric, plus its
**best** and **worst** runs per rubric, each linking to the eval. Those two lists are what turn "3%
of this agent's output is weak" into "read this specific review". They are computed over the agent's
whole history rather than folded from whatever page of evals happens to be loaded, so the worst run
is genuinely the worst run.
The same view appears as an **Evals** tab on the agent's own page in
[Fleet health](/manage-your-agents/optimize/fleet-health), shown once the agent has LLM verdicts to
plot.
## Choosing a sample rate
The rate is the one knob that trades cost for confidence, and both directions have a real failure
mode.
| Rate | When it fits |
| ---------- | ------------------------------------------------------------------------------------------------------------------------------------------------ |
| **10%** | Steady-state monitoring on a high-volume agent. Enough to move a trend, cheap enough to leave on. |
| **25–50%** | An agent you have recently changed, or one whose quality you do not yet trust. |
| **100%** | Low-volume, high-stakes work where every output matters — and while you are calibrating a new judge against runs you have already read yourself. |
Grading costs model calls. A panel of three judges at 100% on a busy agent is three extra runs for
every production run, and it shows up in
[Agent spend](/manage-your-agents/optimize/agent-spend-attribution) like any other agent work.
Sample the everyday rules and pay for depth only where a decision hangs on it.
## Next steps
Grade a candidate against production on the same input, with a blinded judge.
Turn accumulated verdicts into a proposed change, settled as an experiment.
The run a verdict grades, and the trail behind it.
Where a weak rubric usually gets fixed.
## Related topics
- [Model Evaluation](/manage-your-agents/optimize/model-evaluation.md)
- [Install](/get-started/on-prem/install.md)
- [Golden Scenarios](/manage-your-agents/optimize/golden-scenarios.md)
- [Continuous Improvement](/manage-your-agents/optimize/continuous-improvement.md)
- [Change Intelligence & Risk Control](/modules-and-workflows/ops-automation/change-intelligence.md)