> ## Documentation Index > Fetch the complete documentation index at: https://docs.komodor.com/llms.txt > Use this file to discover all available pages before exploring further. # Model Evaluation > Run a candidate agent or model beside production on real traffic and let a blinded judge say which one is better. Changing an agent's model or prompt can regress quality without moving a single operational metric — the run still succeeds, latency holds, and only the answers get worse. **Model Evaluation** in the Komodor Agentic Operation Platform (KAOP) answers the question directly: run the candidate beside production on the same real inputs, and have a blinded judge say which output was better. This page explains how the comparison is set up, how it stays safe, and how to read the result. ## The shape of it ```mermaid theme={null} flowchart LR Req["A real request"] --> CP["Control plane"] CP -->|production run| P["Production arm"] CP -->|shadow run| S["Candidate arm"] P -->|answer| Caller["The caller"] S -.->|answer discarded| X["Nothing"] P --> Pair["Comparison pair"] S --> Pair Pair --> Judge["Blinded judge"] Judge --> Verdict["Winner per rubric"] ``` Every queued request runs on **both** arms. Callers only ever receive the production answer. The two executions are first-class runs, linked as a **comparison pair**, and that pair is what a judge grades. ## Deploying a candidate A candidate is a second deployment of the **same agent**, declaring itself as the shadow arm with one environment variable: ```bash theme={null} AGENTOPS_VARIANT_ROLE=secondary ``` Absence of the declaration means production, so nothing changes on the serving side. The candidate is minted its own worker token, so bringing it up — or taking it down — never rotates the credential your production deployment is using. An unrecognized value for `AGENTOPS_VARIANT_ROLE` fails the worker's boot, and a replica presenting the candidate's token while declaring no arm is refused outright. Both produce a loud, harmless candidate crash loop rather than a silent promotion into the serving fleet — which is the failure this guard exists to prevent. Because both arms share one agent id, [Fleet health](/manage-your-agents/optimize/fleet-health) shows one row per agent. A live candidate makes itself visible there: * The **Instances** cell splits into `N production · M shadow`. * A **Shadow active** facet filters the whole fleet to agents currently running one. * The agent's **Remove shadow** action ends the experiment. A candidate must run the Python SDK — a Go worker always registers as production, so it cannot be the candidate arm. And give the candidate its **own** token: a copy of production's token registers it as an ordinary production replica that answers real callers, and no guard catches that. ## What the candidate cannot do The whole design rests on the candidate being unable to affect anything. Specifically: | Guarantee | How it holds | | ----------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | **Callers never hear it** | Every synchronous caller awaits the production run only. | | **It never serves real traffic** | A candidate replica is handed shadow runs only; a production replica is handed ordinary and production runs only. The backlog that existed before the candidate is drained by production, as before. | | **It causes no side effects** | A shadow run publishes no completion event and no channel reply — so no incident is touched, no review recomputed, no second Slack message posted, no onboarding step ticked. | | **It cannot corrupt conversation state** | It reads the production conversation's snapshot at claim time, so the comparison is on identical history, then writes to its own namespaced session. | | **It cannot flap the agent's definition** | Its presence proves it is alive but never steers the shared agent — card, inputs, labels, skills, and schedules stay production-authored. | | **It cannot slow production down** | The candidate is best-effort and fully isolated; it cannot fail, delay, or overwrite anything the production path causes. | The single deliberate exception is grading. Grading a candidate is the entire point, so a shadow run does publish for grading — with the carve-out that a shadow *judge* run never publishes, so a candidate judge cannot overwrite the production judge's verdict. ## Attaching a judge Fan-out gives you pairs; grading them is an eval rule away. Create a rule in [Evaluations](/manage-your-agents/optimize/evaluations) and set **What to grade** to **Run pair (shadow vs. production)**. Rules configured that way carry a **Run pair** badge in the rules list. A pair rule fires on a *completed pair* rather than on a single terminal run, and both arms are selected together or not at all — sampling them independently would produce half-pairs, which cannot answer a comparison question. ### Why the judge is blinded The judge is handed the two outputs as **Candidate 1** and **Candidate 2**, in randomized order, with nothing in its context naming which arm is which. It sees each side's input, output, status, error, and tool trace, and nothing that hints at role. **Unblinded pairwise comparison is not offered, on purpose.** A judge told which output came from production will favour it — and the resulting number looks exactly like a trustworthy one. Blinding is the only thing that makes a small preference meaningful, so it is not a setting you can turn off. The platform stores the blinding assignment with the grade, and resolves the winner back to production or candidate only after the verdict lands. ### Why a pairwise verdict beats two separate scores Both instruments exist and neither replaces the other. | | Grading each arm separately | Grading the pair together | | ----------------- | ----------------------------------------------- | ----------------------------------- | | Judgments | Two, independent | One | | The judge sees | One output | Both outputs | | Produces | Absolute scores per rubric | A winner per rubric, with reasoning | | Small differences | Lost in judge variance | Detectable | | Bias risk | The judge's internal scale drifts between calls | Controlled by blinding | | Cost | Two grading runs | One grading run | An absolute score is only as stable as the judge's internal scale, and that scale wobbles between independent calls: a `0.72` beside a `0.68` cannot tell you whether the candidate is worse or the judge was stingier the second time. A pairwise verdict deletes the scale — the judge only says which of two things in front of it is better, which is an easier judgment and far more sensitive to a small real difference. The converse matters just as much: **a pairwise winner from two bad answers is still a bad answer.** Keep your per-arm rubric rules running alongside the pair rule. They tell you the absolute bar is met; the pair tells you which version to ship. ## Reading the result Open **Agent Score Results** for the list of agents that have been compared, then an agent for its shadow-comparison history. ### A/B Test Results Each row is one judge's verdicts under one eval rule against one agent's candidate: | What the row carries | Meaning | | ---------------------- | -------------------------------------------------------------------------------------- | | **Result** | **Production wins**, **Shadow wins**, or **Tie**, folded from the individual verdicts. | | **Comparisons** | How many distinct pairs this judge graded. | | **Wins and ties** | Production wins, shadow wins, and ties, counted. | | **Per-rubric tallies** | The same counts held separately per rubric, never blended across rubrics. | Expanding a row lists every graded pair with, for each rubric, the result, both arms' scores, and the tie band that produced the result. A **tie** therefore reads as "within the band" rather than as a coin flip — the two scores are right there. The row-level result is folded by **counting labels**, never by averaging scores, and there is deliberately no percentage delta on it. "The candidate won 61% of 240 pairs" is a countable claim. "The candidate is 6% better" is not, at any sample size the platform can vouch for. A low total count is your own signal to keep collecting. Two charts summarize the same data: **Win distribution** (how often each arm scored higher) and **Score distribution** (both arms' scores over time). ### Cost, tokens, and latency per arm Beside the verdicts sits each arm's rolled-up usage — cost, token breakdown, and mean latency — with the delta expressed as production minus shadow, so a green figure means the candidate used less. Every figure carries its own **sample size**: how many of that arm's runs actually contributed. This matters more than it sounds. A run can carry no usage record at all, and pricing is not resolvable for every model, so a missing cost is reported as unknown rather than as zero — "cost from 3 of 17 runs" instead of a blended figure that reads as "this arm is free". Token counts are kept disjoint for the same reason: fresh input tokens sit beside cached and cache-write tokens rather than being summed into them, because blending fresh with cached input misprices a cache-heavy agent by roughly an order of magnitude. ## Deciding The comparison gives you three independent readings — preference, absolute quality, and marginal cost — and the decision is yours to make and execute. A handful of pairs is an anecdote. Watch the comparison count, not the win rate, until the counts are worth reading. A candidate that wins on one rubric and loses on another has told you something specific. Do not average it away. Confirm the candidate's own per-arm rubric scores clear the level you require. A preference win between two weak answers is not a release. Read the per-arm cost and latency totals, with their sample sizes. A candidate that wins narrowly and costs twice as much is a different decision from one that wins narrowly and costs a tenth. Apply the winning configuration to the production deployment yourself, then **Remove shadow** to end the experiment. Removing the shadow is also what lets an ordinary undeclared replica of that agent register normally again. No result ships itself. Nothing here auto-promotes, auto-blocks a release, or writes into your serving path — the platform produces the evidence and a human makes the change. Every promotion is therefore recorded as an ordinary agent change and shows up in the **Changes** figure on [Fleet health](/manage-your-agents/optimize/fleet-health). ## Comparing without a live candidate Both arms are fanned out from real traffic, so a comparison needs traffic. For a candidate you want to exercise deliberately rather than wait for, drive it locally: the CLI starts a production and a candidate worker side by side, fans a single message out to both, and asserts that each arm ran it on its own worker. ```bash theme={null} aops shadow start --agent my-agent aops shadow trigger --agent my-agent --message "…" aops shadow validate --agent my-agent aops shadow status --agent my-agent aops shadow stop --agent my-agent ``` ## Next steps Eval rules, judges, rubrics, and how a verdict is read. Where a proposed change comes from, and how it gets settled. Spotting a live candidate, and removing it. Both arms of a pair, side by side, with the pair's own id. ## Related topics - [Evaluations](/manage-your-agents/optimize/evaluations.md) - [Golden Scenarios](/manage-your-agents/optimize/golden-scenarios.md) - [Continuous Improvement](/manage-your-agents/optimize/continuous-improvement.md) - [Providers](/manage-your-agents/build/providers.md) - [Optimize — how it works](/manage-your-agents/optimize/how-it-works.md)