> ## Documentation Index > Fetch the complete documentation index at: https://docs.komodor.com/llms.txt > Use this file to discover all available pages before exploring further. # Continuous Improvement > Turn accumulated judge verdicts into a concrete proposed change, then settle it as an experiment against real history. Grading agents produces scores; **Continuous Improvement** is what makes those scores change something. The Komodor Agentic Operation Platform (KAOP) reads accumulated verdicts, finds the repeated weakness behind them, proposes the specific edit that would fix it, and settles that proposal as an experiment against runs that actually happened. This page walks the loop and explains what each stage decides. Find it under **Agent improvement**, which holds two surfaces: **Opportunities** and **Experiments**. ## Why this is a separate loop A product that grades every run and leaves the score on a dashboard has built a report. The point here is the closed part: * A single low verdict is noise. A pattern across forty runs is a weakness. * The proposal is a concrete diff, not advice. * The proposal is tested before it ships — against incidents you actually had, not a synthetic benchmark that has drifted away from production. * Nothing ships itself. Every promotion is a human decision, recorded and reversible. ```mermaid theme={null} flowchart LR V["Judge verdicts
accumulate"] --> O["Opportunity
the repeated weakness"] O --> P["Proposed change
a concrete diff"] P --> E["Experiment
candidate vs production"] E --> S["Settle
quality · cost · time"] S --> D["Promote or reject
a human decides"] ``` ## Opportunities An opportunity is a named weakness with the evidence behind it. Each card carries: | What it shows | Meaning | | -------------------- | ---------------------------------------------------------------------------------------------------------------------- | | **Source** | **Fleet runs** for a quality weakness read from run evidence and verdicts, or **Agent cost** for a spend inefficiency. | | **Severity** | High, medium, or low. | | **Category** | What kind of weakness it is. | | **Impact estimate** | What fixing it is worth. | | **Evidence summary** | Why the platform believes it. | | **Affected runs** | How many runs the pattern appears in — the difference between noise and a weakness. | Filter the list by **All**, **Fleet runs**, or **Agent cost**. **View workload** opens the work the opportunity was read from. Opportunities that already have an experiment attached move into an **Already in experiment** group with a link to it, so the open list stays actionable. Read the affected-run count before the severity. An opportunity across three runs and one across forty are different claims, and the count is the one that tells you which. ## Proposed changes Every opportunity carries its proposal as a file-level diff, under **Proposed changes**. Each change names: * The **path** it touches and the **target** within it. * Whether it is an **Add**, **Modify**, or **Remove**. * A one-line summary of the edit. * **Production (current)** beside **Suggested (variant)** — the before and after, side by side. * An estimated cost impact, where the change moves spend. Typical proposals are the things that actually move an agent's quality: a prompt revision, a different tool binding, a narrower tool scope, or a smaller model where the larger one was not earning its cost. A proposal can also be handed off as a **pull request** against the agent's definition, so the change is reviewed, approved and merged wherever that definition is kept rather than applied from the console. The proposal is deliberately expressed as a reviewable diff rather than as prose, and the pull-request hand-off follows from that. An agent's prompt and tool bindings are production configuration with production blast radius, so a change to them should be read the way a code change is read — as a before and an after you can argue with, in the place your team already reviews changes. No proposal opens a pull request today. The hand-off is the direction this is going, not a path you can wire a review process to — proposals are reviewed and applied from the console, as described above. ## Experiments **Start shadow experiment** turns a proposal into a running comparison: the current agent is the baseline, the proposal is the variant, and both are exercised against real historical work so the comparison is against incidents you actually had. ### Configuring one | Setting | What it controls | | -------------------- | ------------------------------------------------------ | | **Shadow traffic %** | How much of the work the variant is exercised against. | | **Excluded traffic** | Work to keep out of the experiment entirely. | Then the **stop rules**, which decide when the experiment settles. Each is independent, and you can enable any combination: | Stop rule | Field | Effect | | --------------------- | ------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------- | | **Budget** | **Budget (USD)** | Settle when the experiment's own spend reaches the cap. | | **Time** | **End date** | Settle at a fixed date. | | **Sample count** | **Max samples** | Settle once enough comparisons have completed. | | **Score degradation** | **Degradation threshold (%)** and **Min samples** | Settle early if the variant is meaningfully worse than production — but only after enough samples that one bad run cannot trigger it. | Set a budget and a sample count on every experiment. Without them an experiment has no defined end, and an experiment with no end is a second copy of your agent running indefinitely — which shows up in [Agent spend](/manage-your-agents/optimize/agent-spend-attribution) as exactly that. ### Watching one The experiments table reports each experiment's baseline and variant, its **Tokens / cost per run**, **Total cost**, **Spent / budget**, **End date**, **Shadow %**, and status. An experiment is **running**, **completed**, **promoted**, or **discarded**, and a completed one names why it settled — its end date was reached, its budget was exhausted, its sample target was met, the variant degraded, or someone stopped it. ## Settling on three axes A completed experiment produces a verdict, and the verdict is not one number. | Axis | What is compared | | ----------- | ------------------------------------------------------------------------------- | | **Quality** | Baseline against variant score, overall and per dimension, each with its delta. | | **Cost** | Tokens and cost per run, and total cost for each side. | | **Time** | Mean run duration for each side. | Underneath, the experiment keeps the individual comparisons: for each one, the production run and the variant run that replayed the same input, both scores, the delta, and both outputs side by side. That is what makes a verdict arguable — you can read the specific case where the variant did better or worse, rather than accepting an aggregate. **Three axes, because "better" is genuinely three questions.** A variant that scores marginally higher and costs 40% more is a different decision from one that matches on quality and halves the token count. Collapsing them into a single score is how a regression gets promoted — so the verdict names a winner per axis and leaves the trade to you. ## Promotion Promotion is a human action on a settled experiment. Promoting applies the variant, records the change, and stays reversible; rejecting closes the experiment and keeps the baseline. A negative result is still a result — it prices the incumbent, and it stops the same proposal being re-litigated next quarter. Nothing auto-promotes. A settled experiment waits for a decision, however conclusive it looks. A result that shipped itself would be indistinguishable from one nobody read. ## Running the loop unattended Improvement that depends on someone opening a dashboard is improvement that does not happen. An **automation policy** authorizes KAOP to open experiments on its own within bounds you set: | Setting | What it bounds | | -------------------------- | -------------------------------------------------------------------------- | | **Agents in scope** | Which agents the policy may experiment on. Leave it empty for all of them. | | **Purposes** | What it may optimize for — cost, run duration, or accuracy. | | **Max budget** | The spend ceiling per experiment. | | **Samples per experiment** | How much evidence each experiment gathers. | | **Max runtime** | How long an experiment may run. | Experiments started this way are marked as policy-driven and name the policy that authorized them, so an automated experiment is never anonymous. The bounds are the point: automation opens the experiment, and a human still promotes it. ## Reading the loop end to end Continuous Improvement reads what [Evaluations](/manage-your-agents/optimize/evaluations) produced. An agent with no grading has nothing here — set up an eval rule first. Sort by affected runs, not severity. Read the evidence summary and confirm the pattern is real. Before and after. If the proposal is wrong, that is a signal about the opportunity, not just about the change. And a degradation rule, so a bad variant stops itself early instead of burning the budget. Quality, cost, and time. Then read two or three individual comparisons before you trust the aggregate. Either outcome closes the opportunity. A rejection is worth recording — it is what stops the same proposal returning. ## Next steps The verdicts this loop reads, and how to produce them. Comparing a candidate against production on live traffic. The cost axis an experiment settles on. Where a promoted change usually lands. ## Related topics - [Golden Scenarios](/manage-your-agents/optimize/golden-scenarios.md) - [Agent spend attribution](/manage-your-agents/optimize/agent-spend-attribution.md) - [Proactive reliability optimization](/modules-and-workflows/sre/reliability-optimization.md) - [Evaluations](/manage-your-agents/optimize/evaluations.md) - [Model Evaluation](/manage-your-agents/optimize/model-evaluation.md)