Skip to main content
Grading agents produces scores; Continuous Improvement is what makes those scores change something. The Komodor Agentic Operation Platform (KAOP) reads accumulated verdicts, finds the repeated weakness behind them, proposes the specific edit that would fix it, and settles that proposal as an experiment against runs that actually happened. This page walks the loop and explains what each stage decides. Find it under Agent improvement, which holds two surfaces: Opportunities and Experiments.

​
Why this is a separate loop

A product that grades every run and leaves the score on a dashboard has built a report. The point here is the closed part:
  • A single low verdict is noise. A pattern across forty runs is a weakness.
  • The proposal is a concrete diff, not advice.
  • The proposal is tested before it ships — against incidents you actually had, not a synthetic benchmark that has drifted away from production.
  • Nothing ships itself. Every promotion is a human decision, recorded and reversible.

​
Opportunities

An opportunity is a named weakness with the evidence behind it. Each card carries: Filter the list by All, Fleet runs, or Agent cost. View workload opens the work the opportunity was read from. Opportunities that already have an experiment attached move into an Already in experiment group with a link to it, so the open list stays actionable.
Read the affected-run count before the severity. An opportunity across three runs and one across forty are different claims, and the count is the one that tells you which.

​
Proposed changes

Every opportunity carries its proposal as a file-level diff, under Proposed changes. Each change names:
  • The path it touches and the target within it.
  • Whether it is an Add, Modify, or Remove.
  • A one-line summary of the edit.
  • Production (current) beside Suggested (variant) — the before and after, side by side.
  • An estimated cost impact, where the change moves spend.
Typical proposals are the things that actually move an agent’s quality: a prompt revision, a different tool binding, a narrower tool scope, or a smaller model where the larger one was not earning its cost. A proposal can also be handed off as a pull request against the agent’s definition, so the change is reviewed, approved and merged wherever that definition is kept rather than applied from the console.
The proposal is deliberately expressed as a reviewable diff rather than as prose, and the pull-request hand-off follows from that. An agent’s prompt and tool bindings are production configuration with production blast radius, so a change to them should be read the way a code change is read — as a before and an after you can argue with, in the place your team already reviews changes.
No proposal opens a pull request today. The hand-off is the direction this is going, not a path you can wire a review process to — proposals are reviewed and applied from the console, as described above.

​
Experiments

Start shadow experiment turns a proposal into a running comparison: the current agent is the baseline, the proposal is the variant, and both are exercised against real historical work so the comparison is against incidents you actually had.

​
Configuring one

Then the stop rules, which decide when the experiment settles. Each is independent, and you can enable any combination:
Set a budget and a sample count on every experiment. Without them an experiment has no defined end, and an experiment with no end is a second copy of your agent running indefinitely — which shows up in Agent spend as exactly that.

​
Watching one

The experiments table reports each experiment’s baseline and variant, its Tokens / cost per run, Total cost, Spent / budget, End date, Shadow %, and status. An experiment is running, completed, promoted, or discarded, and a completed one names why it settled — its end date was reached, its budget was exhausted, its sample target was met, the variant degraded, or someone stopped it.

​
Settling on three axes

A completed experiment produces a verdict, and the verdict is not one number. Underneath, the experiment keeps the individual comparisons: for each one, the production run and the variant run that replayed the same input, both scores, the delta, and both outputs side by side. That is what makes a verdict arguable — you can read the specific case where the variant did better or worse, rather than accepting an aggregate.
Three axes, because “better” is genuinely three questions. A variant that scores marginally higher and costs 40% more is a different decision from one that matches on quality and halves the token count. Collapsing them into a single score is how a regression gets promoted — so the verdict names a winner per axis and leaves the trade to you.

​
Promotion

Promotion is a human action on a settled experiment. Promoting applies the variant, records the change, and stays reversible; rejecting closes the experiment and keeps the baseline. A negative result is still a result — it prices the incumbent, and it stops the same proposal being re-litigated next quarter.
Nothing auto-promotes. A settled experiment waits for a decision, however conclusive it looks. A result that shipped itself would be indistinguishable from one nobody read.

​
Running the loop unattended

Improvement that depends on someone opening a dashboard is improvement that does not happen. An automation policy authorizes KAOP to open experiments on its own within bounds you set: Experiments started this way are marked as policy-driven and name the policy that authorized them, so an automated experiment is never anonymous. The bounds are the point: automation opens the experiment, and a human still promotes it.

​
Reading the loop end to end

1

Let verdicts accumulate

Continuous Improvement reads what Evaluations produced. An agent with no grading has nothing here — set up an eval rule first.
2

Open the highest-evidence opportunity

Sort by affected runs, not severity. Read the evidence summary and confirm the pattern is real.
3

Read the proposed diff

Before and after. If the proposal is wrong, that is a signal about the opportunity, not just about the change.
4

Start the experiment with a budget and a sample target

And a degradation rule, so a bad variant stops itself early instead of burning the budget.
5

Settle on all three axes

Quality, cost, and time. Then read two or three individual comparisons before you trust the aggregate.
6

Promote or reject, and say why

Either outcome closes the opportunity. A rejection is worth recording — it is what stops the same proposal returning.

​
Next steps

Evaluations

The verdicts this loop reads, and how to produce them.

Model Evaluation

Comparing a candidate against production on live traffic.

Agent spend attribution

The cost axis an experiment settles on.

Skills

Where a promoted change usually lands.