Four lenses, four questions
An agent can be healthy and wrong, correct and ruinously expensive, or cheap and invisible outside KAOP. Those are four separate failure modes, so they get four separate readings.Reliability and quality are the pair worth internalizing. A failing run announces itself — it has
a status, a category, and a place to look. A wrong run does not: it succeeds, costs what it always
costs, and leaves no operational trace at all. Reliability tooling will never find it. That gap is
why quality gets its own instruments rather than being folded into fleet monitoring.
Reliability
Reliability starts from presence and run outcomes, and both are computed rather than declared. Fleet health gives you the fleet’s live state — who is online, who has capacity, what is queued — alongside period figures for volume, cost, latency, error rate, and how many agent versions were deployed. Insights takes the failures and does the grouping for you. Failing runs are categorized from structural proof — the run’s status, the shape of its span tree, the agent’s presence — never by guessing at error text, and identical failures collapse into one ranked issue with its verbatim reason and an agent-produced recommendation.Quality
Quality is graded, not observed. Evaluations puts LLM judges on a sampled share of runs and records a score per rubric with the reasoning that produced it — so a low number comes with the criteria that were checked and what moved it off full marks. Model Evaluation answers the comparative question. Deploy a candidate beside production, let both run every real request, and have a blinded judge say which output was better — which is far more sensitive to a small real difference than diffing two absolute scores. Continuous Improvement closes the loop. Accumulated verdicts become a named weakness, the weakness becomes a concrete proposed diff, and the diff is settled as an experiment against runs that actually happened, on quality, cost, and time together.Cost
Agent spend attribution builds every figure from what each run reported, and keeps the derivation visible: cost is reported by the worker, estimated from model rates and token counts, or unattributed. Spend pivots by label, agent, model, provider, MCP server, or tool, and budgets cap what an agent may spend per window — with the option to stop it when the cap is reached.Portability
Everything above lives in KAOP. Two surfaces make it useful outside it. Observability & OTel export emits agent activity as standard OpenTelemetry traces, logs, and metrics to your own backend, so an agent’s tool call sits in the same trace as the service it touched. Knowledge graph is the estate KAOP has discovered by operating it — the resources, and how incidents, cost findings, and agents attach to them.The order to read them in
1
Is the fleet up?
Fleet health’s live cards. A capacity or presence problem makes every other reading unreliable.
2
What is failing, and is it new?
Insights. Grouped, ranked, and already categorized — with the run evidence one click away.
3
Of what succeeded, what was any good?
Evaluations. This is the reading nothing else substitutes for.
4
What did all of it cost?
Agent spend. Read cost per successful run before total spend — volume flatters a total.
5
Then change something, and prove it
Continuous Improvement for a proposal against history, Model Evaluation for a candidate against
live traffic. Both settle before anything ships.
The pages
Fleet health
Presence, capacity, backlog, reliability, and version churn across the whole fleet.
Evaluations
LLM judges grading runs against explicit rubrics, with the reasoning behind every score.
Model Evaluation
A candidate beside production on real traffic, settled by a blinded judge.
Continuous Improvement
Verdicts to a proposed change to a settled experiment to a human decision.
Agent spend attribution
What the fleet costs per run, agent, and model — and budgets that cap it.
Knowledge graph
The estate your agents have discovered, and how incidents and findings attach to it.
Observability & OTel export
Agent traces, logs, and metrics in your own OpenTelemetry backend.
Next steps
Runs & evidence
The record every reading on these pages is computed from.
Build — how it works
Where a fix lands once you know what to change.
Troubleshooting
The fixes for the failure categories Insights reports.
Limits & quotas
The caps that shape what a fleet can do.