Two places the same activity shows up
Export does not replace the in-product evidence. It puts agent behavior on the same screen as the
services those agents operate on, so an agent’s tool call and the upstream latency spike it caused
sit in one trace.
Enabling export
Instrumentation ships with both the control plane and every worker built on the Python SDK, and it does nothing until you point it at a collector. There is no KAOP-specific configuration format — it reads the standard OpenTelemetry environment variables, so it works with any OTLP-compatible backend and there is nothing to unlearn.1
Point the worker at a collector
Set
OTEL_EXPORTER_OTLP_ENDPOINT on the worker’s process or container. Setting it is what turns
export on.2
Add whatever your backend needs
Most managed backends authenticate with a header.
3
Name the service, if the default is not what you want
A worker defaults its own service name to
agentops-worker-<agent>, derived from the agent’s
name. Set OTEL_SERVICE_NAME to override it.4
Restart the worker and look for the first trace
Trigger a run. The trace appears in your backend under the service name above, rooted at the
run’s own span.
Other standard
OTEL_* variables are honored as well.
Export is off until an endpoint is set, and nothing leaves your worker before then. A worker run
from a laptop with no OTel configuration pays nothing for the instrumentation being present.
What a worker emits
Spans
A run produces a span tree you can read as the agent’s decision path.
Every worker span also carries resource attributes identifying the worker and the agent it runs:
agentops.worker.id, agentops.worker.name, and agentops.agent.id.
Metrics
Every metric a worker emits lives under theagentops. namespace, so one facet in your backend
isolates agent activity from everything else you collect.
The two durations are histograms in seconds (
unit=s). Everything else is a counter and carries
the dimensionless unit=1, so your backend will not label it for you — the count is of whatever the
row above says.
Logs
A worker’s log records are exported alongside its spans, and each record carries the trace and span it belongs to — so you can pivot from a slow span to the lines the agent wrote while it was executing.Choosing what a worker forwards to KAOP
Two variables control the KAOP-side log forwarding, independent of OTel export:
Raise
AGENTOPS_CP_LOG_LEVEL on a chatty worker if run evidence is noisier than it is useful; set
AGENTOPS_CP_LOG_CAPTURE_ALL while debugging a worker whose own libraries hold the answer.
What the control plane emits
The control plane is instrumented the same way and turns on with the same variables. Its spans carry the resolved caller identity, so traces are sliceable per account and per principal, and authorization decisions appear as their own spans with the action, resource, and decision attached. Two attribute namespaces are used and only two:usr.*for human principals — the user id, email, and name.agentops.*for everything else — the account, the principal, the authentication path, and authorization decisions.
Some identity reaches every span by default. The root span carries the full set. Two settings,
both on unless you turn them off, widen part of it to the rest of the trace:
AGENTOPS_OTEL_SPAN_BAGGAGE_ENABLED propagates the non-identifying ids (account and principal), and
AGENTOPS_OTEL_ALL_SPANS_IDENTITY_ENABLED adds the principal kind and display name. Set either to
false to keep it off the child spans.Email and name never leave the root span whatever those two are set to, which bounds how much
personal data reaches your telemetry store.What to build with it
- Alert on agent failure rate the way you alert on service error rate.
agentops.run.failedagainstagentops.run.completed, split by agent. - Watch model latency separately from agent latency. A run that got slower because one provider got slower is a different problem from a run that took more turns.
- Correlate a tool call with the system it touched. The
agentops.tool.callspan sits in the same trace as your own service’s spans when your instrumentation propagates context. - Track token volume as an operational signal. Token counters move before a spend chart does.
Next steps
Runs & evidence
The in-product view of the same activity, and what a span tree contains.
Fleet health
The rollup KAOP builds from run activity.
Agent spend attribution
Where cost comes from and how it is attributed.