> ## Documentation Index > Fetch the complete documentation index at: https://docs.komodor.com/llms.txt > Use this file to discover all available pages before exploring further. # Observability & OTel export > Export agent traces, logs, and metrics over OpenTelemetry so agent activity sits beside the rest of your systems. Every run in the Komodor Agentic Operation Platform (KAOP) already keeps its own durable evidence, but that evidence lives in KAOP. This page shows how to also emit the same activity as standard OpenTelemetry (OTel) traces, logs, and metrics to your own backend, what the spans and metrics contain, and how the two views complement each other. ## Two places the same activity shows up | Where | What you get | Who reads it | | --------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------- | | **In KAOP** | The run's full evidence trail — spans, logs, progress messages, output, artifacts, token usage — plus the fleet and spend rollups built on it. | Anyone operating agents. Nothing to configure. | | **Your OTel backend** | The same agent activity as OTLP traces, logs, and metrics, correlated with the rest of your infrastructure. | Your existing observability practice — the dashboards, alerts, and on-call rotations you already run. | Export does not replace the in-product evidence. It puts agent behavior on the same screen as the services those agents operate on, so an agent's tool call and the upstream latency spike it caused sit in one trace. ## Enabling export Instrumentation ships with both the control plane and every worker built on the Python SDK, and it does nothing until you point it at a collector. There is no KAOP-specific configuration format — it reads the standard OpenTelemetry environment variables, so it works with any OTLP-compatible backend and there is nothing to unlearn. Set `OTEL_EXPORTER_OTLP_ENDPOINT` on the worker's process or container. Setting it is what turns export on. ```bash theme={null} OTEL_EXPORTER_OTLP_ENDPOINT=https://otlp.example.internal:4318 OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf ``` Most managed backends authenticate with a header. ```bash theme={null} OTEL_EXPORTER_OTLP_HEADERS=api-key= ``` A worker defaults its own service name to `agentops-worker-`, derived from the agent's name. Set `OTEL_SERVICE_NAME` to override it. Trigger a run. The trace appears in your backend under the service name above, rooted at the run's own span. | Variable | What it does | | ----------------------------- | --------------------------------------------------------------- | | `OTEL_EXPORTER_OTLP_ENDPOINT` | The collector or backend to send to. Setting it enables export. | | `AGENTOPS_OTEL_ENABLED` | Enables export when you configure the endpoint some other way. | | `OTEL_EXPORTER_OTLP_PROTOCOL` | The transport — `grpc` or `http/protobuf`. | | `OTEL_SERVICE_NAME` | How the process appears in your traces. | | `OTEL_EXPORTER_OTLP_HEADERS` | Headers your backend requires, such as an API key. | Other standard `OTEL_*` variables are honored as well. Export is off until an endpoint is set, and nothing leaves your worker before then. A worker run from a laptop with no OTel configuration pays nothing for the instrumentation being present. ## What a worker emits ### Spans A run produces a span tree you can read as the agent's decision path. | Span | What it covers | | ------------------------ | ------------------------------------------------------------------------------------------------------------- | | `agentops.run` | The whole run. Carries `agentops.agent.id`, `agentops.agent.name`, and `agentops.run.id`. | | `agentops.llm.iteration` | One model turn. Carries the provider, the model, the iteration number, and the input and output token counts. | | `agentops.tool.call` | One tool invocation, by name, with its outcome. A failed call ends with an error status. | | `agentops.skill.invoke` | One skill invocation, by id and name. | Every worker span also carries resource attributes identifying the worker and the agent it runs: `agentops.worker.id`, `agentops.worker.name`, and `agentops.agent.id`. ### Metrics All KAOP metrics live under a single `agentops.` namespace, so one facet in your backend isolates the whole product. | Metric | Unit | What it counts | | ------------------------------- | ----------- | ------------------------------------------------------ | | `agentops.run.completed` | runs | Runs that finished successfully. | | `agentops.run.failed` | runs | Runs that failed, tagged with a failure reason. | | `agentops.run.duration` | seconds | Run duration. | | `agentops.llm.requests` | requests | Model requests, tagged by provider, model, and status. | | `agentops.llm.request.duration` | seconds | Model request latency. | | `agentops.llm.input_tokens` | tokens | Input tokens consumed. | | `agentops.llm.output_tokens` | tokens | Output tokens emitted. | | `agentops.tool.calls` | calls | Tool calls, tagged by tool name and status. | | `agentops.skill.invocations` | invocations | Skill invocations, tagged by skill and status. | ### Logs A worker's log records are exported alongside its spans, and each record carries the trace and span it belongs to — so you can pivot from a slow span to the lines the agent wrote while it was executing. Worker log lines also reach KAOP independently of OTel, which is why a run's evidence shows its logs whether or not you export anything. Komodor has no access to a worker you host, so the worker ships its own lines to the control plane as part of the run. ## Choosing what a worker forwards to KAOP Two variables control the KAOP-side log forwarding, independent of OTel export: | Variable | Default | Effect | | ----------------------------- | ------- | --------------------------------------------------------------------------------------------------------------------- | | `AGENTOPS_CP_LOG_LEVEL` | `INFO` | The minimum level forwarded to the control plane. | | `AGENTOPS_CP_LOG_CAPTURE_ALL` | off | Off forwards only the SDK's own loggers. Set it to attach to the root logger and capture everything the process logs. | Raise `AGENTOPS_CP_LOG_LEVEL` on a chatty worker if run evidence is noisier than it is useful; set `AGENTOPS_CP_LOG_CAPTURE_ALL` while debugging a worker whose own libraries hold the answer. ## What the control plane emits The control plane is instrumented the same way and turns on with the same variables. Its spans carry the resolved caller identity, so traces are sliceable per workspace and per principal, and authorization decisions appear as their own spans with the action, resource, and decision attached. Two attribute namespaces are used and only two: * `usr.*` for human principals — the user id, email, and name. * `agentops.*` for everything else — the account, the principal, the authentication path, and authorization decisions. Identity attributes are stamped on the root span. Two opt-out settings widen that to child spans: `AGENTOPS_OTEL_SPAN_BAGGAGE_ENABLED` propagates non-identifying ids (account and principal) onto every span in the trace, and `AGENTOPS_OTEL_ALL_SPANS_IDENTITY_ENABLED` adds the principal kind and display name. Email and name deliberately stay on the root span only, to bound how much personal data reaches your telemetry store. ## What to build with it * **Alert on agent failure rate the way you alert on service error rate.** `agentops.run.failed` against `agentops.run.completed`, split by agent. * **Watch model latency separately from agent latency.** A run that got slower because one provider got slower is a different problem from a run that took more turns. * **Correlate a tool call with the system it touched.** The `agentops.tool.call` span sits in the same trace as your own service's spans when your instrumentation propagates context. * **Track token volume as an operational signal.** Token counters move before a spend chart does. Do not use exported metrics as your billing record. Spend attribution is computed in KAOP from each run's reported usage, including its cost source classification — see [Agent spend attribution](/manage-your-agents/optimize/agent-spend-attribution). Token counters tell you volume; they do not tell you money. ## Next steps The in-product view of the same activity, and what a span tree contains. The rollup KAOP builds from run activity. Where cost comes from and how it is attributed. Building a worker — instrumentation is included, not added. ## Related topics - [Observability Cost](/modules-and-workflows/cost-optimization/observability-optimization.md) - [Runs & evidence](/manage-your-agents/run/runs-evidence.md) - [Agent spend attribution](/manage-your-agents/optimize/agent-spend-attribution.md) - [Optimize — how it works](/manage-your-agents/optimize/how-it-works.md) - [Built-in integrations](/manage-your-agents/build/built-in-integrations.md)