# APIs Source: https://docs.komodor.com/developer-tools/apis The REST API behind the console — base URL, bearer authentication, error and pagination conventions, and the interactive reference. Everything the console does is available under the REST API at `/api/v1`. Your own account host serves an interactive reference with the exact request and response shape of every endpoint, which is authoritative where this documentation and the schema disagree. ## Base URL Every endpoint lives under your account host. Set it once and the examples throughout these docs work as written: ```bash theme={null} export AGENTOPS_URL=https:// ``` REST endpoints are then `"$AGENTOPS_URL/api/v1/..."`. > [!NOTE] > > The same host exposes a few sibling surfaces: `/internal/mcp` (the > [MCP server](/developer-tools/mcp-server)), `/gateway/mcp` (the > [MCP Gateway](/security-and-governance/architecture-considerations/mcp-gateway)), `/a2a/...` for agent-to-agent traffic, and > `/auth/...`. ## The interactive reference The full, always-current reference is served at **`/api/docs`** on your account host, and the **API** link in the top navigation points there. Because it is generated from the running service, it is authoritative in a way a written page cannot be — you can try requests against your own account from the browser. ## Authentication Send an [API key](/security-and-governance/identity-and-access/api-keys) as a bearer token on every request: ```bash theme={null} curl -H "Authorization: Bearer $AGENTOPS_API_KEY" \ "$AGENTOPS_URL/api/v1/agents" ``` A key acts either as a [service account](/security-and-governance/identity-and-access/service-accounts) — preferred for automation — or as the person who created it. Console requests use the signed session from single sign-on instead. Either way the request is bound to one identity in one account, and what it may do is governed by that identity's [roles and grants](/security-and-governance/identity-and-access/roles-permissions). ## Errors An unauthenticated request gets a JSON `401`. Other failures return a standard status with a JSON body carrying a readable message: ```json theme={null} { "detail": "account agent limit reached (maximum 60); archive or delete an agent before creating a new one" } ``` | Status | Means | | - | - | | `400` | Invalid request | | `401` | Missing or invalid credentials | | `403` | Authenticated, but not permitted | | `404` | Not found — or hidden by account scoping | | `409` | Conflict, such as a quota or a duplicate | | `422` | Validation error | > [!NOTE] > > A `404` can mean "exists, but not in your account". Resources are scoped to the account the > request resolved to, so a cross-account read is a not-found rather than a forbidden — which is > what keeps one account from confirming another's resources exist. ## Pagination List endpoints that can return large result sets accept `limit`, and `offset` where supported. The exact parameters and any cursor or count fields are documented per endpoint in the interactive reference. ## Input shapes * **Invoke and schedule** endpoints wrap the run parameters under an `"input"` key. * **Inbound endpoints are different**: the whole request body is handed to the agent as its input. ## Next steps > **[API keys](/security-and-governance/identity-and-access/api-keys)** > > Issue, reveal, and revoke the tokens these calls use. > **[MCP server](/developer-tools/mcp-server)** > > The same capabilities as tools, for a model to call. > **[Limits & quotas](/reference/limits-quotas)** > > The caps a `409` is telling you about. # Connect coding agents Source: https://docs.komodor.com/developer-tools/connect-coding-agents Wire Claude Code, Codex CLI, or Cursor to your account over MCP. Your local coding agent can operate the platform directly: create and invoke agents, read runs, search knowledge, manage workflows. Find it under **Settings → Connect coding agents**. ## Connecting The settings tab shows one card per supported agent — **Claude Code**, **Codex CLI**, and **Cursor** — each with the command or link that connects it, plus a manual configuration snippet if you would rather edit your own config file. What the card gives you depends on how your deployment authenticates: | Mode | What connecting looks like | | - | - | | **OAuth available** | The command needs no credential. The agent completes sign-in on first use. | | **OAuth not available** | The command and the manual snippet carry a placeholder for an [API key](/security-and-governance/identity-and-access/api-keys) that you fill in. | > [!WARNING] > > Cursor's deep link cannot carry a header, so on a deployment without OAuth the link alone is not > authenticated. Use the manual configuration snippet on that card and add your key there instead. ## What your assistant can do The connection serves the platform's own [MCP server](/developer-tools/mcp-server), so everything there applies — including that a connection can be scoped to particular toolsets, and that every tool call is authorized against the same capability as its HTTP equivalent. That last point is the one worth internalizing: **your assistant acts as the identity the connection authenticated with.** It holds that identity's permissions, its actions are attributed to it, and the governed ones land in the [audit log](/security-and-governance/controls/audit-log) under its name — you, for an OAuth sign-in or a personal key; the service account, when the connection uses one. ## Next steps > **[MCP server](/developer-tools/mcp-server)** > > The toolsets available, and how to narrow a connection. > **[API keys](/security-and-governance/identity-and-access/api-keys)** > > Issuing the key a non-OAuth deployment needs. > **[Accounts & members](/security-and-governance/identity-and-access/accounts-members)** > > Belonging to more than one account. > **[Audit log](/security-and-governance/controls/audit-log)** > > Where your assistant's governed actions are recorded. # MCP server Source: https://docs.komodor.com/developer-tools/mcp-server The platform's own capabilities as MCP tools — grouped into toolsets, scopable per connection, and authorized exactly like the API. The platform exposes its own capabilities as MCP tools, so a model or an assistant can operate it directly rather than through generated HTTP calls. It is the mirror image of the [MCP Gateway](/security-and-governance/architecture-considerations/mcp-gateway), which is the path for exposing *your* tools to agents. ## The two MCP surfaces Getting these the right way round matters: | Surface | Direction | What it is | | - | - | - | | **MCP server** (`/internal/mcp`) | Inbound to the platform | The platform's own capabilities as tools, for a model or assistant to call | | **MCP Gateway** (`/gateway/mcp`) | Outbound from an agent | Your own MCP servers and OpenAPI surfaces, exposed to agents under gateway policy | ## What is on it Entries are tools, resources, and prompts, grouped into **toolsets** by the job they do: | Toolset | What it covers | | - | - | | `core` | Overview, search, get, listing available agents, metrics | | `agents` | Managing agents, schedules, credentials, integrations, hosted agents, notification sinks | | `runs` | Invoking an agent synchronously or asynchronously, waiting on runs, cancelling one, loading an artifact | | `knowledge` | Searching knowledge, searching and saving memory | | `slack_bot` | The tools that act through the Slack bot — sending, reacting, reading threads and channels, user lookups, permalinks | | `incidents` | Managing an incident and its investigation | | `workflows` | Managing workflows, triggers, and channels | | `gateway` | Managing the gateway and adding a server to it | | `goldens` | Managing golden scenarios, versions, and suites, running them, and reading results | | `ui` | Listing and calling console tools | | `onboarding` | Listing and loading platform skills, and onboarding tasks | ## Scoping a connection By default a connection serves everything. Two query parameters subtract from that: ``` /internal/mcp?toolset=slack_bot,knowledge /internal/mcp?tools=manage_agent,search_knowledge ``` | Parameter | Value | | - | - | | `toolset` | A comma-separated list of toolsets | | `tools` | A comma-separated list of individual entry names | > [!TIP] > > Scope deliberately. A model given the whole surface spends context on tool descriptions it will > never call, and a narrow connection is both cheaper and easier to reason about. Point an assistant at > the toolsets its job needs. ## Knowing which account you are in One tool is always served regardless of scoping: **`whoami`**. A connection is bound to exactly one account, nothing else on the surface reports which, and an assistant asked "which account am I operating in?" would otherwise have no way to answer. It returns the active account, the acting identity, and — for a person — every account they belong to, with the current one marked. ## Authorization Every tool call is checked against the same capability as its HTTP equivalent. There is no shortcut here: an assistant connected as you can do what you can do, and nothing more. > [!NOTE] > > Authorization depends on the **arguments**, not only on the tool. A general-purpose tool resolves to > different capabilities depending on what it is asked to act on, so being able to call a tool is not > the same as being able to do everything it can express. Guardrails apply to this surface as well as to the gateway: a rule at the tool-call gate covers both your own connected servers and the platform's own tools. See [Standards & Guardrails](/security-and-governance/controls/standards-guardrails). ## Next steps > **[Connect coding agents](/developer-tools/connect-coding-agents)** > > Wiring Claude Code, Codex CLI, or Cursor to this surface. > **[MCP Gateway](/security-and-governance/architecture-considerations/mcp-gateway)** > > The other direction — your tools, for your agents. > **[Roles & permissions](/security-and-governance/identity-and-access/roles-permissions)** > > The capabilities a tool call is checked against. > **[APIs](/developer-tools/apis)** > > The same capabilities over HTTP. # Overview Source: https://docs.komodor.com/developer-tools/overview The three programmatic surfaces — REST API, MCP server, and a coding-agent connection — and which one to reach for. Everything the Komodor Agentic Operation Platform (KAOP) console does is reachable programmatically, through three surfaces that exist for different jobs: the REST API to automate the platform, the MCP server to let a model operate it, and a coding-agent connection to drive it from your editor. All three resolve to one identity and are checked against the same permissions, so the surface you pick changes the ergonomics rather than what you are allowed to do. ## Which surface | You want to… | Use | Why | | - | - | - | | Script or automate the platform itself — create an agent, invoke it, read a run | [The REST API](/developer-tools/apis) | Everything the console does, under one versioned HTTP surface | | Let a model or an assistant operate the platform | [The MCP server](/developer-tools/mcp-server) | The platform's own capabilities as MCP tools, governed by the same permissions | | Work from your editor or terminal assistant | [Connect coding agents](/developer-tools/connect-coding-agents) | Wire Claude Code, Codex CLI, or Cursor to your account | | Write the code that **is** an agent | [Build from scratch](/manage-your-agents/build/how-it-works) | The worker SDK — a different job from the three above | ## What they share All of these resolve to one identity in one account, and every action is checked against that identity's [permissions](/security-and-governance/identity-and-access/roles-permissions) — including each MCP tool, which is authorized against the same capability as its HTTP equivalent. There is no surface that bypasses the permission model. Governed mutations and selected sensitive reads are audited, and a run's evidence records its instrumented execution — ordinary reads are not audit entries. | Surface | Authenticates with | | - | - | | REST API | An [API key](/security-and-governance/identity-and-access/api-keys) as a bearer token | | MCP server | An API key, or OAuth where the deployment offers it | | Worker SDK | A [worker token](/security-and-governance/identity-and-access/agent-identity) bound to one agent | ## Next steps > **[APIs](/developer-tools/apis)** > > Base URL, authentication, errors, and the interactive reference. > **[MCP server](/developer-tools/mcp-server)** > > The platform's own tools, and how to scope a connection. > **[Connect coding agents](/developer-tools/connect-coding-agents)** > > Claude Code, Codex CLI, and Cursor against your account. # Architecture Source: https://docs.komodor.com/get-started/architecture How the control plane, your agents, and the evidence trail fit together, and the boundaries you can rely on. The Komodor Agentic Operation Platform (KAOP) has three parts: a **control plane** that holds every agent, run, credential and policy; the **SDK** you author an agent with; and the **workers** that execute agents wherever you run them. A worker connects outward to the control plane and receives its work over that connection, so agents run behind NAT, inside a private cluster, or with egress-only networking, and no inbound path is ever needed. ## System overview ![People use the web console and external systems such as alerts, schedules, Slack and CI reach the control plane, which holds agents, runs, evidence, identity and policy. A worker running your agent code with the SDK connects out to the control plane, registers and streams evidence, and receives work over that connection. The worker reaches Datadog, GitHub, AWS, Kubernetes, models and your MCP tools through governed integrations and the MCP Gateway.](https://mintcdn.com/komodor-test/cSgppmqRqLa0smAL/figures/fig-system-overview.svg?fit=max\&auto=format\&n=cSgppmqRqLa0smAL\&q=85\&s=d5ff3503c27030816bb2c5c156a31677) ## The control plane The control plane is the boundary for everything durable. It owns: * **Your agents:** which exist, which are online, and what each one can do. * **Runs:** starting them, tracking them, and streaming their progress to you. * **What starts work:** schedules, inbound endpoints, Slack routing, chat, workflows, and modules. * **Evidence:** everything an agent reports while it runs: the tools it called, its reasoning, its output, its logs, and what it cost. * **Who may do what:** identity, permissions, approvals, and the audit log. ## Agents and workers An **agent** is the governed identity and behavior. A **worker** is the process that runs it, built with the Python or Go SDK. One worker runs one agent. A worker does two things continuously, independent of any run: * **Registers and stays present.** It reports in periodically; the first report registers the agent, and if reports stop, the agent shows as offline. * **Reports as it works.** During a run it streams tool calls, reasoning, output, logs, and usage back to the control plane. What you see in a run is exactly what the worker reported while it ran. > [!NOTE] > > An agent needs **outbound** access to the control plane, and nothing more. You do not have to > expose an inbound endpoint to run agents in a private network. See > [Deployment methods](/get-started/deployment-methods). ## Follow one run, end to end ![Sequence diagram. An alert source posts a webhook to the control plane and receives 202 Accepted. The control plane dispatches the job to a private worker over the worker's own outbound channel. The worker calls a tool through the MCP Gateway, which makes a scoped request to the upstream tool and returns the result. The worker streams spans, usage, output and result back to the control plane.](https://mintcdn.com/komodor-test/Ov9gd2GqI0w5UgVg/figures/fig-16-run-sequence.png?fit=max\&auto=format\&n=Ov9gd2GqI0w5UgVg\&q=85\&s=879715b6aba432015f611fa7b4497658) *One alert-driven run, end to end, from inbound request to recorded evidence.* An alert posts to an endpoint you created; the control plane records the run and answers `202 Accepted`, meaning the run was accepted for background processing, not that the investigation succeeded. The job travels to the worker over the worker's own outbound connection, carrying the prompt, the input, correlation IDs, and the run-scoped credentials it needs. The worker's tool calls go through the MCP Gateway, which adds scoped credentials the agent never sees; spans, usage and output stream back as the run's evidence. Where an action is gated, the run waits for the approval decision and records it. Rotating a stored credential takes effect on the next run without redeploying the agent. ## Account isolation Every agent, run, integration, credential, and record belongs to exactly one **account**, and that boundary is enforced on every read and write. An agent in one account cannot see or reach another's data. Agents themselves are governed principals: an agent's permissions are checked at the moment it acts, not just when someone starts it. See [Agent identity](/security-and-governance/identity-and-access/agent-identity) and [Agent isolation & tenancy](/security-and-governance/architecture-considerations/agent-isolation-tenancy). ## Boundaries you can rely on * Agents never touch the platform's database; only the control plane does. * The console talks only to the control plane. * Agents need outbound HTTPS to the control plane; inbound access to an agent is never required. * No inbound firewall rule, public endpoint, or VPN is needed to run agents in a private network. * Secrets are delivered to a run in memory and never exposed to you or written to its evidence. * When nothing is running, the only traffic between an agent and the control plane is a periodic "still here". ## Next steps > **[Deployment methods](/get-started/deployment-methods)** > > Which parts run on Komodor's infrastructure and which run on yours. > **[Runs & evidence](/manage-your-agents/run/runs-evidence)** > > What the evidence trail contains and how to read it. > **[Developer tools](/developer-tools/overview)** > > Every programmatic surface that talks to the control plane. > **[Security overview](/security-and-governance/how-it-works)** > > How access and actions are governed. # Concepts & glossary Source: https://docs.komodor.com/get-started/concepts The terms you will meet in the console, the API, and the rest of this documentation, each defined once. Every piece of work an agent does in the Komodor Agentic Operation Platform (KAOP), whether it investigates an alert, reviews a pull request, or answers a question in Slack, is recorded as a **run**. The sections below follow one run from start to finish and define each term where it first appears. ![Architecture diagram. A trigger, workflow step, chat or module starts an Agent. The agent runs as a Worker, which produces a Run plus evidence, which is graded by Evaluations. The agent also uses Integrations and the MCP Gateway, reads a Knowledge base, and follows Skills.](https://mintcdn.com/komodor-test/Ov9gd2GqI0w5UgVg/figures/fig-03-agent-architecture.png?fit=max\&auto=format\&n=Ov9gd2GqI0w5UgVg\&q=85\&s=d8240c6eaaeb7888cb903a082e575b41) *One run, and the terms around it.* ## What starts a run A run begins when something asks an agent to work. That can be a person, a schedule, an external system, or a message in Slack. | Term | What it means | Read more | | - | - | - | | **Manual** | A person or a system starts the run directly, from the console, a chat message, or the API. | | | **Endpoint** | An inbound URL you create under Integrations → Endpoints. An external system posts to it and the run starts. This is how alerting and CI tools drive agents. | [Triggers & schedules](/manage-your-agents/build/triggers-schedules) | | **Trigger** | A recurring cron schedule, and the only thing the product calls a trigger. | [Triggers & schedules](/manage-your-agents/build/triggers-schedules) | | **Channel** | A messaging surface, Slack today, where people invoke agents and get results in the thread. Routing rules decide what each message starts. | [Channels](/manage-your-agents/run/channels) | A workflow step, a module, or another agent can also start a run. ## The agent The agent is what you build and configure. It carries its instructions, the tools it may use, and the knowledge you give it. When it runs, a process called a worker does the executing. | Term | What it means | Read more | | - | - | - | | **Agent** | The thing you create, name, deploy and grant access to: its instructions, the tools it may reach, the permissions it holds, and its history. An agent is a **principal**: KAOP authorizes its actions the same way it authorizes a person's. | [How building works](/manage-your-agents/build/how-it-works) | | **Worker** | The process that runs an agent. One worker runs one agent. You configure and inspect the agent; you look at the worker when one is offline or on the wrong version. | [Fleet health](/manage-your-agents/optimize/fleet-health) | | **Skill** | A reusable unit of know-how (a runbook, a procedure, a pattern) attached to an agent. Skills tell it *how* to work. | [Skills](/manage-your-agents/build/skills) | | **Knowledge base** | Documents you upload, indexed so agents can search them at run time and cite what they used. Knowledge tells an agent *what you know*. Nothing is ingested automatically. | [Knowledge base](/manage-your-agents/build/knowledge-base) | | **Memory** | What an agent carries forward between runs, so a later run finds what an earlier one worked out instead of re-deriving it. Memory tells it *what it has already learned*. Writes are reviewed before a later run can read them. | [Memory](/manage-your-agents/build/memory) | ## What the agent reaches To do anything useful, an agent needs access to your systems. Every connection is one of two kinds, and in both the credentials stay with the platform, never with the agent. | Term | What it means | Read more | | - | - | - | | **Built-in integration** | A system from the catalog, such as Datadog, AWS, Azure, GitHub, Slack, Kubernetes or PagerDuty, connected once and shared by the agents you allow. | [Built-in integrations](/manage-your-agents/build/built-in-integrations) | | **MCP Gateway** | Your own tool surface, any MCP server or OpenAPI endpoint, exposed to agents as tools, with the gateway holding the credentials. | [MCP Gateway](/security-and-governance/architecture-considerations/mcp-gateway) | | **Integration group** | A named bundle of integrations you attach to an agent as a set. | [Integration groups](/manage-your-agents/build/integration-groups) | | **Credential** | A stored secret, encrypted at rest, bound to the agents allowed to use it, and delivered to a run without its value ever appearing to you or in the agent's logs. | [Credentials & secrets](/security-and-governance/identity-and-access/credentials-secrets) | | **Outbound webhook** | The reverse direction: KAOP posting events (a run finished, an incident was triaged) to an endpoint you own, signed and retried. | [Outbound webhooks](/manage-your-agents/run/outbound-webhooks) | ## What the run leaves behind When the run finishes you have a complete record of it, and the means to judge whether it was any good and what it cost. | Term | What it means | Read more | | - | - | - | | **Run** | One execution of an agent, with everything it did: the input, each tool call, the reasoning, the output, and the cost. The unit you open to find out what happened. | [Runs & evidence](/manage-your-agents/run/runs-evidence) | | **Evidence** | That record. It is what every term below measures. | [Runs & evidence](/manage-your-agents/run/runs-evidence) | | **Evaluation** | Grading a run against explicit criteria. A **judge** (an LLM or agent) applies a **rubric** (the named criteria) and returns a **verdict** with per-criterion reasoning. A verdict is itself a run you can open. | [Evaluations](/manage-your-agents/optimize/evaluations) | | **Golden scenario** | A fixed input with a known-good result, kept as a regression test for an agent's behavior. | [Evaluations](/manage-your-agents/optimize/evaluations) | | **Model evaluation** | Comparing models on the same work, to decide which one an agent should use. **Shadow** runs a candidate alongside the live agent without serving its output. | [Model evaluation](/manage-your-agents/optimize/model-evaluation) | | **Fleet health** | The operational state of every agent you run: online, failing, drifting, idle. | [Fleet health](/manage-your-agents/optimize/fleet-health) | | **Agent spend** | What each agent costs, per agent and per run, with budgets that can stop an agent at a cap. | [Agent spend](/manage-your-agents/optimize/agent-spend-attribution) | ## Workflows and modules Most operations work takes more than one agent, and it has a shape: first correlate the alerts, then investigate, then propose a fix, then ask a person before applying it. A **workflow** is that sequence of steps, with agents assigned to each. A **module** packages Komodor's expertise in one operational domain, such as incident management or Kubernetes cost, as the workflows and specialist agents for that domain. You enable a module; its workflows do the work. For example, the module **Incident management & troubleshooting** contains the workflow **Production Alert Investigation**. That workflow has four steps: source, dedupe and correlate, investigation, remediation. The investigation step is handled by the Datadog, Kubernetes and AWS investigators; the remediation step proposes a fix and waits for a person to approve it. When you open a module you see its workflows; when you open a workflow you see its steps and which agents handle each. | Term | What it means | Read more | | - | - | - | | **Module** | Komodor's expertise in one operational domain, such as "Incident management & troubleshooting" or "Kubernetes Cost", packaged as workflows and the specialist agents they use. You enable it for your account. It answers *which domain is this?*; its workflows answer *how does the work get done?* | [Modules & workflows](/modules-and-workflows/overview) | | **Workflow** | A sequence of steps inside a module that carries out one piece of that domain's work. The steps are fixed by the workflow's template; you choose which agents handle each step. | [Orchestrator](/manage-your-agents/build/catalog/orchestrator) | | **Step** | One named task in a workflow, with a goal, an input, and an output. A step is carried out by the agents assigned to it, by KAOP itself with no model involved, or by a person choosing between options. | | | **Orchestrator** | The catalog agent that works a step through the agents assigned to it, so several specialists can contribute to one task. | [Orchestrator](/manage-your-agents/build/catalog/orchestrator) | ## Who may do what Everything above happens inside an account, under permissions that apply to people and agents alike. | Term | What it means | Read more | | - | - | - | | **Account** | The tenant boundary: its members, agents, and data. Everything belongs to exactly one, and you act in one at a time. | [Accounts & members](/security-and-governance/identity-and-access/accounts-members) | | **Member** | A person with access to an account. One person can belong to several. | [Accounts & members](/security-and-governance/identity-and-access/accounts-members) | | **Role** | A bundle of permissions assigned to a member, a service account, or an agent. | [Roles & permissions](/security-and-governance/identity-and-access/roles-permissions) | | **Service account** | A non-human identity for automation, with its own grants and API keys. | [Service accounts](/security-and-governance/identity-and-access/service-accounts) | | **API key** | A credential that authenticates a person or a service account to the API. | [API keys](/security-and-governance/identity-and-access/api-keys) | | **Agent identity** | The principal an agent authenticates as, separate from whoever deployed it. | [Agent identity](/security-and-governance/identity-and-access/agent-identity) | | **Guardrail** | A rule evaluated at a gate that blocks an action, holds it for approval, redacts a matched value, or records it. | [Standards & guardrails](/security-and-governance/controls/standards-guardrails) | | **Approval** | A risky action paused for a human decision, with the evidence needed to make it. | [Approvals](/security-and-governance/controls/approvals) | | **Audit log** | The append-only record of sensitive actions and access decisions. | [Audit log](/security-and-governance/controls/audit-log) | | **Access explorer** | Who, or which agent, can reach a given resource, and by which grant. | [Access explorer](/security-and-governance/identity-and-access/access-explorer) | | **Configuration cluster** | Configuration managed as code and applied to the platform, rather than edited in the console. | [Configuration clusters](/security-and-governance/identity-and-access/configuration-clusters) | ## Next steps > **[Architecture](/get-started/architecture)** > > How the control plane, SDK and workers fit together. > **[Modules & workflows](/modules-and-workflows/overview)** > > The module catalog, and how modules contain workflows. > **[Build an agent](/manage-your-agents/build/how-it-works)** > > Author your own or deploy one from the catalog. > **[Runs & evidence](/manage-your-agents/run/runs-evidence)** > > What a run records and how to read it. # Deployment methods Source: https://docs.komodor.com/get-started/deployment-methods Where your agents run, where the control plane runs, and how to choose. The Komodor Agentic Operation Platform (KAOP) has three parts: the control plane, the SDK, and the agents. Two of them are deployed somewhere and matter for a network and security review: the **agents**, which do the work and need access to the systems they work on, and the **control plane**, the service behind the console that runs them and keeps the record. The SDK is a library inside each agent, not something you run. Each of the two can run on Komodor's infrastructure or on yours, and the choices are independent. ## Komodor-hosted agents ![Komodor runs both the control plane and the agents; your network has nothing to install. Hosted agents reach public APIs such as Datadog, AWS and PagerDuty.](https://mintcdn.com/komodor-test/RZpNEefdbWe1K_UP/figures/fig-deploy-hosted-agents.svg?fit=max\&auto=format\&n=RZpNEefdbWe1K_UP\&q=85\&s=036e7b7167260b782082eac8f2238bf8) Deploy from the console; Komodor runs the agent. Nothing to install. Choose this when everything the agent needs is reachable over public APIs: Datadog, your cloud provider, GitHub. Most catalog agents run this way. ## Self-hosted agents ![Agents run in your network beside the private systems they reach, with one outbound HTTPS connection to the Komodor-hosted control plane.](https://mintcdn.com/komodor-test/RZpNEefdbWe1K_UP/figures/fig-deploy-self-hosted-agents.svg?fit=max\&auto=format\&n=RZpNEefdbWe1K_UP\&q=85\&s=438a56b9a347861419a612adc4ba3a5c) Run the agent in your own cluster or VM; it connects out to the control plane and appears in your fleet like any other. Choose this when the agent needs something only reachable from inside your network. The agent needs outbound HTTPS to the control plane and nothing else. No inbound firewall rule, no VPN. The choice is per agent, so hosted and self-hosted agents share one fleet and one evidence trail. ## Self-hosted control plane ![Control plane, agents and private systems all run in your network; outbound traffic goes only to the model endpoint and the systems you connect.](https://mintcdn.com/komodor-test/RZpNEefdbWe1K_UP/figures/fig-deploy-self-hosted-control-plane.svg?fit=max\&auto=format\&n=RZpNEefdbWe1K_UP\&q=85\&s=c6aed740e66ab35acaf5f00776bdcdf0) For organizations that cannot use a hosted service, the control plane itself runs in your own environment: everything the platform stores stays with you, and the only outbound traffic is to the model and the systems you connect. It ships as an umbrella Helm chart that bundles every component, so it installs as a single release. > [!NOTE] > > **Beta.** Self-hosting the control plane is available under agreement, not as a self-service > install. Talk to your account team before planning a rollout around it. ## Next steps > **[Network & egress control](/security-and-governance/architecture-considerations/network-egress-control)** > > What agent traffic may reach, and how to scope it. > **[Data handling & redaction](/security-and-governance/architecture-considerations/data-handling-redaction)** > > What a run records, and what never leaves your network. > **[Architecture](/get-started/architecture)** > > How the control plane and agents communicate. # Onboarding Source: https://docs.komodor.com/get-started/onboarding The six steps that take an account from a first run to agents working against your systems, your model, your alerts, and your team. The [Quickstart](/get-started/quickstart) proves the loop on a fresh Komodor Agentic Operation Platform (KAOP) account. The six steps below connect it to your systems, your model, your alerts, and your team. They are ordered by dependency: an agent needs something to reach before it is worth creating, and a workflow needs agents to bind to. You act inside one account at a time. If you belong to more than one, use the account switcher at the foot of the sidebar. Anything your role does not grant appears locked. See [Accounts & members](/security-and-governance/identity-and-access/accounts-members). ## 1. Connect your systems An agent that cannot see your systems cannot do operations work. Connect Datadog, AWS, GitHub, Kubernetes, Slack and the rest from the catalog, or expose your own tools through the MCP Gateway. Both paths keep credentials away from the agent. See [Integrations overview](/manage-your-agents/build/integrations-overview). ## 2. Connect a model provider Agents call models through KAOP, never with a provider key of their own. Register your provider account and the models on it, or use the models Komodor manages for your account; an agent can be pointed at either. See [Providers](/security-and-governance/architecture-considerations/providers). ## 3. Create your agents There are three ways to create one: **Start from catalog** to deploy a Komodor-built agent, **Build from scratch** to author a worker with the SDK, or **Import** to bring in an agent that already exists. Whichever you choose, the agent gets its own identity, permissions, and run history. See [How building works](/manage-your-agents/build/how-it-works). ## 4. Create an endpoint for alerts and CI to post to An endpoint is an inbound URL an external system posts to in order to start work: an alerting tool, a CI job, a webhook from your own service. Create one under **Integrations → Endpoints**; the requests it receives start runs with their origin recorded. See [Triggers & schedules](/manage-your-agents/build/triggers-schedules). ## 5. Chain agents into a workflow Most operations work is too big for one agent. A workflow is an ordered set of steps, each delivering a task to one or more agents. One step correlates alerts, the next investigates, the next proposes a remediation. See [Orchestrator](/manage-your-agents/build/catalog/orchestrator). ## 6. Connect Slack A channel puts agents where the team already works. Connect Slack and people invoke agents from a channel and get results in the thread; routing rules decide what each inbound message targets: an agent, a workflow, or an incident pipeline. See [Channels](/manage-your-agents/run/channels). ## Next steps > **[Deployment methods](/get-started/deployment-methods)** > > Run agents inside your own network against the hosted control plane. > **[Modules & workflows](/modules-and-workflows/overview)** > > Komodor's expertise in incidents, cost and change, packaged as workflows you enable per account. > **[Chat & history](/manage-your-agents/run/chat-history)** > > Work conversationally with any chat-capable agent; every turn is a recorded run. > **[Roles & permissions](/security-and-governance/identity-and-access/roles-permissions)** > > Who can do what, before you invite the rest of the team. # Overview Source: https://docs.komodor.com/get-started/overview What the Komodor Agentic Operation Platform is, who it is for, what teams use it for, and where to start. The Komodor Agentic Operation Platform (KAOP) runs AI agents on your production operations work: investigating incidents, reviewing changes, finding wasted spend. It does so with the controls your security team will ask about: each agent has its own identity and permissions, reaches your systems without ever seeing a credential, and leaves a complete record of what it did and what it cost. Under the hood it has three parts: a **control plane** Komodor operates, the **SDK** you write agents with, and the **workers** that run them, on Komodor's infrastructure or inside your own network. ## Who it is for Platform engineers, SREs, and developers responsible for production operations, who need agent work to be governed, auditable, and measurable. ## What teams use it for Komodor packages its operational expertise as **modules**, in three families. A module holds the workflows and specialist agents for one domain; you enable it per account. | Family | Modules | What it does | | - | - | - | | **[AI SRE](/modules-and-workflows/sre/incidents)** | Incident management & troubleshooting · Alert Intelligence · Proactive reliability optimization | Detect, investigate, and remediate production incidents; classify which monitors justify paging; convert error budgets and recurring failures into concrete work. | | **[Cost Optimization](/modules-and-workflows/cost-optimization/k8s-cost)** | Kubernetes Cost · Cloud Cost · Observability Cost | Attribute Kubernetes spend to the teams that cause it; catch cloud billing anomalies; find monitoring spend that buys nothing. | | **[AI Software Operations](/modules-and-workflows/ops-automation/change-intelligence)** | Change Intelligence & Risk Control · Production Readiness & Standards · CI/CD Health & Remediation | Decide whether a shipped change is good and roll it back if not; score services against production standards; keep the merge path trustworthy. | Where no module covers the work, you build the agent yourself with the SDK and govern it through the same identity, evidence, and policy model. See [Build from scratch](/manage-your-agents/build/how-it-works). ## What you get | | | | - | - | | **Agents that already know your stack** | The catalog is Komodor-built and validated agents for Kubernetes, AWS, Azure, Google Cloud, Datadog, Grafana and GitHub: investigators, remediators, code reviewers, a cost analyzer. Deploy one in a couple of minutes; it arrives with its own identity, permissions, and run history. | | **Expertise, not just a runtime** | Modules package how experienced SREs work a domain as ready workflows: an incident detected, grouped, investigated and remediated; alerts sorted by whether they justify paging; Kubernetes spend attributed to the teams causing it; a bad change caught and rolled back. You enable the module; the workflows run. | | **Agents that act on production, safely** | Agents do not only read. They fix a live resource, open the pull request, roll back the deploy. A risky action pauses for a human with the evidence to decide; guardrails block or redact; every step lands in the audit log. | | **Proof for every run** | Every run records what the agent was asked, which tools it called, how it reasoned, what it concluded, and what it cost. Grade that against your own criteria, compare models on the same work, and see spend per agent and per team. | | **Your own agents, same guarantees** | Where no catalog agent fits, write one with the SDK or import one you already run. It gets the same identity, evidence, approvals and audit trail as Komodor's. Nothing is bolted on afterwards. | Where the agents run and where the control plane runs are separate choices. See [Deployment methods](/get-started/deployment-methods). ## How you drive it | Interface | Use it to | Who reaches for it | | - | - | - | | **Web console** | Build agents, watch runs, operate modules, manage access | Everyone; the primary surface | | **Slack** | Invoke agents and receive results where the team already works | On-call and responders | | **REST API** | Automate anything the console does | Platform engineers, CI, internal tooling | | **MCP server** | Let a coding assistant or another agent inspect the account and invoke agents | Claude Code, Cursor, Codex | | **MCP Gateway** | Expose *your* tools to agents | Platform engineers | The programmatic surfaces are covered in [Developer tools](/developer-tools/overview); Slack in [Channels](/manage-your-agents/run/channels). ## Give your AI agent the full context Download all published documentation as a single Markdown file to give your AI agent context about Komodor's platform, features, and workflows. Each download includes the latest published pages. > **[Download docs for AI (.md)](https://docs.komodor.com/llms-full.txt)** ## Where to go next > **[Quickstart](/get-started/quickstart)** > > Sign in, get a first run from the built-in assistant, deploy a catalog agent. About ten minutes. > **[Onboarding](/get-started/onboarding)** > > The six steps that connect an account to your systems, your model, your alerts, and your team. > **[Deployment methods](/get-started/deployment-methods)** > > Where agents run, where the control plane runs, and what leaves your network. > **[Concepts & glossary](/get-started/concepts)** > > The objects and vocabulary the rest of this documentation uses. # Quickstart Source: https://docs.komodor.com/get-started/quickstart Sign in, get a first run out of the built-in assistant, deploy one catalog agent and run it, and read the evidence every run records, in about ten minutes. A new Komodor Agentic Operation Platform (KAOP) account has no agents and nothing connected. In about ten minutes you can get a first run out of the built-in assistant, deploy an agent from the catalog, run it, and read the evidence it produced. Everything else in KAOP extends that loop. ## Before you start * The account URL your Komodor contact sent you. Sign-in is by **single sign-on**; there is no separate KAOP password, and your access is provisioned on first sign-in. * Optionally, one system to connect (Kubernetes, Datadog, AWS, or GitHub) so the catalog agent you deploy has something real to look at. ## Your first run **1. Sign in** Open your account URL and sign in. You land on **Overview**: a chat box, counts of agents online and recent activity, and anything needing attention. The sidebar groups the console by job: **Build**, **Operate**, **Improve**, **Connect**, with **Settings** in the footer. **2. Ask the built-in assistant something** Type a question into the chat box on Overview. *What can you do?* is a fine start. The built-in chat assistant works before you have deployed or connected anything, and because it is an agent like any other, your question and its answer are recorded as a run. **3. Open the run** Open **History**. It lists every run and chat session, newest first. Click the question you just asked: you see what you asked, which tools the assistant used, how it reasoned, what it answered, and what it cost. This is what every run in KAOP records, whoever or whatever started it. ## Deploy a catalog agent and run it The catalog holds Komodor-built agents you deploy as they are: read-only investigators for Kubernetes, AWS, Azure, Google Cloud, Datadog and Grafana, code review and GitHub agents, remediation agents, an AWS cost analyzer, and an orchestrator. **1. Pick an agent** Open **Build → Agents**, choose **Add agent**, then **Start from catalog** and **Browse catalog**. Each card says what the agent reads and whether it is **Ready to deploy** on your account or **Needs setup**, in which case the wizard collects what is missing, usually a credential for the system it investigates. If you connected a system in *Before you start*, deploy the investigator for that system. It has the shortest path from deploy to a useful run. **2. Deploy it** Press **Deploy to Komodor cloud**. Komodor runs the process for you; nothing to install. The agent shows as **deploying**, then **online** once it reports in, usually within a couple of minutes. It now appears in **Fleet** with its own identity, permissions, and run history. **3. Run it** On the agent's row in **Fleet**, choose **Run agent** and describe the task in plain text. For a Kubernetes investigator, something like *why is the checkout deployment in namespace prod restarting?* **4. Read the evidence** The run lands in **History**. Open it. Every tool the agent called, every argument and result, the reasoning between them, the final output, and the cost of the run are there. It is the same record you saw for the chat turn, now against your own infrastructure. ## Next steps > **[Onboarding](/get-started/onboarding)** > > Connect your systems, your model, your alerts, and Slack: the six steps to production use. > **[Agent catalog](/manage-your-agents/build/catalog/overview)** > > Every Komodor-built agent: what it reads, what it costs in time, where it stops. > **[Runs & evidence](/manage-your-agents/run/runs-evidence)** > > What the evidence trail contains and how to read it. > **[APIs](/developer-tools/apis)** > > Invoke an agent and read a run programmatically. # Integrations list Source: https://docs.komodor.com/manage-your-agents/build/built-in-integrations The catalog of ready-made connections — cloud, observability, source control, ticketing, chat — with the credentials each needs and how to test one. A built-in integration is a connection the Komodor Agentic Operation Platform (KAOP) already knows how to make. You supply credentials once; the tools, the auth handling, and the connection health come with it. ## Where the catalog lives Everything on this page lives under **Integrations**. The section rail splits it into: | Section | What it holds | | - | - | | **Integrations** | The provider catalog and your live connections | | **Integration groups** | Named bundles you attach to an agent — see [Integration groups](/manage-your-agents/build/integration-groups) | | **Endpoints (inbound)** | URLs external systems post to in order to start work | | **Notification sinks** | KAOP posting events to an endpoint of yours | | **Channels** | Slack surfaces where people invoke agents | ## The catalog | Provider | Category | How it authenticates | | - | - | - | | **AWS** | Cloud | Access keys, or a cross-account IAM role | | **Azure** | Cloud | Application credentials | | **Google Cloud** | Cloud | Service account key | | **Datadog** | Observability | API key | | **Dynatrace** | Observability | API tokens | | **New Relic** | Observability | API key | | **Grafana Cloud** | Observability | Service account token | | **Grafana** | Observability | Service account token | | **GitHub** | Source control | App install | | **GitLab** | Source control | OAuth | | **Buildkite** | CI/CD | API key | | **PagerDuty** | Incident management | OAuth | | **Opsgenie** | Incident management | API key | | **Jira** | Ticketing | OAuth | | **Linear** | Ticketing | OAuth | | **Sentry** | Error tracking | OAuth | | **LaunchDarkly** | Feature flags | API key | | **Slack** | Communication | OAuth, or your own app's tokens | | **Microsoft Teams** | Communication | OAuth | | **Google Workspace** | Productivity | OAuth | | **Komodor** | Kubernetes | OAuth | > [!NOTE] > > Some providers are unavailable until this deployment has the corresponding OAuth or app registration > configured. The catalog says so on the tile rather than letting you start a connect that cannot > finish. ### Providers with sub-selections Two providers let you narrow what the connection covers, so an agent is not handed more surface than it needs: * **Google Cloud** — choose the products to include: Cloud Logging, Cloud Monitoring, Error Reporting, Cloud Trace, Service Health, Cloud Quotas, Compute Engine, Google Kubernetes Engine, Cloud Run, Cloud Billing, Recommender, Cloud Asset Inventory, Network Management, and Policy Troubleshooter. * **Google Workspace** — choose from Drive, Docs, Sheets, Slides, Calendar, Chat, Contacts, and Gmail. Because each product is its own connection, one organization legitimately has several Google Workspace connections. ## The auth modes Four shapes of connection exist, and which one you get is decided by the provider, not by you. **API key** You paste the provider's key or token into the connect dialog and press **Connect**. For API-key providers the credential is checked as part of connecting: an authentication failure refuses the create outright, so you never end up with a connection that was never going to work. **OAuth** You press **Authorize with \** and complete the provider's own sign-in in a popup. KAOP stores the resulting tokens and refreshes them as needed — there is nothing for you to rotate by hand. **App install** GitHub connects by installing a GitHub App: press **Install GitHub** and choose which organization and repositories it covers. The permissions the app requests are shown before you install, and recorded on the connection afterwards. **Cross-account role** AWS also offers an IAM-role variant instead of long-lived keys. You deploy a CloudFormation stack that creates a role KAOP may assume, then paste the created role back to complete the connection. The external id used in the trust policy is derived from your account and cannot be supplied by the caller, which is what stops one account naming another's role. > [!TIP] > > Prefer the AWS IAM-role variant over access keys. It avoids storing long-lived credentials, and the > role can be scoped read-only at creation time. ## Connect one **1. Open the catalog** Go to **Integrations** and press **Add integration**. Search or browse by category. **2. Pick the provider** Press **Connect** on its tile. If you already have a connection to that provider, the tile reads **Add another** instead — connecting twice is normal when you have two accounts or two regions. Slack is the exception: its tile opens the **Connect Slack** dialog described in [Connecting Slack](/manage-your-agents/run/channels#connecting-slack). **3. Choose the variant, if it has one** AWS asks whether to authenticate with access keys or an IAM role. The choice decides what the connection can do, so make it deliberately. **4. Name it and supply credentials** A display name is required and must be unique for that provider — it is how you tell two connections apart everywhere else in the product. Fill in the provider's fields, then complete the connect, authorize, or install. **5. Confirm it works** Use **Test connection** from the connection's menu. See [Test and reconnect](#test-and-reconnect) for how to read the result. **6. Attach it to an agent** A connection existing does not give any agent access to it. Attach it on the agent — directly, or as part of an [integration group](/manage-your-agents/build/integration-groups). ## What each provider asks for Field labels are the provider's own vocabulary, which is what you will see in the dialog. | Provider | Required | Optional | | - | - | - | | **AWS** (access keys) | Access key ID · Secret access key | AWS region | | **AWS** (IAM role) | AWS account ID | Role name | | **Azure** | Client secret · Application (client) ID · Directory (tenant) ID · Subscription ID | — | | **Google Cloud** | Service account key (base64) | Products | | **Datadog** | API key · Application key · Datadog site | Toolsets | | **Dynatrace** | Platform token · Environment URL | Classic API token | | **New Relic** | User API key | New Relic region | | **Grafana Cloud** | Service account token · Grafana URL | — | | **Grafana** | Service account token · Grafana URL | — | | **Buildkite** | API Access Token | — | | **LaunchDarkly** | API access token | — | | **Opsgenie** | API key | — | | **GitHub** | Nothing to paste — the app install carries the grant | — | | **Slack** | KAOP's Slack app: nothing to paste. [Your own Slack app](/manage-your-agents/run/slack-own-app): its App ID and tokens | — | | **Microsoft Teams** · **PagerDuty** · **Jira** · **Linear** · **Sentry** · **GitLab** · **Google Workspace** · **Komodor** | Nothing to paste — OAuth carries the grant | — | > [!WARNING] > > The **Datadog site** and the **Dynatrace environment URL** are not cosmetic — they select which > regional endpoint the connection talks to. Getting them wrong produces authentication failures that > look like a bad key. ## Test and reconnect A connection carries two independent pieces of state, and reading them as one is the most common source of confusion. | State | Question it answers | Values | | - | - | - | | **Status** | Did someone finish connecting this, and is it paused? | Pending · Active · Error · Revoked · Disabled | | **Last check** | Do the credentials actually work? | Passed · Failed · Skipped | **Skipped is not a pass.** Not every provider has a check KAOP can make, and for those the result is skipped rather than a green tick you should not trust. A failure is additionally classified so you know where to look: authentication, network, timeout, a tool error, a missing credential, or no available check. ### Testing Use **Test connection** from the connection's menu. The test makes one real call to the provider — an identity or a light read — and records the verdict on the connection. > [!WARNING] > > Testing runs when you ask for it. Nothing re-checks a connection in the background, so a credential > that expires after a passing test keeps reading as passed until someone tests again. A result older > than a day is shown as verified earlier rather than as current — treat that as a prompt to re-test > before relying on it. ### Reconnecting When a connection's last check has failed, a **Reconnect** button appears directly on the row. When it is not failing, the same action lives in the menu as **Re-authorize** — use it to replace an expired credential or extend a grant. * Reconnecting an OAuth connection is pinned to the account it already serves. Authorizing a different account is refused rather than silently repointing the connection. * Replacing a credential clears the previous verdict, so test again afterwards. ### The other actions | Action | What it does | | - | - | | **Test connection** | Run the check and record the verdict | | **View details** | Inspect the connection, including its id | | **Add server** | Expose the provider's MCP tools through this connection | | **Re-authorize** / **Reconnect** | Replace or refresh the credential | | **Disconnect** | Remove the connection. Your own Slack app works differently: see [Disconnect](/manage-your-agents/run/slack-own-app#disconnect) | > [!WARNING] > > Disconnecting is refused while something still depends on the connection, and the refusal names what > depends on it. Detach or repoint those first rather than forcing it — an agent left pointing at a > removed connection loses the tools it was relying on. ## Exposing a provider's tools Connecting a provider stores a credential. For providers that ship an MCP surface, the setup wizard also stands up the tool server that uses it, in three steps — **Connect account**, **Expose tools**, and **Route**. On the way through you choose whether to expose every discovered tool and which integration group the result belongs to. Started from a Slack connection (its row's **Add server**, or **Set up the Slack MCP** after connecting), the setup opens on **Expose tools** with that connection chosen, and **Test and discover tools** adds the server. Nothing is added before that. Some providers instead authenticate against their MCP server's own authorization server, in which case signing in *is* the setup and no separate stored credential is involved. For your own tools rather than a provider's, see the [MCP Gateway](/security-and-governance/architecture-considerations/mcp-gateway). ## Next steps > **[Integration groups](/manage-your-agents/build/integration-groups)** > > Bundle connections and attach the set to an agent. > **[Credentials & secrets](/security-and-governance/identity-and-access/credentials-secrets)** > > How stored secrets reach a run without being exposed. > **[MCP Gateway](/security-and-governance/architecture-considerations/mcp-gateway)** > > The other connection path — your own tools. > **[Triggers & schedules](/manage-your-agents/build/triggers-schedules)** > > Endpoints, and why they are not triggers. # APM Source: https://docs.komodor.com/manage-your-agents/build/catalog/apm The read-only investigators for Datadog and Grafana — what each reads, the paths to reach it, what to ask it, and where each stops. Two catalog agents investigate a regression through application performance telemetry: **Datadog Investigator** and **Grafana Investigator**. Both run the same aggregate → pivot → trace flow — narrow on the failing slice, read the flame graph of one bad trace within it, correlate against deploys — and return structured findings with a deeplink on every claim. Both are strictly read-only. Neither can mute a monitor, silence an alert, close an incident or edit a dashboard. ## Datadog Investigator ### Overview An on-call engineer with a latency regression and a Datadog account has the evidence but not the time: localising it means narrowing on the failing slice, opening one bad trace, reading its flame graph, then checking what deployed. That is twenty minutes of retrieval before the thinking starts, repeated every incident. This agent does that retrieval. Hand it an incident description and the questions you want answered, and it returns structured findings with a Datadog permalink on every claim — so the engineer starts from evidence rather than assembling it. **For:** SREs and on-call engineers who already run Datadog as their primary telemetry. **Not for:** infrastructure-level causes — that is a cloud provider investigator's job. ### Tools supported | Tool surface | What it can call | | - | - | | Traces and spans | Search, aggregate, and fetch one trace's spans | | Logs | Query scoped to a service and window | | Metrics | Timeseries query | | Monitors | Read definition and state transitions | | Events | Read deploys and changes in a window | Read-only. There is no mutating tool in its surface, so it cannot mute, acknowledge, close or edit anything in Datadog. ### Scenario examples In chat, or as a run's prompt: **Latency regression** ```text theme={null} Checkout-api p99 latency tripled between 09:00 and 09:40 UTC. Find where the time is going, whether it is us or a downstream dependency, and whether anything deployed in that window. Link the traces you used. ``` **Error-rate spike** ```text theme={null} payments-api error rate went from 0.2% to 6% at 11:20 UTC. Which endpoint, which error, and is it correlated with a deploy or a dependency? ``` **Post-remediation verification** ```text theme={null} We rolled back checkout-api at 15:05 UTC. Confirm whether p99 and error rate have returned to their pre-incident baseline, and say what still has not recovered. ``` When a workflow drives it instead of a person, it takes a structured envelope: | Field | What it is | | - | - | | `incident_description` | What is happening, in a sentence or two | | `questions` | The specific questions to answer — the agent scopes its work to these | | `affected_services` | Service names, to narrow the search | | `investigation_start` | An ISO-8601 timestamp; it looks no further back | | `context` | Enrichment already gathered — alerts, errors, deploys | | `budget_seconds` | A shorter or longer wall-clock budget for this run | ### Prerequisites | Requirement | Detail | Where to set it up | | - | - | - | | Integrations | A Datadog connection. An integration group is **optional** here, unlike the other investigators — the agent is scoped to Datadog's own MCP server, which the connection itself credentials, and the wizard derives the connection from that. Add a group only to give it Datadog alongside other tools. | [Built-in integrations](/manage-your-agents/build/built-in-integrations) | | Permissions | Read-only Datadog access spanning traces, logs, metrics, monitors and events. A connection missing one of those narrows what it can conclude rather than failing. | — | | Model access | The managed model gateway, or your own Claude API key | [Credentials & secrets](/security-and-governance/identity-and-access/credentials-secrets) | | Deployment | Komodor cloud or self-hosted. Neither is required — there is no in-cluster dependency. | — | **Catalog ID:** `datadog-investigator` — what the deploy API takes. > [!NOTE] > > An account can hold several Datadog connections. Provider presence alone does not identify the > organization or environment you meant, so select the specific connection this agent should use > rather than assuming the only active one is the right one. ### As part of a workflow A specialist, not a lead. An orchestrator delegates the Datadog slice of an incident to it and synthesizes its findings with other specialists'. Because it reads monitors as well as telemetry, it also works as the verification step *after* a remediation — asking whether the signal that fired has actually recovered. See [Orchestrator](/manage-your-agents/build/catalog/orchestrator) for how a step delivers work to it, and [Incidents](/modules-and-workflows/sre/incidents) for the module that drives it. ### Limitations * **It writes nothing to Datadog.** No muting, acknowledging, closing or editing. * **It is bounded by what Datadog holds.** If the span was sampled away or the retention window has passed, it reports that it could not verify the step rather than inferring. * **It sees only Datadog.** A cause in the cloud provider, the cluster, or the code is outside its reach — pair it with the relevant investigator. * **A partial report is possible.** Budget exhaustion returns what it has, so read the findings rather than assuming completeness. ### Recommended model `claude-sonnet-4-6` — the shipped default. Investigation is long-context, multi-step tool use where the work is retrieval and correlation rather than novel reasoning, so Sonnet's speed and cost suit it better than a larger model; a faster model degrades the correlation step, which is where the value is. ## Grafana Investigator ### Overview The same problem in a different stack. A regression visible in Prometheus means pivoting to Loki for the error text, to Tempo for the trace, and to Pyroscope when the cost is CPU or memory rather than a downstream call — four tools and four query languages before the diagnosis starts. Hand it an incident description and the questions you want answered. It runs the aggregate → pivot → trace flow across those data sources and returns structured findings with Grafana deeplinks, so each claim can be checked. **For:** SREs and on-call engineers whose telemetry lives in the Grafana stack, on Grafana Cloud or their own Grafana. **Not for:** infrastructure-level causes — that is a cloud provider investigator's job. ### Tools supported | Tool surface | What it can call | | - | - | | Prometheus | Rates, latencies and saturation across the incident window | | Loki | The error text behind a failure, scoped to the failing slice | | Tempo | Traces, and the flame graph of one bad trace | | Pyroscope | Profiles, when the regression is in CPU or memory rather than a dependency | | Alerts and incidents | What fired, when, and what it was watching | | Annotations | Deploys and changes to correlate against | | Sift | Grafana's own automated investigations, where they have already run | Read-only across all of them. It cannot silence an alert, close an incident, edit a dashboard or add an annotation. ### Scenario examples In chat, or as a run's prompt: **Error-rate spike** ```text theme={null} Error rate on payments-api climbed from 0.2% to 6% starting 11:20 UTC. Localise it: which route, which dependency, and does the profile show anything. Link the Grafana views you used. ``` **Saturation, not errors** ```text theme={null} search-api is slow but nothing is erroring, since about 08:30 UTC. Check whether it is CPU, memory or a downstream call, and use the profile if the traces do not settle it. ``` **Did the alert tell the truth** ```text theme={null} The HighLatency alert on cart-api fired at 02:10 UTC and resolved itself. Was there a real regression in that window, or is the monitor mistuned? ``` When a workflow drives it instead of a person, it takes the same structured envelope the other investigators take — `incident_description`, `questions`, `affected_services`, `investigation_start`, `context` and `budget_seconds`. ### Prerequisites | Requirement | Detail | Where to set it up | | - | - | - | | Integrations | A path to Grafana — one of the three below | [Built-in integrations](/manage-your-agents/build/built-in-integrations) | | Permissions | Read access to the data sources you point it at. It reports an unreachable source rather than working around it. | — | | Model access | The managed model gateway, or your own Claude API key | [Credentials & secrets](/security-and-governance/identity-and-access/credentials-secrets) | | Deployment | Komodor cloud or self-hosted; the path you pick depends on which. | — | **Catalog ID:** `grafana-investigator` — what the deploy API takes. #### Three ways to reach Grafana | Path | How it works | Offered for | | - | - | - | | The agent's built-in Grafana tooling | It talks to Grafana Cloud or your own Grafana over a stored Grafana connection. Komodor is never in the data path. | Either deployment | | Your own in-cluster server | The agent uses an MCP server you already run, which holds its own Grafana credentials. Nothing is stored in Komodor. | Self-hosted | | The MCP Gateway | The gateway is bound to a registered remote MCP server, such as Grafana Cloud's own endpoint. | Komodor cloud | > [!NOTE] > > The Grafana connection is listed as optional because the second path replaces it: a self-hosted > agent pointed at your own in-cluster server needs no connection here. Add one only to use the > agent's built-in Grafana tooling. The wizard asks where the agent runs and then disables the paths that target does not support, rather than hiding them — so a choice that is unavailable is visibly unavailable. ### As part of a workflow A specialist, not a lead. An orchestrator delegates the Grafana slice of an incident to it and synthesizes its findings with the other specialists'. Because it reads alerts as well as telemetry, it also works as the verification step *after* a remediation — asking whether the signal that fired has actually recovered. See [Orchestrator](/manage-your-agents/build/catalog/orchestrator) for how a step delivers work to it. ### Limitations * **It writes nothing to Grafana.** No silencing an alert, closing an incident, editing a dashboard or adding an annotation. * **It is bounded by retention, and by what its path can reach.** A self-hosted agent pointed at your own in-cluster server sees exactly what that server exposes, and it reports an unreachable data source rather than working around it. * **It sees only Grafana.** A cause in the cloud provider, the cluster, or the code is outside its reach — pair it with the relevant investigator. * **A partial report is possible.** Budget exhaustion returns what it has, so read the findings rather than assuming completeness. ### Recommended model `claude-sonnet-4-6` — the shipped default. Investigation is long-context, multi-step tool use where the work is retrieval and correlation rather than novel reasoning, so Sonnet's speed and cost suit it better than a larger model; a faster model degrades the correlation step, which is where the value is. ## Next steps > **[Agent catalog](/manage-your-agents/build/catalog/overview)** > > Every deployable agent, and what each one is for. > **[Cloud providers](/manage-your-agents/build/catalog/cloud-providers)** > > The AWS, Azure and Google Cloud investigators. > **[Integrations list](/manage-your-agents/build/built-in-integrations)** > > Connecting Datadog and Grafana. > **[Integration groups](/manage-your-agents/build/integration-groups)** > > The group-scoped gateway endpoint Datadog needs. # AWS Cost Analyzer Source: https://docs.komodor.com/manage-your-agents/build/catalog/aws-cost-analyzer Scans a connected AWS account for waste across six use cases and reports findings ranked by estimated monthly impact, each with its evidence. AWS Cost Analyzer scans a connected AWS account for waste across six use cases. Findings come back ranked by estimated monthly impact, each carrying the evidence behind the estimate and a concrete step to act on. ## Overview A cost review usually produces a long list nobody can rank. The instance that looks idle may be a warm standby; the tagging gap may be worth more than the idle instance. What is missing is not findings but the evidence to sort them by what fixing them is worth. Deploy it to get that ranking. You name the use cases you want run, with any scoping — regions, tag keys, thresholds — and it scans the connected account, sorting what it finds by what fixing it is worth. | Use case | What it finds | | - | - | | Idle resources | Provisioned capacity that is not doing work | | Service and tag cost trends | Where spend is moving, broken down by service or tag | | Tagging gaps | Spend that cannot be attributed because tags are missing | | Cross-AZ network cost | Traffic crossing availability zones, and what is causing it | | Reserved-instance coverage | Committed-use gaps against actual steady-state usage | | Savings toggles | Opt-in savings settings that are available but not switched on | **For:** whoever owns AWS spend in one account, on a schedule rather than during an incident. **Not for:** acting on what it finds — every finding is a recommendation for someone else to carry out. > [!NOTE] > > It runs only the use cases you name. A prompt that names none gets the list back rather than a full > scan — deliberately, because a blind scan of every use case across every region is the slowest > possible way to start. ### The preflight Every run, regardless of which use cases you chose, first checks whether the account's opt-in cost and data features are enabled — Cost Explorer, Cost and Usage Reports, VPC Flow Logs, Cost Anomaly Detection, Budgets and Compute Optimizer. This is not optional, and it is worth reading before the findings. Several use cases are blocked or degraded without those features, and a thin result is much more often a disabled feature than a clean account. ## Tools supported | Tool surface | What it can call | | - | - | | Cost Explorer | Spend, trends and the estimates behind each finding | | CloudWatch | The utilisation evidence that decides whether capacity is idle | | EC2, EBS, RDS, ELB, EKS | The provisioned resources themselves, and their configuration | | S3 | Storage footprint and its cost | | Tagging APIs | Attribution, and the gaps in it | Read-only. There is no mutating tool in its surface. ## Scenario examples In chat, or as a run's prompt: **Targeted analyses** ```text theme={null} Run the idle-resources and cross-AZ network analyses against eu-west-1 and us-east-1. Ignore anything under $50/month. Show me the evidence behind each estimate. ``` **Explain a jump** ```text theme={null} AWS spend rose about 18% month over month. Break down where the increase came from and which of it is recoverable. ``` **Before a commitment** ```text theme={null} We are about to buy a one-year Savings Plan. Tell me what our steady-state baseline actually is, and what we would be over-committing on. ``` ## Prerequisites | Requirement | Detail | Where to set it up | | - | - | - | | Integrations | An AWS connection, unless the pod supplies its own identity through IRSA | [Integrations list](/manage-your-agents/build/built-in-integrations) | | Permissions | Read-only across Cost Explorer, CloudWatch, S3, EC2, EBS, RDS, ELB, EKS and the tagging APIs. The account's opt-in cost features also have to be enabled — see the preflight above. | — | | Model access | The managed model gateway, or your own Claude API key | [Credentials & secrets](/security-and-governance/identity-and-access/credentials-secrets) | | Deployment | Komodor cloud or self-hosted. IRSA is self-hosted only. | — | **Catalog ID:** `aws-cost-analyzer` — what the deploy API takes. The AWS connection offers the same two authentication modes as the AWS investigator, and which you see depends on where the agent runs: | Mode | How it works | Offered for | | - | - | - | | Cross-account IAM role | Komodor assumes a read-only role in your account per run, from a CloudFormation stack you deploy. No long-lived keys are stored. | Komodor cloud | | IRSA | The pod inherits AWS access from its Kubernetes ServiceAccount's IAM role. | Self-hosted | ## As part of a workflow It is a scheduled analysis rather than an incident specialist: give it a trigger, point it at one account, and let the findings accumulate where someone reviews them. Where you run it across several accounts, deploy one per account so each run's findings carry an unambiguous owner. See [Triggers & schedules](/manage-your-agents/build/triggers-schedules) and [Cloud optimization](/modules-and-workflows/cost-optimization/cloud-optimization). ## Limitations * **It finds waste; it never acts on it.** There is no mutating tool in its surface, so it cannot stop an instance, delete a volume, change a tag or buy a reservation. Every finding is a recommendation for someone else to carry out. * **It is bounded by the preflight.** Where an opt-in feature is disabled, the analyses that depend on it are degraded or unavailable, and it reports that rather than estimating around the gap. * **It scans one account.** Run one instance per account so each finding carries an unambiguous owner. * **It runs only what you name.** A prompt naming no use cases returns the list, not a full scan. ## Recommended model `claude-sonnet-5` — the shipped default. The work is judgement over evidence someone else gathered, where a weaker model produces confident-sounding conclusions that do not hold. ## Next steps > **[Cloud optimization](/modules-and-workflows/cost-optimization/cloud-optimization)** > > The module that turns these findings into tracked work. > **[Triggers & schedules](/manage-your-agents/build/triggers-schedules)** > > Run the analysis on a schedule. > **[AWS Infrastructure Investigator](/manage-your-agents/build/catalog/cloud-providers)** > > The same account, read for incidents rather than waste. > **[Agent catalog](/manage-your-agents/build/catalog/overview)** > > Every catalog agent, side by side. # Cloud providers Source: https://docs.komodor.com/manage-your-agents/build/catalog/cloud-providers The read-only investigators for AWS, Azure and Google Cloud — what each reads, what to connect, what to ask it, and where each stops. Three catalog agents investigate incidents inside a cloud account: **AWS Infrastructure Investigator**, **Azure Investigator**, and **Google Cloud Investigator**. Each takes an incident description and the questions you want answered, reads that provider's own evidence, and returns structured findings that cite what they are based on. All three are strictly read-only. None has a mutating tool in any deployment mode, so none can stop a resource, change a policy, or roll anything back — turning a finding into an action is [Remediation Executor](/manage-your-agents/build/catalog/remediation-executor)'s job. ## AWS Infrastructure Investigator ### Overview An incident with no deploy behind it sends an engineer through CloudTrail by hand, looking for the change that explains it — and the change is rarely in the service that broke. It is in a policy, a security group, or a dependency two hops away. Hand it an incident description and the questions you want answered. It traces root causes through configuration changes, access-control layers and resource dependencies, then returns findings that cite the CloudTrail event, metric or policy behind each claim. Findings come back structured, so an orchestrator can synthesize them with other specialists' work rather than re-reading prose. Run it on its own and you get the same report in the run's output. **For:** SREs and on-call engineers investigating an incident inside one AWS account, especially when nothing deployed. **Not for:** acting on what it finds — that is [Remediation Executor](/manage-your-agents/build/catalog/remediation-executor)'s job — or for a cause that lives in a different account. ### Tools supported | Tool surface | What it can call | | - | - | | CloudTrail | Who changed what, and when — the change timeline behind an incident | | CloudWatch | Metrics and alarms around the incident window | | IAM | Policies, roles and trust relationships, when access is the suspect | | Resource APIs | The dependencies between resources, and their current configuration | Read-only. The investigator has no mutating tool, in any deployment. ### Scenario examples In chat, or as a run's prompt: **A change you cannot see** ```text theme={null} Checkout-api started returning 5xx at 14:10 UTC today. Nothing deployed in that window. Tell me what changed in AWS in the two hours before, and whether any of it touches checkout-api's dependencies. Cite the evidence. ``` **Access denied out of nowhere** ```text theme={null} orders-worker started getting AccessDenied on S3 at 09:30 UTC with no deploy. Work out which policy, role or trust relationship changed and when. ``` **Capacity or configuration** ```text theme={null} The prod RDS instance hit connection limits at 16:00 UTC. Tell me whether something changed in its configuration or a dependent service started opening more connections. ``` When a workflow drives it instead of a person, it takes a structured envelope: | Field | What it is | | - | - | | `incident_description` | What is happening, in a sentence or two | | `questions` | The specific questions to answer — the agent scopes its work to these | | `affected_services` | Service names, to narrow the search | | `investigation_start` | An ISO-8601 timestamp; it looks no further back. Defaults to 24 hours ago | | `context` | Enrichment already gathered — alerts, errors, deploys | | `budget_seconds` | A shorter or longer wall-clock budget for this run | ### Prerequisites | Requirement | Detail | Where to set it up | | - | - | - | | Integrations | An AWS connection, unless the pod supplies its own identity through IRSA | [Built-in integrations](/manage-your-agents/build/built-in-integrations) | | Permissions | Read-only across CloudTrail, CloudWatch, IAM and the resource APIs, scoped to one account. | — | | Model access | The managed model gateway, or your own Claude API key | [Credentials & secrets](/security-and-governance/identity-and-access/credentials-secrets) | | Deployment | Komodor cloud or self-hosted. IRSA is self-hosted only. | — | **Catalog ID:** `aws-investigator` — what the deploy API takes. The AWS connection offers two authentication modes, and which ones you see depends on where the agent runs: | Mode | How it works | Offered for | | - | - | - | | Cross-account IAM role | Komodor assumes a read-only role in your account per run, from a CloudFormation stack you deploy. No long-lived keys are stored. You supply the 12-digit account ID. | Komodor cloud | | IRSA | The pod inherits AWS access from its Kubernetes ServiceAccount's IAM role, so there is no connection to bind. You supply the role ARN. | Self-hosted | > [!NOTE] > > The AWS connection is listed as optional because IRSA replaces it. If you deploy self-hosted with a > ServiceAccount that already carries the role, the agent needs no connection at all. > [!WARNING] > > A connection is validated as configuration, not as working AWS access. A well-formed role ARN with > insufficient permissions deploys cleanly and fails on the first run, so read that run rather than > treating a successful deploy as proof. ### As part of a workflow A specialist, not a lead. In an incident workflow it takes a scoped sub-task from an orchestrator, investigates the AWS slice of the problem, and hands its findings back to be synthesized with the other specialists' work. Add a second investigator for another system when an incident spans both — they run independently and their findings are combined. See [Orchestrator](/manage-your-agents/build/catalog/orchestrator) for how a step delivers work to it, and [Incidents](/modules-and-workflows/sre/incidents) for the module that drives it. ### Limitations * **It changes nothing in AWS.** There is no mutating tool in its surface, in either deployment mode, so it cannot stop an instance, edit a policy or roll back a change — it can only tell you which one it thinks is responsible. * **It sees only what its credentials reach.** An investigation scoped to one account cannot observe a cause that lives in another, and it reports the gap rather than inferring across it. * **A clean deploy is not proof of access.** A well-formed role ARN with insufficient permissions fails on the first run, not at deploy time. * **A partial report is possible.** Budget exhaustion returns what it has, so read the findings rather than assuming completeness. ### Recommended model `claude-sonnet-4-6` — the shipped default. Investigation is long-context, multi-step tool use where the work is retrieval and correlation rather than novel reasoning, so Sonnet's speed and cost suit it better than a larger model; a faster model degrades the correlation step, which is where the value is. ## Azure Investigator ### Overview Azure spreads the evidence for one incident across Resource Graph, Monitor, Log Analytics and the cluster's own metadata, each with its own query surface. Assembling that by hand is the slow part of an Azure incident. Hand it an incident description and the questions you want answered. It locates the affected resources, reads their metrics and logs, and correlates what it finds into structured findings — each one carrying a portal deeplink, so the evidence can be opened rather than taken on trust. **For:** SREs and on-call engineers running workloads in an Azure subscription, including AKS. **Not for:** acting on what it finds, or for a cause outside the subscription the connection reaches. ### Tools supported | Tool surface | What it can call | | - | - | | Resource Graph | Finding the affected resources and their relationships | | Monitor | Metrics across the incident window | | Log Analytics | Log queries, in KQL | | AKS | Cluster metadata, when the workload is on Kubernetes | | Cost Management | Spend signals, when the symptom is financial rather than operational | Read-only. The five capabilities above were measured against a plain Reader role, not assumed from documentation. ### Scenario examples In chat, or as a run's prompt: **Readiness failure** ```text theme={null} The orders-api AKS workload in subscription prod-weu started failing readiness at 08:30 UTC. Check what changed on the cluster and its dependencies, read the Log Analytics errors for that window, and tell me what is responsible. ``` **Cost signal, not an outage** ```text theme={null} Spend in subscription prod-weu jumped about 40% week over week with no release. Find what is responsible. ``` **Where did the logs go** ```text theme={null} We stopped seeing logs from the billing container in prod-weu after 21:00 UTC. Is the workload down, or is it the log pipeline? ``` When a workflow drives it instead of a person, it takes the same structured envelope the other investigators take — `incident_description`, `questions`, `affected_services`, `investigation_start`, `context` and `budget_seconds`. ### Prerequisites | Requirement | Detail | Where to set it up | | - | - | - | | Integrations | An Azure connection with read-only access across the five capabilities above | [Built-in integrations](/manage-your-agents/build/built-in-integrations) | | Permissions | A plain **Reader** role is enough — the five capabilities were measured against it. | — | | Model access | The managed model gateway, or your own Claude API key | [Credentials & secrets](/security-and-governance/identity-and-access/credentials-secrets) | | Deployment | Komodor cloud or self-hosted. | — | **Catalog ID:** `azure-investigator` — what the deploy API takes. > [!NOTE] > > Unlike the Datadog and Grafana investigators, this agent has no MCP Gateway path. Azure publishes no > hosted MCP endpoint, so there is no remote server for the gateway to be bound to and no group to > pick — the agent's Azure tooling runs alongside it, configured from the connection you select. ### As part of a workflow A specialist, not a lead. An orchestrator delegates the Azure slice of an incident to it and synthesizes its findings with other specialists' work. Where a workload spans Azure and Kubernetes, pairing it with a cluster investigator covers both halves. See [Orchestrator](/manage-your-agents/build/catalog/orchestrator) for how a step delivers work to it. ### Limitations * **It changes nothing in Azure.** It has no mutating tool, so it cannot restart a resource, edit a configuration or adjust a scaling rule. * **It sees one subscription's worth of evidence**, bounded by what the connection's role permits. Where a capability is not granted, it reports the gap instead of inferring around it. * **There is no MCP Gateway path.** Azure publishes no hosted MCP endpoint, so the gateway is not an option for reaching it. * **A partial report is possible.** Budget exhaustion returns what it has, so read the findings rather than assuming completeness. ### Recommended model `claude-sonnet-4-6` — the shipped default. Investigation is long-context, multi-step tool use where the work is retrieval and correlation rather than novel reasoning, so Sonnet's speed and cost suit it better than a larger model; a faster model degrades the correlation step, which is where the value is. ## Google Cloud Investigator ### Overview Free-text log search is the wrong instrument for a Google Cloud incident: it finds occurrences, not the first one. The useful question is when the failure actually started and what changed immediately before it, which means narrowing structurally rather than by keyword. Hand it an incident description and the questions you want answered. It narrows the log stream by resource, log name and severity, pins the earliest failing entry, and correlates it against the audit log to find the change that preceded it. Findings cite the exact filter and time window behind each claim, so a conclusion can be reproduced by running the same query yourself. **For:** establishing when a failure began in one Google Cloud project, and what changed just before it. **Not for:** metrics, traces, resource configuration or billing — none of those are in its reach. ### Tools supported | Tool surface | What it can call | | - | - | | Cloud Logging | The failing entries, narrowed by resource, log name and severity | | Cloud audit logs | The change that preceded the first failure | Read-only, and one source. There is nothing else in its surface. ### Scenario examples In chat, or as a run's prompt: **Earliest failure** ```text theme={null} Something started failing in project acme-prod around 03:40 UTC. Find the earliest failing log entry, tell me which resource it belongs to, and check the audit log for what changed just before it. Give me the filters you used. ``` **What preceded it** ```text theme={null} Cloud Run service checkout started throwing 503s around 11:00 UTC in acme-prod. Find the first failing entry and whatever changed in the audit log just before it. ``` **A quiet failure** ```text theme={null} Nothing is alerting but the nightly export job in acme-prod produced no output. Find where it stopped and what the last successful step was. ``` When a workflow drives it instead of a person, it takes the same structured envelope the other investigators take — `incident_description`, `questions`, `affected_services`, `investigation_start`, `context` and `budget_seconds`. ### Prerequisites | Requirement | Detail | Where to set it up | | - | - | - | | Integrations | A Google Cloud connection, plus an MCP group — the agent reaches Cloud Logging through a group-scoped gateway endpoint | [Built-in integrations](/manage-your-agents/build/built-in-integrations) · [Integration groups](/manage-your-agents/build/integration-groups) | | Permissions | Read-only Cloud Logging for the project the service-account key belongs to. | — | | Model access | The managed model gateway, or your own Claude API key | [Credentials & secrets](/security-and-governance/identity-and-access/credentials-secrets) | | Deployment | Komodor cloud or self-hosted. | — | **Catalog ID:** `gcp-investigator` — what the deploy API takes. > [!NOTE] > > The connection is required rather than optional, because the gateway is this agent's only path. > There is no alternative tooling bundled alongside it, so without a connection it has no tools at > all. ### As part of a workflow A specialist with a narrow surface, which makes it a good first step rather than a sole one: it establishes when the failure started and what changed around it, and an orchestrator takes that timestamp to the specialists that hold the rest of the picture. See [Orchestrator](/manage-your-agents/build/catalog/orchestrator) for how a step delivers work to it. ### Limitations * **It writes nothing to Google Cloud**, and it reads nothing outside Cloud Logging. Metrics, traces, resource configuration and billing are all outside its reach — where a cause lives in one of those, it establishes the timeline and says plainly that the rest is not something it can see. * **It is scoped to a single project** — the one its service-account key belongs to. * **Without a connection it has no tools at all**, because the group-scoped gateway is its only path. * **A partial report is possible.** Budget exhaustion returns what it has, so read the findings rather than assuming completeness. ### Recommended model `claude-sonnet-4-6` — the shipped default. Investigation is long-context, multi-step tool use where the work is retrieval and correlation rather than novel reasoning, so Sonnet's speed and cost suit it better than a larger model; a faster model degrades the correlation step, which is where the value is. ## Next steps > **[Agent catalog](/manage-your-agents/build/catalog/overview)** > > Every deployable agent, and what each one is for. > **[APM](/manage-your-agents/build/catalog/apm)** > > The Datadog and Grafana investigators. > **[Integrations list](/manage-your-agents/build/built-in-integrations)** > > Connecting AWS, Azure and Google Cloud. > **[Remediation Executor](/manage-your-agents/build/catalog/remediation-executor)** > > Turning a finding into an action, with approval. # Code Review Source: https://docs.komodor.com/manage-your-agents/build/catalog/code-review Code Reviewer reviews pull requests. GitHub Agent locates a problem in a repository and, once a person approves, opens a pull request for a configuration fix. Two catalog agents work against your repositories: **Code Reviewer**, which reviews pull requests as they change, and **GitHub Agent**, which finds where a characterised problem lives in the code. Code Reviewer writes only its review comment. GitHub Agent writes only a pull request for a Kubernetes configuration value, and only after a person approves it. ## Code Reviewer ### Overview Review latency is where small pull requests go to wait. Most of what a maintainer catches on a first pass is mechanical — a correctness slip, an unhandled path, a name that misleads — and finding it costs a context switch that the author then waits on. On each push to a pull request, this agent reads the diff and reports what a maintainer needs to know before merging. Findings come back in three buckets, and each one names a file, a line, and the concrete consequence rather than a style preference: | Bucket | What belongs in it | | - | - | | Blocking | Something that should stop the merge | | Should-fix | Worth fixing, but not a reason to hold the change | | Nit | Minor, and explicitly optional | It comments once per commit rather than amending a single review, so the history of what was flagged and when stays readable on the pull request itself. **For:** teams who want a consistent first pass on every pull request before a human reads it. **Not for:** replacing the human review, or satisfying a branch-protection rule — it never casts a GitHub approval. #### The verdict The worst finding in the review decides the verdict, and the agent does not get to choose it — the buckets are what it produces, and the verdict follows from them: | Worst finding | Verdict | The comment reads | | - | - | - | | A Blocking finding | Changes requested | **Code Reviewer — Changes requested** | | A Should-fix finding | Comment | **Code Reviewer — Comments** | | Only Nits, or nothing | Approve | **Code Reviewer — Looks good** | Finding nothing is a valid and common result. The review says so rather than padding itself with observations to look thorough. ### Tools supported | Tool surface | What it can call | | - | - | | The pull request diff | The whole basis of the review | | Repository metadata | Locating the file and line a finding refers to | Read access, plus the one review comment it posts. It reviews against its own rubric: there is no policy file to write and no repository convention to configure — which is the trade, consistent reviews everywhere rather than reviews tuned per repository. ### Scenario examples **This is the one catalog agent you do not prompt, and do not give a schedule.** It is driven by the repositories bound to the review workflow, so a pull request arriving is what starts a run — there is no cron trigger to set in the wizard, and the **Triggers** step has nothing to collect. What a run produces is the review comment. With one Blocking and one Nit finding: ```markdown theme={null} #### Code Reviewer — Changes requested Adds retry handling to the payment webhook consumer. One correctness issue in the retry path, and a naming nit. **Blocking** - `consumers/webhook.py:88` Retry loop re-sends before the idempotency key is refreshed — a duplicate charge on the second attempt. **Nit** - `consumers/webhook.py:12` `n` reads better as `attempt`. ``` With nothing to report, the body is the summary followed by `No blocking or should-fix findings.` ### Prerequisites | Requirement | Detail | Where to set it up | | - | - | - | | Integrations | A GitHub connection with read access to the pull request, and permission to post the review comment | [Built-in integrations](/manage-your-agents/build/built-in-integrations) | | Permissions | The connection gates the only path this agent has to GitHub. Without one it cannot read the pull request at all: it fails closed and returns a verdict saying so, rather than reviewing blind. | — | | Model access | The managed model gateway, or your own Claude API key | [Credentials & secrets](/security-and-governance/identity-and-access/credentials-secrets) | | Deployment | Komodor cloud or self-hosted. | — | **Catalog ID:** `code-reviewer` — what the deploy API takes. > [!NOTE] > > This agent is enabled per account. If it is not in your catalog, an account administrator can > request it for your account. ### As part of a workflow It is the review step itself rather than a specialist another agent delegates to. A pull request opens or is pushed to, the step runs, and the comment lands on the pull request — the workflow's output is visible in GitHub rather than only in the console. ### Limitations * **It never casts a GitHub approval.** Its verdict is reported in its own comment — "Looks good" is a verdict, not an approving review — so it cannot satisfy a branch-protection rule that requires one, and it never merges. A maintainer still reviews and merges. * **Posting that comment is the only thing it writes.** It does not push, branch, or change a file. * **A review the clock stopped never comes back as approve.** Where the time budget cut a review short, an otherwise-empty result is downgraded to Comment and says plainly that it is incomplete — so "no findings" from a partial review cannot be mistaken for "nothing wrong". * **It reviews only what the diff shows.** A change that is wrong because of something in a file it did not touch is outside what the review can see — the review is a filter in front of a human, not a replacement for one. ### Recommended model `claude-sonnet-5` — the shipped default. The work is judgement over evidence someone else gathered, where a weaker model produces confident-sounding conclusions that do not hold. ## GitHub Agent ### Overview An investigation that ends at "the retry path double-charges" has answered the incident but not the fix. Someone still has to find the code — and a grep against a monorepo returns the string in forty places, ranked by nothing that matters. Give it a characterised problem — an incident, an error message, a failing service. It searches code for the error's own text, opens every candidate to confirm it rather than trusting the search ranking, and checks what changed there just before the problem started. What comes back is a ranked shortlist of files with the evidence for each, and the durable code change that shortlist points to. When the fix is a Kubernetes configuration value kept in git — a resource limit, an image tag, an environment variable, a replica count, a probe, a label or annotation, an HPA or ConfigMap value — it can also open the pull request that changes it. It opens one, on its own branch, with an edit limited to that value, and only after a person approves it. It never merges: nothing reaches the cluster until someone merges the pull request and the change is applied. **For:** turning an investigation's conclusion into the file and the change that explains it, and, for a Kubernetes configuration value, into a pull request a person can review. **Not for:** a vague problem statement, and not for deciding what to do during the incident itself. > [!NOTE] > > It runs on concrete evidence. Given a vague problem statement with no error text, no service and no > timeframe, there is nothing distinctive to search for — so pass it the output of an investigation > rather than the original alert. ### Tools supported | Tool surface | What it can call | | - | - | | Repository code | Searching for the error's strings, and opening candidates to confirm them | | File contents | Reading the surrounding implementation before recommending a change | | Recent history | What changed in a candidate file just before the problem appeared | | Pull requests | Opening one pull request for a Kubernetes configuration value, after a person approves it | Reading needs read access. Opening a pull request also needs write access to contents and pull requests on the GitHub connection, and an approval. ### Scenario examples In chat, or as a run's prompt: **Locating a failure in code** ```text theme={null} Our checkout service is throwing "connection pool exhausted while acquiring lease" under load since the 14:00 UTC deploy. Find where that message comes from, what governs the pool size, and what changed there recently. ``` **From an investigation's conclusion** ```text theme={null} An investigation concluded the retry path in the webhook consumer double-charges on the second attempt. Find the code that does it and what changed there recently. ``` **Where is this configured** ```text theme={null} Something is capping our outbound HTTP connections at 100 and I cannot find where. Search for the setting and show me what governs it. ``` The more of the investigation you pass through — the exact error text, the service, the window — the narrower its search, and the better the shortlist. ### Opening a pull request A workflow or another agent asks for a pull request; a prompt sent straight to this agent does not. The request names the change — the resource, the field, the current value and the new one — and carries the investigation behind it. An orchestrator sends it only when the cause is established, the investigation is confident and cites its evidence, and the agent has located the file that defines the value. **1. It checks the request** It opens nothing, and says why, when the change is a Secret, an RBAC object, a resource quota or a limit range, when it would lower a resource limit, or when the investigation is not confident or cites no evidence. **2. It prepares the edit** It reads the file from the base branch and replaces exactly the text that holds the current value. An edit that changes anything but the declared value, or that touches a file defining a Secret or an RBAC object, is refused. **3. A person approves** The call that opens the pull request is held until a person who can approve it answers in the **Actions** inbox, with the edit in front of them. If nobody answers within the agent's wait — an hour by default — no pull request is opened. **4. It opens the pull request** On its own branch, against the base branch as it stands when the approval arrives. The description carries the reason, the runs that support it and the confidence in words. Asking again for the same change finds the open pull request instead of opening a second one. What comes back says which of these happened: the pull request's link, an existing one, the reasons it declined, or — when it was prepared but not approved — the edit itself, so a person can apply it by hand. The person who deploys the agent from the catalog is its approver. Add others under Fleet → the agent → Permissions, in **Who can approve or deny**. > [!NOTE] > > Approvals for tool calls are enabled per account. Where they are off, the agent refuses to open a > pull request rather than opening one nobody approved. ### Prerequisites | Requirement | Detail | Where to set it up | | - | - | - | | Integrations | A GitHub connection with read access to the repositories it searches. To open pull requests, the connection also needs write access to contents and pull requests. | [Built-in integrations](/manage-your-agents/build/built-in-integrations) | | Permissions | The connection gates the only path this agent has to GitHub. Without one it cannot search anything at all — it fails closed and says so, rather than returning an empty result that reads like "nothing found". The agent is deployed with a tool permission that asks a person before it opens a pull request. | — | | Model access | The managed model gateway, or your own Claude API key | [Credentials & secrets](/security-and-governance/identity-and-access/credentials-secrets) | | Deployment | Komodor cloud or self-hosted. | — | **Catalog ID:** `github-file-finder` — what the deploy API takes. > [!NOTE] > > This agent is enabled per account. If it is not in your catalog, an account administrator can > request it for your account. ### As part of a workflow It sits downstream of an investigation, not in place of one. A workflow that has localised a failure to a service passes the findings here to turn "this is broken" into "this file is why" — and its shortlist is what a remediation step then acts on. An orchestrator asks it for a pull request as the last step, once an investigation has named a Kubernetes configuration value as the cause and the agent has located the file. A pull request is a proposed fix, not an applied one: the workflow reports it beside what to do now, and does not treat the incident as resolved because one was opened. See [Orchestrator](/manage-your-agents/build/catalog/orchestrator) for how a step delivers work to it. ### Limitations * **It opens a pull request only for a Kubernetes configuration value, and only after a person approves it.** It never merges, never pushes to the base branch, and never opens more than one pull request for a change. Everything else it reads and recommends. * **It never edits a Secret, an RBAC object, a resource quota, a limit range, application code or Terraform, and it never lowers a resource limit.** For a fix that is not a Kubernetes configuration value, [Remediation Executor](/manage-your-agents/build/catalog/remediation-executor) can open a pull request against the source. * **A pull request changes the declared value and nothing else.** It touches three files at most and changes at most 40 lines, so the person approving reads the whole change. * **It does not tell you what to do right now.** Its recommendation is the durable code change that stops a problem recurring, which is a different question from how to restore service during an incident. * **A vague problem gives it nothing to search for.** Without error text, a service or a window, there is no distinctive string to anchor on. * **It sees only what the connection reaches.** Repositories outside it are outside the shortlist. ### Recommended model `claude-sonnet-5` — the shipped default. The work is judgement over evidence someone else gathered, where a weaker model produces confident-sounding conclusions that do not hold. ## Next steps > **[Agent catalog](/manage-your-agents/build/catalog/overview)** > > Every deployable agent, and what each one is for. > **[Integrations list](/manage-your-agents/build/built-in-integrations)** > > Connecting GitHub. > **[Remediation Executor](/manage-your-agents/build/catalog/remediation-executor)** > > Turning a recommendation into a pull request. > **[Approvals](/security-and-governance/controls/approvals)** > > The decision boundary before anything is written. # Kubernetes Source: https://docs.komodor.com/manage-your-agents/build/catalog/kubernetes Three catalog agents for Kubernetes — one resource, one cluster with telemetry and GPU signals, or a question across every cluster Komodor covers. Three catalog agents investigate Kubernetes, and which to deploy depends on how much you want it to see: **Kubernetes RCA** for one resource in one cluster, **Cluster Investigator** for one cluster correlated against Grafana and GPU health, and **Klaudia Investigator** for a question spanning every cluster your Komodor account covers. The first two run inside a cluster and see only that cluster. Deploying one instance and expecting it to answer for a fleet is the mistake worth avoiding before you install rather than after. ## Kubernetes RCA ### Overview Reading events off a broken Deployment tells you it is broken. It does not tell you that the Kafka consumer group is stuck, or that the Istio sidecar never became ready — that takes knowing how the specific thing fails, which is knowledge most on-call engineers hold for a handful of workloads and nobody holds for all of them. Name a resource — a kind and a name — and it investigates that resource: reconstructing what changed around it, reading the state of its dependencies, and working through a domain-specific method for whatever it is. It is built from Klaudia's root-cause knowledge, which is why it can diagnose a Kafka consumer group or a stuck Istio sidecar rather than only reading events. It returns a structured root-cause analysis for that one resource. **For:** root-causing one named resource, in the one cluster the agent runs in. **Not for:** a fleet. One instance does not answer for more than its own cluster — use [Klaudia Investigator](#klaudia-investigator) for that. > [!WARNING] > > This agent investigates **only the cluster it is deployed in**. One instance does not answer for a > fleet, and that is the mistake worth avoiding before you install rather than after. For > multi-cluster work, use [Klaudia Investigator](#klaudia-investigator) instead. > > The wizard states this on its first step and holds Next until you confirm it, and it recommends > putting the cluster name in the agent's name for exactly this reason. ### Tools supported Read-only `kubectl` against the cluster it runs in. There is no connection to bind, because the ServiceAccount's own permissions are simultaneously its only data source and its only security boundary. | Tool surface | What it can call | | - | - | | Namespaced resources | The resource under investigation and everything it depends on | | Cluster-scoped reads | Nodes, persistent volumes, storage classes, API services and metrics | | Events and change history | Reconstructing what changed before the failure | | In-pod diagnostics | State that exists nowhere else, for a short list of workload types | ### Scenario examples This agent takes a resource, not a question. A run's input names what to investigate: | Field | What it is | | - | - | | `kind` | The resource kind — `Deployment`, `StatefulSet`, `Pod`, `Job` | | `name` | The resource name | | `namespace` | Its namespace; omit for cluster-scoped kinds | | `budget_seconds` | A shorter or longer wall-clock budget for this run | **A stuck StatefulSet** ```json theme={null} { "kind": "StatefulSet", "name": "kafka-broker", "namespace": "streaming" } ``` **A deployment that will not roll out** ```json theme={null} { "kind": "Deployment", "name": "checkout-api", "namespace": "prod" } ``` **A job that failed overnight** ```json theme={null} { "kind": "Job", "name": "nightly-reconcile", "namespace": "billing", "budget_seconds": 900 } ``` ### Prerequisites | Requirement | Detail | Where to set it up | | - | - | - | | Integrations | None. There is no connection to bind — the ServiceAccount is both the data source and the boundary. | — | | Permissions | Cluster RBAC, granted by the rendered install; see below | — | | Model access | The managed model gateway, or your own Claude API key | [Credentials & secrets](/security-and-governance/identity-and-access/credentials-secrets) | | Deployment | Self-hosted only — there is no Komodor cloud option, and one instance per cluster | [Deploy an agent](/manage-your-agents/build/how-it-works) | **Catalog ID:** `kubernetes-rca` — what the deploy API takes. The self-hosted install renders a values file that binds the agent's ServiceAccount to Kubernetes' own aggregated `view` role, plus a supplemental role for the cluster-scoped reads `view` omits. Leaning on the built-in role means custom resources are covered as they are installed, rather than by a list that goes stale. #### The one grant that is not read-only `pods/exec` is granted cluster-wide, because several domain methods diagnose from inside the pod — that state exists nowhere else, and at install time nobody knows which namespace an incident will land in. It is a separate value from the read access, so you can narrow or remove it: | Value | Effect | | - | - | | `rbac.exec.namespaces` | Restrict in-pod diagnostics to named namespaces | | Removing the exec block | Those methods fall back to resource state; the rest are unaffected | What may actually run through that grant is decided by the agent itself, not only by RBAC. It admits a short list of read-only diagnostics and refuses everything else, interactive sessions included. No database client is on that list, so a Postgres incident that turns on a grant or a row-security policy is out of reach — and the agent names that as a blind spot rather than approximating. > [!NOTE] > > The `view` role excludes Secrets, so Helm release history is unavailable and the change-timeline > method falls back to its other sources. Two grants are deliberately off by default and need an > edit to the values file: reads on custom resource groups that do not aggregate to `view`, and > cluster-wide Secret reads. ### As part of a workflow A specialist bound to one cluster, so a workflow step that targets it has already decided which cluster the problem is in. Where a fleet-wide investigation is the starting point, an orchestrator reaches for a multi-cluster investigator first and delegates here once the cluster is known. See [Orchestrator](/manage-your-agents/build/catalog/orchestrator) for how a step delivers work to it. ### Limitations * **It changes nothing.** Its read access is the whole of its reach, and the one non-read grant is confined to a fixed list of diagnostics by the agent's own classifier. * **It sees exactly one cluster and no external context.** No Komodor backend, no metrics vendor and no cloud provider API — a cause in any of those is outside what it can observe, and it says so rather than inferring. * **Some evidence is off by default.** The `view` role excludes Secrets, so Helm release history is unavailable unless you edit the values file. * **No database client is admitted through exec**, so a Postgres incident that turns on a grant or a row-security policy is out of reach — named as a blind spot rather than approximated. ### Recommended model `claude-sonnet-5` — the shipped default. The work is judgement over evidence someone else gathered, where a weaker model produces confident-sounding conclusions that do not hold. ## Cluster Investigator ### Overview A pod restarting because of a GPU fault looks like an ordinary crash loop until something reads both the cluster and the hardware. The same is true of a latency regression that is visible in traces but explained by a node: the answer sits between two consoles, and nobody correlates them under pressure. Describe what is wrong and it investigates across all three signal sources at once: reading cluster state through read-only `kubectl`, pulling the metrics, logs and traces around the incident from Grafana, and checking GPU health where the workload depends on it. It reconstructs the change timeline and returns a structured root-cause analysis. **For:** one cluster where the cause could be the workload, the telemetry or the GPU — especially ML serving. **Not for:** a fleet. Run one per cluster, and name each after the cluster it watches. ### Tools supported | Tool surface | What it can call | | - | - | | Kubernetes | Cluster state, dependencies, events, and the change timeline | | Prometheus, Loki, Tempo | Metrics, logs and traces around the incident window | | GPU health signals | XID and SXid errors, DCGM metrics, and inference-engine crash patterns | Read-only, apart from the in-pod diagnostics noted under Prerequisites. It carries the same domain knowledge as [Kubernetes RCA](#kubernetes-rca) — twenty-three workload-specific methods — plus five GPU-specific ones. ### Scenario examples In chat, or as a run's prompt: **Restart loop with GPU in play** ```text theme={null} The inference-gateway deployment in namespace ml-serving has been restarting since 06:00 UTC. Check the pods, the GPU health on the nodes they land on, and the Grafana metrics for that window. Tell me whether this is the workload or the hardware. ``` **Slow, not failing** ```text theme={null} The recommendations deployment in ml-serving is healthy but its p99 doubled after the Tuesday rollout. Correlate the pod metrics against the Grafana traces and tell me where the time went. ``` **Node-level suspicion** ```text theme={null} Three pods across ml-serving restarted within a minute of each other at 04:12 UTC. Was that the workload, the node, or the GPU? ``` ### Prerequisites | Requirement | Detail | Where to set it up | | - | - | - | | Integrations | A Grafana connection, optional — see below | [Built-in integrations](/manage-your-agents/build/built-in-integrations) | | Permissions | Cluster RBAC, granted by the rendered install, including in-pod diagnostics | — | | Model access | The managed model gateway, or your own Claude API key | [Credentials & secrets](/security-and-governance/identity-and-access/credentials-secrets) | | Deployment | Self-hosted only, one instance per cluster | [Deploy an agent](/manage-your-agents/build/how-it-works) | **Catalog ID:** `cluster-investigator` — what the deploy API takes. > [!NOTE] > > The Grafana connection is optional because the agent can instead reach your own in-cluster MCP > server, or run its own bundled Grafana tooling. Add a connection only to use the built-in path. > Without any of the three, it still investigates the cluster — it simply has no telemetry to > correlate against. The install grants the same cluster access as Kubernetes RCA: Kubernetes' aggregated `view` role, a supplemental role for the cluster-scoped reads it omits, and `pods/exec` for the workload methods that diagnose from inside the pod. Exec is a separate value, so you can narrow it to named namespaces or remove it and let those methods fall back to resource state. ### As part of a workflow A specialist bound to one cluster, so a step targeting it has already established which cluster the problem is in. Because it covers three signal sources, it often replaces two steps that would otherwise run in parallel and need synthesizing afterwards. See [Orchestrator](/manage-your-agents/build/catalog/orchestrator) for how a step delivers work to it. ### Limitations * **It changes nothing** in the cluster or in Grafana. Its non-read grant is confined to a fixed list of read-only in-pod diagnostics, and everything else is refused. * **It sees exactly one cluster.** Deploying one instance and expecting it to answer for a fleet is the mistake worth avoiding — run one per cluster, and name each after the cluster it watches. * **Without a Grafana path it still investigates**, but it has no telemetry to correlate the cluster state against. * **A partial report is possible.** Budget exhaustion returns what it has, so read the findings rather than assuming completeness. ### Recommended model `claude-sonnet-4-6` — the shipped default. Investigation is long-context, multi-step tool use where the work is retrieval and correlation rather than novel reasoning, so Sonnet's speed and cost suit it better than a larger model. ## Klaudia Investigator ### Overview The in-cluster investigators each answer for one cluster, which is the wrong shape for a question that starts "which of our clusters…". Answering that by deploying an agent everywhere and collating the results is work nobody wants to do during an incident. Ask a Kubernetes question or describe an incident. The agent puts it to Klaudia, who investigates across the clusters your Komodor account already covers, and relays the answer back — as prose for a question, or as a structured root-cause analysis when that is what she produces. **For:** questions that span clusters, or an incident whose cluster is not yet known. **Not for:** answering an approval or a choice on your behalf, and not a substitute for an investigator where you already know the cluster. ### Tools supported It reads nothing directly. Its entire surface is a small, fixed set of tools for holding a conversation with Klaudia — asking a question, collecting the result, and nothing else. The investigation happens on Komodor's side, and its reach is whatever your Komodor account's agent already sees. Because that surface is fixed by the agent itself, the wizard's tool checklist has no effect on it: what it can do is decided by the implementation, not by what you tick. ### Scenario examples In chat, or as a run's prompt: **One workload, one cluster** ```text theme={null} The payments deployment in the prod-eu cluster has been unhealthy since 12:15 UTC. What happened, and what changed before it? ``` **Across the fleet** ```text theme={null} Which clusters are running a version of ingress-nginx with the known config-reload issue, and which of them have actually hit it? ``` **A question, not an incident** ```text theme={null} What changed in the prod-eu cluster in the last 24 hours that touched anything in the payments namespace? ``` ### Prerequisites | Requirement | Detail | Where to set it up | | - | - | - | | Integrations | An MCP group with a Komodor server — the agent reaches Klaudia through a group-scoped gateway endpoint | [Integration groups](/manage-your-agents/build/integration-groups) | | Permissions | Whatever your Komodor account's agent already covers; there is nothing extra to grant | — | | Model access | The managed model gateway, or your own Claude API key | [Credentials & secrets](/security-and-governance/identity-and-access/credentials-secrets) | | Deployment | Komodor cloud or self-hosted. | — | **Catalog ID:** `klaudia-investigator` — what the deploy API takes. > [!NOTE] > > There is no Komodor connection to create. The Komodor MCP server authenticates itself on the gateway > server object, configured once in gateway administration, so the Integrations step has nothing to > collect for this agent — only the MCP group needs binding. ### As part of a workflow The multi-cluster counterpart to the in-cluster investigators: where a step needs a Kubernetes answer but the cluster is not known in advance, this is the specialist to route to. An orchestrator can also use it as a first pass, then delegate to a cluster-bound investigator once the affected cluster is identified. See [Orchestrator](/manage-your-agents/build/catalog/orchestrator) for how a step delivers work to it. ### Limitations * **It cannot answer an approval or a choice on your behalf.** That is enforced in the agent rather than by configuration, so a gate Klaudia raises always comes back to a human. * **It performs no investigation of its own.** Everything it reports comes from Klaudia, so what it can see is exactly what your Komodor account covers — no more, and no less. * **The wizard's tool checklist does not apply.** Its surface is fixed by the implementation. ### Recommended model `claude-sonnet-5` — the shipped default. The work is judgement over evidence someone else gathered, where a weaker model produces confident-sounding conclusions that do not hold. ## Next steps > **[Agent catalog](/manage-your-agents/build/catalog/overview)** > > Every deployable agent, and what each one is for. > **[APM](/manage-your-agents/build/catalog/apm)** > > The Datadog and Grafana investigators. > **[Integration groups](/manage-your-agents/build/integration-groups)** > > The gateway endpoint Klaudia Investigator needs. > **[Kubernetes Cost](/modules-and-workflows/cost-optimization/k8s-cost)** > > The module that owns Kubernetes spend rather than incidents. # Orchestrator Source: https://docs.komodor.com/manage-your-agents/build/catalog/orchestrator The catalog agent that routes work to other agents — running a declarative workflow over your fleet, or executing one orchestrated step of a module workflow. The **KAOP Workflow Orchestrator** is the one catalog agent that does no specialist work itself. It decides which agents should, delegates to them over the control plane, and combines what comes back into a single answer. ## Overview An incident that spans the cluster, the cloud account and a deploy needs three specialists, and someone to decide which three. Run them by hand and you are the router: choosing who to ask, waiting, and reconciling three answers that do not agree. It runs a declarative workflow over your fleet, and it is also the Komodor Agentic Operation Platform (KAOP) account's executor for a module workflow's `orchestrated` steps. It delegates over agent-to-agent invocation through the control plane, so every specialist call is authorized and recorded like any other run rather than becoming an unobserved side channel. **For:** questions that need more than one specialist, and for the module workflows whose steps dispatch to it. **Not for:** doing the investigation itself — it has no specialist surface of its own. ### Running a workflow A workflow is steps that fan out to several specialists at once, run in order, or loop until the work is done. | Behaviour | What it means | | - | - | | **Named or chosen** | A step can name the specialist to call, or hand the choice to a lead agent that picks from a roster you define. | | **Failure is contained** | A specialist that fails or times out does not fail the workflow — its step records the failure and the rest carries on. | | **Roster without a workflow** | Given only a roster and no workflow, it loops, deciding for itself who to call, until it judges the goal met. | The contained-failure behaviour is the one to design around: a workflow finishing does not mean every step succeeded, so read the steps rather than the outcome. ### Executing an orchestrated step A module workflow's steps come in two kinds, and the orchestrator handles one of them: | Step kind | What runs it | | - | - | | **`orchestrated`** | A run of the step-orchestrator agent, scoped to the specialists that step binds. Each specialist call is its own trace span. | | **`deterministic`** | A registered handler, invoked in process — not this agent. | Either way the outcome is written back onto the step run, which is the only place a caller learns what happened. > [!NOTE] > > An account has one step executor: whichever agent holds the `step-orchestrator` role is the one > every orchestrated step dispatches to. Deploying a second does not give you two — it gives you a > question about which one holds the role. You do not usually invoke this path directly. It is what makes a module's workflow steps run, and it matters when you are reading a workflow's traces or working out why a step routed the way it did. ## Tools supported | Tool surface | What it can call | | - | - | | The agent registry | Which specialists exist, what each advertises, and which are online | | A specialist's returned findings | The material it synthesizes into one answer | | The workflow definition | The steps, their order, and the roster each may route among | It reaches no external system. Its whole tool surface is the fleet, through agent-to-agent invocation over the control plane — which is why every delegated call is authorized and recorded like any other run. ## Scenario examples In chat, or as a run's prompt: **Let it choose the specialists** ```text theme={null} Checkout-api is failing in eu-west-1 and I do not know whether it is the cluster, the AWS side, or something that deployed at 14:00. Work out which, using whichever specialists you need, and give me one answer with the evidence behind it. ``` **Scope the roster yourself** ```text theme={null} Investigate the payments outage using only the Kubernetes and Datadog investigators. Combine what they find into one answer and tell me where they disagree. ``` **Verify after the fact** ```text theme={null} We remediated the checkout incident an hour ago. Ask the relevant specialists whether the symptoms have actually cleared, and flag anything that has not. ``` Given a roster and no workflow it decides for itself who to call, and loops until it judges the goal met. Given a declarative workflow it follows that instead. ## Prerequisites | Requirement | Detail | Where to set it up | | - | - | - | | Integrations | None of its own. It reaches other agents over the control plane rather than external systems, so the tool surface it needs is the fleet, not a provider. | — | | Permissions | Specialists to route to — the agents it delegates to must already be deployed and online, since it reaches only agents that are actually up | [Agent catalog](/manage-your-agents/build/catalog/overview) | | Model access | The managed model gateway, or your own Claude API key | [Credentials & secrets](/security-and-governance/identity-and-access/credentials-secrets) | | Deployment | Komodor cloud or self-hosted. | — | **Catalog ID:** `workflow-orchestrator` — what the deploy API takes. ## As part of a workflow It is the lead rather than a specialist — the agent a workflow runs *on*, not one a step delegates to. A module's `orchestrated` steps dispatch to whichever agent holds the `step-orchestrator` role, and that is this one. Pair it with specialists rather than replacing them: it decides who to call and combines what comes back, and every sub-task it delegates is its own run with its own evidence. ## Limitations * **It does no specialist work itself** — it routes, delegates, and synthesizes, which is what keeps its output auditable: each sub-task is its own run with its own evidence. * **It reaches only agents that are online.** A specialist that is down is not queued for later; the step records that it could not be reached and the rest carries on. * **It cannot make a workflow succeed that its specialists failed.** A finished workflow does not mean every step succeeded — read the steps rather than the outcome. * **An account has one step executor.** Deploying a second orchestrator does not give you two; it gives you a question about which one holds the `step-orchestrator` role. ## Recommended model `claude-sonnet-5` — the shipped default. The work is judgement over evidence someone else gathered, where a weaker model produces confident-sounding conclusions that do not hold. ## Next steps > **[Agent catalog](/manage-your-agents/build/catalog/overview)** > > Every deployable agent, and what each one is for. > **[Modules & workflows](/modules-and-workflows/overview)** > > The packaged workflows these steps belong to. > **[Runs & evidence](/manage-your-agents/run/runs-evidence)** > > Reading a delegated run and the spans that link it. # Agent catalog Source: https://docs.komodor.com/manage-your-agents/build/catalog/overview Reference for every Komodor-built agent you can deploy from the catalog — what each one reads, what it costs in time, and where it stops. The catalog holds Komodor-built agents that you deploy as-is, supply credentials for, and run. This section is the reference for each one: what it reads, what to connect before deploying it, how to ask it for something useful, and the boundary it will not cross. For browsing the catalog and walking the deploy wizard, see [Use specialized agents](/manage-your-agents/build/use-specialized-agents). Agents are grouped by domain, and every agent is written to the same seven sections — **Overview**, **Tools supported**, **Scenario examples**, **Prerequisites**, **As part of a workflow**, **Limitations** and **Recommended model** — so two agents can be compared side by side, whether or not they share a page. > [!NOTE] > > The catalog is curated per account, so the entries you see in your own console are the > authoritative list. An agent documented here may not be offered to your account, and the console is > the place to check. ## Choosing one Start from the system that already holds your evidence. An investigator is only as useful as the data it can reach, so the shortest path from deploy to a useful run is the one pointed at a system you already operate. | If you want to… | Deploy | | - | - | | Investigate an incident in AWS, Azure or Google Cloud | [Cloud providers](/manage-your-agents/build/catalog/cloud-providers) | | Investigate from Datadog or Grafana telemetry | [APM](/manage-your-agents/build/catalog/apm) | | Root-cause one Kubernetes resource, one cluster, or ask across every cluster | [Kubernetes](/manage-your-agents/build/catalog/kubernetes) | | Review pull requests, locate a problem in a repository, or open a pull request for a configuration fix | [Code Review](/manage-your-agents/build/catalog/code-review) | | Route work to other agents, or run a workflow over your fleet | [Orchestrator](/manage-your-agents/build/catalog/orchestrator) | | Decide what to do about a finding | [Remediation](/manage-your-agents/build/catalog/remediation) | | Carry out an approved action | [Remediation Executor](/manage-your-agents/build/catalog/remediation-executor) | | Find and act on AWS cost | [AWS Cost Analyzer](/manage-your-agents/build/catalog/aws-cost-analyzer) | ## The domains > **[Cloud providers](/manage-your-agents/build/catalog/cloud-providers)** > > AWS, Azure and Google Cloud — one read-only investigator each. > **[APM](/manage-your-agents/build/catalog/apm)** > > Datadog and Grafana, across traces, logs, metrics and profiles. > **[Kubernetes](/manage-your-agents/build/catalog/kubernetes)** > > One resource, one cluster with GPU signals, or every cluster Komodor covers. > **[Code Review](/manage-your-agents/build/catalog/code-review)** > > Reviewing pull requests, locating a problem in the code, and opening a pull request for a configuration fix. > **[Orchestrator](/manage-your-agents/build/catalog/orchestrator)** > > Agents that route work to other agents rather than doing it. > **[Remediation](/manage-your-agents/build/catalog/remediation)** > > Deciding what to do, and — separately — carrying it out. ## What every catalog agent has in common A few facts hold for all of them, so the individual pages do not repeat them. | | | | - | - | | Behavior | Sealed to the deployed version. There is no prompt to edit and no revert. | | Model access | Either the managed gateway or your own Claude API key, chosen at deploy. | | Credentials | Held in the encrypted credential store and delivered per run, never baked into the image. | | Evidence | Every run produces a full transcript, tool calls and output in [Runs & evidence](/manage-your-agents/run/runs-evidence). | | Identity | Each deployed agent has its own identity and is governed by the same permissions as any other. | | Time budget | A wall-clock bound on a single run, not a spend cap. Past it, the agent reports what it has. | > [!WARNING] > > A time budget bounds how long one run may take. It does not cap what a run costs — the Komodor > Agentic Operation Platform (KAOP) reports spend per run rather than refusing an expensive one. > See [Agent spend attribution](/manage-your-agents/optimize/agent-spend-attribution). ## Next steps > **[Use specialized agents](/manage-your-agents/build/use-specialized-agents)** > > Browse the catalog and walk the deploy wizard. > **[Integrations list](/manage-your-agents/build/built-in-integrations)** > > Connect what an agent needs before you deploy it. > **[Modules & workflows](/modules-and-workflows/overview)** > > Bind these agents to workflow steps instead of running them one at a time. # Remediation Source: https://docs.komodor.com/manage-your-agents/build/catalog/remediation Reads a finished investigation and returns a typed decision — propose a fix, decide no action is warranted, or ask the questions that would settle it. Remediation is the decision step between an investigation and an action. It reads a finished investigation and returns one of three typed decisions — a proposed fix, no action warranted, or the questions that would settle it. ## Overview An investigation that ends in a finding creates pressure to act on it, and the finding is not always strong enough to deserve that. The judgement call — is this evidence, or is it a confident-sounding inference — is the one most often skipped when an incident is still open. Deploy it to make that judgement explicit. You hand it a finished investigation, and it reads the findings and the run transcripts behind them — separating what was actually observed from what was inferred — and returns one of three typed decisions: | Decision | When it returns this | | - | - | | A proposed remedy | The evidence supports a specific fix | | No action warranted | The finding does not justify changing anything | | Questions | Something is missing, and it names what would settle it | Reading the transcripts rather than only the summary is the point. A confident-sounding finding built on an inference is a different thing to act on than one built on an observation, and the distinction survives into its decision. **For:** teams who want a decision step between investigating and changing something. **Not for:** carrying the change out — despite the name, this agent plans. [Remediation Executor](/manage-your-agents/build/catalog/remediation-executor) is the one that acts. ## Tools supported | Tool surface | What it can call | | - | - | | The investigation's findings | What was concluded | | The investigation's run transcripts | What was actually observed, versus inferred | It holds no cloud, cluster or repository credentials, and has no shell. Its only reach is back into the platform for the investigation it was given. ## Scenario examples It takes a completed investigation rather than a free-text question, so it is normally driven by a workflow step rather than by a person. Where you do invoke it directly, pass the investigation you want a decision on: **Decide on one investigation** ```text theme={null} Take the AWS investigation from run 4f2a and tell me whether it justifies a change. If it does, propose one. If the evidence is not there, say what would settle it. ``` **When you expect no action** ```text theme={null} Investigation 8c11 says a pod restarted once and recovered. Tell me plainly whether that warrants a change, and if not, say so rather than proposing one. ``` **When the evidence is thin** ```text theme={null} Read investigation 22d9 and tell me what is actually observed versus inferred. If the conclusion rests on an inference, say what would confirm it. ``` ## Prerequisites | Requirement | Detail | Where to set it up | | - | - | - | | Integrations | None. Because it is plan-only, there is no integration to bind and no credential for a target system. | — | | Permissions | Nothing beyond platform access to the investigation it is given | — | | Model access | The managed model gateway, or your own Claude API key | [Credentials & secrets](/security-and-governance/identity-and-access/credentials-secrets) | | Deployment | Komodor cloud or self-hosted, equally — there is nothing private for a self-hosted deploy to unlock. | — | **Catalog ID:** `remediation` — what the deploy API takes. ## As part of a workflow It is the decision step between investigation and action. An incident workflow investigates, passes the result here, and routes on what comes back: a proposed remedy goes to an executor or to a human for approval, questions go back to the specialists that can answer them, and no-action closes the loop without a change. That routing is what makes the third answer useful. An agent that could only propose fixes would propose one for every finding. See [Orchestrator](/manage-your-agents/build/catalog/orchestrator) and [Approvals](/security-and-governance/controls/approvals). ## Limitations * **It never mutates a live system.** Not as a policy setting, and not as a prompt instruction — it has no credentials for a target system and no shell, so there is nothing to mutate with. A proposed remedy is a recommendation until something else carries it out. * **Its name invites the opposite assumption**, which is worth stating plainly to anyone reviewing its access: this agent plans, and [Remediation Executor](/manage-your-agents/build/catalog/remediation-executor) is the one that acts. * **It decides on what it was given.** Its reach is the investigation passed to it, not the systems that investigation looked at. * **It takes an investigation, not a question.** A free-text prompt with no run behind it has nothing for it to read. ## Recommended model `claude-sonnet-5` — the shipped default. The work is judgement over evidence someone else gathered, where a weaker model produces confident-sounding conclusions that do not hold. ## Next steps > **[Remediation Executor](/manage-your-agents/build/catalog/remediation-executor)** > > Carry a proposed remedy out. > **[Approvals](/security-and-governance/controls/approvals)** > > Put a human decision in front of an action. > **[Agent catalog](/manage-your-agents/build/catalog/overview)** > > Every catalog agent, side by side. # Remediation Executor Source: https://docs.komodor.com/manage-your-agents/build/catalog/remediation-executor Carries a finished investigation into action — briefing Klaudia to fix the live resource, or opening the pull request that fixes the source. Remediation Executor is the step that carries a decision out. It takes a completed investigation and delegates the fix — briefing Klaudia to act on the live resource, or opening the pull request that fixes the source. ## Overview Between a decision and a change sits the part nobody wants to automate carelessly: something has to touch production, or open the pull request that will. The risk is not the action itself but who gets to approve it. Deploy it when a decision has been made and something has to actually change. It consumes the same input envelope as [Remediation](/manage-your-agents/build/catalog/remediation) — a completed investigation's findings — and hands the work to a system that can carry it out. It does not re-investigate; it instructs, and relays a typed outcome back. | Target | What it does | | - | - | | The live resource | Opens a Klaudia chat session and instructs it to act on the findings | | The source | Opens a pull request against the code that caused the problem | Which one it uses is a configured choice, not something it decides mid-run. **For:** the last step of an incident workflow, once a decision and an approval are behind it. **Not for:** deciding *whether* to change something, and never for answering the approval that guards the change. ### The approval boundary This is the part worth understanding before deploying it. When Klaudia pauses for an approval, this agent **cannot answer it**. The tool that would respond is denied to it in its own code, so a gate Klaudia raises comes back to a human every time. That holds regardless of how the agent is configured or what its instructions say — it is not a setting that can be relaxed. See [Approvals](/security-and-governance/controls/approvals) for where those decisions land. ## Tools supported | Tool surface | What it can call | | - | - | | The investigation's findings | The basis for what it instructs | | Klaudia | A chat session, instructed to act on those findings against the live resource | | GitHub | Reading the file being fixed, and opening the pull request that fixes it | The wizard's tool checklist has no effect here: the agent fixes its own tool surface in the implementation, so what it can do is decided there rather than by what you tick. ## Scenario examples It takes a completed investigation rather than a free-text instruction, so it is normally driven by a workflow step. Where you invoke it directly, pass the investigation to act on: **Execute with an approval stop** ```text theme={null} Take the findings from run 4f2a and remediate. Use the cluster path, and stop for approval before anything changes. ``` **Propose the change only** ```text theme={null} Take the remediation proposed in run 9b03 and prepare it, but do not apply anything — I want to see the exact change and its blast radius first. ``` **A code fix rather than a cluster one** ```text theme={null} The fix in run 5e77 is a code change. Open the pull request against the right repository with the change and the evidence behind it, and stop there. ``` ## Prerequisites | Requirement | Detail | Where to set it up | | - | - | - | | Integrations | An MCP group with a Komodor server — how it reaches Klaudia. A GitHub connection is optional, and required only for the source path. | [Integration groups](/manage-your-agents/build/integration-groups) · [Integrations list](/manage-your-agents/build/built-in-integrations) | | Permissions | Whatever the target system enforces: the guardrails on the live cluster are Klaudia's, and the guardrail on the source is that a pull request still needs a reviewer. | — | | Model access | The managed model gateway, or your own Claude API key | [Credentials & secrets](/security-and-governance/identity-and-access/credentials-secrets) | | Deployment | Komodor cloud or self-hosted. | — | **Catalog ID:** `remediation-executor` — what the deploy API takes. > [!NOTE] > > GitHub is optional because the cluster path does not use it. If you only ever remediate through > Klaudia, there is nothing to connect. Configure the source path without a connection behind it and > the agent refuses at validation rather than failing mid-run. ## As part of a workflow It is the final step, downstream of an investigation and usually of a decision. A typical incident workflow investigates, asks [Remediation](/manage-your-agents/build/catalog/remediation) whether a change is warranted, routes the proposal through an approval, and only then reaches here. Putting it directly after an investigation skips the step that decides whether anything should change at all — which is a choice, but worth making deliberately. ## Limitations * **It cannot approve or reject on a person's behalf.** The tool that would answer an approval is denied to it in its own code, and no configuration relaxes that. * **It does not decide what the fix should be.** It carries out what an investigation established, through a system that has its own controls. * **Its guardrails are the target system's.** On the live cluster they are Klaudia's; on the source, the pull request still needs a reviewer. * **The target is configured, not chosen.** It does not switch between the cluster and the source mid-run. ## Recommended model `claude-sonnet-5` — the shipped default. The work is judgement over evidence someone else gathered, where a weaker model produces confident-sounding conclusions that do not hold. ## Next steps > **[Remediation](/manage-your-agents/build/catalog/remediation)** > > Decide whether a change is warranted first. > **[Approvals](/security-and-governance/controls/approvals)** > > Where a paused decision lands, and who may answer it. > **[Klaudia Investigator](/manage-your-agents/build/catalog/kubernetes)** > > The same Klaudia, used to investigate rather than to act. > **[Agent catalog](/manage-your-agents/build/catalog/overview)** > > Every catalog agent, side by side. # How it works Source: https://docs.komodor.com/manage-your-agents/build/how-it-works What building an agent involves in KAOP — the two authoring paths, and everything you attach to an agent before it runs. Building an agent in the Komodor Agentic Operation Platform (KAOP) means deciding what it should do, what it may reach, and what starts it — then letting the platform run it as a governed worker. This page orients the Build stage: the difference between an agent and the worker that runs it, the two paths for getting an agent in the first place, and each thing you attach along the way. ## What "building an agent" means here An agent is not a script you deploy and forget. In KAOP an agent is a **governed identity and behavior**: its instructions, the tools it may call, the credentials it may resolve, the knowledge it may read, the schedules that start it, and the permissions enforced when it acts. Building one is therefore mostly composition. You define behavior once, then attach capability to it: ```mermaid theme={null} flowchart TB A["Agent
instructions · tools · handler"] A --> C["Connections
built-in integrations or MCP Gateway"] A --> S["Skills
reusable know-how"] A --> U["Use cases
typed actions others call"] A --> K["Knowledge base
documents and past incidents"] A --> T["Triggers
cron schedules"] A --> D["Deploy"] D --> W["Worker
the running process"] ``` Nothing on that list is mandatory. An agent with instructions and no connections is a valid agent — it just cannot see anything outside itself. ## Agent versus worker These are two different things, and the distinction matters as soon as you start debugging. | | **Agent** | **Worker** | | - | - | - | | What it is | The governed identity and behavior | The process that runs it | | Where it lives | The control plane's agent registry | Wherever you run it, or on Komodor's infrastructure | | What you do with it | Author it, attach capability, grant it access | Start it, watch it register, check whether it is online | | Cardinality | One agent | One worker per agent | You build an agent. You deploy a worker. Everything in the console — runs, evidence, spend, permissions — is organized around the agent; the worker is the thing that shows as online or offline. See [Architecture](/get-started/architecture) for how the two talk to each other. ## Creating an agent Writing the code is one half; the agent also has to exist in the control plane. Three surfaces do that, and they produce the same agent. **Console** **Create an agent** opens on a starting-point picker with three choices — **Start from catalog**, **Build from scratch**, and **Import an existing agent**. Choose a starting point and the wizard walks you through **Agent card**, **Where it runs**, **Agent instructions**, **Model**, **MCP tools**, and **Triggers**, then an **Activate** step that registers the agent, mints a worker token — shown once — and hands you what your worker needs to start. Your progress is saved as a draft as you go, so you can leave and come back. **API** `POST /api/v1/agents` creates an agent, and `POST /api/v1/agents/worker-token` mints a worker token for one. Use these when agent creation is part of your own provisioning. See [APIs](/developer-tools/apis). **MCP** The `agents` toolset's `manage_agent` tool creates and mints a worker token the same way, for a model or assistant connected to the [MCP server](/developer-tools/mcp-server) to do it directly. Either way the agent starts life as a **draft** and goes live on its worker's first heartbeat. There is no separate activation step. If the agent already runs on your own infrastructure, the third path connects it instead of deploying anything — see [Import an existing agent](/manage-your-agents/build/import-an-agent). ## Making it available in chat Chat is opt-in per agent, advertised on its agent card: ```python theme={null} AGENT_SPEC = AgentSpec.from_dir( Path(__file__).parent, agent_card={"capabilities": {"chat": True, "streaming": True}}, ) ``` See [Chat & history](/manage-your-agents/run/chat-history). ## Validating the deploy These checks prove the agent is live and doing what you expect: **1. It registered** The agent appears in **Fleet** and reads as online. Registration happens on the worker's first heartbeat — if it stays offline, the worker is not reaching the control plane, and `AGENTOPS_URL` being unreachable *from where the worker runs* is the usual cause. **2. It advertises what you expect** Open the agent and read its card: capabilities, tools, input schema, triggers. This is what the spec actually declared, which is not always what you meant to declare. **3. A real run completes** Invoke it once — **Run agent** on its row in Fleet, or the API — and let it finish. A run that errors here but passed locally is almost always credentials or tool access, not logic. Every agent advertises two built-in checks for this, without declaring anything: | Built-in | What it proves | | - | - | | **Check connectivity** | A canned, deterministic reply with no model, no tools and no integrations. It short-circuits before your handler runs, so it proves only that the platform can reach the worker and that runs and transcripts are flowing. | | **Sample run** | A real run through the agent and its model with a canonical prompt, producing a live transcript. It proves the whole path, including the model. | Run them in that order to localize a failure: connectivity failing is deployment or networking; connectivity passing while the sample run fails points at the model endpoint. **4. Its evidence is readable** Open the run. The input, the tool calls, the reasoning, the output and the token cost should all be there. Missing spans mean the agent is doing work outside the platform's instrumentation; missing cost means usage is not being reported. Both are worth fixing before the agent does anything real. See [Runs & evidence](/manage-your-agents/run/runs-evidence). **5. Its permissions are right** Confirm what it can reach is what you intended, not what was convenient during development. See [Roles & permissions](/security-and-governance/identity-and-access/roles-permissions). ## Everything in this section > **[Use specialized agents](/manage-your-agents/build/use-specialized-agents)** > > Browse the catalog and deploy a Komodor-built agent without writing code. > **[Upgrade a self-hosted agent](/manage-your-agents/build/upgrade-a-self-hosted-agent)** > > See which agents you run yourself are behind, and move them to the newest release. > **[Integrations overview](/manage-your-agents/build/integrations-overview)** > > The two connection paths, and which one your case calls for. > **[Integrations list](/manage-your-agents/build/built-in-integrations)** > > The catalog of ready-made connections, and how to connect and test one. > **[Integration groups](/manage-your-agents/build/integration-groups)** > > Bundle connections into a named set and attach the set to an agent. > **[MCP Gateway](/security-and-governance/architecture-considerations/mcp-gateway)** > > Expose your own tools through an MCP server or an OpenAPI surface. > **[Credentials & secrets](/security-and-governance/identity-and-access/credentials-secrets)** > > Store a secret once, bind it to the agents allowed to use it, deliver it without exposing it. > **[Skills](/manage-your-agents/build/skills)** > > Package your runbooks as reusable know-how any agent can load. > **[Knowledge base](/manage-your-agents/build/knowledge-base)** > > Give agents your documentation and past incidents as a cited source. > **[Triggers & schedules](/manage-your-agents/build/triggers-schedules)** > > Start runs on a recurring cron schedule. ## Next steps > **[Concepts & glossary](/get-started/concepts)** > > The vocabulary this section assumes. > **[Run — how it works](/manage-your-agents/run/how-it-works)** > > What happens once an agent is built and something starts it. # Import an existing agent Source: https://docs.komodor.com/manage-your-agents/build/import-an-agent Connect an agent you already run — Google ADK, LangChain, Claude Agent SDK or Agno — by generating the wrapper that registers it and reports its evidence. Import connects an agent you are already running to the Komodor Agentic Operation Platform (KAOP), rather than rebuilding it here. Your agent's behaviour stays in your code and keeps running where it runs; KAOP generates a thin wrapper that registers it, receives its work, and reports its evidence. ## Prepare before you start | | | | - | - | | **A working agent** | It already runs and produces useful output. Import connects it; it does not fix it. | | **A supported framework, or none** | Google ADK, LangChain, Claude Agent SDK and Agno each have an adapter. Anything else uses the bare worker API and you wire the call yourself. | | **Python 3.10 or later** where the agent runs | The wrapper is a Python package, even when the agent it wraps is not. | | **Somewhere the agent can reach the control plane from** | Outbound only. Nothing needs to reach into your environment. | | **A model provider** | The imported agent keeps calling whatever model it already calls, unless you point it at a registered provider. See [Providers](/security-and-governance/architecture-considerations/providers). | > [!NOTE] > > The agent does not have to be reachable from KAOP, or have a public address. The wrapper opens the > connection outward and claims work over it. > [!NOTE] > > The steps below describe the console wizard, but they are not the only way in. A coding agent > connected to the [MCP server](/developer-tools/mcp-server) can load the platform's own import skill > from the `onboarding` toolset and carry out the import from your editor or terminal instead. ## The steps **1. Framework** Pick the framework your agent is built on. The choice decides which adapter the generated wrapper uses and what it captures automatically: | Framework | What the adapter does | | - | - | | **Google ADK** | Wraps your `Runner`. Tool calls and sub-agents arrive as spans. | | **LangChain** | Wraps a chain or agent executor through the callback handler. | | **Claude Agent SDK** | Wraps `query()`. Hooks carry tool use and cost. | | **Agno** | Wraps a team or a single agent run handler. | | **Something else** | A custom loop, or a framework with no adapter yet. You get the bare worker API and connect it yourself. | **2. Agent card** Name the agent and give it an id and a description. This is the identity KAOP governs — what appears in Fleet, what permissions attach to, and what runs are attributed to. It is independent of whatever your code calls itself. **3. Where it runs** Keep the agent where it already runs, or move it to Komodor-hosted. Importing usually means keeping it in place, which is the point: the code and its network access do not move. **4. Model, tools and triggers** The same wizard steps as any other agent — **Model**, **MCP tools**, **Integrations**, **Triggers**, **Permissions**. An imported agent can skip most of them and keep using its own model and tools, or adopt KAOP's so that credentials, guardrails and spend attribution apply to it too. **5. Generate the wrapper** KAOP produces the install line and the wrapper source for your framework: ```bash theme={null} pip install "komodor-agentops[langchain]" ``` The wrapper builds its spec with `AgentSpec.from_dir`, which reads `agent-spec.yaml` from the directory alongside it. That path — rather than constructing the spec in code — is what applies the `AGENTOPS_AGENT_ID` override and picks up `agent.md`, labels and provenance. **6. Set three variables and start it** The wrapper authenticates with three environment variables: | Variable | What it is | | - | - | | `AGENTOPS_URL` | Your account host. Must be reachable **from where the agent runs**. | | `AGENTOPS_AGENT_ID` | The agent id from the card step. | | `AGENTOPS_WORKER_TOKEN` | Issued during import, and shown once. | Start the agent as you normally would. The wrapper registers it on the first heartbeat. ## Validate the import The **Verify** step watches for the agent rather than asking you to confirm it, and updates on its own as soon as the agent connects: | Signal | What it means | | - | - | | **First heartbeat received** | The wrapper reached the control plane and registered. Until this arrives, the agent shows as offline. | | **Roles applied** | Permissions attached on connect. | | **What your agent says it is** | The card the wrapper advertised — its capabilities, tools and input schema. Compare it with what you expected; a mismatch here means the wrapper is reading a different spec than you think. | If the first heartbeat does not arrive, check these in order: 1. **All three variables are set**, and `AGENTOPS_AGENT_ID` matches the id the token was minted for. A token bound to a different agent authenticates as that one. 2. **`AGENTOPS_URL` is reachable from where the agent runs** — not only from your own machine. This is the most common cause: the value was tested from a laptop and the agent runs in a cluster. 3. **The spec is the one you think it is.** If it is built in code rather than loaded from `agent-spec.yaml`, the environment override does not apply and the agent registers under a different id. Once the heartbeat lands, run it once and read the evidence. Its input, tool calls, output and cost are recorded the same way as any other agent's — that is the import working, not just connecting. ## Next steps > **[Runs & evidence](/manage-your-agents/run/runs-evidence)** > > What the imported agent now records on every run. > **[Fleet health](/manage-your-agents/optimize/fleet-health)** > > Whether it stays online once it is in. > **[Upgrade a self-hosted agent](/manage-your-agents/build/upgrade-a-self-hosted-agent#imported-agents-and-your-own-images)** > > Move the wrapper to a newer `komodor-agentops` release when one is available. # Integration groups Source: https://docs.komodor.com/manage-your-agents/build/integration-groups Bundle your connected tool servers into a named group, narrow which tools it grants, and attach the whole set to an agent in one move. An integration group is a named bundle of tool servers that you attach to an agent as a set. It exists so that scoping an agent's reach is one decision instead of a dozen, and so that several agents can share exactly the same tool surface. ## What a group is The Komodor Agentic Operation Platform (KAOP) reaches external systems through tool servers — the ones stood up behind your [built-in integrations](/manage-your-agents/build/built-in-integrations), and the ones you register yourself through the [MCP Gateway](/security-and-governance/architecture-considerations/mcp-gateway). A group is a list of those servers plus rules about which of their tools it grants. | A group has | Meaning | | - | - | | A **name** | Unique within your account, and the key the group is addressed by. Letters, digits, `_` and `-`, starting with a letter or digit | | **Member servers** | Which tool servers belong to the group | | **Allow** and **Deny** | Which of the members' tools the group grants | Groups are found under **Integrations → Integration groups**. In the product's own words, a group bundles servers behind a per-agent gateway endpoint: each group has its own endpoint, and binding an agent to the group is what points the agent at it. ## Create one **1. Open Integration groups** Go to **Integrations**, then the **Integration groups** section. **2. Add the group** Press **Add integration group** and give it a name that says what it is for — `buildkite-triage` reads better in an agent's configuration than `group-2`. **3. Choose member servers** Pick the tool servers the group covers. A server can belong to several groups, so you can build a broad group and a narrow one over the same connections. **4. Narrow the tools, if you need to** Use the allow and deny lists to trim the members' tools down to the ones this group should grant. **5. Attach it to an agent** See [Attach a group to an agent](#attach-a-group-to-an-agent). ## How allow and deny narrow the set A group's rules layer on top of the rules each member server already has, and they can only ever narrow. Two consequences follow, and both are worth stating plainly: * A tool that the owning server denies stays denied. A group cannot grant back what the server refuses. * **Deny wins.** If a tool is on both lists, it is denied. This is what makes a group safe to hand to more than one agent: adding an agent to a group can never widen what that group's servers permit. ## Attach a group to an agent A group reaches an agent from the agent's own tool configuration. In the agent's **MCP tools** step, choose the scope: | Scope | Effect | | - | - | | **One server** | Pick a single server and choose its tools for this agent alone | | **Integration group** | The group's servers and tools, shared by every agent bound to it | Once a group is selected, the step lists the tools the group grants as a read-only summary — those tools belong to the group, so you change them by editing the group, not the agent. An agent belongs to **at most one** group. If an agent needs the union of two groups, make a group that covers both rather than trying to attach two. > [!WARNING] > > An agent with no integration group is **not** scoped to nothing — it is unscoped, which means it can > reach every tool server in the account. Unscoped is the common starting state, not an edge case, so > if an agent should only reach a subset, it needs a group. Attaching one is how you narrow it. ## When to use a group | Situation | Use a group? | | - | - | | Several agents need the same tools | Yes — build it once, attach it to each | | One agent needs one server's tools | No — scope it to that server directly | | You want a narrower view of a broad server | Yes — a group with a deny list is the tool for it | | You want an agent to reach everything | No group needed; that is the unscoped default | **Example.** Say three agents all take part in incident response: one summarizes the alert, one digs through logs and metrics for a root cause, one posts status updates. All three need the same read access to PagerDuty, Datadog, and Slack, but none of them should be able to, say, resolve the incident or delete a monitor. Rather than scoping each agent by hand and keeping the three copies in sync, create one `incident-response` group — member servers PagerDuty, Datadog, and Slack, with a deny list that strips write actions like acknowledge, resolve, and delete — and attach all three agents to it. Widening or narrowing what the trio can do is then one edit to the group instead of three edits to three agents. ## Next steps > **[Integrations list](/manage-your-agents/build/built-in-integrations)** > > Connect the providers whose servers a group bundles. > **[MCP Gateway](/security-and-governance/architecture-considerations/mcp-gateway)** > > Register your own servers, then group them. > **[Integrations overview](/manage-your-agents/build/integrations-overview)** > > The two connection paths a group draws from. # Integrations overview Source: https://docs.komodor.com/manage-your-agents/build/integrations-overview The two ways an agent reaches anything external — the built-in integration catalog or the MCP Gateway — and how to choose between them. An agent that cannot see your systems cannot do operations work. The Komodor Agentic Operation Platform (KAOP) gives it exactly two ways to reach anything external: a **built-in integration** from the catalog, or the **MCP Gateway** for tools you expose yourself. ## The two paths > **[Integrations list](/manage-your-agents/build/built-in-integrations)** > > A ready-made connection to a system KAOP already knows — Datadog, AWS, Azure, GitHub, > Kubernetes, PagerDuty, Slack and more. You supply credentials; the tools, auth handling, and > connection health come with it. > **[MCP Gateway](/security-and-governance/architecture-considerations/mcp-gateway)** > > A connection to a tool surface you own — any MCP server, or an OpenAPI surface KAOP turns into > tools. You define what the tools are; the gateway governs who may call them. There is no third path. Anything an agent reaches outside itself arrives through one of these two, which is what makes the question "what can this agent touch?" answerable at all. ```mermaid theme={null} flowchart LR A[Agent] --> I["Built-in integration
from the catalog"] A --> G["MCP Gateway"] I --> S1["Datadog, AWS, GitHub,
Kubernetes, Slack, …"] G --> M["Your MCP server"] G --> O["Your OpenAPI surface"] C[(Credentials)] -.-> I C -.-> G ``` ## How to choose Start with the catalog. Reach for the gateway when the catalog does not cover the system, or when the capability you want is genuinely yours. | Question | Built-in integration | MCP Gateway | | - | - | - | | Is the system in the catalog? | Yes — use it | No, or the catalog connection does not expose what you need | | Who defines the tools? | KAOP does | You do | | What do you supply? | Credentials, and any account or region detail the provider needs | A reachable MCP server or an OpenAPI surface, plus its auth | | Typical case | Read metrics from Datadog, list resources in AWS, comment on a GitHub pull request | Call your internal deployment service, query your own inventory API, wrap a homegrown CLI | | Ongoing work | Connect once; test and reconnect when credentials change | Keep your server or spec available and current | > [!TIP] > > The two are not exclusive, and most real agents use both. A remediation agent might read Datadog > through a built-in integration and open a change through your own service via the gateway. ### Cases worth naming > **The system is in the catalog, but I want a different tool than it offers** > > Use the built-in integration for what it covers, and expose the extra capability through the MCP > Gateway. They coexist on the same agent, and each keeps its own credentials. > **I already run an MCP server for my developers** > > Register it with the MCP Gateway. The gateway is the governance boundary — it decides which > agents may call which of its tools — so you get policy over a surface you already maintain. > **I only have a REST API and no MCP server** > > Register the OpenAPI surface with the gateway instead. You do not have to build an MCP server to > give agents access to an existing API. > **I need to reach several systems for one agent** > > Connect each one, then bundle them with an > [integration group](/manage-your-agents/build/integration-groups) and attach the group to the > agent as a set. ## What both paths share Whichever path a connection takes, the same three mechanisms apply. ### Credentials Secrets are never part of an agent's code or image. You store a credential once, bind it to the agents allowed to use it, and it is delivered to a run in memory without its value being exposed to you or written into the run's evidence. Both paths resolve their secrets this way. See [Credentials & secrets](/security-and-governance/identity-and-access/credentials-secrets). ### Governed access A connection existing is not the same as an agent being allowed to use it. An agent reaches a connection only when that connection — or the group containing it — is attached to the agent, and its actions are still checked against policy at the moment it acts. See [Standards & Guardrails](/security-and-governance/controls/standards-guardrails). ### Evidence Every call an agent makes through either path is recorded in the run: which tool, with what arguments, and what came back. This is what lets you audit a conclusion instead of trusting it. See [Runs & evidence](/manage-your-agents/run/runs-evidence). ## Next steps > **[Integrations list](/manage-your-agents/build/built-in-integrations)** > > The catalog, the auth modes, and how to connect and test one. > **[MCP Gateway](/security-and-governance/architecture-considerations/mcp-gateway)** > > Register your own MCP server or OpenAPI surface. > **[Integration groups](/manage-your-agents/build/integration-groups)** > > Attach a set of connections to an agent in one move. > **[Credentials & secrets](/security-and-governance/identity-and-access/credentials-secrets)** > > The store both paths use. # Knowledge base Source: https://docs.komodor.com/manage-your-agents/build/knowledge-base Give agents your runbooks, postmortems, and service docs as a searchable source they retrieve from and cite during a run. The knowledge base is your organization's written knowledge, indexed so agents can search it while they work and cite what they used. Where skills tell an agent *how* to do a job, the knowledge base tells it *what your organization knows* — how this service behaves, what happened last time, who owns what. > [!NOTE] > > A knowledge base holds what **you** put in it. Nothing lands here on its own — a resolved > incident does not become a knowledge-base document, and neither does a run's output. What an agent > learns for itself lives in [Memory](/manage-your-agents/build/memory) instead. ## What it is The Komodor Agentic Operation Platform (KAOP) hosts the knowledge base as a wiki: you add pages, KAOP indexes them, and agents search them during a run. The pages are grouped by what they are — Runbooks, Postmortems, Service docs, Policies, Decisions, Guides, Concepts, and Incidents — so the library stays legible as it grows. A page with no doc type lands in **Pages**, and an unrecognized type becomes a group of its own rather than being dropped. Find it under **Build → Knowledge**. ## Add a document **1. Open Knowledge and add a page** Press **Add page**, or drag files onto the page. **2. Choose the files** Markdown, plain text, and PDF are supported. Up to 20 files per upload, each up to 10 MB. **3. Describe them** The metadata you set applies to the whole batch and is what makes retrieval precise later. **4. Wait for indexing** A document is **Indexing** while it is being processed and **Indexed** once it is searchable. A document that could not be processed shows **Error**. ### The metadata, and why it matters | Field | What it is for | | - | - | | **Title** | How the page is listed and cited. Set it for a single upload; a batch takes each file's own title or filename | | **Owner** | Who maintains this page | | **Service** | Which service it is about — the most useful filter an agent has | | **Environment** | Which environment it describes | | **Doc type** | What kind of page it is: `runbook`, `postmortem`, `service-doc`, `policy`, `decision`, `guide`, `concept`, `incident`. This is what groups the sidebar | | **Sensitivity** | How restricted the content is | | **Tags** | Free-form, comma-separated | > [!TIP] > > Setting **Service** and **Doc type** is the highest-value thing you can do at upload time. An agent > investigating `payment-api` can narrow retrieval to that service's runbooks instead of searching > everything, which is the difference between a precise citation and a plausible-sounding one. > [!NOTE] > > A Markdown file with a YAML frontmatter block has its metadata read from it — owner, environment, > sensitivity, type, tags, and the first listed component as the service. Anything you type in the > upload form wins over the file's own frontmatter, and the frontmatter block is stripped before > indexing so it does not pollute retrieval. ## How a document becomes searchable This determines how to write documents that retrieve well. **1. Text is extracted** Markdown and plain text are read directly; PDF and Word documents have their text pulled out, with Word headings preserved as Markdown headings. **2. It is split into sections** A Markdown document is split on its headings, so each section becomes a separately retrievable chunk that remembers which heading it came from. A long section is split further, with a small overlap so a sentence is never cut in half between chunks. **3. Each chunk is indexed twice** Once as a vector for meaning-based search, and once for keyword search. Retrieval combines both rankings, which is what lets it find the right page whether you remember the exact phrase or only the idea. > [!TIP] > > Use headings generously. Because sections are the unit of retrieval, a document with clear `##` > headings surfaces the *relevant part* of a long runbook rather than the whole thing — and the > heading is what appears in the citation. If you change a document outside KAOP and re-upload it, or a document failed to process, you can reindex it from its detail view. ## How an agent retrieves from it Retrieval is a tool the agent calls, not something injected into every prompt. An agent that opts into KAOP's own tool surface gets `search_knowledge`, and calls it when it decides it needs background. A search takes a query, optionally narrowed by service, environment, doc type, or sensitivity, and returns **five** passages by default — an agent can ask for as few as one or as many as ten. Each hit carries: | Field | What it is | | - | - | | Document title | The page the passage came from | | Section | The heading it sits under | | Body | The passage itself | | Score | How well it matched | | Document and chunk id | Exactly which passage, so a citation is checkable | That is the citation payload: when an agent tells you "per the payment-api runbook", the run's evidence shows which page and which section it actually read. See [Runs & evidence](/manage-your-agents/run/runs-evidence). > [!NOTE] > > Every Komodor-built catalog agent has knowledge search enabled, and so does the runtime behind agents > you create in the console. If you author your own worker, opting in is one SDK call — see > [Build from scratch](/manage-your-agents/build/how-it-works). > [!WARNING] > > An agent with knowledge search enabled can search the knowledge bases in its own account. The > account is the boundary — a document is never reachable from another account — but the knowledge > base is not the place for a secret only one agent should see. Secrets belong in > [credentials](/security-and-governance/identity-and-access/credentials-secrets). > > One agent can be narrowed further: give it a **`kb-tags`** label and it reads only pages whose tags > overlap that label, plus every untagged page. An agent without the label reads everything, so the > narrowing is opt-in — and it sorts pages by audience, which is not the same as keeping a secret. ## Searching it yourself The **Knowledge** page has its own search across page titles, services, and labels, plus an **Attention** filter for documents that need a look. This is a librarian's view for finding and maintaining pages — the semantic retrieval described above is what agents use during a run. ## Past incidents Incidents your agents have worked are retained by the module that produced them, with their full evidence, and are read there rather than through the knowledge base. See [Incident management & troubleshooting](/modules-and-workflows/sre/incidents). What the knowledge base adds is the *written* record: a postmortem you upload as a page becomes retrievable and citable alongside your runbooks, so the next investigation can find what you concluded last time. Uploading postmortems with the `postmortem` doc type and the affected **Service** set is the single highest-leverage thing you can put in the knowledge base. ## Writing documents agents can use * **One subject per page.** A page about one service retrieves better than a page about six. * **Headings that name the situation** — "Elevated 5xx after deploy" beats "Troubleshooting". * **Concrete thresholds and commands.** An agent can act on "scale to 6 replicas if queue depth exceeds 500"; it cannot act on "scale up as needed". * **Say when a page does not apply.** Explicit non-applicability stops an agent citing the wrong runbook confidently. * **Keep it current.** A stale runbook is worse than a missing one, because it retrieves. ## Next steps > **[Skills](/manage-your-agents/build/skills)** > > The other half — procedures an agent follows, rather than facts it looks up. > **[Runs & evidence](/manage-your-agents/run/runs-evidence)** > > Where an agent's citations show up. # Memory Source: https://docs.komodor.com/manage-your-agents/build/memory What an agent learned in one run and wrote down for its future self — reviewed before it is published, and searchable by the agents you bind to it. Memory is how an agent carries something forward. A run that worked out why a service was slow can write that down, and a later run of the same agent finds it instead of re-deriving it. Agents reach it through two tools — one to search, one to save — and call them on their own judgement. Find it under **Build → Memory**. > [!NOTE] > > Memory is per agent, not per account. What one agent learned is invisible to another until they > share a space. ## Setting it up Three things have to be true before an agent remembers anything. **1. Create a space** A **space** holds the memories for one area of work, under a name you choose — for example *Cloud and observability* or *Kubernetes RCA*. Name it for the work rather than the agent, since more than one agent can end up sharing it. **New space** creates one; **Manage spaces** lists them with their memory and agent counts. **2. Bind the agents that should share it** **Choose agents** on a space binds them. An agent writes to exactly one space and cannot be bound to a second, so sharing runs the other way: putting **two agents in one space** is how one agent's learning reaches another. Bind nothing and an agent's memories are its own. **3. Deploy the Memorizer, and point the space at it** Every memory is reviewed before anyone can read it, and the reviewer is itself an agent: **Memorizer**, from the agent catalog. Deploy **one per account** — it needs no integrations, only the managed LLM credential, and it is never chatted with or invoked by hand. It arrives already carrying the label the platform resolves it by, so a space left on **Find by role** finds it with nothing else to configure. Choose a specific agent on the space only if you want that space reviewed by a different one. > [!WARNING] > > **A space with no reviewer records nothing.** Memories stay unreviewed drafts, invisible to search, > while agents go on saving — silently, since saving still looks like it works. The space flags it, > and waiting drafts are reviewed as soon as a reviewer exists. ## The four statuses | Status | On screen | What it means | | - | - | - | | **Draft** | *Awaiting review* | Where every write lands. A draft reaches nobody — not another run, and not even the run that wrote it. | | **Active** | *Retrievable* | Published. The only status a search can return. | | **Superseded** | *Folded into another memory* | Replaced by a later memory that says it better. | | **Archived** | *Archived* | Retired, not wrong — out of retrieval but still readable, and restorable, for 90 days. | An agent often saves the same lesson several times in one run, worded differently each time. Drafts are what stop all of them being published: the Memorizer sees them together and keeps one. ## What happens after a run Three stages, and each one is an ordinary run you can open and read. **1. The agent is asked what to remember** The same agent resumes a **copy** of its own thread and is asked what should be remembered. It answers with the full context of the work it just did, rather than a summary of it. Because the thread is copied, the conversation you were having is never touched. **2. The drafts are reviewed** The Memorizer is handed the drafts, the run's transcript, and any existing memories the drafts may duplicate — so the decision is made with the alternatives in view. **3. The decision is applied** Each draft is promoted, rewritten, merged into an existing memory, or rejected. The space's activity reports the outcome plainly: *2 created, 1 combined, 1 removed*. ### Writing the same thing twice writes it once Review catches near-duplicates. Exact ones never get that far: a write whose text already exists in the agent's space returns the existing memory rather than adding a second copy, so an agent that re-learns the same fact verbatim does not inflate the space. ### A coherence pass runs on the space itself Every six hours a sweep asks a question no single run can: **is this space still coherent?** Two memories written weeks apart that say the same thing, one that contradicts another, wording that has aged badly — none of that is visible to a pass looking at one run's output. A space with nothing new in the window is skipped — but a single memory is enough to sweep on. The duplicate it needs comparing against is found by searching the space, not by being handed over beside it. ## Checking that it is working **Activity** is where you find out whether memory is earning its keep. | Card | What it answers | | - | - | | **Sessions that added memories · 7d** | How many sessions taught the fleet something, against how many ran at all. The bars split *Learned something* from *Looked, kept nothing*. | | **What became of them · 7d** | The memories written this week, by what the Memorizer decided about each: active, awaiting review, archived, or folded into another. | | **Most read memories · all time** | How often an agent opened a memory while working. Lifetime rather than windowed, because a read is counted but not dated. | > [!TIP] > > **A lopsided first card is healthy** — most sessions should teach nothing, and a fleet that learns > from every run is writing noise. Worry instead when nothing is being read, which **Most read > memories** shows. **What memory is doing** is the running log — each sweep and each review, with what it kept and the run behind it. ## Curating what it remembers **Manage memories** is the librarian's view: every memory, filtered by space and by status, with **Edit** and **Archive** on each. Archive the ones that have aged out; edit the ones that are nearly right. Each memory carries a **kind** the agent chose when it wrote — *technique*, *pointer*, *constraint*, *fact*, *feedback*, or *preference* — and **labels** as `key: value` pairs, like `topic: slack-formatting` or `tool: datadog`. Labels are not decoration: a hit that matches on label ranks above one that does not. **Labels** governs that vocabulary, per space, in two lists: * **Entities** — dimensions you have declared, through **Declare dimension**. You give each a key and a description of what it holds. * **Agent labels** — keys agents invented while writing, held separately so you can see what the fleet reached for that you never declared. **Declaring one changes what gets written.** Your keys and their descriptions are handed to the agent as it decides what to remember, so it labels with your vocabulary instead of inventing a near-synonym for something you already track — which is how `service` and `svc` end up as two dimensions. A match on a declared key also counts for a little more, enough to settle a tie between two equally rare labels and no more. So a key agents keep inventing is a dimension your space was missing: promote it and they start using it. Going the other way, **Stop tracking** keeps every label already applied — it only stops the key being offered to writers. ## How a search finds a memory A search runs two ways at once — keyword matching and meaning-based matching — and the results are fused into one ranking. Ranking is by each hit's **position** in that fused list rather than by its raw score, because the two methods produce scores on scales an order of magnitude apart; ranking on the raw number put every meaning-matched memory above every keyword-matched one regardless of the evidence. > [!WARNING] > > **An active memory never expires.** Nothing retires one by age — that is your call, or the coherence > pass's — so a space needs an owner rather than cleaning itself. Retiring one starts a clock instead: > an **archived** or **superseded** memory stays readable and restorable for **90 days**, then is > deleted for good. ## Next steps > **[Knowledge base](/manage-your-agents/build/knowledge-base)** > > Documents you upload, as opposed to what an agent learned on its own. > **[Runs & evidence](/manage-your-agents/run/runs-evidence)** > > The digest and review stages are runs — open them like any other. > **[Skills](/manage-your-agents/build/skills)** > > Know-how you author deliberately, rather than something learned. # Skills Source: https://docs.komodor.com/manage-your-agents/build/skills Package your runbooks and procedures as reusable, versioned skills, attach them to any agent, and see which ones a run actually used. A skill is a reusable unit of know-how — a runbook, a procedure, a pattern — that any agent can load. Skills are how you give the Komodor Agentic Operation Platform (KAOP) your team's operational knowledge without rewriting an agent's instructions every time the procedure changes. ## Why skills instead of instructions An agent's instructions describe what it is. A skill describes how a particular job is done. Keeping them apart buys you three things: * **Reuse.** One escalation procedure, attached to every agent that might need it. * **Change without redeploy.** Publishing a new version of a skill updates the agents attached to it. Nothing restarts. * **Attribution.** A run records which skills were available to it and which it actually used, so you can tell a procedure that helps from one that is never opened. ## The two kinds **Skills** lists both, and the kind filter separates them. | Kind | Where it comes from | Who versions it | | - | - | - | | **Authored** | Written in KAOP — by you, in the console or through the API | KAOP, with an integer version per publish | | **Self-reported** | Published by a worker from files next to its own code | Your repository, by whatever you deploy | Authored skills are the ones you manage as a library. Self-reported skills are visible so that you can see what your workers are carrying — and if you want to take one under management, **Claim as authored** copies it into the registry and versions it from then on. ## Author a skill **1. Open Skills and create one** Go to **Skills** and press **Create skill**. **2. Name it for the job it does** **Name** and **Description** are what an agent sees when deciding whether a skill is relevant, so write the description as a one-line statement of when to use it — not a title restated. **3. Write the content** **SKILL.md content** is Markdown. Say what the agent should do and when, concretely: the checks to run, the order to run them in, the thresholds that matter, what to do with the result. **4. Label it** **Labels** are key/value metadata. They group the library and can be used to control who sees a skill. **5. Create it** Creating with content publishes version 1. > [!TIP] > > Write for an agent that has your tools but not your context. The most useful skills are specific — > "if error rate exceeded 2% for five minutes, check the last three deploys before looking at > infrastructure" beats "investigate carefully". ### Authoring in a worker's repository instead If you would rather keep a skill next to the code that uses it, put it in the worker's `skills/` directory and it registers when the worker connects. Two layouts work: ```text theme={null} skills/ triage-checklist.md # the skill id is the filename rca/ SKILL.md # the skill id is the directory name postgres-queries.md # any sibling Markdown becomes a reference ``` Frontmatter carries three fields, all optional: ```markdown theme={null} --- name: Root cause analysis description: Use when an incident needs a documented root cause. tags: [incident, rca] --- # Root cause analysis ... ``` > [!NOTE] > > This is the same `SKILL.md` convention Claude Code uses, and a worker also reads a `.claude/skills` > directory if it finds one. Skills you already keep for a coding agent transfer as a copy — no format > change. ## Versions Authored skills are versioned with plain increasing integers, and the distinction that matters is what does and does not create a version: | Change | Creates a version? | | - | - | | Editing the name, description, tags, or labels | No | | Publishing new content | Yes — **Publish as v2**, and so on | | Rolling back | Yes — the old content is republished as a new latest version | Rollback is append-only on purpose. Version history stays a true record of what was in force when, which is what makes it useful for auditing a run that happened three weeks ago. ## Attach a skill to an agent Attaching is a binding between an agent and a skill, and it carries one decision: track latest, or pin a version. | Binding | Behavior | | - | - | | **Latest** | The agent picks up every new version as it is published | | **Pinned** | The agent stays on the version you pinned until someone repins it | Three places attach a skill, and they write the same binding: * **From the skill** — the *Currently attached* list on the skill's editor. * **From the agent** — its **Skills & tools** tab, using **Attach a skill**. * **While creating the agent** — the wizard's skills section. > [!WARNING] > > Publishing a new version moves every agent bound at *latest* immediately. If a skill is load-bearing > for an agent you cannot afford to surprise, pin that agent and repin it deliberately. ### Declared versus discovered, on the agent An agent's **Skills & tools** tab splits its skills the same way: * **Declared** — explicitly attached and versioned. These are the ones you manage. * **Discovered** — self-reported by the running agent. **Promote to declared** brings one under management. ## How an agent finds the right skill at run time Skills are not all pasted into the prompt. An agent gets a short index of everything attached to it — each skill's name, description, and tags — and the content of the most relevant few. The rest are available for the agent to open during the run when it decides it needs them. Two consequences for how you write them: 1. **The description is the routing decision.** It is what the agent reads when choosing, so a vague description means a skill that never gets opened. 2. **Long skills are fine.** Length costs nothing until the skill is actually loaded, so put the detail in rather than compressing it into something ambiguous. > [!NOTE] > > A published version reaches a running agent on its next run. Skills reload live — unlike an agent's > instructions or its tool configuration, a skill change does not need a restart. ## See which skills a run used Open a run and look at its **Skills** card. It lists every skill that was available to the run and marks the ones the agent actually invoked. Over time this is the honest signal about your library: skills that are consistently available but never used are either badly described or not needed. Skills are also graded alongside your agents — **Skills** sorts by rating, and a skill that has not been graded says so. See [Evaluations](/manage-your-agents/optimize/evaluations). ## Delete a skill Deleting is refused while a skill is still attached to an agent — detach it first. Once deleted, the skill and its version history are removed from the library while the runs that used it keep their record of having done so. ## Next steps > **[Knowledge base](/manage-your-agents/build/knowledge-base)** > > Skills tell an agent how to work; the knowledge base tells it what you know. > **[Evaluations](/manage-your-agents/optimize/evaluations)** > > How a skill's contribution gets graded. # Triggers & schedules Source: https://docs.komodor.com/manage-your-agents/build/triggers-schedules Run an agent on a recurring cron schedule from the trigger registry — and where inbound webhook endpoints live instead. A trigger in the Komodor Agentic Operation Platform (KAOP) is a recurring cron schedule that starts an agent run. That is the whole of it, and the distinction matters: if you want an external system to start work by posting to a URL, that is an **endpoint**, it lives under Integrations, and it is not a trigger. > [!WARNING] > > **Triggers are cron schedules.** Inbound webhook endpoints live under **Integrations → > Endpoints (inbound)**. They are a separate surface with separate authentication, and nothing in the > trigger registry manages them. See [Inbound endpoints](#inbound-endpoints-are-not-triggers). ## Where triggers live The trigger registry is account-wide, on the **Schedules** tab of **Fleet**. It lists every schedule in the account rather than only one agent's, which is what makes it useful for answering "what runs overnight?" The registry describes itself the same way: *recurring agent runs on a cron schedule*. ## Create a schedule **1. Open Fleet → Schedules** Press **Create schedule**. **2. Choose the agent** One schedule targets one agent. The agent cannot be changed afterwards — to move a schedule, make a new one and delete the old. **3. Write the cron expression** A standard five-field expression: minute, hour, day of month, month, day of week. **4. Set the timezone** The zone the schedule's wall-clock time is read in. The dialog defaults to your browser's zone; through the API the default is UTC. **5. Give it input** **Input (JSON)** is a JSON object handed to the agent on every fire. Leave it empty and the agent runs with no input. **6. Create it** The schedule appears in the registry, enabled. ### Cron expressions | Expression | When it runs | | - | - | | `0 6 * * *` | Every day at 06:00 | | `*/15 * * * *` | Every fifteen minutes | | `0 9 * * 1` | Mondays at 09:00 | | `0 0 1 * *` | The first of every month, at midnight | An expression KAOP cannot schedule is rejected when you save it, with the reason, rather than failing silently later. > [!NOTE] > > Some cron shapes do not survive a timezone. KAOP runs schedules by converting your local expression > into its UTC equivalent, so an expression with no single UTC equivalent — a date-specific schedule > that would shift across a day boundary, for instance — is refused at save time. Write the schedule in > UTC if you need one of those shapes. ### Timezones and daylight saving Set the timezone to the zone your team thinks in. A schedule written as 09:00 in `Europe/Berlin` is converted to the equivalent UTC schedule, and that conversion is re-derived as the offset changes — so the run stays at 09:00 local across a daylight-saving transition rather than drifting by an hour. ### What the run receives The **Input (JSON)** object is passed to the agent verbatim on every fire. Two shapes are common: **A prompt** For an agent that takes a natural-language instruction, use the `prompt` key: ```json theme={null} { "prompt": "Summarize the last 24 hours of production alerts." } ``` **A structured payload** For an agent that declares an input schema, supply the fields it expects: ```json theme={null} { "environment": "production", "lookback_hours": 24, "dry_run": false } ``` A scheduled run is marked as having come from a schedule, and records which trigger fired it, so you can trace any run back to the schedule responsible. ## Operate a schedule The registry shows each schedule's **Trigger ID**, its agent, its cron expression, when it **last fired**, and whether it is **enabled**, alongside counts of how many schedules are enabled and disabled. | Action | How | | - | - | | Disable one schedule | Turn off its **Enabled** switch. The schedule stays, but stops firing | | Change the cron, timezone, or input | Edit the schedule | | Delete it | From the schedule's actions | ### Two independent switches A schedule fires only when **both** it and its agent are enabled. That gives you two levers with different reach: * **Disable the schedule** to stop one recurring job. * **Disable the agent** to stop *all* of its triggers at once, while leaving manual runs and chat working. See [Manage a deployed agent](/manage-your-agents/optimize/fleet-health). Only one of those two switches is visible on the schedule. If the agent is disabled, its schedules still show as **Enabled** in the registry — they simply stop firing, and **Last fired** stops advancing. An enabled schedule that has not fired recently is the signal to check whether its agent is enabled. > [!NOTE] > > There is no minimum interval. A schedule can run every minute — which also means nothing stops you > from scheduling an expensive agent far more often than you meant to. Check > [Agent spend](/manage-your-agents/optimize/agent-spend-attribution) after adding a frequent schedule. ## Schedules an agent declares for itself An agent can declare its own schedules in its `agent-spec.yaml`, and those register automatically when the worker connects: ```yaml theme={null} triggers: - id: nightly type: schedule name: Nightly summary cron: "0 6 * * *" ``` These appear in the registry marked **YAML-managed** and are read-only there: the switch is disabled and there is no delete control. That is deliberate — the schedule's source of truth is the agent's repository, so change it there and redeploy. An edit made in the console would be overwritten on the agent's next heartbeat. > [!TIP] > > Declare a schedule in the spec when it is part of what the agent *is* — a nightly digest agent that > makes no sense without its nightly run. Create it in the registry when it is an operational choice > you may want to change without a deploy. The registry also lists the Slack channels an agent subscribes to through its spec, read-only, for the same reason. Channel routing itself is configured under [Channels](/manage-your-agents/run/channels). ## Inbound endpoints are not triggers If something outside KAOP should start a run by making an HTTP request, you want an **endpoint**. Endpoints live under **Integrations → Endpoints (inbound)** and are a genuinely different object: they authenticate the sender, they can capture payloads before they route anywhere, and their URL is stable independently of what they point at. | | **Trigger** | **Endpoint** | | - | - | - | | What starts the run | A cron schedule KAOP owns | An HTTP request an external system makes | | Where it is managed | Fleet → Schedules | Integrations → Endpoints (inbound) | | What you configure | Cron, timezone, input | Sender, authentication, field mapping, routing | | Authentication | None needed — KAOP is the caller | A token, a signing secret, or a bearer token | | Input | The JSON you set on the schedule | The request body the sender posts | Creating one walks through the sender and its name, then **Authentication**, then a **Test & mapping** step where you can fire a real request and see what arrives, and finally **Routing** — which agent or workflow it starts. > [!NOTE] > > Routing is the last step and it is optional. An endpoint with no target accepts deliveries and > captures them without running anything, which is the right way to confirm a sender's payload shape > before you point it at an agent. You can attach the target later without the URL changing. > [!WARNING] > > An endpoint's token is shown once, when it is created or rotated. A GitHub endpoint validates every > delivery against a signing secret, so an endpoint created without one rejects everything while > looking configured — set the secret as part of creating it. For KAOP posting events *out* to a URL of yours, see [Outbound webhooks](/manage-your-agents/run/outbound-webhooks). ## Everything else that can start a run Schedules and endpoints are two of several. For the full picture — chat, channels, workflow steps, modules, agent-to-agent invocation, and manual runs — see [Run — how it works](/manage-your-agents/run/how-it-works). ## Next steps > **[Runs & evidence](/manage-your-agents/run/runs-evidence)** > > What a scheduled run leaves behind. > **[Integrations list](/manage-your-agents/build/built-in-integrations)** > > Where endpoints and outbound webhooks live. # Upgrade a self-hosted agent Source: https://docs.komodor.com/manage-your-agents/build/upgrade-a-self-hosted-agent Find a self-hosted agent running an older version, and move it to the newest release with the command for how it was deployed. An agent you run yourself is upgraded by you, with the command that matches how it was deployed. The Komodor Agentic Operation Platform (KAOP) compares the version each live replica reports with the newest one available, and marks the agent **Update available** when it is behind. Agents on Komodor cloud show no version indicator. ## How to tell an agent is behind In **Fleet**, the **Version** column shows the version each self-hosted agent runs, followed by an **Update available** badge when a newer one exists. The agent's own page shows the same version in its header, with **Up to date** or **Update available** beside it. On the agent's page, select **Update available** for the upgrade command for that agent and a link to the matching section below. If no command is shown, use the section that matches how you deployed the agent. What is compared depends on the agent: | Agent | Compared | Newest available | | - | - | - | | Catalog agent | The version of the catalog agent's image | The newest release of that catalog agent | | Any other self-hosted agent | The version of the `komodor-agentops` SDK inside the worker | The SDK version your KAOP control plane ships with | * **Replicas are checked one by one.** If any live replica is behind, the agent shows **Update available**, with how many of its replicas are behind. A replica ahead of the newest release, such as a development build, counts as up to date. * **No badge means nothing to compare.** An offline agent, a draft, or a worker whose version is not a release number (`dev`, a commit hash) shows no badge rather than a guess. The indicator is enabled per account. > [!NOTE] > > Both Helm commands below use the release name you installed with — the agent id, unless you changed > it — and the namespace you installed it into, passed with `-n`. Without `-n`, Helm looks in your > current namespace and fails with `has no deployed releases`. The command on the agent's page fills > in the namespace the agent's production replicas report; when they report different namespaces or > none, it shows `` for you to replace. > > They also use `--reuse-values`, which keeps everything you set at install time, including the > worker token and any model API key — KAOP does not hold a copy of either and cannot put them back > for you. ## Catalog agents A self-hosted catalog agent runs a released image of that agent. The install values the console generates set its image tag to `latest`. A running pod keeps the image it pulled when it started, and a pod that is recreated pulls `latest` again, so replicas can end up on different versions. Upgrade by setting the tag to the new release explicitly. **1. Note the version to upgrade to** Hover **Update available** to see the running and the newest release, for example `0.4.12`. The upgrade command on the agent's page has it filled in already. **2. Upgrade the release** Run this against the cluster the agent is installed in: ```sh theme={null} helm upgrade oci://ghcr.io/komodorio/charts/agentops-agent-base \ -n --reuse-values --set image.tag= ``` **3. Confirm it** The new workers register on their first heartbeat and the badge clears. An agent with several replicas shows how many are still behind until the rollout finishes. > [!WARNING] > > Applying a values file generated by the agent's **Edit** wizard sets the image tag back to `latest`. > Run the command above again afterwards to keep the agent on a fixed release. ## Agents you built in KAOP An agent created with **Build from scratch** and installed in your own cluster runs the KAOP agent base image, and the base image carries the `komodor-agentops` SDK. Its values do not pin an image tag, so the chart's own version decides which base image runs: upgrading the chart upgrades the agent. **1. Upgrade the release** ```sh theme={null} helm upgrade oci://ghcr.io/komodorio/charts/agentops-agent-base \ -n --reuse-values ``` Without `--version`, Helm installs the newest chart, which brings the newest base image. **2. Confirm it** The new workers register with the newer SDK version and the badge clears. > [!NOTE] > > If you set your own image when you built the agent, the chart no longer decides which image runs. > Rebuild that image — on a newer agent base image, or with a newer `komodor-agentops` as described > in the next section — and deploy the new tag. ## Imported agents and your own images An imported agent, or any agent that runs from an image you build, carries whichever `komodor-agentops` version you installed into it. KAOP cannot replace it, so you rebuild and redeploy with your own tooling. **1. Raise the SDK version** Set `komodor-agentops` to the newest release shown on **Update available**, keeping the extra for your framework: ```text requirements.txt theme={null} komodor-agentops[langchain]>= ``` With `pip` directly: ```sh theme={null} pip install --upgrade "komodor-agentops[langchain]>=" ``` **2. Rebuild and redeploy** Build the image again and roll it out the way you normally deploy the agent. Its environment — `AGENTOPS_URL`, `AGENTOPS_AGENT_ID` and `AGENTOPS_WORKER_TOKEN` — stays as it is. **3. Confirm it** The new replicas report the newer SDK version on their first heartbeat and the badge clears. ## Next steps > **[Fleet health](/manage-your-agents/optimize/fleet-health)** > > The SDK version mix across your live workers, and whether each agent stays online. > **[Use specialized agents](/manage-your-agents/build/use-specialized-agents)** > > Deploy a catalog agent, on Komodor cloud or in your own cluster. > **[Import an existing agent](/manage-your-agents/build/import-an-agent)** > > Connect an agent you already run with the `komodor-agentops` wrapper. > **[Troubleshooting](/reference/troubleshooting#an-agent-appears-offline)** > > What to check when an agent does not come back online after an upgrade. # Use specialized agents Source: https://docs.komodor.com/manage-your-agents/build/use-specialized-agents Deploy a Komodor-built specialized agent from the catalog — validated, ready to run, and live in minutes without writing code. The fastest way to get real work out of the Komodor Agentic Operation Platform (KAOP) is to deploy an agent someone has already built and validated. The catalog holds Komodor-built specialized agents — investigators, remediators, orchestrators, evaluators — that you deploy as-is, supply credentials for, and run. ## When to use a catalog agent | Use a catalog agent when | Author your own when | | - | - | | The job is a common operations problem — investigate an alert, trace a cost spike, review a PR | The job depends on your systems, your data, or your runbooks | | You want a working evidence trail today | You need behavior no catalog entry has | | You would rather not run the process yourself | You need the agent to run inside your own network | The two are not exclusive. A common shape is a catalog investigator doing the reading and an agent you wrote doing the part that is specific to you. ## Browse the catalog **Add agent** opens the starting-point picker. On the **Start from catalog** card, press **Browse catalog** — the card itself is not clickable, the button on it is. Browsing opens a grid of cards, one per agent. Each card tells you what you need to know before committing: | On the card | What it tells you | | - | - | | Name, category, and description | What the agent does | | **Integrations** | Which connections and credentials it needs | | **Image** | The image tag you would deploy | | Readiness | Whether *your account* already has what this agent needs. The pill in the card's corner reads **Ready to deploy** when everything is connected, or **Needs setup** when something is missing — the wizard will collect it | Readiness is about your account, not about the agent: the same entry can read **Ready to deploy** for one account and **Needs setup** for another. > [!NOTE] > > The catalog is curated and can differ between accounts, so treat the entries in your own console as > the authoritative list. The [Agent catalog](/manage-your-agents/build/catalog/overview) reference > documents each agent in depth — what it reads, what to connect, its model and time budget, and where > it stops. ### What the catalog covers The catalog is organized by what an agent is for. | Category | What these agents do | | - | - | | **Investigators** | Gather evidence and produce a root-cause conclusion from a specific system — AWS, Azure, Google Cloud, Datadog, Grafana, Kubernetes, GitHub, and combined cluster investigators | | **Remediators** | Take an investigation and propose or carry out a fix, routed for approval | | **Assistants** | Orchestrators that route a problem to the right specialists in your fleet, and run workflow steps across them | | **Code review** | Review a pull request from its diff and report findings by severity, with file and line | | **Cost optimization** | Scan a connected cloud account for waste and attribute spend | | **Quality** and **Evaluators** | Author and run test scenarios, and grade other agents' runs against a rubric | > [!TIP] > > If you are starting from an alert-driven use case, deploy an investigator for whichever observability > system you already trust. It is the entry with the shortest path from deploy to a useful run, because > the evidence it needs is already in a system you have connected. ## Deploy one **1. Pick the entry** Selecting a card opens the wizard on that agent, with a shorter step list than authoring your own — the agent's instructions and behavior are already decided. **2. Name it** On **Agent card**, set the display name and, if you want, labels. The default name is the catalog entry's. A few entries whose scope is narrower than their name suggests also show a notice here that you must tick before **Next** will move — it is your acknowledgement of what that agent can and cannot see, so read it rather than clicking past it. **3. Choose where it runs** Most entries run on Komodor cloud. Some can also be installed in your own cluster, and a few — an in-cluster Kubernetes investigator, for instance — only run there. This step appears only when the entry has more than one option; when it has only one, the wizard skips the step and uses it. **4. Choose the model** The **Model** step asks which model the agent runs on and how it authenticates to it: a model credential already in your account, or one you supply here. Entries whose model is fixed do not show this step. **5. Supply its integrations** The **Integrations** step collects the connections this entry declared it needs, plus any secrets it uses at run time — not the model credential, which you set above. The wizard checks them before anything is created, so a credential that is missing or a connection that is inactive is reported here rather than at the first run. Some entries also let you choose MCP tools on this step. **6. Add triggers** **Triggers** adds a cron schedule if the agent should run on its own. **7. Activate** Nothing is created until this step. On Komodor cloud, press **Deploy to Komodor cloud** and it is usually live within a couple of minutes. For an entry you are installing yourself, this step generates the worker token and install command instead — see [Self-hosted catalog agents](#self-hosted-catalog-agents). Steps appear only when the entry needs them, so two catalog agents rarely have the same list. An entry with no integrations to bind has no **Integrations** step at all. ### What "deploy to Komodor cloud" means Komodor runs the process for you. There is nothing to install, no cluster to prepare, and no worker to keep alive. KAOP provisions the agent, brings it up, and manages its replicas. The agent shows as **deploying** while it comes up, then **online** once it reports in. If provisioning fails you get **deploy failed** with the reason, rather than an agent that silently never appears. > [!NOTE] > > Secrets are not baked into the deployment. The values live in the encrypted credential store and are > delivered to each run — which is why rotating a credential takes effect without redeploying the > agent. ### Self-hosted catalog agents Some entries publish a public image you can run yourself. Choosing self-hosted mints a worker token — shown once — and hands you the command and values to install it. From there it behaves like any agent you run: see [Deploy an agent](/manage-your-agents/build/how-it-works). > [!WARNING] > > Credentials are validated as configuration, not as working credentials. A well-formed but wrong API > key deploys cleanly and fails on the first run — so read that first run rather than assuming a > successful deploy means a working agent. ## What you can change afterwards This is the real trade for the convenience, and it is worth understanding before you deploy. A catalog agent's **behavior is sealed to the version you deployed**. Its instructions and its reasoning are part of the validated image, so there is no Edit for them and no revert — the console does not offer either on a catalog agent. What you get in exchange is an agent whose behavior does not drift and whose runs are comparable over time. | Change | Possible? | Where | | - | - | - | | Its instructions, prompt, or model behavior | No — sealed to the deployed version | — | | Its connections' credentials | Yes — rotate them, no redeploy | The credential store | | Pointing it at a *different* connection | Redeploy from the catalog | — | | Its triggers and schedules | Yes | Fleet → Schedules | | Its labels | Yes | **Labels** card, agent page | | Its permissions | Yes | **Permissions** tab, agent page | | The secrets it may use at run time | Yes | **Used secrets** card, Permissions tab | | Which specialists an orchestrator may delegate to | Yes | **Delegatable workers** card, agent page | | Its version, when you host it yourself | Yes — upgrade its Helm release to the new version | [Upgrade a self-hosted agent](/manage-your-agents/build/upgrade-a-self-hosted-agent#catalog-agents) | | Enable, disable, archive | Yes | Buttons on the agent page | | Tear it down | Yes | **Uninstall** on Komodor cloud; **Delete** for one you host yourself, once it is archived and offline | There is no Edit, so nothing above is behind one — each lives on its own surface on the agent's page. The **Delegatable workers** card behaves as an allowlist: leave it empty and the orchestrator may use any agent that is online; select some and anything left out is refused on every run, and cannot be picked as a specialist in an incident workflow that uses that orchestrator. If you need behavior an entry does not have, author your own agent instead. See [Build from scratch](/manage-your-agents/build/how-it-works). ## After it is live A catalog agent is an agent like any other from here on: it appears in Fleet, produces runs with full evidence, is governed by the same permissions and guardrails, and shows up in spend and quality. > **[Run it once](/manage-your-agents/run/how-it-works)** > > Invoke it manually and read the run before you schedule it. ## Next steps > **[Agent catalog](/manage-your-agents/build/catalog/overview)** > > Every catalog agent in depth — what it reads, its limits, and where it stops. > **[Integrations list](/manage-your-agents/build/built-in-integrations)** > > Connect what a catalog agent needs before you deploy it. > **[Triggers & schedules](/manage-your-agents/build/triggers-schedules)** > > Give it a recurring schedule. > **[Modules & workflows](/modules-and-workflows/overview)** > > Bind specialists to workflow steps instead of running them one at a time. # Agent spend attribution Source: https://docs.komodor.com/manage-your-agents/optimize/agent-spend-attribution What the fleet costs per run, agent, and model — how each figure is derived, and how to cap spend with budgets. **Agent spend** answers what your agents cost, where the money went, and whether it is trending up. The Komodor Agentic Operation Platform (KAOP) builds every figure from the usage each run reports, so a total on this page can always be traced back to specific runs. Figures are reported from the usage each run emits, not sampled or estimated after the fact, so any total can be traced back to the specific runs behind it. Find it under **Improve → Cost**. ## What you see Pick a window of **Last 7 days**, **Last 30 days**, or **Last 90 days** — 30 by default. | Figure | What it reports | | - | - | | **Total spend** | Attributed spend in the window, with its estimated and reported split. | | **vs previous period** | The same figure against the preceding window of equal length. | | **Cost per successful run** | Total spend divided by runs that succeeded — the closest thing to a unit cost. | | **Spend over time** | A daily series, as a line or bars. | | **Spend breakdown** | Attributed cost by dimension, each entry with its share of spend, its run count, and its cost per success. | | **Runs in window** | How many runs the spend is drawn from. | | **Most expensive runs** | The top runs by attributed cost. | The breakdown can be pivoted by **Agent**, **Model**, or **Provider**. > [!NOTE] > > Cost also appears where you are already looking: the account **Overview** shows **Spend (30d)**, > [Fleet health](/manage-your-agents/optimize/fleet-health) shows cost per finished run across the > fleet and window cost for each agent, [Insights](/manage-your-agents/optimize/insights) shows known > cost and the per-model mix, and every run's detail shows its own usage. ## Where the numbers come from Every run carries a **usage** record: the provider, the model, the token breakdown, a cost amount, a currency, and a cost source. The usage is merged from what the worker reports during the run — token events, the run's output, and its metadata. Each run's cost then falls into one of three classes. | Class | How it is derived | | - | - | | **Reported** | The worker supplied a cost amount directly, from its own model client's accounting. A reported cost is authoritative and is never downgraded by a later estimate. | | **Estimated** | No reported cost, but the run named a model and carried token counts — so the control plane prices it from the model's published rates: input tokens, cached input, cache writes, and output tokens each at their own rate. | | **Unattributed** | A cost amount exists but carries no source classification. | Runs with no usage record at all contribute runs but no cost. > [!NOTE] > > **Why the split is shown rather than hidden.** A single "total spend" number silently mixes figures a > provider billed you for with figures KAOP computed from token counts and published rates. Those have > different error bars, and treating them as one is how a cost review ends in an argument. The split is > on the tile so you always know how much of the total is measured and how much is modelled. ### When a worker's own figure is discarded If a run names a provider KAOP does not price directly and the model's rate is resolvable, the worker's own cost figure is discarded and the cost is re-estimated from those rates. That is deliberate: a worker-supplied figure for a provider the platform cannot corroborate is not more trustworthy than a rate-based estimate, and mixing the two conventions within one total is worse than either. Where cache rates are not published for a model, they are derived from its input rate — cached reads at a fraction of it, cache writes at a premium to it. > [!WARNING] > > If no rate can be resolved for a model at all, that run's cost is **left unset** rather than guessed. > It shows up as runs with no cost, which is a data-quality signal about the model, not evidence that > the work was free. ## Making your agents' cost show up Agents built on the KAOP SDK adapters report token usage automatically. If you build a worker yourself, include a usage object in the run's output or metadata. Several field aliases are accepted, so most existing usage shapes work unchanged: | Field | Accepted keys | | - | - | | Model | `model`, `model_id`, `model_name` | | Provider | `provider`, `llm_provider` | | Input tokens | `input_tokens`, `prompt_tokens`, `prompt_token_count`, `input_token_count` | | Output tokens | `output_tokens`, `completion_tokens`, `completion_token_count`, `output_token_count` | | Cost | `cost_amount`, `cost`, `cost_usd`, `total_cost`, plus an optional `currency` | Report a cost amount to get **reported** cost. Report a model plus token counts to get an **estimate**. Report neither and the run contributes volume but no spend. > [!TIP] > > Reporting model and tokens is usually the better choice even when you could compute a cost yourself. > It keeps one pricing convention across the fleet, so agent-to-agent comparisons on the breakdown are > comparisons of usage rather than of two teams' accounting. ## Reading spend through the API The **Spend** view is backed by the account activity endpoint, which you can call directly. ```bash theme={null} # Account totals, daily buckets, per-agent summaries with model mix, # and the top-cost runs for the window GET /api/v1/runs/activity/workspace?window_days=30 # The same summary for a single agent GET /api/v1/runs/activity/agents/{agent_id}?window_days=30 ``` Both return run counts split into succeeded, failed, and active; cost totals split by source; daily buckets; and the per-model mix. `window_days` accepts 1 to 365. ## Budgets A budget caps what an agent may spend in a time window, and optionally stops it when the cap is reached. You set one on the agent itself. Go to **Operate → Fleet**, open the agent, and pick its **Budgets** tab — it offers **Add first budget** until that agent has one. Budgets are per agent, so there is no fleet-wide cap: an agent with no budget is uncapped regardless of what the others have. Every agent's budgets are also listed together on the **Budgets** view, which summarizes them across the fleet and is where you compare them. > [!NOTE] > > Budgets are enabled per account. If the **Budgets** tab is not on your agents, ask your Komodor > contact to turn it on — see [Support](/reference/support). ### Setting one **1. Pick the windows** Under **Spend cap by window**, enable any combination of **Hourly**, **Daily**, **Weekly**, and **Monthly**, each with its own cap amount. Each window is an independent budget that resets on its own clock, all of them in UTC — hourly at the top of the hour, daily at midnight, weekly on Monday, monthly on the first. **2. Set the notification threshold** A percentage of the cap. Crossing it is what makes the budget read **Near cap** rather than **OK**, before it is actually reached. **3. Choose what happens when the cap is hit** **Notify only** or **Block new runs**. ### What "Block new runs" does The behavior is specific, and worth reading before you enable it: * The agent's **schedule, webhook, and channel triggers stop firing** until the window resets. A run that was already queued is cancelled. * A run **already in progress is terminated** when the cap is hit. * **Manual invocation still works.** You can always run the agent yourself from the console. > [!WARNING] > > Blocking is the right default for a runaway-cost guard and the wrong default for anything on your > critical path. A blocked agent stops responding to its triggers entirely, which for an > incident-facing agent means it is absent exactly when it is needed. Start with **Notify only**, and > move to blocking once you know the agent's normal spend shape. ### Reading the Budgets view Three tiles summarize the picture: agents with a budget, the combined cap across all windows, and how many budgets are at or near their cap. The table then lists each budget with its **Cap & window**, **Spend this period**, **Status**, and **Resets in**. | Status | Meaning | | - | - | | **OK** | Below the notification threshold. | | **Near cap** | At or past the threshold, not yet at the cap. | | **Exhausted** | At or past the cap. Blocking budgets are enforcing. | Spend against a budget counts finished runs, so a run still in flight is not yet charged against the window — with one exception: the enforcement check for a blocking budget includes runs currently executing, because a cap you can exceed by starting enough runs at once is not a cap. > [!NOTE] > > A budget cannot be created for an agent that brings its own model key. KAOP does not see that > agent's spend, so a cap it could neither measure nor enforce is refused rather than accepted and > quietly ignored. ## Bringing spend down **1. Find the concentration** Pivot the breakdown by **Agent**, then by **Model**. Spend is almost never evenly spread — one agent or one model usually dominates. **2. Read cost per success, not total** A high-volume agent with a low unit cost is working. A low-volume agent with a high unit cost is the one to look at. **3. Open the most expensive runs** Each one links to its evidence. An outlier run usually explains itself — a retry storm, an oversized context, or a tool loop. **4. Settle any change as an experiment** Swapping to a cheaper model is a quality decision, not a billing one. Prove it against real history before you apply it. ## Next steps > **[Fleet health](/manage-your-agents/optimize/fleet-health)** > > Cost beside reliability, per agent and per window. > **[Runs & evidence](/manage-your-agents/run/runs-evidence)** > > A single run's usage record, and what it did to earn it. > **[Observability & OTel export](/manage-your-agents/optimize/observability-otel-export)** > > Token volume as an operational signal in your own backend. # Evaluations Source: https://docs.komodor.com/manage-your-agents/optimize/evaluations Grade agent runs with LLM judges against explicit rubrics, and read exactly why a score is what it is. A run that succeeded can still be wrong. **Evaluations** is how the Komodor Agentic Operation Platform (KAOP) grades what agents actually produced — an LLM judge reads a run and returns a score per rubric with its reasoning attached. Find it under **Improve → Quality Lab**, which opens on **Runs**. ## The tabs | Tab | What it answers | | - | - | | **Runs** | Every evaluated run, one record each, with every judge's verdict — the surface you work in day to day, and where Quality Lab lands. Its rail filters these **evals** by rubric, evaluator, and agent. | | **Evaluators** | The eval rules, and the judges doing the grading. This is where grading is set up. | | **Overview** | How each rubric is trending, which agents moved, and who did the evaluating. | | **Agent Score Results** | Every agent running a candidate beside production, and how the two compare. See [Model Evaluation](/manage-your-agents/optimize/model-evaluation). | > [!NOTE] > > **Grading is itself agent work.** A judge is an agent, and its grading pass is a real run against > your control plane — with an input, tool calls, and an output. That is why a verdict can carry > reasoning rather than a bare number: the reasoning is something an agent produced and the platform > stored, not a label the platform inferred. ## Setting up grading Grading starts when you create an **eval rule** — the configuration that decides which agents' runs get graded, how often, and by whom. Rules live on the **Evaluators** tab alongside the judges themselves: **New eval rule** opens the form, and each existing rule has **Edit**. **1. Name the rule and choose what to grade** **Single run** grades each matching run on its own — the default. **Run pair (shadow vs. production)** grades a completed comparison pair together and belongs to [Model Evaluation](/manage-your-agents/optimize/model-evaluation). **2. Pick the agents** A rule names the agents whose runs it grades. Nothing outside that list is touched. **3. Set the sample rate** **10%**, **25%**, **50%**, or **100%**, defaulting to 10%. The rate is per rule: the whole panel of judges evaluates the same fraction of matching runs, so their verdicts are comparable. It is the one knob that trades cost for confidence — see [Choosing a rate](#choosing-a-rate) below. **4. Choose the panel of judges** One or several. A **judge** is an ordinary agent that advertises the grading capability, listed on the **Evaluators** tab; each one you select produces its own independent verdict. **5. Add rule instructions** Free text telling the judges what "good" means for these agents specifically — for example, that a weekly report must lead with the total, or that a table missing a share-of-total column is a defect. Optional, and the most under-used field on the form. **6. Activate** The rule starts matching new runs immediately, samples them at the rule's rate, and dispatches one grading task per judge. Edits apply to new matching runs from then on. A rule can be **Active** or **Paused**, and pausing it stops grading without losing anything already recorded. > [!NOTE] > > **What each judge measures lives in the judge, not in the rule.** A rule says whose runs to grade > and how often; the judge decides which rubrics it reports and how it computes them, and the rule's > instructions give it your nuance. That is why two judges on the same rule can return different > rubrics — and why adding a judge that measures something new needs no change to the rule. By default a rule grades runs that **succeeded** — a failed run is not sampled. Grading also keeps a shadow candidate separate from the agent it is being compared against: the candidate's runs advance their own sampling counter, and its verdicts stay out of the production agent's average. A live experiment therefore cannot quietly move the numbers it is being judged against. ### Choosing a rate The rate is the one knob that trades cost for confidence, and both directions have a real failure mode. | Rate | When it fits | | - | - | | **10%** | Steady-state monitoring on a high-volume agent. Enough to move a trend, cheap enough to leave on. | | **25–50%** | An agent you have recently changed, or one whose quality you do not yet trust. | | **100%** | Low-volume, high-stakes work where every output matters — and while you are calibrating a new judge against runs you have already read yourself. | > [!WARNING] > > Grading costs model calls. A panel of three judges at 100% on a busy agent is three extra runs for > every production run, and it shows up in > [Agent spend](/manage-your-agents/optimize/agent-spend-attribution) like any other agent work. > Sample the everyday rules and pay for depth only where a decision hangs on it. ## Reading a verdict A **verdict** is one judge's score on one rubric for one run, with its reasoning — where a **rubric** is the named dimension the score is reported against: `quality`, `precision`, `recall`, and whatever else a judge measures. The score runs from **0 to 1**, shown to two decimals with its rubric name attached — `safety 0.92`. The chip is coloured by how strong the score is, but never labelled with a word for it: calling a number "good" or "poor" is a firmer claim than one score supports. A run graded by several judges has one entry on **Runs** and opens into one review per judge. Open it and each review expands into three parts: | Section | What it holds | | - | - | | **How it was scored** | The judge's own method, in one line — for example, precision and recall at 50 points each, equal weight. | | **What was checked** | The individual criteria, each with an outcome of **pass**, **partial**, **fail**, or **info**, its reasoning, and the evidence behind it. | | **What moved the score** | The adjustments that took the number off full marks — each with its signed points, why it applied, and where. | Below those sits the judge's analysis prose, and a link straight to the run itself so you can read what the agent actually did. > [!WARNING] > > The points under **What moved the score** are in the judge's own scale, not the 0-to-1 score — a > judge marking out of 50 reports them out of 50. Two judges' points are not comparable with each > other, and KAOP never adds them up. ### Two rubrics stay two numbers If a judge measures `precision` and `recall`, you see two scores — never their mean. A blend cannot distinguish a reviewer that invents findings from one that misses them, which is precisely the distinction two rubrics exist to preserve. Where a single headline is genuinely needed, KAOP shows the **lowest** rubric with its name attached, which is the honest summary of "is this run's evaluation good". ### Human grades are kept separate A person can grade a run directly, on a 1-to-5 star scale, and those grades are shown in their own amber styling rather than the score colouring. They are never blended into a judge's average and never move an agent's rubric scores, which are built from LLM verdicts only. They are not discarded, though: human grades roll up into their own **Quality** figure on the agent's page — a star rating out of five, over both runs graded individually and the agent graded directly. So the two live side by side and neither moves the other. > [!NOTE] > > A human grade is one person's spot check; an LLM verdict at a known sample rate is a measurement. > Reporting them as one number would let machine grading quietly inflate or mask what your team > actually thinks — which is why they are kept apart rather than averaged. ## Trending quality over time Pick a window of **7d**, **30d**, or **90d** — 30 by default — and optionally narrow to one rubric, one agent, or one judge. Every filter narrows the whole **Overview** tab together, so the tiles always describe the same slice the chart is drawing. * **Tiles** — **Evaluations** in the window, **Agents evaluated**, **Active evaluators**, and the **Weakest rubric** named alongside its average. Never a cross-rubric mean. * **Score over time** — a daily series per rubric, with volume. A day with no verdict is omitted rather than plotted as zero: a zero would draw a cliff on every quiet weekend that looks identical to a quality collapse. * **Movers** — the agents whose rubric average moved most against the preceding window, split into **Dropped** and **Improved**. A mover needs a baseline on both sides, so an agent first evaluated this window is not reported as the period's biggest improvement. * **Evaluators** — each judge with its volume, its average, and when it first evaluated anything, so a newly added judge is visible as new and its early verdicts can be sanity-checked against the incumbent's. > [!TIP] > > Counts on Overview are of LLM verdicts only. That is deliberate — the tab exists to trend a > measurement, and a human spot check is not one. ### Per-agent quality An agent that has been evaluated carries its own score-over-time series per rubric, plus its **best** and **worst** runs per rubric, each linking to its review. Those two lists are what turn "3% of this agent's output is weak" into "read this specific review". They are computed over the agent's whole history rather than folded from whatever page of results happens to be loaded, so the worst run is genuinely the worst run. The same view appears as an **Evals** tab on the agent's own page in [Fleet health](/manage-your-agents/optimize/fleet-health), shown once the agent has LLM verdicts to plot. ## Next steps > **[Model Evaluation](/manage-your-agents/optimize/model-evaluation)** > > Grade a candidate against production on the same input, with a blinded judge. > **[Runs & evidence](/manage-your-agents/run/runs-evidence)** > > The run a verdict grades, and the trail behind it. > **[Skills](/manage-your-agents/build/skills)** > > Where a weak rubric usually gets fixed. # Fleet health Source: https://docs.komodor.com/manage-your-agents/optimize/fleet-health Read the whole agent fleet at a glance — what is online, what went quiet, what is failing, and what changed. **Fleet** is where you find out whether your agents are actually working, without opening them one at a time. The Komodor Agentic Operation Platform (KAOP) computes presence, capacity, reliability, and version churn from the same run records that back everything else, so the numbers here and the numbers on a run agree. Find it under **Operate → Fleet**. ## The three tabs | Tab | What it answers | | - | - | | **Analytics** | How is the fleet doing over a period — volume, cost, latency, error rate, version churn — and what is its live state right now. | | **Agents** | Which specific agent is in which state, filterable and sortable. This is where Fleet lands when you open it with no tab chosen. | | **Schedules** | Every recurring schedule in the account, when each last fired, and whether it is enabled — the registry is account-wide, not per agent. Creating and operating one is covered in [Triggers & schedules](/manage-your-agents/build/triggers-schedules). | Retired agents sit behind the **Archived (N)** toggle in the Agents toolbar. Archiving keeps an agent's history — its runs and evidence stay readable. ## Agent state, precisely An agent's state is computed from presence, not declared. The control plane pings each open worker connection; the worker's reply is the only thing that updates presence. If replies stop arriving for **90 seconds**, the agent reads as offline. | Status | What it means | | - | - | | **Online** | The worker replied inside the presence window. | | **Offline** | It did not. The agent still exists, its history is intact, and it comes back on the next reply. | | **Draft** | The agent has never reported in. A draft goes live automatically on its worker's first contact — there is no separate activation step. | | **Deploying** | A first deploy is in flight. | | **Deploy Failed** | A first deploy did not complete. | Several qualifiers can sit beside the status, because they are independent of presence: * **Disabled** — triggers will not fire for this agent. Manual and chat invocations still work. * **Archiving** / **Archived** — retiring, and retired. An agent reads *Archiving* until its last replica is genuinely gone, then *Archived*. * **New version deploying**, **Restarting**, **Reverting** — a rollout on an agent that is already live, so its status stays *Online* while the badge shows what is happening. * **Deploy failed** — a rollout onto an already-live agent failed. The previous version keeps serving. > [!NOTE] > > "Offline" and "unreachable" are different facts. Offline means presence lapsed. An agent can be > heartbeat-online while its message channel is not reachable — Analytics calls that out > separately, because the fix is different. ## The Analytics tab ### Choosing a period The window selector offers **Today**, **Last 7 days**, and **Last 30 days**, and defaults to 30 days. It is calendar-day inclusive: *Today* means today so far, not a rolling 24 hours. Days are **UTC** days, both for the window and for the daily grouping inside it — so if you are far from UTC, *Today* will not line up with your own. The selector governs this tab's period figures only. ### The four headline figures | Card | What it reports | | - | - | | **Outcomes & reliability** | Finished runs in the window, split into succeeded and failed. | | **Cost** | Spend in the window, and cost per finished run. Links through to spend detail. | | **Latency (mean)** | Mean completed-run duration. | | **Changes** | How many new agent versions were deployed in the window. | > [!TIP] > > **Changes** is the figure most teams under-use. A reliability dip that lines up with a spike in > deployed versions is a rollout problem, not an agent problem — and that correlation is one glance > here rather than an investigation. ### Trends and leaderboards Three charts plot the window day by day: **Runs & errors over time**, **Cost over time**, and **Latency over time**. Splitting runs from errors on separate rows is deliberate — an outage day stands out without hiding the throughput underneath it. Three panels rank what needs attention: * **Worst error rate** — agents ordered by failure rate, with the run count behind each rank so a rate comes with its volume. A minimum-run floor applies to the denominator, so a single bad run on a barely used agent cannot top the board. On a short window the floor can legitimately exclude everything, and the panel says so rather than implying a healthy fleet. * **Slowest agents** — ordered by mean duration, with no floor. A slow agent is worth seeing on its first few runs. * **Model mix** — which models the fleet actually used, sized by run volume and labelled with spend. Two entries in the model mix are structural rather than models a worker reported: | Entry | What it is | | - | - | | `(other)` | Everything past the top models, folded into one row that carries the tail's runs and cost — so the slices still sum to the fleet totals. | | `(unattributed)` | Runs whose usage record names no model at all. This is a signal about your workers' reporting, not about the fleet's model choice. | ### Live state Three cards describe right now rather than the window, and refresh on their own. > **Fleet composition** > > Four exhaustive buckets — **online**, **offline \<24h**, **offline >24h**, and **draft** — > each linking into the filtered agent list. Splitting recent from long-standing offline matters: > a worker that stopped an hour ago is an incident, and one that stopped last month is a cleanup > task. > > A **Just went offline (last hour)** call-out names the agents that changed state most recently, > which is almost always what you came to find. > **Capacity & backlog** > > Live **replicas**, how many are **busy** versus free, how many runs are **queued**, and the age > of the **oldest queued** run. Oldest-queued is the number to watch: a growing backlog with idle > replicas is a routing problem, and a growing backlog with no free replicas is a capacity one. > > An **Online, 0 live replicas** call-out flags an agent that reads online with nothing serving — > which happens when a candidate arm keeps the agent present while its production arm is dead. > **Reachability & version mix** > > Which online agents cannot currently be reached over their channel, headed > **Heartbeat-online, channel-unreachable**, plus the **SDK version mix** across live workers. > The version mix is how you find the worker that never got upgraded; > [Upgrade a self-hosted agent](/manage-your-agents/build/upgrade-a-self-hosted-agent) covers moving > it to the newest release. > [!NOTE] > > Live figures and period figures refresh on different cadences — presence is polled every 10 seconds > and the window metrics every 60 — so a period number beside a presence row can be up to a minute > older than it. ## The Agents tab ### The list | Column | What it shows | | - | - | | **Name** | The agent, with badges for how it is hosted and run. | | **Status** | The status and its qualifiers, as above. | | **Last run** | The most recent run's outcome and when it happened. | | **Last change** | When the agent's definition last changed. | | **Instances** | Live replicas, split into busy and free. When a candidate arm is live, a second line reads `N production · M shadow`. | | **Activity (30d)** | Four figures in one cell — runs and cost on the first line, error rate and mean latency on the second. This column is pinned to 30 days regardless of the Analytics selector. | Click any sortable header to rank by it. Because the activity cell holds four numbers, its header offers a **Sort by** choice of **Runs**, **Cost**, **Error rate**, or **Avg latency** — so the ranking always names what it ranked by. Error-rate sorting suppresses agents with too few runs in the window, for the same reason the leaderboard has a floor. > [!WARNING] > > An em dash in an activity cell means "not measured", which is not the same as zero. An agent created > in the last minute has no measurement yet; a route that does not request window metrics shows none > at all. ### Narrowing the fleet * **Search** matches an agent's name, id, worker id, description, tools, and labels. * **Status** facets: **All agents**, **Online**, **Offline**, and **Draft**. Draft appears only once you have a draft agent — an always-visible `Draft 0` would be noise on the accounts that never have one — and stays up while it is selected even if another facet narrows its count to nothing. * **Shadow**: **Shadow active** filters to agents currently running a candidate arm — see [Model Evaluation](/manage-your-agents/optimize/model-evaluation). * **Labels**: one facet group per label key your agents declare, so `team`, `env`, or `tier` become filters without anyone configuring them. * **Clear filters** returns you to the whole fleet. Use it when you arrive from a composition card's link — those can select combinations the facet buttons cannot express, which leaves the buttons reading inactive. > [!NOTE] > > Search scopes the Agents tab only. Analytics deliberately ignores it, so that a query typed on > one tab cannot silently narrow numbers on another with no visible control to clear it. ## An individual agent Opening an agent gives you the same picture for one agent, plus what only it can tell you: * **Run history** and **Run activity** — its runs, and its own outcome, cost, latency, error, and change trends. * **Instances** — every live replica, including which arm each belongs to. * **Versions** — its deployed versions, with invocations split into production and shadow. * **Evals** — its judge verdicts and score trend, when it has been evaluated. See [Evaluations](/manage-your-agents/optimize/evaluations). * **Observability** — where its time went, per run. * **Budgets** — its spend caps, when budgets are enabled for your account. See [Agent spend attribution](/manage-your-agents/optimize/agent-spend-attribution). * **Skills & tools**, **Metadata**, **Permissions** — what it knows, what it is, and what it may do. The header carries the lifecycle actions: **Disable** / **Enable**, **Archive** / **Unarchive**, **Run agent**, **Chat**, **Edit**, and **Revert**. Disabling stops triggers from firing while leaving manual and chat invocation working, which makes it the right way to quiet a misbehaving agent without deleting anything. > [!TIP] > > Metadata carries **Last heartbeat**. When an agent reads offline and you want to know whether it > just went quiet or has been gone for a week, that is the field to read — before you go looking at > the worker. ## Reading the fleet in practice **1. Start on Analytics at 30 days** Look at Outcomes & reliability and Changes together. A dip that coincides with version churn points at a rollout. **2. Check live state before period state** Fleet composition and the just-went-offline call-out tell you whether you are looking at a current outage or at history. **3. Name the agents** Use Worst error rate and Slowest agents to get from "the fleet is degraded" to two agent names. **4. Switch to Agents and sort by error rate** Confirm the ranking against volume, then open the worst agent. **5. Read one failing run** The run's evidence says what actually happened. If the same failure repeats, [Insights](/manage-your-agents/optimize/insights) has already grouped it. ## Next steps > **[Runs & evidence](/manage-your-agents/run/runs-evidence)** > > What a single run records, and how to read it. > **[Agent spend attribution](/manage-your-agents/optimize/agent-spend-attribution)** > > Where the Cost figure comes from and how it is attributed. # Optimize — how it works Source: https://docs.komodor.com/manage-your-agents/optimize/how-it-works The four lenses on a running fleet — reliability, quality, cost, and portability — and which surface answers each. Building and running agents gets work done; **Optimize** is how you find out whether it was done well. The Komodor Agentic Operation Platform (KAOP) measures a running fleet through four independent lenses, each answering a question the others cannot. The console groups most of these under **Improve** — Insights, Quality Lab, and Cost. Fleet health is the exception: it sits under **Operate → Fleet**. **Quality Lab** carries both quality surfaces — Evaluations on its **Runs** tab, Model Evaluation on **Agent Score Results**. ## Four lenses, four questions An agent can be healthy and wrong, correct and ruinously expensive, or cheap and invisible outside KAOP. Those are four separate failure modes, so they get four separate readings. | Lens | The question | Where you answer it | | - | - | - | | **Reliability** | Is it running, and is it failing? | [Fleet health](/manage-your-agents/optimize/fleet-health) · [Insights](/manage-your-agents/optimize/insights) | | **Quality** | Was the output actually good? | [Evaluations](/manage-your-agents/optimize/evaluations) · [Model Evaluation](/manage-your-agents/optimize/model-evaluation) | | **Cost** | What did it cost, and is that worth it? | [Agent spend attribution](/manage-your-agents/optimize/agent-spend-attribution) | | **Portability** | Can the rest of my tooling see this? | [Observability & OTel export](/manage-your-agents/optimize/observability-otel-export) · Knowledge graph | > [!NOTE] > > **Reliability and quality are the pair worth internalizing.** A failing run announces itself — it has > a status, a category, and a place to look. A wrong run does not: it succeeds, costs what it always > costs, and leaves no operational trace at all. Reliability tooling will never find it. That gap is > why quality gets its own instruments rather than being folded into fleet monitoring. ## Reliability Reliability starts from presence and run outcomes, and both are computed rather than declared. **Fleet health** gives you the fleet's live state — who is online, who has capacity, what is queued — alongside period figures for volume, cost, latency, error rate, and how many agent versions were deployed. [**Insights**](/manage-your-agents/optimize/insights) takes the failures and does the grouping for you. Failing runs are categorized from structural proof — the run's status, the shape of its span tree, the agent's presence — never by guessing at error text, and identical failures collapse into one ranked issue with its verbatim reason and an agent-produced recommendation. ## Quality Quality is graded, not observed. **Evaluations** puts LLM judges on a sampled share of runs and records a score per rubric with the reasoning that produced it — so a low number comes with the criteria that were checked and what moved it off full marks. **Model Evaluation** answers the comparative question. Deploy a candidate beside production, let both run every real request, and have a blinded judge say which output was better — which is far more sensitive to a small real difference than diffing two absolute scores. ## Cost **Agent spend attribution** builds every figure from what each run reported, and keeps the derivation visible: cost is **reported** by the worker, **estimated** from model rates and token counts, or **unattributed**. Spend pivots by label, agent, model, provider, MCP server, or tool, and budgets cap what an agent may spend per window — with the option to stop it when the cap is reached. ## Portability Everything above lives in KAOP. **Observability & OTel export** is what makes it useful outside it, emitting agent activity as standard OpenTelemetry traces, logs, and metrics to your own backend — so an agent's tool call sits in the same trace as the service it touched. ## The order to read them in **1. Is the fleet up?** Fleet health's live cards. A capacity or presence problem makes every other reading unreliable. **2. What is failing, and is it new?** [Insights](/manage-your-agents/optimize/insights). Grouped, ranked, and already categorized — with the run evidence one click away, and a recommendation produced by an agent that investigated your account. **3. Of what succeeded, what was any good?** Evaluations. This is the reading nothing else substitutes for. **4. What did all of it cost?** Agent spend. Read cost per successful run before total spend — volume flatters a total. **5. Then change something, and prove it** Model Evaluation. Run the candidate beside production on live traffic and let a blinded judge settle it before anything ships. ## The pages > **[Fleet health](/manage-your-agents/optimize/fleet-health)** > > Presence, capacity, backlog, reliability, and version churn across the whole fleet. > **[Insights](/manage-your-agents/optimize/insights)** > > Failures grouped and ranked, each with its verbatim reason and an agent-produced next step. > **[Evaluations](/manage-your-agents/optimize/evaluations)** > > LLM judges grading runs against explicit rubrics, with the reasoning behind every score. > **[Model Evaluation](/manage-your-agents/optimize/model-evaluation)** > > A candidate beside production on real traffic, settled by a blinded judge. > **[Agent spend attribution](/manage-your-agents/optimize/agent-spend-attribution)** > > What the fleet costs per run, agent, and model — and budgets that cap it. > **[Observability & OTel export](/manage-your-agents/optimize/observability-otel-export)** > > Agent traces, logs, and metrics in your own OpenTelemetry backend. ## Next steps > **[Runs & evidence](/manage-your-agents/run/runs-evidence)** > > The record every reading on these pages is computed from. > **[Build — how it works](/manage-your-agents/build/how-it-works)** > > Where a fix lands once you know what to change. > **[Troubleshooting](/reference/troubleshooting)** > > The fixes for the failure categories Insights reports. > **[Limits & quotas](/reference/limits-quotas)** > > The caps that shape what a fleet can do. # Insights Source: https://docs.komodor.com/manage-your-agents/optimize/insights Failures across your agents, grouped and ranked, each with the evidence behind it and an agent-produced next step. **Insights** answers the question you have right after agents start failing: what is failing, why, and what to do about it. The Komodor Agentic Operation Platform (KAOP) groups related failures into ranked issues, shows the reason each one reported verbatim, and attaches a recommendation produced by a real agent investigating your account. Find it under **Improve → Insights**. ## What the page tells you The header figures cover the last 30 days of account activity: | Card | What it reports | | - | - | | **30-day runs** | Total runs, with the succeeded count beneath. | | **Failures** | Failed runs, broken down by the top failure categories. | | **Success Rate** | Outcome rate over the last 100 terminal runs. | | **Known Cost** | Usage that carried a cost figure. Full attribution lives on [Agent spend](/manage-your-agents/optimize/agent-spend-attribution). | Below that, a 30-day activity heatmap plots run volume and failures day by day, so a bad day is a visible block rather than an averaged-away dip. Five charts break the same window down further: * **Run trend** — recent volume. * **Run status** — the current outcome distribution across `Queued`, `Claimed`, `Running`, `Succeeded`, `Failed`, and `Cancelled`. * **Agent activity** — runs by agent, each row split into succeeded, failed, and active. * **Model breakdown** — the run mix by model. * **Failure causes** — failures by category, using the categories below. > [!NOTE] > > Insights is fixed to a 30-day window. If you need a shorter or longer period for reliability, the > [Fleet health](/manage-your-agents/optimize/fleet-health) Analytics tab offers Today, 7 days, and 30 > days. ## How failures are categorized This is the part worth understanding, because it is what makes the rest trustworthy. KAOP segments a failing or stuck run **only from structural signals it can prove** — the run's status, the shape of its span tree, and the agent's presence. It never guesses a cause by matching substrings in an error message. | Category | The structural proof behind it | | - | - | | **Agent run failed** | The run failed *with* an agent span that ended in error. The handler ran and raised. | | **Worker unreachable** | The run failed with *no agent span at all*. The agent never started. | | **Tool call failed** | A tool span ended with an error status. | | **Stuck unclaimed** | The run sat queued past its lease window with no worker claiming it. | | **Worker offline** | Work is queued or claimed while the agent's presence is down. | | **Unclassified failure** | The run failed with no structural proof of a cause. | > [!NOTE] > > **Why prose matching is refused.** An error string is written by whatever failed, in whatever > wording it chose, and it changes without warning. A classifier built on it produces confident labels > that are sometimes wrong, and a wrong label is worse than no label — you go and check the thing it > named instead of the thing that broke. So a run KAOP cannot prove stays **Unclassified failure**, > with its real error text attached, rather than being forced into a plausible bucket. ### The reason is shown verbatim Instead of re-labelling what a run reported, KAOP shows the failure reason exactly as it arrived — `Missing Datadog credentials. Set DD_API_KEY and DD_APP_KEY…` reads as that sentence, not as "configuration error". The category tells you *where* to look; the verbatim reason tells you *what* happened. Grouping uses the category together with the normalized reason, so identical failures collapse into one issue instead of a hundred rows. ## Needs attention The **Needs attention** panel is the ranked list of issues — "ranked failure clusters with an agent-backed recommended fix". Each card carries: * The **category** badge and a **confidence** badge (**High**, **Medium**, or **Low confidence**). * A **Growing** marker when the issue is getting worse rather than sitting still. * The issue title and why it happened. * The scale: how many runs, how many agents, and when it was last seen. * The recommendation, once produced. Ranking is by severity first, then by how many runs an issue affects. Severity is derived, not assigned: an issue is high severity when it is growing or affects at least five runs, medium when it was last seen inside the window, and low otherwise. High-severity issues are treated as incidents. An issue carries a lifecycle you can read on it — **open**, **investigating**, or **resolved** — which moves as an investigation starts and finishes. ## Recommendations come from a real agent run A recommendation is not a lookup in a table of canned advice. When you ask for one, KAOP starts a run on a chat-capable agent, which investigates the issue **using your account's own tools** and returns two things: a one- or two-sentence tagline for the card, and a full root-cause write-up behind **Details**. **1. Pick who recommends** The **Recommendations by** control selects which chat-capable agent investigates. Your choice is remembered, so the agent that knows your environment best stays selected. **2. Ask for a recommendation** Press **Recommend**. The card shows *Investigating…* while the agent works. **3. Read the tagline, then the detail** The tagline is the answer in one line. **Details** expands the full root-cause analysis, with the evidence the agent used. **4. Continue interactively** **Chat** — or **Continue in chat** from inside the details — takes you into the same session, prefilled with the failure context. You are continuing the investigation, not restarting it. > [!TIP] > > Because a recommendation is a run, it has an evidence trail like any other run. If a recommendation > looks wrong, you can read exactly which tools the investigating agent called and what they returned > — which is the difference between an opinion and a traceable conclusion. ## Acting on each category The category tells you where the fix lives, and each issue links to the right place. | Category | Where to go | | - | - | | **Agent run failed** | The failing run's evidence. The spans and logs show what the agent was doing when it raised. | | **Worker unreachable** | [Fleet health](/manage-your-agents/optimize/fleet-health) — confirm the worker is running and can reach the control plane. | | **Tool call failed** | The integration behind the failing tool — check its connection and health. | | **Stuck unclaimed** | Fleet health — usually there is no online worker for that agent. | | **Worker offline** | Fleet health — bring the fleet back or scale it up. | | **Unclassified failure** | The run's evidence. The verbatim reason and the logs are the whole lead. | > [!WARNING] > > KAOP recommends and never remediates on its own here. There are no automatic retries, restarts, > credential rotations, or scale-ups behind an issue. Every fix is yours to make, which is why the > recommendation names the place to make it. ## Where else issues appear The account **Overview** carries a compact form of the same panel — the top reliability issues from the last 30 days, with a **View insights** link. It is deliberately read-only: it never starts a recommendation run, so opening your home page costs nothing. ## Next steps > **[Fleet health](/manage-your-agents/optimize/fleet-health)** > > Live presence, capacity, and the per-agent reliability ranking. > **[Runs & evidence](/manage-your-agents/run/runs-evidence)** > > The span tree the categories are proved from. > **[Troubleshooting](/reference/troubleshooting)** > > The fixes for each of these categories, step by step. > **[Evaluations](/manage-your-agents/optimize/evaluations)** > > A run that succeeded can still be wrong. This is how you find out. # Model Evaluation Source: https://docs.komodor.com/manage-your-agents/optimize/model-evaluation Run a candidate agent or model beside production on real traffic and let a blinded judge say which one is better. Changing an agent's model or prompt can regress quality without moving a single operational metric — the run still succeeds, latency holds, and only the answers get worse. **Model Evaluation** in the Komodor Agentic Operation Platform (KAOP) answers the question directly: run the candidate beside production on the same real inputs, and have a blinded judge say which output was better. Find it under **Improve → Quality Lab → Agent Score Results**. ## The shape of it ```mermaid theme={null} flowchart LR Req["A real request"] --> CP["Control plane"] CP -->|production run| P["Production arm"] CP -->|shadow run| S["Candidate arm"] P -->|answer| Caller["The caller"] S -.->|answer discarded| X["Nothing"] P --> Pair["Comparison pair"] S --> Pair Pair --> Judge["Blinded judge"] Judge --> Verdict["Winner per rubric"] ``` Every queued request runs on **both** arms. Callers only ever receive the production answer. The two executions are first-class runs, linked as a **comparison pair**, and that pair is what a judge grades. ## Deploying a candidate A candidate is a second deployment of the **same agent**, and KAOP walks you through setting one up. **1. Open the agent and edit it** From **Operate → Fleet**, open the agent you want to test and choose **Edit**. Change whatever the candidate should do differently — its model, its instructions, its tools. **2. On the Review step, choose Deploy as a shadow** The last step of the wizard offers two outcomes. **Update the main agent** saves the change to the agent you already run. **Deploy as a shadow** leaves that agent untouched and sets the change up as a candidate instead. **3. Deploy the candidate with the environment you are given** KAOP cannot reach your infrastructure, so it hands you what only it can provide: a worker token minted for the shadow alone, and the declaration the SDK needs. Run your candidate image as a **second** deployment alongside the one you already run, with that environment. ```bash theme={null} AGENTOPS_VARIANT_ROLE=secondary ``` **The token is shown once.** It belongs to the shadow alone — the main agent's token is untouched, so bringing the candidate up or taking it down never rotates the credential your production deployment is using. Both deployments carry the same agent id, and that shared id is what pairs the two arms. Absence of the `AGENTOPS_VARIANT_ROLE` declaration means production, so nothing changes on the serving side. > [!WARNING] > > An unrecognized value for `AGENTOPS_VARIANT_ROLE` fails the worker's boot, and a replica presenting > the candidate's token while declaring no arm is refused outright. Both produce a loud, harmless > candidate crash loop rather than a silent promotion into the serving fleet — which is the failure > this guard exists to prevent. Because both arms share one agent id, [Fleet health](/manage-your-agents/optimize/fleet-health) shows one row per agent. A live candidate makes itself visible there: * The **Instances** cell splits into `N production · M shadow`. * A **Shadow active** facet filters the whole fleet to agents currently running one. * The agent's **Remove shadow** action ends the experiment. > [!NOTE] > > **Use the token the wizard minted, not a copy of production's.** The guard above keys on the > credential: a replica on the shadow's token that declares no arm is refused, but one on > **production's** token is a production pod whatever else it declares — it answers real callers, and > nothing catches it. A candidate also has to run the Python SDK, since that is what reads > `AGENTOPS_VARIANT_ROLE`. ## What the candidate cannot do The whole design rests on the candidate being unable to affect anything. Specifically: | Guarantee | How it holds | | - | - | | **Callers never hear it** | Every synchronous caller awaits the production run only. | | **It never serves real traffic** | A candidate replica is handed shadow runs only; a production replica is handed ordinary and production runs only. The backlog that existed before the candidate is drained by production, as before. | | **It causes no side effects** | A shadow run publishes no completion event and no channel reply — so no incident is touched, no review recomputed, no second Slack message posted, no onboarding step ticked. | | **It cannot corrupt conversation state** | It reads the production conversation's snapshot at claim time, so the comparison is on identical history, then writes to its own namespaced session. | | **It cannot flap the agent's definition** | Its presence proves it is alive but never steers the shared agent — card, inputs, labels, skills, and schedules stay production-authored. | | **It cannot slow production down** | The candidate is best-effort and fully isolated; it cannot fail, delay, or overwrite anything the production path causes. | The single deliberate exception is grading. Grading a candidate is the entire point, so a shadow run does publish for grading — with the carve-out that a shadow *judge* run never publishes, so a candidate judge cannot overwrite the production judge's verdict. ## Attaching a judge Fan-out gives you pairs; grading them is an eval rule away. Create a rule in [Evaluations](/manage-your-agents/optimize/evaluations) and set **What to grade** to **Run pair (shadow vs. production)**. Rules configured that way carry a **Run pair** badge in the rules list. A pair rule fires on a *completed pair* rather than on a single terminal run, and both arms are selected together or not at all — sampling them independently would produce half-pairs, which cannot answer a comparison question. ### Why the judge is blinded The judge is handed the two outputs as **Candidate 1** and **Candidate 2**, in randomized order, with nothing in its context naming which arm is which. It sees each side's input, output, status, error, and tool trace, and nothing that hints at role. > [!NOTE] > > **Unblinded pairwise comparison is not offered, on purpose.** A judge told which output came from > production will favour it — and the resulting number looks exactly like a trustworthy one. Blinding > is the only thing that makes a small preference meaningful, so it is not a setting you can turn off. The platform stores the blinding assignment with the grade, and resolves the winner back to production or candidate only after the verdict lands. ### Why a pairwise verdict beats two separate scores Both instruments exist and neither replaces the other. | | Grading each arm separately | Grading the pair together | | - | - | - | | Judgments | Two, independent | One | | The judge sees | One output | Both outputs | | Produces | Absolute scores per rubric | A winner per rubric, with reasoning | | Small differences | Lost in judge variance | Detectable | | Bias risk | The judge's internal scale drifts between calls | Controlled by blinding | | Cost | Two grading runs | One grading run | An absolute score is only as stable as the judge's internal scale, and that scale wobbles between independent calls: a `0.72` beside a `0.68` cannot tell you whether the candidate is worse or the judge was stingier the second time. A pairwise verdict deletes the scale — the judge only says which of two things in front of it is better, which is an easier judgment and far more sensitive to a small real difference. The converse matters just as much: **a pairwise winner from two bad answers is still a bad answer.** Keep your per-arm rubric rules running alongside the pair rule. They tell you the absolute bar is met; the pair tells you which version to ship. ## Reading the result Open **Agent Score Results** for the list of agents that have been compared, then an agent for its shadow-comparison history. The same report also appears under **Operate → Fleet → agent → Versions**, against the version that produced it — the better entry point when you are asking "was this change any good" rather than "how are my candidates doing". ### A/B Test Results Each row is one judge's verdicts under one eval rule against one agent's candidate: | What the row carries | Meaning | | - | - | | **Result** | **Production wins**, **Shadow wins**, or **Tie**, folded from the individual verdicts. | | **Comparisons** | How many distinct pairs this judge graded. | | **Wins and ties** | Production wins, shadow wins, and ties, counted. | | **Per-rubric tallies** | The same counts held separately per rubric, never blended across rubrics. | Expanding a row lists every graded pair with, for each rubric, the result, both arms' scores, and the tie band that produced the result. A **tie** therefore reads as "within the band" rather than as a coin flip — the two scores are right there. > [!WARNING] > > The row-level result is folded by **counting labels**, never by averaging scores, and there is > deliberately no percentage delta on it. "The candidate won 61% of 240 pairs" is a countable claim. > "The candidate is 6% better" is not, at any sample size the platform can vouch for. A low total > count is your own signal to keep collecting. Two charts summarize the same data: **Win distribution** (how often each arm scored higher) and **Score distribution** (both arms' scores over time). ### Cost, tokens, and latency per arm Beside the verdicts sits each arm's rolled-up usage — cost, token breakdown, and mean latency — with the delta expressed as production minus shadow, so a green figure means the candidate used less. Every figure carries its own **sample size**: how many of that arm's runs actually contributed. This matters more than it sounds. A run can carry no usage record at all, and pricing is not resolvable for every model, so a missing cost is reported as unknown rather than as zero — "cost from 3 of 17 runs" instead of a blended figure that reads as "this arm is free". Token counts are kept disjoint for the same reason: fresh input tokens sit beside cached and cache-write tokens rather than being summed into them, because blending fresh with cached input misprices a cache-heavy agent by roughly an order of magnitude. ## Deciding The comparison gives you three independent readings — preference, absolute quality, and marginal cost — and the decision is yours to make and execute. **1. Wait for volume** A handful of pairs is an anecdote. Watch the comparison count, not the win rate, until the counts are worth reading. **2. Read preference per rubric** A candidate that wins on one rubric and loses on another has told you something specific. Do not average it away. **3. Check the absolute bar** Confirm the candidate's own per-arm rubric scores clear the level you require. A preference win between two weak answers is not a release. **4. Price it** Read the per-arm cost and latency totals, with their sample sizes. A candidate that wins narrowly and costs twice as much is a different decision from one that wins narrowly and costs a tenth. **5. Promote deliberately** Edit the agent again, make the change the candidate was carrying, and take **Update the main agent** on the Review step this time. For an agent KAOP deployed, that saves the config and gives you an updated `values.yaml` and a `helm upgrade` command to run against its existing release. For an agent you deploy yourself, it saves the config and what reaches the running agent is down to how you ship it. **6. Then tear down** **Remove shadow** on the agent ends the experiment, and you delete the candidate deployment yourself — KAOP cannot reach your infrastructure. Removing the shadow is also what lets an ordinary undeclared replica of that agent register normally again. > [!NOTE] > > No result ships itself. Nothing here auto-promotes, auto-blocks a release, or writes into your > serving path — the platform produces the evidence and a human makes the change. Every promotion is > therefore recorded as an ordinary agent change and shows up in the **Changes** figure on > [Fleet health](/manage-your-agents/optimize/fleet-health). ## Next steps > **[Evaluations](/manage-your-agents/optimize/evaluations)** > > Eval rules, judges, rubrics, and how a verdict is read. > **[Fleet health](/manage-your-agents/optimize/fleet-health)** > > Spotting a live candidate, and removing it. > **[Runs & evidence](/manage-your-agents/run/runs-evidence)** > > Both arms of a pair, side by side, with the pair's own id. # Observability & OTel export Source: https://docs.komodor.com/manage-your-agents/optimize/observability-otel-export Export agent traces, logs, and metrics over OpenTelemetry so agent activity sits beside the rest of your systems. Every run in the Komodor Agentic Operation Platform (KAOP) already keeps its own durable evidence, but that evidence lives in KAOP. Export sends the same activity to your own backend as standard OpenTelemetry traces, logs and metrics, so agent work appears alongside the rest of your systems rather than only in a separate console. ## Two places the same activity shows up | Where | What you get | Who reads it | | - | - | - | | **In KAOP** | The run's full evidence trail — spans, logs, progress messages, output, artifacts, token usage — plus the fleet and spend rollups built on it. | Anyone operating agents. Nothing to configure. | | **Your OTel backend** | The same agent activity as OTLP traces, logs, and metrics, correlated with the rest of your infrastructure. | Your existing observability practice — the dashboards, alerts, and on-call rotations you already run. | Export does not replace the in-product evidence. It puts agent behavior on the same screen as the services those agents operate on, so an agent's tool call and the upstream latency spike it caused sit in one trace. ## Enabling export Instrumentation ships with both the control plane and every worker built on the Python SDK, and it does nothing until you point it at a collector. There is no KAOP-specific configuration format — it reads the standard OpenTelemetry environment variables, so it works with any OTLP-compatible backend and there is nothing to unlearn. **1. Point the worker at a collector** Set `OTEL_EXPORTER_OTLP_ENDPOINT` on the worker's process or container. Setting it is what turns export on. ```bash theme={null} OTEL_EXPORTER_OTLP_ENDPOINT=https://otlp.example.internal:4318 OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf ``` **2. Add whatever your backend needs** Most managed backends authenticate with a header. ```bash theme={null} OTEL_EXPORTER_OTLP_HEADERS=api-key= ``` **3. Name the service, if the default is not what you want** A worker defaults its own service name to `agentops-worker-`, derived from the agent's name. Set `OTEL_SERVICE_NAME` to override it. **4. Restart the worker and look for the first trace** Trigger a run. The trace appears in your backend under the service name above, rooted at the run's own span. | Variable | What it does | | - | - | | `OTEL_EXPORTER_OTLP_ENDPOINT` | The collector or backend to send to. Setting it enables export. | | `AGENTOPS_OTEL_ENABLED` | Enables export when you configure the endpoint some other way. | | `OTEL_EXPORTER_OTLP_PROTOCOL` | The transport — `grpc` or `http/protobuf`. | | `OTEL_SERVICE_NAME` | How the process appears in your traces. | | `OTEL_EXPORTER_OTLP_HEADERS` | Headers your backend requires, such as an API key. | Other standard `OTEL_*` variables are honored as well. > [!NOTE] > > Export is off until an endpoint is set, and nothing leaves your worker before then. A worker run > from a laptop with no OTel configuration pays nothing for the instrumentation being present. ## What a worker emits ### Spans A run produces a span tree you can read as the agent's decision path. | Span | What it covers | | - | - | | `agentops.run` | The whole run. Carries `agentops.agent.id`, `agentops.agent.name`, and `agentops.run.id`. | | `agentops.llm.iteration` | One model turn. Carries the provider, the model, the iteration number, and the input and output token counts. | | `agentops.tool.call` | One tool invocation, by name, with its outcome. A failed call ends with an error status. | | `agentops.skill.invoke` | One skill invocation, by id and name. | Every worker span also carries resource attributes identifying the worker and the agent it runs: `agentops.worker.id`, `agentops.worker.name`, and `agentops.agent.id`. ### Metrics Every metric a worker emits lives under the `agentops.` namespace, so one facet in your backend isolates agent activity from everything else you collect. | Metric | What it counts | | - | - | | `agentops.run.completed` | Runs that finished successfully. | | `agentops.run.failed` | Runs that failed, tagged with a failure reason. | | `agentops.run.duration` | Run duration. | | `agentops.llm.requests` | Model requests, tagged by provider, model, and status. | | `agentops.llm.request.duration` | Model request latency. | | `agentops.llm.input_tokens` | Input tokens consumed. | | `agentops.llm.output_tokens` | Output tokens emitted. | | `agentops.tool.calls` | Tool calls, tagged by tool name and status. | | `agentops.skill.invocations` | Skill invocations, tagged by skill and status. | The two durations are histograms in **seconds** (`unit=s`). Everything else is a counter and carries the dimensionless `unit=1`, so your backend will not label it for you — the count is of whatever the row above says. ### Logs A worker's log records are exported alongside its spans, and each record carries the trace and span it belongs to — so you can pivot from a slow span to the lines the agent wrote while it was executing. > [!TIP] > > Worker log lines also reach KAOP independently of OTel, which is why a run's evidence shows its logs > whether or not you export anything. Komodor has no access to a worker you host, so the worker ships > its own lines to the control plane as part of the run. ## Choosing what a worker forwards to KAOP Two variables control the KAOP-side log forwarding, independent of OTel export: | Variable | Default | Effect | | - | - | - | | `AGENTOPS_CP_LOG_LEVEL` | `INFO` | The minimum level forwarded to the control plane. | | `AGENTOPS_CP_LOG_CAPTURE_ALL` | off | Off forwards only the SDK's own loggers. Set it to attach to the root logger and capture everything the process logs. | Raise `AGENTOPS_CP_LOG_LEVEL` on a chatty worker if run evidence is noisier than it is useful; set `AGENTOPS_CP_LOG_CAPTURE_ALL` while debugging a worker whose own libraries hold the answer. ## What the control plane emits The control plane is instrumented the same way and turns on with the same variables. Its spans carry the resolved caller identity, so traces are sliceable per account and per principal, and authorization decisions appear as their own spans with the action, resource, and decision attached. Two attribute namespaces are used and only two: * `usr.*` for human principals — the user id, email, and name. * `agentops.*` for everything else — the account, the principal, the authentication path, and authorization decisions. > [!NOTE] > > **Some identity reaches every span by default.** The root span carries the full set. Two settings, > both **on unless you turn them off**, widen part of it to the rest of the trace: > `AGENTOPS_OTEL_SPAN_BAGGAGE_ENABLED` propagates the non-identifying ids (account and principal), and > `AGENTOPS_OTEL_ALL_SPANS_IDENTITY_ENABLED` adds the principal kind and display name. Set either to > `false` to keep it off the child spans. > > Email and name never leave the root span whatever those two are set to, which bounds how much > personal data reaches your telemetry store. ## What to build with it * **Alert on agent failure rate the way you alert on service error rate.** `agentops.run.failed` against `agentops.run.completed`, split by agent. * **Watch model latency separately from agent latency.** A run that got slower because one provider got slower is a different problem from a run that took more turns. * **Correlate a tool call with the system it touched.** The `agentops.tool.call` span sits in the same trace as your own service's spans when your instrumentation propagates context. * **Track token volume as an operational signal.** Token counters move before a spend chart does. > [!WARNING] > > Do not use exported metrics as your billing record. Spend attribution is computed in KAOP from each > run's reported usage, including its cost source classification — see > [Agent spend attribution](/manage-your-agents/optimize/agent-spend-attribution). Token counters > tell you volume; they do not tell you money. ## Next steps > **[Runs & evidence](/manage-your-agents/run/runs-evidence)** > > The in-product view of the same activity, and what a span tree contains. > **[Fleet health](/manage-your-agents/optimize/fleet-health)** > > The rollup KAOP builds from run activity. > **[Agent spend attribution](/manage-your-agents/optimize/agent-spend-attribution)** > > Where cost comes from and how it is attributed. # Attention Source: https://docs.komodor.com/manage-your-agents/run/attention The operator inbox — held calls waiting on approval, parked selections waiting on a choice, and runs that stopped and need a look. Agents that run on their own still reach moments only a person can settle. **Attention** is where those land: one inbox for everything waiting on you, across every agent and workflow. ## Three kinds, because the next move differs | Item | What is waiting | What you do | | - | - | - | | **Awaiting approval** | A guardrail held an action rather than letting it run | Approve it or refuse it | | **Awaiting your selection** | A workflow reached a point with more than one way forward | Pick the branch | | **Needs attention** | A run stopped and nobody has looked at it | Open it and decide | They are separated deliberately. Answering a held call, choosing a branch, and investigating a stopped run are different jobs, and a single undifferentiated list would make you re-read each row to work out which one you were being asked to do. When you decide, you can attach a note — *why this branch*, or the values the option you chose needs. The note travels with the decision into the run's evidence, so the reasoning is recoverable later rather than living in someone's memory. ## Held calls come from guardrails An **Awaiting approval** item is a guardrail doing its job. A rule with the **Hold for approval** action pauses the matching call and puts it here rather than refusing it outright — the human-in-the-loop posture for work that mutates something. Approving it from this screen releases the call through the same path the guardrail held it on. ## It tells you when it cannot see everything The inbox reads from several sources, and the screen is explicit when one of them does not answer rather than quietly showing a shorter list: * **"One source did not answer"** — the list is incomplete, and it names which section is affected. * **"Can't confirm the inbox is empty"** — nothing is showing, but the screen will not claim you are clear when it could not read everything. * When there is more than the screen holds, **only the oldest items are shown** — work through them and reload for the rest. > [!NOTE] > > That last distinction is worth trusting. An empty inbox and an inbox that could not be read look the > same on most screens; here they do not, so "nothing needs you" means it. When it is genuinely clear, it says so plainly: *no held calls, no parked selections, and no stopped runs.* ## Next steps > **[Approvals](/security-and-governance/controls/approvals)** > > How an approval is requested, and what the decider is shown. > **[Standards & Guardrails](/security-and-governance/controls/standards-guardrails)** > > The rules that hold an action in the first place. > **[Runs & evidence](/manage-your-agents/run/runs-evidence)** > > The run behind a stopped item, and the trail your decision joins. # Channels Source: https://docs.komodor.com/manage-your-agents/run/channels Connect Slack so your team starts agents, workflows, and incident pipelines from the conversation they are already having. A **channel** connects the Komodor Agentic Operation Platform (KAOP) to a place your team already talks — today, Slack. Once it is connected, routing rules decide what an inbound message starts, and the work shows up either in-thread or in KAOP depending on what was started. ## Connecting Slack Every Slack connect button opens the same **Connect Slack** dialog: **Add channel** under **Integrations → Channels**, **Add integration → Slack**, **Connect Slack** in the Slack MCP setup and in a notification sink, and the Slack connection field when you add an agent. It asks which app connects your Slack, then what the app is for (**Chat**, **Slack MCP** or **Both**), and says what each choice creates before anything is connected. A step with only one possible answer is skipped. | | KAOP's Slack app | Your own Slack app | | - | - | - | | Setup | Approve it in Slack. Nothing to set up. | You create it in Slack from KAOP's manifest and paste its tokens. | | Serves | Chat | Chat, the Slack MCP, or both | | How Slack reaches KAOP | Slack sends events to KAOP over HTTPS | Socket Mode: KAOP opens an outbound connection to Slack, so nothing reaches KAOP from the internet | | Name and permissions | KAOP's | Yours | | Available | Every account | Accounts where Komodor has turned it on | On a self-hosted control plane, your own Slack app is the only one, so the dialog starts at what the app is for. Choose your own app when you need the Slack MCP, want to own the app's name and permissions, or work under a security policy that allows Socket Mode apps only. For the Slack MCP, a Slack admin must approve the app: Slack serves its MCP only to internal apps and apps published in the Slack Marketplace. Chat runs through one app at a time, but for the Slack MCP alone you can connect your own app and keep chat on KAOP's app. ### KAOP's Slack app **1. Choose KAOP's Slack app** Choose **KAOP's Slack app**, then **Authorize in Slack**. **2. Approve the app in Slack** Approve KAOP in the Slack tab that opens. KAOP connects Slack as soon as you approve. If Slack is already connected to your account, the Slack tab says so and there is nothing else to do. If setup fails, the tab says why and what to do next. **3. The channel appears in KAOP** The dialog picks the connection up automatically, and the channel is ready to route messages. ### Your own Slack app You create the app in Slack from KAOP's manifest, install it, and paste its tokens into the dialog. KAOP checks each token with Slack before saving anything. The steps, who the Slack MCP acts as, replacing tokens and disconnecting are in [Your own Slack app](/manage-your-agents/run/slack-own-app). ### Channel status One channel represents one Slack account connection. You can rename it, pause it, resume it, or delete it from its page. While connected, a channel on your own Slack app has **Replace tokens** and **Disconnect** instead of delete. | Status | Meaning | | - | - | | `active` | Connected and processing messages | | `paused` | Connected, not processing inbound messages right now | | `error` | The connection needs attention | | `revoked` | The connection ended: the app was uninstalled, its tokens were revoked, or you disconnected your own Slack app | `revoked` is distinct from `paused` on purpose: **Resume** brings a paused channel back, while a revoked one needs Slack connected again. KAOP verifies its Slack credentials with a live check and reacts to Slack's own revocation events, so a dead connection surfaces as `revoked` rather than failing quietly. ## Routing rules A channel's **Routing rules** decide what an inbound message starts. Rules are evaluated in priority order, top of the list first, and **the first match wins** — one message starts exactly one thing. **A rule can route to an agent, a workflow, or an incident pipeline.** All three appear in the rule list, labelled by kind. ### Rule types Four kinds, matching the form's own options: | Rule type | Matches | | - | - | | **Private** | Direct messages to the bot | | **Channel** | Messages posted in the Slack channels you name | | **Keyword** | Messages containing any of the keywords you list, optionally narrowed to specific Slack channels | | **Mention** | Messages that @-mention the bot | Channel, keyword, and mention rules are scoped by Slack channel ID in the rule form. A keyword rule with no channel scope matches account-wide. Two toggles decide what a route accepts: * **Allow bot posts.** Posts from other apps — a Datadog or PagerDuty alert, a deploy notification — are ignored unless a rule explicitly opts in. That opt-in is how you route machine-generated alerts to an investigation agent. * **Default route.** One rule can be marked the default: it catches messages that match no other rule, so the bot always has an answer path. **A default rule ignores its own matching conditions** — mark a keyword rule as the default and its keywords stop applying, so it catches everything that reached it. Only the bot-post setting still applies. An incident-pipeline route cannot be the default. You can reorder rules with the arrows on each row, enable or disable one individually, edit it, or delete it. ### Rules you cannot edit here Two kinds of rule appear in the list read-only, each with a chip saying where it comes from: | Chip | Owner | How to change it | | - | - | - | | **From agent spec** | The agent's own `agent-spec.yaml` — its declared Slack channel subscriptions | Edit the agent's spec; changes made in the channel list are overwritten on its next check-in | | **From incident pipeline** | An incident pipeline's own Slack triggers | Add or remove a Slack trigger on that incident pipeline | For these rows the reorder arrows, the enable toggle, and delete are all disabled — editing them here would be undone by whichever system owns them. > [!NOTE] > > **A route to an incident pipeline is not created here.** It comes from that incident pipeline's own Slack > triggers, so the rule form offers **Agent** and **Workflow** as targets only. Incident-pipeline routes > show up in the list because they are real routes on this channel, and they are read-only because the > incident pipeline owns them. ### Precedence in practice Agent-spec rules and rules you create in the console are ordered against each other in one list, and a console-created rule usually lands near the top. Rather than reasoning about it, read the real precedence off the channel page: the Routing rules list is sorted by priority, and whatever is nearest the top wins. To change it, use the reorder arrows. **The default rule is the exception.** It is considered only after every other rule has failed to match, wherever it sits in the list — so a default near the top still loses to any rule below it that matches. Because the first match wins, two agents that both listen in the same Slack channel do **not** both get the message — the higher rule wins, and the overlap is ambiguous by construction. If two agents must both respond, give them distinct channels or model the fan-out inside one workflow rather than in the routing table. ## Who the agent is acting for The first time someone messages the bot, KAOP prompts them in Slack to connect their Slack identity to their KAOP user with a quick sign-in link. Once linked, runs started from their messages are attributed to them rather than to the bot, which is what keeps the audit trail honest. Until a sender links, their messages get a connect prompt instead of a run. Two exceptions, both deliberate: a bot post cannot link an identity, so bot-triggered runs act under the agent's own principal; and a message routed to an incident pipeline creates an incident regardless, because a human mentioning an outage should not be blocked on their account linkage. ## What happens when a message matches **1. KAOP reacts to the message** A reaction is added so the sender knows it was picked up. This happens for every kind of target — unless the rule routes to a **disabled** agent, in which case nothing happens at all: no reaction, no reply, and no run. An agent that has stopped answering in Slack is worth checking for that first. **2. The target is invoked with the message as input** Any preset input on the rule is merged in. An agent target becomes a run recorded with channel `slack`. A workflow target starts a workflow, whose agent steps are their own runs. An incident-pipeline target opens an incident and runs its orchestrator. **3. An agent target replies in-thread, live** A threaded reply is posted immediately and **updated as the agent streams its progress**, so the thread shows the work happening. The final reply renders the run's human-readable output. ### Only an agent target replies in-thread This is the single most important behavioural difference between the three target kinds: | Target | Threaded reply in Slack | Where you follow the work | | - | - | - | | **Agent** | Yes — posted immediately and updated live as the run streams | In the thread, and in the run | | **Workflow** | **No** | The reaction tells you it was picked up; follow the workflow run in KAOP | | **Incident pipeline** | **No** | The reaction tells you it was picked up; follow the incident in KAOP. The incident pipeline separately posts its own conclusion to the Slack channel it is configured with | **Evidence is recorded for every target kind.** The run, or the workflow's step runs, or the incident's orchestrator and specialist runs, all land in KAOP with their full evidence trail — so the operational record is complete even where Slack shows only a reaction. That is the point worth internalising: silence in the thread is not silence in the platform. If an agent run fails, the threaded reply says so and links to the run so you can read the full evidence rather than guessing from a one-line error. ### Messages that match nothing A message that explicitly addresses the bot — a mention or a DM — and matches no rule gets one in-thread reply saying so, rather than silence. It is best-effort and stays quiet in the cases where a reply would be wrong: a bot post that named the bot does not draw a public answer, and a paused, errored, or revoked channel does not post at all. ## Who answered, and who really answered Routing decides which agent *receives* a message. The two diverge as soon as that agent delegates, which is normal for a general-purpose assistant that hands off to a specialist. Every channel reply therefore carries a one-line attribution footer naming the agent that answered and, when it delegated, the agent that handed off — with each run linked. The footer is added by KAOP's reply path, not written by the agent. A footer the agent authored would be exactly as trustworthy as the paraphrase it exists to make checkable. If you want one run and one bill instead of two, write a rule that routes straight to the specialist. ## Next steps > **[Your own Slack app](/manage-your-agents/run/slack-own-app)** > > Create the app, connect it for chat or the Slack MCP, and manage its tokens. > **[Runs & evidence](/manage-your-agents/run/runs-evidence)** > > Messages from a channel produce ordinary runs — here is what they record. > **[Incident management & troubleshooting](/modules-and-workflows/sre/incidents)** > > Incident pipelines, and the Slack triggers that own their channel routes. > **[Roles & permissions](/security-and-governance/identity-and-access/roles-permissions)** > > Who can connect a channel and manage its routing. # Chat & history Source: https://docs.komodor.com/manage-your-agents/run/chat-history Work conversationally with your agents in resumable sessions, and read every run and chat in one ledger. Not all agent work is triggered and structured — a lot of it starts as a question. Chat in the Komodor Agentic Operation Platform (KAOP) is a conversation with a real agent, not a generic model wrapper, and every turn is recorded with the same evidence as any other run. ## Two kinds of record, one timeline KAOP keeps two session-like records, and they answer different questions: | Record | What it is | What you do with it | | - | - | - | | **Run** | One execution of an agent, with its full evidence trail | Open it and read what happened | | **Chat session** | A saved conversation with a chat-capable agent | Resume it and keep going | **History** presents both in one ledger, newest first. A chat row resumes the conversation; a run row opens the evidence. ## Chatting with an agent The responses in a chat come from an agent's worker — the same worker that would serve any other invocation — so an agent answers in chat with the tools, permissions, knowledge, and skills it has everywhere else. The built-in chat assistant is available from the start, and any agent you register joins the chat picker by advertising chat capability on its agent card: ```json theme={null} { "capabilities": { "chat": true } } ``` `chat` is the only key that decides eligibility. Replies stream whether or not a card advertises anything else. In Fleet, a chat-capable agent gets chat actions alongside **Run**; an agent that does not advertise chat keeps the structured run-and-evidence flow. The capability is checked on every send, not only when a session is created, so an agent that stops advertising chat also stops serving sessions that already exist. You can switch the model for a session from the chat header. The list is curated by the control plane rather than free text, and every requested model is validated. ## What a turn actually does Sending a message is not a shortcut around the platform — it goes through it: **1. Your message is persisted** Before anything is invoked, so a refresh mid-answer never loses what you asked. **2. The session's agent is invoked, as a run** The turn creates an ordinary run against that agent, marked as a chat turn and carrying the session it belongs to. **3. The reply streams back as it is produced** Incremental text, including for agents that are only reachable over their live worker connection. **4. The assistant message is persisted, and the run keeps its evidence** The conversation is resumable, and the turn's traces, logs, and tool calls are available exactly as they would be for any other invocation. The consequence worth internalising: **a chat answer is as auditable as a scheduled one.** There is no lower-fidelity path for conversational work. ## Tool calls and evidence in the transcript The transcript is the primary object, not a chrome-free chat box. As an agent works, what it does appears inline: * **Tool calls** render as their own rows — which tool, with what arguments, and what came back — interleaved with the agent's narration in the order they happened. * **Consecutive tool calls are grouped into rounds**, so parallel calls issued in one model turn read as one round of work rather than a flat list, and each result stays with the call it answers. * **Data the agent pulled and drafts it produced** appear as their own transcript entries rather than being flattened into prose. * **Timestamps are stamped when an event is emitted**, not when it reaches the control plane, so the order and spacing you read is the order and spacing that happened. History's own list shows a per-run tool-call count alongside each row, counted once per tool call. For the full trace tree, the logs, and the artifacts behind a turn, open its run — see [Runs & evidence](/manage-your-agents/run/runs-evidence). ## Resumable sessions Sessions are addressable, so you can bookmark one or send someone a link to it: | URL | Opens | | - | - | | `/chat` | The built-in chat assistant, or your first chat-capable agent | | `/chat?agent=` | The session list scoped to one agent | | `/chat?agent=&session=` | That exact saved session, resumed | Resuming is not a replay: the transcript is loaded from the control plane with each message's status, so a session reopens in the state it was actually left in — complete, errored, or interrupted. You can rename a session, change its model, or delete it. ## Waiting for a worker A turn's run is created queued and claimed by a worker, which means a chat can be waiting on capacity rather than on the model. That wait is stated rather than shown as a frozen cursor, and it travels on the stream as its own frames. The streaming endpoints emit `text/event-stream` frames: ```text theme={null} data: {"type":"queued","queued_behind":2} data: {"type":"dequeued"} data: {"type":"text-delta","delta":"Checking the latest deploy…"} data: {"type":"message","message_id":"msg_…"} data: [DONE] ``` The two transport frames behave differently, which matters if you are building against the API: * **`queued_behind` is a queue position**, not a duration — the number of runs for the same agent that a worker would claim before this one. `0` therefore means nothing is ahead of it: the wait is for a worker to come free or come online, not for a queue to drain. * **The frame is resent whenever that position changes**, so a client can count it down. A queue that is not moving sends nothing. * **Nothing is sent for the wait's first look at the run.** Every run starts out queued, so reporting the first look would put a queue notice in front of the reader on every single message. The trade-off is deliberate: the notice appears one poll into a real wait rather than instantly. * **`dequeued` marks the end of the wait**, sent when a worker claims the run. A claimed run is still silent while it starts up and thinks, so the end of the wait has to be announced or the notice outlives the queue. * **A queued run that never reaches a worker sends no `dequeued`.** Cancelled or failed outright, there was no handoff to report, and the terminal frame that follows carries its own content. > [!WARNING] > > Neither frame carries assistant text. Render both as transport state — a status line, a spinner > label — and never as part of the reply. Folding them into the message persists them into the > transcript, where they get replayed on every reload. Errors arrive as `{"type":"error","message":"…"}` in place of the terminal message frame. ## When an agent cannot answer A failure is shown, not swallowed. When a turn cannot produce an answer — a model-provider billing, credential, or rate-limit problem, or a worker runtime failure — the transcript renders an error card explaining what went wrong and who can fix it, instead of an empty reply. The underlying run keeps its classified failure alongside whatever evidence it collected first. ## Continuing from a run A finished run's page carries a composer beneath its output: **Continue this investigation**. You type a follow-up, pick who answers, and pick where it continues. * **In KAOP chat.** The new session opens as a real transcript rather than one machine-formatted prompt: the run's own input becomes the prior user turn, its output — or, for a failed run, its failure reason — becomes the prior assistant turn, and your follow-up is the next message. A trailing line names the run so the agent can fetch its full transcript, logs, and trace rather than working from an excerpt. * **By re-running the agent.** When the run saved a session snapshot it is *revivable*: **Re-run the agent** re-executes it with its tools and the earlier context, producing a new run in the same thread. That is different from talking about the run, and it is the right choice when you want the work redone rather than explained. * **By handing it to a local coding agent.** Claude Code or Codex, or the prompt copied to your clipboard. These handoffs carry only your follow-up, the run id, and the run's URL — never the evidence itself. The coding agent fetches the run over the KAOP MCP server under your own permissions, so nothing sensitive lands in shell history or a clipboard manager. See [Connect coding agents](/developer-tools/connect-coding-agents). Any chat-capable agent may answer, defaulting to the run's own agent when it can chat. When a different agent has to answer instead, the composer says so inline — the substitution is never silent. ## The History ledger History combines chat sessions and runs, newest first, and filters by type (all, chats, runs), run status, agent, and free-text search. Two behaviors to know: * **Chat rows absorb their turns.** The per-turn runs behind a conversation are folded into the session's row rather than listed a second time, so a long chat does not bury everything else. * **The list is capped, and says so.** When a page comes back full, the table prints a truncation notice rather than implying you are looking at everything. Search is evaluated on the server — over run and session ids, a row's title, and the agent's display name or slug — so looking for something older than the current page finds it. ## Chat API Chat endpoints live under `/api/v1/chat`: | Endpoint | Purpose | | - | - | | `GET /api/v1/chat/models` | The curated model list | | `GET /api/v1/chat/sessions` | List sessions, optionally `?agent_id=` or `?q=` | | `POST /api/v1/chat/sessions` | Create an empty session for an agent and model | | `POST /api/v1/chat/sessions/start` | Create a session and persist its opening messages in one step | | `GET /api/v1/chat/sessions/{session_id}` | Load a session and its messages | | `PATCH /api/v1/chat/sessions/{session_id}` | Rename it, or change its model | | `DELETE /api/v1/chat/sessions/{session_id}` | Delete it | | `POST /api/v1/chat/sessions/{session_id}/messages` | Send a message and stream the reply | | `POST /api/v1/chat/sessions/{session_id}/responses` | Stream a reply for an already-persisted message | | `POST /api/v1/chat` | One-shot streaming chat with no persisted session | You can pass a friendly agent slug wherever an `agent_id` is expected — KAOP resolves it to the agent's stable id. ## Next steps > **[Runs & evidence](/manage-your-agents/run/runs-evidence)** > > The evidence behind every chat turn. > **[Channels](/manage-your-agents/run/channels)** > > The same conversational pattern, in Slack. > **[Connect coding agents](/developer-tools/connect-coding-agents)** > > Hand a run to Claude Code or Codex with your own permissions. # How Run works Source: https://docs.komodor.com/manage-your-agents/run/how-it-works Every way work reaches an agent, what one execution records, and where to look when you need to know what happened. Building an agent is only half of it — the other half is work actually arriving and getting done. Eight paths start a run in the Komodor Agentic Operation Platform (KAOP), and whichever one fires, the result is the same object with the same evidence behind it. ## Eight ways work starts Work arrives from wherever your team already is. Whatever starts it, the result is the same object: a **run**. | Path | What starts it | Where it is set up | | - | - | - | | **Manual invocation** | You open an agent and start it with a prompt or a JSON payload | The agent's page in Fleet, or the API | | **Cron trigger** | A recurring schedule fires | [Triggers & schedules](/manage-your-agents/build/triggers-schedules) | | **Inbound endpoint** | An external system posts to a URL you created | Integrations → **Endpoints (inbound)** | | **Channel** | A Slack message matches a routing rule | [Channels](/manage-your-agents/run/channels) | | **Chat** | You send a message in a conversation with a chat-capable agent | [Chat & history](/manage-your-agents/run/chat-history) | | **Workflow step** | A step delivers its task to the agents bound to it | [Orchestrator](/manage-your-agents/build/catalog/orchestrator) | | **Module** | One of a module's workflows reacts to something in its domain | [Modules & workflows](/modules-and-workflows/overview) | | **Another agent** | An agent delegates to an agent it is allowed to reach | [Orchestrator](/manage-your-agents/build/catalog/orchestrator) | Every run records the surface that started it, so an alert-driven investigation is never confused with a manual one: | Recorded channel | Started by | | - | - | | `ui` | A person in the console | | `api` | A direct API call | | `webhook` | An inbound endpoint | | `cron` | A schedule | | `slack` | A channel routing rule | | `agent` | Another agent, including a workflow step | > [!NOTE] > > A **trigger** in KAOP is a recurring cron schedule. Inbound webhook URLs are a separate object — > they live under Integrations → **Endpoints (inbound)** and are covered with the rest of > [Integrations](/manage-your-agents/build/integrations-overview). ## What a run is A run is one execution of one agent, recorded durably from the moment it is created to the moment it reaches a terminal state. It carries its own input, its own evidence, its own output, and its own cost, and it is addressable forever after. Runs move through a small set of statuses: | Status | Meaning | | - | - | | `queued` | Created, waiting for a worker to claim it | | `claimed` | A worker has taken it and is starting | | `running` | The agent is working and streaming progress | | `succeeded` | It finished and produced its output | | `failed` | It stopped with an error | | `cancelled` | Someone stopped it before it finished | Work is never pushed at a worker — a worker claims runs from the control plane's queue. That is why an agent needs only outbound access to KAOP, and why a run that is waiting is reported as `queued` rather than stalling silently. See [Architecture](/get-started/architecture). ## What a run leaves behind Nothing about a run is reconstructed after the fact. The agent streams its evidence back to the control plane while it works, and that stream is what you read later: * **Transcript** — the human-readable narration: progress notes, tool calls and their results, data the agent pulled, drafts it produced. * **Trace timeline** — the structured span tree of reasoning, model calls, and tool calls, with the time each took. * **Logs** — timestamped log lines, at the usual levels. * **Output** — the final result, in two channels: machine-readable fields for automation, plus an optional human-readable markdown rendering. * **Artifacts** — discrete files the run produced, which you can open and download. * **Usage** — model, token counts by class, and cost. * **Guardrail decisions** — any action a policy blocked, redacted, or held for approval. That collection is what makes an agent's conclusion auditable instead of something you take on trust. [Runs & evidence](/manage-your-agents/run/runs-evidence) is the page on how to read it. ## Where the work is visible * **History** lists every run and every chat session in one ledger, newest first, filterable by type, status, agent, and free-text search. * **A run's own page** holds its evidence, and the composer beneath a finished run's output lets you ask a follow-up, re-run the agent, or hand the run to a local coding agent. * **Attention** lists everything currently waiting on a person — a held action to approve, a choice to make, a workflow run that stopped and needs review. ## Explore Run > **[Chat & history](/manage-your-agents/run/chat-history)** > > Conversational work with agents, resumable sessions, and the unified ledger. > **[Runs & evidence](/manage-your-agents/run/runs-evidence)** > > What a run records, and how to read it well enough to trust or challenge it. > **[Outbound webhooks](/manage-your-agents/run/outbound-webhooks)** > > Have KAOP post signed, retried events to your own systems as agents work. > **[Channels](/manage-your-agents/run/channels)** > > Slack — connect an account, route messages, and get answers in-thread. > **[Your own Slack app](/manage-your-agents/run/slack-own-app)** > > Connect a Slack app you create, for chat over Socket Mode, the Slack MCP, or both. ## Next steps > **[Triggers & schedules](/manage-your-agents/build/triggers-schedules)** > > Put an agent on a cron schedule. > **[Approvals](/security-and-governance/controls/approvals)** > > How a risky action pauses for a human decision. > **[Modules & workflows](/modules-and-workflows/overview)** > > The prebuilt outcomes whose workflows drive most automated runs. > **[Fleet health](/manage-your-agents/optimize/fleet-health)** > > Read the fleet once runs are flowing. # Outbound webhooks Source: https://docs.komodor.com/manage-your-agents/run/outbound-webhooks Have KAOP post signed, retried, auditable events to your own endpoints so external systems react to what your agents do. Agents produce facts your other systems need — a run finished, a deployment failed, an incident was triaged. An **outbound webhook** is the Komodor Agentic Operation Platform (KAOP) posting those events to an endpoint you own, signed and retried, with a durable record of every attempt. ## Outbound is not inbound Two objects, opposite directions, both under **Integrations**: | | **Endpoints (inbound)** | **Notification sinks** | | - | - | - | | Direction | Your systems post to KAOP | KAOP posts to your systems | | Purpose | Start a run or a workflow from an external event | React to something an agent did | | You supply | Nothing — KAOP gives you a URL and a token | The destination URL and a signing secret | | Covered in | [Integrations](/manage-your-agents/build/integrations-overview) | This page | If your goal is "an alert should start an investigation", you want an inbound endpoint. If your goal is "our ticketing system should know when the investigation is done", you want an outbound webhook. ## Setting one up Outbound webhooks live under **Integrations → Notification sinks**. **New notification sink** opens a four-step wizard. **1. Destination** Give it a name, then choose where events go. * **Webhook** — signed POST requests to any HTTPS endpoint. You supply the destination URL; KAOP generates the signing secret and reveals it once so you can verify deliveries on your side. * **Slack** — posts messages to a channel using your existing Slack connection. Without one, **Connect Slack** opens the dialog described in [Connecting Slack](/manage-your-agents/run/channels#connecting-slack). **2. Events** Pick which event types this sink should receive. The catalog is the same regardless of which sources feed it. **3. Payload & test** Inspect exactly what will be sent — always JSON — and use **Send test event** to fire a real request at your endpoint before anything depends on it. **4. Subscriptions** Choose which sources feed this sink: specific agents, specific workflows, specific incident pipelines, or the whole account. A sink can also be picked while you are creating an agent, from the Triggers step's **When the run finishes** section — the same object, reached from where you were already working. ## What you can subscribe to Three families of event, in one catalog: | Group | Events | | - | - | | **Agent lifecycle** | `agent.created`, `agent.updated`, `agent.archived`, `deployment.succeeded`, `deployment.failed`, `agent.unreachable` | | **Run lifecycle** | `run.finished` (any terminal status), `run.succeeded`, `run.failed`, `run.cancelled`, `run.started` | | **Incident lifecycle** | `incident.triaged` (investigation complete), `incident.errored`, `incident.resolved` | Events and sources are separate axes, which is what makes fan-in useful. One sink can take `run.failed` from a single agent; another can take `run.finished` from every source in the account; a third can take incident events from two specific incident pipelines. `agent.created` is account-scoped by nature — there is no per-agent source for an agent that does not exist yet. ## What gets delivered One fixed JSON envelope, versioned, with no templating — so there is exactly one shape to write a receiver against: ```json theme={null} { "event": "run.finished", "status": "failed", "source": { "type": "agent", "id": "agent-id" }, "run_id": "run_abc123", "duration_ms": 1234, "run_url": "https://app.example.com/runs/run_abc123" } ``` The one thing you control about the shape is **Include run output**, which is off by default because a run's output can be large or sensitive. The toggle is read at send time, so flipping it takes effect on the very next delivery — you do not re-subscribe. Payloads are capped. When an envelope would exceed the cap, the output field is replaced with a truncation marker rather than being cut mid-structure, so your receiver always parses valid JSON and can tell that something was withheld. ### The request | Property | Value | | - | - | | Method | `POST` | | `Content-Type` | `application/json` | | `X-AgentOps-Signature` | `sha256=` — HMAC-SHA256 of the raw request body, keyed by the sink's signing secret | | `X-AgentOps-Notification-Envelope-Version` | The envelope version this payload was rendered at | Verify the signature over the **raw bytes** you received, before parsing. Re-serializing the JSON and signing that will not match. Two things KAOP will not do, both deliberate: * **Redirects are never followed.** Any `3xx` response is treated as a failed attempt, because a redirect to an internal address is a way around the destination you configured. * **Private and loopback destinations are refused** — and the destination is re-resolved and re-checked on every send, not only when the sink was created, so a name that starts resolving internally later is caught at delivery time. ### Rotating the signing secret **Rotate signing secret** issues a new one and shows it to you once. Rotation is an audited, elevated action, so plan for the cutover: your receiver should be ready to accept the new secret before you rotate. ## How delivery behaves | Outcome | Meaning | | - | - | | `pending` | The attempt was recorded and the request is in flight | | `success` | Your endpoint returned a success status | | `failed` | A connection error, a timeout, a redirect, or a non-success status | | `rate_limited` | Your endpoint returned `429` or `503` | | `exhausted` | Every retry was spent with no success | A failed attempt is retried with backoff, up to a bounded number of attempts, and then dead-lettered — which marks the delivery `exhausted` in the log. There is no automatic follow-up after that, so `exhausted` is the state to alert on if these events are load-bearing for you. Rate limiting is handled differently from failure, and in your favour: a `429` or `503` is retried after the delay your `Retry-After` header asks for, and that wait does not consume the retry budget. So back-pressure from your side costs you a delay rather than a dropped event. A disabled or deleted sink is a silent no-op — events for it are dropped rather than queued up for a destination that no longer wants them. ## Auditing what was sent Every sink row carries a **History** link. It opens the delivery log for that sink: when each attempt happened, which event it carried, which attempt number it was, the result, and the response body your endpoint returned (captured up to a few kilobytes — enough for the diagnostic, which is almost always at the front). The log is durable and per-account, so "did this fire, and what came back?" has an answer well after the fact. Configuration and deliveries are recorded in two different places on purpose: * **Configuration changes are audited** — creating, updating, or deleting a sink, rotating its secret, and subscribing or unsubscribing a source all write audit entries. * **Deliveries are not audit entries.** They are operational data, and they live in the delivery log above. ## Who can manage them Reading the list of sinks is an account-scoped permission. Creating, updating, deleting, rotating a secret, and sending a test event are **administrative** actions held at a deliberately higher bar — an outbound webhook is an account-chosen arbitrary URL, which puts it in the same class as managing credentials. See [Roles & permissions](/security-and-governance/identity-and-access/roles-permissions). ## Next steps > **[Runs & evidence](/manage-your-agents/run/runs-evidence)** > > What the `run.*` events are describing. > **[Integrations](/manage-your-agents/build/integrations-overview)** > > Inbound endpoints, and the connection catalog. > **[Audit log](/security-and-governance/controls/audit-log)** > > Where configuration changes to a sink are recorded. > **[Channels](/manage-your-agents/run/channels)** > > The conversational route into and out of Slack. # Runs & evidence Source: https://docs.komodor.com/manage-your-agents/run/runs-evidence Every execution is a run with a full evidence trail — inputs, tool calls, reasoning, output, and cost. Here is how to read it. An agent's conclusion is only as good as your ability to check it. In the Komodor Agentic Operation Platform (KAOP), every execution is a **run** that records what it was asked, everything it did, what it produced, and what it cost. ## The lifecycle of a run | Status | Meaning | | - | - | | `queued` | The run exists and is waiting for a worker to claim it | | `claimed` | A worker has accepted it and is starting | | `running` | The agent is working and streaming progress | | `succeeded` | It finished and produced its output | | `failed` | It stopped with an error before producing a result | | `cancelled` | Someone stopped it deliberately | `cancelled` is kept distinct from `failed` on purpose: an operator stopping a run and an agent breaking are different facts, and metrics that conflate them mislead. Every run also records the surface that started it — `ui`, `api`, `webhook`, `cron`, `slack`, or `agent` — so you can always tell an alert-driven investigation from a manual one. See [How Run works](/manage-your-agents/run/how-it-works) for the full set of starting paths. ## What a run was asked A run's input is one JSON object, however it was supplied. From the console's run dialog you can provide it two ways: * **Prompt** — free text, submitted as `{"prompt": "…"}`, for agents driven by a natural-language instruction. * **JSON input** — an arbitrary JSON object, for agents invoked with a structured payload. JSON mode also takes an optional instruction that rides along as the `prompt` key. An agent that advertises an input schema opens in JSON mode with the editor pre-seeded from that schema and the agent's own sample values, and you can read the advertised schema in place. An agent with no schema opens in prompt mode. Either way, the stored input is the same shape, so a run started from a schedule, an endpoint, or another agent is indistinguishable in structure from one you typed. ## What a run records Evidence is streamed to the control plane by the agent's worker while the run is in progress, in batches. Nothing on this list is reconstructed afterwards — which is why it is still complete for a run that failed halfway. | Evidence | What it gives you | | - | - | | **Transcript** | The narration: progress notes, tool calls and their results, data pulled, drafts produced — in the order they happened | | **Trace timeline** | The structured span tree — reasoning, model calls, tool calls — with the duration of each | | **Logs** | Timestamped lines at the usual levels, for detailed diagnostics | | **Output** | The final result, in a structured channel and an optional human-readable one | | **Artifacts** | Files the run produced, openable and downloadable | | **Usage** | Model, token counts by class, and cost | | **Guardrail decisions** | Any action a policy blocked, redacted, or held for approval | | **Skills** | The skills the run actually loaded | ## Reading a run Open a run and you get its status, its input, and a set of tabs. Tabs appear only when there is something in them, so the shape of a run's page tells you something before you read a word of it. | Tab | What to look for | | - | - | | **Run Output** | The answer. Start here, then go looking for how it got there | | **Transcript** | The readable story of the run, tool calls grouped into rounds | | **Trace Timeline** | Where the time went, call by call | | **Logs** | Detail the transcript deliberately leaves out | | **Skills** | Which runbooks and procedures were in play | | **Artifacts** | Reports and generated files, with a count | | **Guardrails** | Every policy decision that touched this run, with a count | | **Metadata** | The run's own facts — ids, thread, timings, the surface that started it | ### Output has two channels A run's output is one JSON object serving two audiences, and knowing which is which stops a lot of confusion: * **Structured output** — the top-level machine-readable fields, the substrate that automation reads: findings, counts, verdicts, flags. * **Descriptive output** — a single reserved `text` key holding human-readable markdown. The console, Slack replies, and History render it. It is a projection of the structured data for people, never a replacement for it. An automation-only agent may emit no `text` at all, and consumers then fall back to the structured payload. An agent that owes a machine-readable answer can declare its output as a schema, in which case a shape mismatch is corrected by the agent inside the same turn rather than discovered as a parse failure afterwards — and a run that produced no structured answer fails, rather than succeeding with nothing in it. ### Time breakdown A finished run shows where its wall-clock time went, by category: **Model**, **MCP tools**, **Other tools**, **Overhead**, and — only when it is non-zero — **Not measured**. The last two are the ones worth understanding, because they are different kinds of fact: * **Overhead** is the agent's own real time that no model or tool call covers — prompt assembly, framework glue, serialization, and time spent waiting on a delegated child run. For an orchestrator this is usually mostly the waiting, which is why a large number there is not automatically a problem with your code. * **Not measured** is a call that started and never reported its end. It is missing measurement, not idle time — the call may have been cut short when the run ended. It is never folded into another category and never hidden below a threshold, at any magnitude. The breakdown is computed when you read it, not stored. A run's span tree is not necessarily complete the instant the run reaches a terminal state, so anything cached would quietly miss late spans. ### What a run cost Usage counts tokens in four disjoint classes, so summing across them is honest: | Class | What it counts | | - | - | | `input_tokens` | Prompt tokens billed fresh | | `cached_input_tokens` | Prompt tokens served from the prompt cache — the cheapest class | | `cache_write_input_tokens` | Prompt tokens written into the cache — the dearest input class | | `output_tokens` | Tokens the model generated | `total_tokens` is their sum, and is derived rather than authoritative. Alongside the counts, a run records its model, provider, and cost, and whether that cost was reported by the worker or estimated. Account and per-agent roll-ups build on the same numbers — see [Agent spend](/manage-your-agents/optimize/agent-spend-attribution). > [!WARNING] > > If you read usage off the raw event stream rather than the run's recorded total: every token count on > a model span is the run's **cumulative** total so far, not that turn's delta. Take the last value; > never sum the series. ## When a run fails A failed run keeps its error alongside every piece of evidence collected before the failure, so the question "how far did it get?" is answerable. Runs that failed because the model provider could not serve the request — billing, credentials, rate limits — carry classified failure metadata, so the page tells you *why* it failed and who can fix it. That is deliberately distinguished from an agent that ran perfectly well and found a problem. > [!NOTE] > > A run can succeed and still be useless — an agent blocked by unbound tools, an unauthenticated > integration, or a bad credential finishes `succeeded`. Status alone is not a quality signal; read the > output. Grading output quality systematically is what > [Evaluations](/manage-your-agents/optimize/evaluations) is for. ## Following a fan-out Orchestrating agents delegate, and delegation is recorded as parentage rather than inferred: * **One conversation thread** — the runs of a multi-turn exchange, oldest first. * **One run's children** — every run that this run delegated to, newest first. So an orchestrator's run page lets you walk down into each specialist it called, and each specialist's run is a first-class run with its own evidence. Where a channel reply came from an agent that delegated, the reply itself names both agents and links both runs — attribution is added by the platform, not written by the agent, so it stays correct even when the answer is not. ## Following up on a run Beneath a finished run's output sits **Continue this investigation**. From there you can ask a follow-up in KAOP chat, hand the run to a local coding agent, or — when the run saved a session snapshot — pick **Re-run the agent** to re-execute it with its tools and its earlier context, which creates a new run in the same thread. See [Chat & history](/manage-your-agents/run/chat-history) for what each destination carries. ## Why the trail is trustworthy Five properties, each of which is a design decision rather than a side effect: * **It is emitted, not inferred.** Everything you read arrived from the agent while it was working. * **Secrets never enter it.** A credential is delivered into a run in memory and is never exposed to you or written into its evidence. See [Secrets & credential handling](/security-and-governance/architecture-considerations/secrets-credential-handling). * **Policy decisions are part of the record.** An action a guardrail blocked, redacted, or held shows up on the run that attempted it — the absence of an action is evidence too. * **Runs are durable and scoped to one account.** Every read and write is checked against that boundary, so a run is retrievable later and only by the account that owns it. * **Identity is recorded from the request, not the payload.** Who started a run, and who approved or answered anything inside it, comes from the authenticated caller. See [Audit log](/security-and-governance/controls/audit-log). ## Finding runs **History** is the ledger: every run and chat session, newest first, filterable by type, status, and agent, and searchable over run and session ids, a row's title, and the agent's name or slug. The `GET /api/v1/runs` API supports the same narrowing — one agent, one conversation thread, or the children of one run — which is how you walk an orchestrator's fan-out programmatically. ## Next steps > **[Chat & history](/manage-your-agents/run/chat-history)** > > The ledger, and conversational work with the same evidence. > **[Evaluations](/manage-your-agents/optimize/evaluations)** > > Grade runs against explicit criteria instead of reading them one by one. > **[Observability & OTel export](/manage-your-agents/optimize/observability-otel-export)** > > Get run telemetry into your own observability stack. > **[Outbound webhooks](/manage-your-agents/run/outbound-webhooks)** > > Have KAOP tell your systems when a run finishes. # Your own Slack app Source: https://docs.komodor.com/manage-your-agents/run/slack-own-app Connect a Slack app you create from KAOP's manifest, for chat over Socket Mode, the Slack MCP, or both, and manage its tokens and connection. The Komodor Agentic Operation Platform (KAOP) can connect to Slack through a Slack app you create and own, instead of KAOP's app. Its chat runs over Socket Mode, so nothing reaches KAOP from the internet, and it is the app that gives agents the Slack MCP. Komodor turns your own app on for your account; until it does, the **Connect Slack** dialog offers KAOP's app only. The two apps are compared in [Connecting Slack](/manage-your-agents/run/channels#connecting-slack). ## What the app serves * **Chat**: messages, mentions, slash commands and buttons reach your agents, as with KAOP's app. Needs the app's bot token and app-level token. * **Slack MCP**: agents call Slack's own tools as one Slack user. Needs the app's User OAuth Token. * **Both**, from one app. ## Create the app and connect it **1. Choose your own app and what it is for** Open **Connect Slack** from any Slack connect button. Choose **Your own Slack app**, then **Chat**, **Slack MCP** or **Both**. **2. Copy the manifest** Copy or download the manifest the dialog shows. It is built for what the app is for. **3. Create the app in Slack** For the Slack MCP, first sign in to Slack as the user it should act as (see [Who the Slack MCP acts as](#who-the-slack-mcp-acts-as)). Open [api.slack.com/apps](https://api.slack.com/apps) and choose **Create New App**, then **From a manifest**. Pick where to create it, paste the manifest, and choose **Create**. **4. Install the app** Open **Install App**, install the app, and approve it. For the Slack MCP, a Slack admin must also approve the app: Slack serves its MCP only to internal apps and apps published in the Slack Marketplace. **5. Paste its values** The dialog asks only for the values the use needs: * **App ID**: Basic Information. * **App-level token** (chat): on Basic Information, choose **Generate Token and Scopes** and add the `connections:write` scope. * **Bot User OAuth Token** (chat): OAuth & Permissions. * **User OAuth Token** (Slack MCP): OAuth & Permissions. **6. Connect** Choose **Connect**. KAOP checks every token with Slack before saving anything. If Slack refuses one, the dialog shows Slack's own reason and keeps what you typed. The result shows where the app is installed, what it serves, and who the Slack MCP acts as. Connecting an app again also lists the MCP servers that use the connection, and warns if they now act as a different Slack user than before. For chat, people link their Slack identity the first time they message the app, as described in [Who the agent is acting for](/manage-your-agents/run/channels#who-the-agent-is-acting-for). > [!NOTE] > > Type the tokens into the dialog yourself. The co-pilot can open the dialog and choose the app, the use > and the App ID, but it never asks for or handles a token. ## Set up the Slack MCP Choose **Set up the Slack MCP** on the result, or **Add server** on the connection's row in **Integrations**. The MCP setup opens on **Expose tools** with this connection chosen. **Test and discover tools** adds the server and tests it; nothing is added before that. ## Who the Slack MCP acts as > [!WARNING] > > The User OAuth Token belongs to the Slack user who installed the app, and every agent that uses the > Slack MCP acts as that user. It can read and search every public channel in your Slack, joined or not, > plus the private channels and DMs that user is in, with their files, canvases and lists, and every > member's profile and email. It can also post, react, upload files, open DMs, create and edit canvases > and lists, and create and manage channels as that user. Install the app with a dedicated Slack > account, kept out of the private channels and DMs agents should not read. ## Replace tokens On the channel's page, choose **Replace tokens** and paste any of the three tokens. Leave the others blank to keep them. The app reconnects within about 10 seconds. A channel that shows a connection error is fixed this way. An app that serves only the Slack MCP has no channel: replace its User OAuth Token from **Integrations**, on its row's menu. ## Disconnect On the channel's page, choose **Disconnect**. For an app that serves both, choose **The whole app**, **Chat only** or **The Slack MCP only**. When chat stops (**The whole app** or **Chat only**), the channel stops as if the app were uninstalled, and its routing rules are kept for **14 days**. When the Slack MCP stops (**The whole app** or **The Slack MCP only**), KAOP first lists the MCP servers that use the connection. They stay set up and stop working. Connecting the same app again brings back the same connection: within those 14 days its routing rules come back, and its MCP servers work again. After disconnecting the whole app, agents the connection was attached to do not get it back: attach it to them again. **Connect again**, on the channel's page or on the app's row in **Integrations**, opens the dialog on your own app with its App ID and what it served filled in. In **Integrations**, **Disconnect** on the app's row does the same, and is how you disconnect an app that serves only the Slack MCP. On an app that is already disconnected, it removes the connection for good. ## Move chat from KAOP's app Chat runs through one app at a time. For the Slack MCP alone there is nothing to move: connect your own app for **Slack MCP** and keep chat on KAOP's app. To move chat from KAOP's app to your own: 1. In **Integrations**, disconnect KAOP's Slack app. Its channel keeps its routing rules, and agents the connection was attached to lose it. 2. Within 14 days, connect your own app for **Chat** or **Both**, installed in the same Slack. The channel comes back with its rules, now on your app. 3. Invite your app to every Slack channel it should read or post in. It is a new Slack user, so it is in none of them yet, and **Channel** and **Keyword** rules match nothing in a channel it is not in. 4. Attach your app's connection to the agents that had KAOP's. Deleting the channel from its page instead drops the rules. ## Next steps > **[Channels](/manage-your-agents/run/channels)** > > Route Slack messages to agents and workflows, and compare the two Slack apps. > **[Built-in integrations](/manage-your-agents/build/built-in-integrations)** > > Where the Slack MCP connection sits, and how a provider's tools are exposed. # Cloud Cost Source: https://docs.komodor.com/modules-and-workflows/cost-optimization/cloud-optimization Track cloud vendor spend, catch the anomaly the week it happens, and convert waste into remediation actions with an owner attached. Cloud Cost owns your cloud vendor bill: what it is made of, why it moved, and what to do about the parts that buy nothing. It breaks spend down by service, account, and tag, detects spend anomalies as new billing data arrives, and turns findings into remediation you approve. ## Module overview A cloud bill is discovered rather than managed. Finance forwards the invoice, someone opens the cost console the week after, deletes the three most obvious things, and does not come back for a quarter. Nothing about that is difficult — it is perpetual, and perpetual work loses to whatever is currently on fire. What accumulates in the gap is boring and expensive: a volume that detached when its instance was terminated and has billed ever since, a gateway in a development network charging by the hour with nothing routed through it, a load balancer that outlived the service behind it, a storage tier full of data nobody has read in two years. None of it pages anyone, because paying too much is not an outage. Good looks like most spend attributable to a named owner, a spend jump traced to its cause within a day rather than on the next invoice, and a reversible action attached to every finding. ## Module enablement | Requirement | Why | Where to set it up | | - | - | - | | Cloud billing export | Every workflow here starts from your own cost and usage data | [Integrations list](/manage-your-agents/build/built-in-integrations) | | Cloud read access | So a finding can be verified against the resource itself, not just the invoice line | [Integrations list](/manage-your-agents/build/built-in-integrations) | | A scoped write credential | Cleanup and lifecycle changes are writes, bound to the agent allowed to make them | [Credentials & secrets](/security-and-governance/identity-and-access/credentials-secrets) | | A ticketing target | So customer-facing remediation lands as a ticket with an owner rather than as an action | [Integrations list](/manage-your-agents/build/built-in-integrations) | ### Turning it on **1. Open Modules → Overview** Cloud Cost appears in the Cost Optimization family with the three workflows it contains. **2. Turn it on** Enabling the module adds Cloud Cost to your sidebar, with its waste findings list. **3. Connect billing and set the autonomy you want** Point the ingest steps at your billing export, then decide which cleanups may run in scope and which must become a ticket. ## The workflows This module contains **3 workflows**. ### AWS spend analyzer | Step | What happens | | - | - | | **1 · CUR ingest** | Pulls the AWS Cost & Usage Report. | | **2 · Spend breakdown** | Attributes spend by service, account, and tag, and identifies the top movers. | | **3 · Recommendation** | Proposes right-sizing or commitment coverage with projected savings — commitment advice where the baseline is steady, right-sizing where the workload is spiky. | | **4 · Act** | Executes reversible cleanup within scope, and opens a remediation ticket for anything customer-facing. | ### S3 optimization | Step | What happens | | - | - | | **1 · Bucket scan** | Inventories buckets, sizes, storage classes, and access patterns. | | **2 · Cold data detection** | Flags cold, idle, or unused data with a waste estimate. | | **3 · Recommendation** | Proposes lifecycle rules or storage-class transitions. | | **4 · Apply** | Applies the policy within autonomy scope — approve-then-run for anything that deletes or transitions data. | ### Cost anomaly detection | Step | What happens | | - | - | | **1 · Billing ingest** | Pulls daily cost and usage across vendors. | | **2 · Anomaly detection** | Detects statistically significant spend spikes and rates their severity. | | **3 · Triage** | Attributes the anomaly to a service, team, or change, and names an owner where it can. | | **4 · Recommend / alert** | Alerts the owner and proposes containment — alert-only for an expected ramp, an action for a sustained spike. | > [!NOTE] > > Anomaly triage is the same investigation shape as an incident: a signal moved, something changed, > correlate the two, produce the evidence. Only the trigger and the evidence store differ. ## What you see when it runs The module lists waste findings and anomalies together, each with the spend it explains, the evidence behind it, the recommendation and its projected savings, and whatever was done about it — an executed cleanup, an applied policy, or a ticket with an owner. Because every conclusion is a run, you can open any finding and read what the agent read: the billing lines, the resource state, and the reasoning that connected them. ## Next steps > **[Kubernetes Cost](/modules-and-workflows/cost-optimization/k8s-cost)** > > The cluster layer of the same bill. > **[Observability Cost](/modules-and-workflows/cost-optimization/observability-optimization)** > > Attribute and reduce spend in your monitoring stack. > **[Approvals](/security-and-governance/controls/approvals)** > > How a deletion waits for a human decision. # Kubernetes Cost Source: https://docs.komodor.com/modules-and-workflows/cost-optimization/k8s-cost Attribute Kubernetes spend by cluster, namespace and workload, and cut waste with policy-driven right-sizing and pod placement. Kubernetes Cost attributes cluster spend down to the workload and turns the gap between requests and real usage into changes it applies for you. Each cluster runs a **K8s Cost agent**: a Helm-installed daemon that reports metrics and carries the admission controller applying the changes. Right-sizing and placement mutate pod specs on admission rather than your manifests, so nothing drifts from what is in Git. ## Module overview Kubernetes makes over-provisioning the rational choice for every individual team. Requests are set generously because the cost of being wrong downward is a page and the cost of being wrong upward is invisible. Multiply that by every workload and you get a cluster bill nobody can attribute and nobody feels responsible for. Two kinds of waste follow from that, and this module owns both. Workloads reserve more CPU and memory than they ever use, and pods that cannot be evicted pin otherwise-empty nodes in place so the autoscaler can never remove them. Good looks like a cluster bill where every line has an owner, requests that match what workloads actually use, and an autoscaler free to remove a node once nothing needs it. ## How the module works Three capabilities, one per tab: cost allocation answers where the money goes, right-sizing and pod placement are the two ways it gets the money back. ### Cost allocation Node cost is derived from instance type, region, operating system and lifecycle, then split across CPU and memory and attributed to the pods scheduled on the node. Resource **requests** drive the attribution, not usage, so the number a team sees is the capacity it reserved. The **Allocation** tab groups that by cluster, namespace or service, over a time window you pick. Each row carries CPU and memory hours, share of allocated cost, unallocated resources and total cost. Unallocated is node capacity no pod requested — it appears as its own column when grouping by cluster and as an aggregated row otherwise. ### Right-sizing Per-workload CPU and memory recommendations are calculated over a 30-day sliding window from the percentile your policy sets, plus that policy's buffers and limits. A workload needs at least seven days of history before it gets a recommendation. The **Right Sizing** tab lists workloads with their recommendations, active and potential savings, how many of their pods are already right-sized, and which policy governs them. Where a workload has no recommendation the row says why: it is still collecting data, it has not been seen in two days, it is managed by an HPA, or its QoS class is excluded by the policy. Applying is done by the admission controller, which mutates the pod on admission and labels it. Deployments and other controllers are never edited. ### Pod placement Pods that cannot be evicted — those using local storage, those protected by a PodDisruptionBudget with no disruptions allowed, and those carrying `cluster-autoscaler.kubernetes.io/safe-to-evict: "false"` or `karpenter.sh/do-not-evict: "true"` — block a node from being scaled down even when the rest of it is idle. The admission controller detects them at creation and adds a **soft** node affinity that prefers nodes already labelled `komodor.com/unevictable=true`, grouping them together and freeing the rest of the cluster for scale-down. The affinity is a preference, so a pod that cannot satisfy it is scheduled normally; existing affinities and taints are never overridden, and DaemonSets are ignored. The **Pod Placement** tab shows per-cluster potential and active savings, and carries the enable and disable action. Placement is turned on per cluster, not per workload. ## Configuring right-sizing Right-sizing is driven by policies, under **K8s Cost → Settings → Right-Sizing Policies**. A policy selects workloads and states how aggressively and when to resize them. | Setting | What it does | | - | - | | **Priority** | An integer from 0 to 1000, unique across your policies. When a workload matches more than one policy, the highest priority wins. | | **Scope** | Statements over cluster, namespace, resource type and workload name, combined with AND. Each accepts explicit values or wildcard patterns with includes and excludes, so `prod-*` covers every matching cluster without a policy per cluster. | | **Optimization preset** | Sandbox, Development, Staging, Production, or Custom. The preset sets the percentile and guardrail defaults; Production leaves the most headroom. | | **Guardrails** | Which resources the policy manages (CPU and memory, requests and optionally limits), CPU and memory buffers, caps on how far a single change may move a value, and whether right-sizing up is allowed. | | **Apply protocol** | **Apply Immediately** patches running pods in place as well as on admission. **Only when pod is created/restarted** waits for the next pod. | | **HPA right-sizing** | Off by default. When on, the HPA's scaling metric is switched from `Utilization` to `AverageValue` before requests are changed, so resizing does not feed back into scaling decisions. | Apply Immediately needs Kubernetes 1.33 or later with the `InPlacePodVerticalScaling` feature gate. Workloads that cannot take an in-place resize — Windows pods, init and ephemeral containers, and any change that would alter the pod's QoS class — fall back to on-creation behaviour rather than being restarted. To take a single workload out of automation regardless of policy, turn off its **Automated** toggle on the Right Sizing tab. To exclude it at the source, annotate it: ```yaml theme={null} metadata: annotations: komodor.io/rightsizing-opt-out: "true" ``` ## Module enablement | Requirement | Why | Where to set it up | | - | - | - | | A K8s Cost agent per cluster | Reports the node, pod and usage metrics every tab reads | **K8s Cost → Settings → Configured Clusters** | | Its admission controller | Applies right-sizing and pod placement; without it the module reports but changes nothing | Installed with the agent | | Kubernetes 1.33 with `InPlacePodVerticalScaling` | Only for Apply Immediately; other clusters right-size on pod creation | Your cluster | | Your enterprise discount rate | Public cloud list prices otherwise overstate what you actually pay | **K8s Cost → Settings → Discount Settings** | > [!NOTE] > > Cloud pricing is read from the providers' public price lists and spot price history, refreshed daily > for on-demand and hourly for spot. Where a price cannot be resolved, **K8s Cost → Settings > → Discount Settings** also holds the fallback CPU and memory monthly costs used instead. ### Turning it on **1. Open Modules → Overview** Kubernetes Cost appears in the Cost Optimization family. It is enabled per account. **2. Connect your first cluster** **K8s Cost → Settings → Configured Clusters → Add cluster** walks three steps: name the cluster, run the Helm command it gives you, and wait while it confirms the agent reporting in. The command carries the installation key already filled in, with a copy button. Asking the co-pilot to connect a cluster opens the same three steps. The agent installs in cost mode: metrics and live cost data only. It does not read pod logs or exec into containers. **3. Wait for data** Cost data becomes available 24 hours after your first cluster connects. Right-sizing needs seven days of history per workload; until then a banner names the clusters still collecting. **4. Create a right-sizing policy and enable placement** Add a policy under **K8s Cost → Settings → Right-Sizing Policies** to start applying recommendations, and enable pod placement per cluster from the **Pod Placement** tab. > [!WARNING] > > The installation key is shown once and cannot be read back. Connecting the same cluster again mints > a second key and leaves the first installation holding its own — reconnect only after a key rotation > or a reinstall. Access is governed by four capabilities, assigned through [roles and permissions](/security-and-governance/identity-and-access/roles-permissions): | Capability | Grants | | - | - | | `k8s_cost.read` | Every K8s Cost tab, and the module's entry in the sidebar | | `k8s_cost.connect_cluster` | **Add cluster** | | `k8s_cost.policy.manage` | Creating and editing right-sizing policies, the pod-placement toggle, and discount settings | | `k8s_cost.service.assign_policy` | Pinning a workload to a policy, and excluding one from automation | ## What you see when it runs **Overview** carries total monthly cost split into allocated and unallocated, potential savings from right-sizing and pod placement, and the impact already realised — right-sized savings, automated workload count, placement savings, automated cluster count, and under-provisioned containers fixed. **Allocation**, **Right Sizing** and **Pod Placement** each hold the table behind one of those numbers, filterable by cluster and exportable. A right-sizing row expands into per-container usage against its recommendation, so you can see the evidence for a number before automating the workload. **K8s Cost → Settings → Configured Clusters** lists every K8s Cost agent reporting in with its version, when it was connected and how long ago it last sent a heartbeat. Agents that have stopped reporting are hidden until you turn on **Show inactive agents**. Opening a row shows the agent's id, chart version, Kubernetes version and full reported Helm values, with credential-shaped values redacted. Every policy change and placement toggle is recorded in the audit log. ## Next steps > **[Cloud Cost](/modules-and-workflows/cost-optimization/cloud-optimization)** > > The same loop one layer down, on your cloud vendor bill. > **[Roles & permissions](/security-and-governance/identity-and-access/roles-permissions)** > > Which role carries each cost capability. # Observability Cost Source: https://docs.komodor.com/modules-and-workflows/cost-optimization/observability-optimization Find the observability spend that buys nothing — high-cardinality metrics, unread logs, unused dashboards — with an agent configured to manage that provider. Observability Cost owns what your monitoring stack costs and what that money buys. It attributes vendor spend to the teams and services generating it, finds the volume nobody queries, and proposes the retention, sampling, and filter changes that remove the waste. ## Module overview Observability spend grows by default. Every new service ships metrics, every incident adds a dashboard, and one high-cardinality label added in a hurry can move a bill by a noticeable percentage. Nothing removes any of it, because deleting telemetry feels like deleting the thing that will explain the next outage. That fear is reasonable, which is why this module leads with attribution rather than deletion: knowing which tool, which service, and which team a line of spend belongs to turns a blunt budget conversation into a specific one. Then waste detection does the unglamorous part — finding the metric with cardinality nobody queries, the dashboard nobody opens, the log source producing volume nobody reads — and every proposal states what you lose by accepting it. Good looks like observability spend that maps to owners, a ranked list of what to cut, and no investigation ever failing because this module removed something. ## Module enablement | Requirement | Why | Where to set it up | | - | - | - | | Observability vendor access | The module reads usage and volume per tool, and the assets that consume it | [Integrations list](/manage-your-agents/build/built-in-integrations) | | Vendor billing or usage export | Attribution needs the spend side of each tool | [Integrations list](/manage-your-agents/build/built-in-integrations) | | Service ownership | So spend lands on a team rather than on a tag nobody recognizes | [Knowledge base](/manage-your-agents/build/knowledge-base) | | A scoped write credential | Applying a retention or filter change is a write into your monitoring configuration | [Credentials & secrets](/security-and-governance/identity-and-access/credentials-secrets) | ### Turning it on **1. Open Modules → Overview** Observability Cost appears in the Cost Optimization family with the two workflows it contains. **2. Turn it on** Enabling the module adds it to your sidebar, with its waste findings list. **3. Connect your tools and decide who acts** Point the ingest steps at each observability vendor, then decide whether a change is applied by an agent with coverage for that tool or filed as an opportunity for the owning team. ## The workflows This module contains **2 workflows**. ### Observability cost allocation | Step | What happens | | - | - | | **1 · Billing ingest** | Pulls vendor invoices and usage exports. | | **2 · Attribution** | Allocates spend by tool, service tags, and owning team. | | **3 · Showback** | Produces a per-team showback report, and a recurring digest where you want one. | ### Observability spend reduction | Step | What happens | | - | - | | **1 · Usage ingest** | Pulls metrics, logs, and trace volume with the billing per tool. | | **2 · Waste detection** | Flags high-cardinality metrics, unused dashboards, and log spam, with a waste estimate. | | **3 · Recommendation** | Proposes retention, sampling, or drop-filter changes with projected savings. | | **4 · Apply** | Applies the change, or files it as an opportunity for the owning team. | > [!NOTE] > > Any retention or sampling change is approve-then-run: it is applied only after a human decision, > because it changes what you will be able to query later. ## What you see when it runs The module lists waste findings — the metric, dashboard, or log source, what it costs, and what the proposal would save — plus the showback records the allocation workflow produces. Each finding carries the run behind it, so a recommendation to drop something is reviewable rather than a suggestion out of nowhere. ## Next steps > **[Cloud Cost](/modules-and-workflows/cost-optimization/cloud-optimization)** > > The cloud-bill context around observability spend. > **[Alert Intelligence](/modules-and-workflows/sre/alert-hygiene)** > > The quality side: which of these signals is worth keeping. > **[Observability & OTel export](/manage-your-agents/optimize/observability-otel-export)** > > Sending the platform's own telemetry to the tools you already pay for. > **[Approvals](/security-and-governance/controls/approvals)** > > How a retention change waits for a human decision. # Change Intelligence & Risk Control Source: https://docs.komodor.com/modules-and-workflows/ops-automation/change-intelligence Decide whether the change that just shipped is good — and undo it if not — with a verdict backed by the signals inside its blast radius. Change Intelligence & Risk Control owns the minutes and hours around a change landing in production. It watches the signals inside that change's blast radius, compares a canary against its baseline, and returns a ship, hold, or roll back verdict with the evidence behind it. ## Module overview The verdict on a deploy already exists — it is just made informally. Someone merges, watches a dashboard for two minutes, sees nothing obviously red, and moves on. The judgement is real work being done with no baseline, no idea which of forty panels sits inside the change's blast radius, and no record of the decision. The failures that slip through are specific: the regression that only appears under the traffic pattern arriving at 4pm, the change that is fine at p50 and doubles p99, the deploy that is correct on its own and interacts badly with a migration from an hour earlier. Change Intelligence & Risk Control makes the verdict explicit — a stated comparison against a baseline, on the signals that belong to this change, producing a decision you can audit and a rollback you can approve. Good looks like a trustworthy verdict within minutes of a deploy, most rollbacks starting from that verdict rather than from an incident, and a false-rollback rate low enough that nobody overrides the module by week two. ## Module enablement | Requirement | Why | Where to set it up | | - | - | - | | A deploy event source | Something has to say "reality just changed" — an endpoint your GitOps or CD system posts to | [Integrations list](/manage-your-agents/build/built-in-integrations) | | Observability access | So the verdict is computed from your own latency, error, and cost signals | [Integrations list](/manage-your-agents/build/built-in-integrations) | | Rollback credentials | A rollback is a write to production, so the agent that performs one is bound to a narrowly scoped credential | [Credentials & secrets](/security-and-governance/identity-and-access/credentials-secrets) | | Evaluation judges | A verdict is a rubric-scored decision; judges are what make it reviewable | [Evaluations](/manage-your-agents/optimize/evaluations) | ### Turning it on **1. Open Modules → Overview** Change Intelligence & Risk Control appears in the AI Software Operations family with the three workflows it contains. **2. Turn it on** Enabling the module adds Change Intelligence & Risk Control to your sidebar. **3. Point your delivery system at it** Create the endpoint your CD system posts deploy events to, then bind an agent to each step you expect to run. ## The workflows This module contains **3 workflows**. ### Post-deploy verdict Starts when your delivery system reports that something shipped. It identifies the blast radius of the change, watches the signals inside it over a soak window, and returns ship, hold, or roll back with the comparison that produced the answer. ### Canary judgement Starts at a canary step in a progressive rollout. It compares the canary against its baseline on latency, errors, and cost together, then promotes or aborts — so a canary that is marginally faster and substantially more expensive does not pass unnoticed. ### Change blast-radius brief Runs before the change ships. It states what this change can break, who owns those services, and which signals to watch — the pre-flight brief nobody writes by hand. > [!NOTE] > > Because a verdict is a watch-and-compare loop over a soak window rather than a single request, each > of these workflows spans several steps and several runs. The whole sequence is durable: it survives > a restart mid-soak. ## What you see when it runs The module lists every change it has judged. Each record carries the change that triggered it, the blast radius it derived, the signals it compared and over what window, and the verdict — with the runs behind each stage available to open. A roll back verdict is a proposed production action, so it arrives as a decision for a human rather than something already done. You approve or reject it with the comparison in front of you. ## Next steps > **[CI/CD Health & Remediation](/modules-and-workflows/ops-automation/ci-cd-pipeline-health)** > > Keep the delivery path healthy before and after a change merges. > **[Incident management & troubleshooting](/modules-and-workflows/sre/incidents)** > > Where a missed verdict ends up. > **[Approvals](/security-and-governance/controls/approvals)** > > How a rollback waits for a human decision. > **[Triggers & schedules](/manage-your-agents/build/triggers-schedules)** > > How a workflow starts, on a schedule or from an endpoint. # CI/CD Health & Remediation Source: https://docs.komodor.com/modules-and-workflows/ops-automation/ci-cd-pipeline-health Keep the merge path trustworthy — flaky stages quarantined, queue time cut, and a broken delivery stage caught before it blocks everyone. CI/CD Health & Remediation owns whether your merge path can be trusted. It quarantines the stages that fail for no reason, attacks the queue time that makes people stop running things, and catches a broken delivery stage early. ## Module overview A merge pipeline degrades quietly. One stage starts failing intermittently, people learn to re-run it, and re-running becomes reflex — at which point the suite has stopped being a signal and started being a toll. Queue time grows the same way: nobody notices the day it crosses from two minutes to twenty, but by then engineers batch their pushes and review cycles get slower. Neither shows up as an incident, and both are measurable. This module treats the delivery path as something with health worth tracking: which stages fail, how often, for real reasons or not, and what that costs in waiting. Good looks like a suite whose failures are believed, a queue time nobody works around, and a quarantine list that shrinks because quarantined stages get fixed rather than forgotten. ## Module enablement | Requirement | Why | Where to set it up | | - | - | - | | Your build system | The module reads stage results, durations, and queue times | [Integrations list](/manage-your-agents/build/built-in-integrations) | | Your source host | So a quarantine or configuration change arrives as a reviewable pull request | [Integrations list](/manage-your-agents/build/built-in-integrations) | | Stage ownership | A quarantined stage needs an owner, or it becomes permanent | [Knowledge base](/manage-your-agents/build/knowledge-base) | | A scoped write credential | Where you let the module apply a quarantine rather than propose it | [Credentials & secrets](/security-and-governance/identity-and-access/credentials-secrets) | ### Turning it on **1. Open Modules → Overview** CI/CD Health & Remediation appears in the AI Software Operations family with the two workflows it contains. **2. Turn it on** Enabling the module adds it to your sidebar. **3. Point it at your merge path** Connect the build system and the repositories it serves, then decide whether a quarantine is applied automatically or proposed for review. ## The workflows This module contains **2 workflows**. ### Flaky stage quarantine Runs on a schedule you set. It identifies the stages failing intermittently rather than meaningfully, quarantines them so they stop blocking merges, and keeps the list visible so a quarantine is a temporary state with an owner rather than a permanent exemption. ### Queue-time & failure-rate triage Runs on a schedule. It reports where time actually goes in the merge path — queueing versus running — and which stages carry the failure rate, then proposes the change that removes the biggest share. ## What you see when it runs Each finding is a record: the stage, its failure rate, whether those failures were real, the time it costs, and what the module proposes. Quarantines carry an owner and the date they started, so the list reads as debt rather than as configuration. ## Next steps > **[Change Intelligence & Risk Control](/modules-and-workflows/ops-automation/change-intelligence)** > > Judge the production impact of a change after it leaves the merge path. > **[Production Readiness & Standards](/modules-and-workflows/ops-automation/production-readiness)** > > Gate release candidates against the standards required for production. > **[Approvals](/security-and-governance/controls/approvals)** > > How a quarantine reaches a human decision. # Production Readiness & Standards Source: https://docs.komodor.com/modules-and-workflows/ops-automation/production-readiness Score services and release candidates against production standards, block unsafe releases, and manage evidence-backed waivers. Production Readiness & Standards decides whether a service is ready to enter or remain in production. It scores services and release candidates against standards you define, and records the evidence behind each score so a gate or a waiver is a reviewable decision rather than a judgement call. ## Module overview Production standards tend to live across templates, review checklists, and platform knowledge. That makes readiness depend on who reviews a release and whether they remember every requirement. This module applies a named standards pack consistently, gathers evidence for every failure, and routes exceptions through an explicit waiver. Good looks like every production service carrying an accountable owner, operational evidence, and a current readiness verdict before traffic depends on it. ## Module enablement | Requirement | Why | Where to set it up | | - | - | - | | Source and delivery systems | The module reads release candidates and posts gate results back to the delivery path | [Integrations list](/manage-your-agents/build/built-in-integrations) | | Service configuration | Standards are scored against the service's deployment and operational metadata | [Integrations list](/manage-your-agents/build/built-in-integrations) | | Standards knowledge | The selected standards pack defines the checks and evidence required for each service | [Knowledge base](/manage-your-agents/build/knowledge-base) | | Notification channel | Owners receive regressions, failed gates, and waiver requests where they work | [Channels](/manage-your-agents/run/channels) | ### Turning it on **1. Open Modules → Overview** Production Readiness & Standards appears in the AI Software Operations family with the three workflows it contains. **2. Review the configured module** Open the module from your sidebar and choose the standards pack and service scope for each workflow. **3. Set gate and waiver policy** Choose whether a failing score blocks, warns, or only records a result, then set who may grant a waiver and when that waiver expires. ## The workflows This module contains **3 workflows**. ### Pre-release readiness gate Starts when a release candidate is ready. It selects the standards that apply to the service, scores the candidate, gathers evidence for each failed standard, and either passes the release, blocks it, or routes a scoped waiver for approval. ### Standards drift sweep Runs nightly across production services. It selects the applicable standards for each service, compares the new score with the previous run, and notifies owners when a service regresses after its initial release. ### New service onboarding review Starts from a new-service request. It checks the requirements needed before traffic is enabled, records the evidence behind the verdict, and creates owned follow-up work for any gaps. ## What you see when it runs The module lists readiness reviews and standards regressions. Each record shows the service, standards pack, score, verdict, failed requirements, and evidence collected for each one. A blocked release that permits an exception carries the requested waiver, approver, and expiry alongside the original failure. ## Next steps > **[CI/CD Health & Remediation](/modules-and-workflows/ops-automation/ci-cd-pipeline-health)** > > Keep the delivery path trustworthy before applying a production gate. > **[Change Intelligence & Risk Control](/modules-and-workflows/ops-automation/change-intelligence)** > > Judge the effect of a change after it reaches production. > **[Approvals](/security-and-governance/controls/approvals)** > > Control who may approve a waiver or blocked release. > **[Standards & Guardrails](/security-and-governance/controls/standards-guardrails)** > > Define the controls that govern production work. # Modules & workflows Source: https://docs.komodor.com/modules-and-workflows/overview A module is an operational outcome you own; the workflows inside it are the automation that delivers it. Modules are how the Komodor Agentic Operation Platform (KAOP) ships operational work you can turn on rather than build. A module is an outcome you own; a workflow is a sequence of steps inside it, and each step delivers a task to one or more agents. ## The one distinction that matters **A module is an outcome you own. A workflow is a sequence of steps inside it. A module contains workflows.** | | Module | Workflow | | - | - | - | | Answers | *What am I accountable for?* | *How does that get done?* | | Example | Incident management & troubleshooting | "Production Alert Investigation" | | Scope | A domain — incidents, cost, drift | One configured path through that domain | | You | Enable it for an account | Operate it, and read what it produces | | Contains | One or more workflows | Steps, each delivering a task to one or more agents | So "Incident management & troubleshooting" is not a workflow, and "Production Alert Investigation" is not a module. The module is the domain; the workflow is the automation. When you open a module you see its workflows; when you open a workflow you see its steps. ```mermaid theme={null} flowchart TD M[Module — Incident management & troubleshooting] --> W1[Workflow — Production Alert Investigation] W1 --> S1[1 · Source] W1 --> S2[2 · Dedupe & correlate] W1 --> S3[3 · Investigation] W1 --> S4[4 · Remediation] S3 --> A1[Datadog Investigator] S3 --> A2[Kubernetes Investigator] S3 --> A3[AWS Investigator] ``` ## What a workflow is made of A workflow is an ordered set of **steps**, and each step delivers a task to one or more **agents**. A step names the work; the agents bound to it do the work. Because binding is explicit, you can see exactly which agent handles which stage — and a step with nothing bound still carries its task, it just has no agent to execute it. That structure is what lets a module coordinate specialists: one step correlates the incoming signal, the next fans out to domain investigators, the next proposes a remediation and asks a human to approve it. ## The three families Every module belongs to one family, which tells you the kind of outcome it owns. > **[AI SRE](/modules-and-workflows/sre/incidents)** > > Keep production healthy — incident response, alert intelligence, and proactive reliability. > **[Cost Optimization](/modules-and-workflows/cost-optimization/k8s-cost)** > > Spend less across Kubernetes, cloud infrastructure, and observability. > **[AI Software Operations](/modules-and-workflows/ops-automation/change-intelligence)** > > Control change risk, production readiness, and CI/CD health. ## Enabling a module Modules are enabled per account. **Modules → Overview** lists the modules configured for yours, across the three families above. Each entry shows its family, what it owns, and how many workflows it contains. **1. Open Modules → Overview** Find the module under **Configured**. **2. Open the module** Configured modules appear in your sidebar with their workflows ready to operate. **3. Connect what it needs** A module reads and writes through integrations. Its page lists what to connect before its workflows can do useful work. > [!NOTE] > > Disabling a configured module moves it to **Available** and removes it from the sidebar. Enable it > again from the catalog when you want to operate its workflows. ## What a module produces An enabled module gives you a place to work: a list of the records it has produced — incidents, findings, drifts, verdicts — each carrying the evidence behind it, the runs that generated it, and any decision it needs from a human. That record list is the module's home in the sidebar. ## Next steps > **[Concepts & glossary](/get-started/concepts)** > > The full vocabulary, in one place. > **[Incident management & troubleshooting](/modules-and-workflows/sre/incidents)** > > The module most teams start with. > **[Approvals](/security-and-governance/controls/approvals)** > > How a workflow asks a human before it acts. # Alert Intelligence Source: https://docs.komodor.com/modules-and-workflows/sre/alert-hygiene Classify which monitors are worth waking someone for, and propose the threshold, dedupe, and suppression fixes that quiet the rest. Alert Intelligence owns the quality of your alerting itself: which monitors fire usefully, which fire constantly and get ignored, and what to change about each one. Every monitor ends up carrying a classification and the rationale behind it, so the list doubles as a work queue. ## Module overview Every alerting setup accumulates monitors nobody trusts. A threshold was right eighteen months ago, the service scaled, and now it fires nine times a week and gets acknowledged without being read. The on-call learns which pages to ignore, and the one that mattered arrives in the same stream as the ones that never do. Fixing it is unglamorous, evidence-heavy work: pull each monitor's firing history, decide whether its alerts were true, false, or simply not actionable, and then argue for a specific change. Alert hygiene does that work continuously and hands you the proposal — a tuned threshold, a dedupe rule, a suppression window — with the history that justifies it. Good looks like a monitor set where every page is worth reading, and where the count of monitors under review keeps falling instead of quietly growing. ## Module enablement | Requirement | Why | Where to set it up | | - | - | - | | A monitoring provider | The module reads monitor definitions and firing history — Datadog, PagerDuty, Grafana, and the other catalog providers | [Integrations list](/manage-your-agents/build/built-in-integrations) | | Provider credentials | Each ingest agent needs read access to the provider it polls, and nothing more | [Credentials & secrets](/security-and-governance/identity-and-access/credentials-secrets) | | Classification agents | The specialists bound to the analysis and optimization steps | [Use specialized agents](/manage-your-agents/build/use-specialized-agents) | ### Turning it on **1. Open Modules → Overview** Alert Intelligence appears in the AI SRE family with the workflow it contains. **2. Turn it on** Enabling the module adds Alert Intelligence to your sidebar, alongside its monitor list. **3. Connect a provider and bind agents** Point the ingest step at your monitoring provider and bind an agent to each step you expect to run. ## The workflows This module contains **1 workflow**. ### Alert noise reduction Reads your monitors' firing history, classifies each one, and proposes the fix. | Step | What happens | | - | - | | **1 · Alert ingest** | Backfills each monitor's fire, page, and flap history from the connected providers. This step is shared with the Incident management & troubleshooting workflow, so editing it there changes it here too. | | **2 · Signal analysis** | Classifies every monitor — true positive, false positive, actionable or not — and records the rationale behind the call. | | **3 · Noise optimization** | Proposes the change: threshold and window tuning, a dedupe rule, or a suppression window, with the trade-off each one carries. | Each step delivers its task to the agents bound to it. The analysis step is where the specialists matter most: one classifies signal quality, one detects flapping patterns, one finds duplicate signals firing on the same underlying condition. > [!NOTE] > > Steps 2 and 3 are graded by evaluation judges, so a classification and a proposed fix each carry a > verdict you can open. See [Evaluations](/manage-your-agents/optimize/evaluations). ## What you see when it runs The module's sidebar entry lists your monitors, each one carrying what the workflow concluded about it: * **Firing history** — how often it fired and flapped over the last seven days, and how often anyone acknowledged it. * **The classification**, and the rationale for it. * **The proposed fix**, with its trade-offs stated, so you can see what you lose by quieting it. Every monitor carries its review status and the disposition of its recommendations, so the list doubles as your work queue. Because each conclusion is produced by a run, you can open any monitor and read exactly what the agent saw before it made the call. ## Next steps > **[Incident management & troubleshooting](/modules-and-workflows/sre/incidents)** > > Where the alerts you keep end up being investigated. > **[Approvals](/security-and-governance/controls/approvals)** > > How a proposed monitor change waits for a human decision. > **[Evaluations](/manage-your-agents/optimize/evaluations)** > > The judges that grade a classification and a proposed fix. # Incident management & troubleshooting Source: https://docs.komodor.com/modules-and-workflows/sre/incidents Detect, group, investigate, remediate, and learn from production incidents — with the evidence behind every conclusion. The Incident management & troubleshooting module owns the path from an incoming alert to an understood, remediated production incident. Detection, grouping, investigation and remediation each run as steps that leave their own evidence, so the conclusion can be audited rather than trusted. ## Module overview An alert fires. Somebody acknowledges it, opens three dashboards, checks what deployed recently, asks in Slack whether anyone else is seeing it, and eventually forms a theory. Most of that is retrieval, and most of it is repeated on the next alert. Incident management & troubleshooting takes that work. It groups related alerts into one incident, dispatches domain specialists to gather evidence in parallel, produces a root-cause conclusion with the evidence attached, and proposes a remediation for a human to approve. What you own is the decision; what the module owns is everything leading up to it. Good looks like: one incident per real problem rather than one per alert, a conclusion you can audit, and a remediation you approve rather than research. ## Module enablement | Requirement | Why | Where to set it up | | - | - | - | | An alert source | Something has to start the workflow — an endpoint your monitoring posts to, or a connected provider | [Integrations list](/manage-your-agents/build/built-in-integrations) | | Observability access | So investigators can read metrics, logs, and traces around the alert | [Integrations list](/manage-your-agents/build/built-in-integrations) | | Investigation agents | The specialists bound to the investigation step | [Use specialized agents](/manage-your-agents/build/use-specialized-agents) | | Slack (optional) | Post the conclusion where the on-call is already looking | [Channels](/manage-your-agents/run/channels) | ### Turning it on **1. Open Modules → Overview** Incident management & troubleshooting appears in the AI SRE family with the workflow it contains. **2. Turn it on** Enabling the module adds Incident management & troubleshooting to your sidebar. **3. Connect a source and bind agents** Point an alert source at the workflow and bind an agent to each step you expect to run. ## The workflows This module contains **1 workflow**. ### Production Alert Investigation Correlates production alerts, runs a multi-agent root-cause analysis, and governs remediation. | Step | What happens | | - | - | | **1 · Source** | Which alerts start this workflow, and from where. | | **2 · Dedupe & correlate** | Collapse repeats and related alerts into a single incident. | | **3 · Investigation** | Domain specialists gather evidence in parallel — the observability, Kubernetes, and cloud investigators each contribute what they can see. | | **4 · Remediation** | Propose the fix and route it for approval before anything is applied. | Each step delivers its task to the agents bound to it, so you can see exactly which agent handled which stage of an investigation. > [!NOTE] > > A step with no agents bound still carries its task — it just has nothing to execute it. Bind an > agent to every step you expect to run. ## What you see when it runs The module's sidebar entry lists every incident it has produced. Each one carries: * **The alert that started it**, and any alerts folded into it during correlation. * **The investigation** — which specialists ran, what each found, and the runs behind them. * **The conclusion**, with the evidence that supports it. * **The proposed remediation**, and whether it is waiting on a human. Because every step is a run, an incident is auditable end to end: you can open any stage and read exactly what the agent saw and did. ## Next steps > **[Runs & evidence](/manage-your-agents/run/runs-evidence)** > > How to read the evidence trail behind a conclusion. > **[Approvals](/security-and-governance/controls/approvals)** > > How remediation waits for a human decision. > **[Alert Intelligence](/modules-and-workflows/sre/alert-hygiene)** > > Fix the noise upstream so fewer alerts become incidents. # Proactive reliability optimization Source: https://docs.komodor.com/modules-and-workflows/sre/reliability-optimization Turn indicators, error budgets, and recurring failure patterns into concrete reliability improvements — including the agent that eliminates the recurrence. Proactive reliability optimization owns the work that happens when nothing is on fire: reading what keeps going wrong and turning it into a specific improvement. It triages where the error budget is going and proposes the fix for a failure that has now happened often enough to be a pattern. ## Module overview Incident response competes on speed of resolution. This module changes the axis to elimination — the recurring cause that is handled well every time and never removed. Nobody gets to that work, because it is never the most urgent thing in the week, and the person who would do it is the person the incidents already consumed. The economics are the interesting part: the highest-return work is the work that runs when nothing is wrong. A cause that produces a page every three weeks costs more over a quarter than the change that would remove it, but the change never gets prioritized against a live incident. This module puts the proposal in front of you at the moment the evidence is strongest — right after the incident that proves it. Good looks like a falling count of repeat causes, and each closed one traceable to a specific change you approved. ## Module enablement | Requirement | Why | Where to set it up | | - | - | - | | Incident history | Recurrence is only visible across incidents, so the module reads what has already happened | [Incident management & troubleshooting](/modules-and-workflows/sre/incidents) | | Observability access | So burn triage is computed from your own indicators | [Integrations list](/manage-your-agents/build/built-in-integrations) | | The systems the fix touches | A proposed maintenance agent needs access to whatever it maintains, scoped to that job | [Credentials & secrets](/security-and-governance/identity-and-access/credentials-secrets) | | A knowledge base | Holds what has already been ruled out or accepted, so the same proposal is not made every month | [Knowledge base](/manage-your-agents/build/knowledge-base) | ### Turning it on **1. Open Modules → Overview** Proactive reliability optimization appears in the AI SRE family with the two workflows it contains. **2. Turn it on** Enabling the module adds it to your sidebar. **3. Point it at your history** Give it the incident history and indicators to read, then bind an agent to each step you expect to run. ## The workflows This module contains **2 workflows**. ### Error-budget burn triage Reads where reliability budget is actually going and ranks it — which service, which cause, and how much of the budget each consumed. The output is an ordered list of what to fix, rather than a dashboard you have to interpret. ### Recurring failure → fix propose Runs when a failure pattern repeats. It identifies the recurrence, then proposes the concrete improvement — frequently an agent that performs the maintenance the recurrence needs, configured with its trigger, its permitted actions, and a dry run before it is live. > [!NOTE] > > After an incident resolves, a **Reliability optimization** insight appears on the incident itself > with the recommended next agent and a link straight into building it, so the proposal arrives while > the incident is still fresh. ## What you see when it runs The module lists the improvements it has proposed, each with the pattern that justifies it: how often the failure recurred, what it cost in budget, and what specifically it proposes to change. Where the proposal is an agent, you get its configuration to review — trigger, permitted actions, and a dry run — before anything is live. Nothing here changes production on its own. Every proposal is a decision you take, with the evidence that motivated it attached. ## Next steps > **[Incident management & troubleshooting](/modules-and-workflows/sre/incidents)** > > Where recurring production failures and their evidence are recorded. > **[Runs & evidence](/manage-your-agents/run/runs-evidence)** > > How to inspect the history behind a reliability proposal. # What's new Source: https://docs.komodor.com/reference/changelog The latest features, improvements, and fixes in KAOP. See what's new in the Komodor Agentic Operation Platform (KAOP). Updates are listed by release date, with the newest first. They are the same updates you see under **What's new** in the product. *** **October 2, 2026** ### Improved: A dataset's page leads with its latest replay A dataset's header now shows four things: the replay agent, the cases (and how many are switched on for replay), the **latest replay** with its verdict, and when it was created. The default replay agent and **Minimum coverage** moved into **Settings** in the dataset's menu, next to **Copy dataset ID**. The Live/Candidate filter on Run history is gone — each replay already names the agent version it ran against. [Try it in KAOP](https://kaop.komodor.com/quality-lab/datasets) **October 1, 2026** ### Improved: Add a run to a dataset and fix its output in one step **Add to dataset** on a run now shows the run's output, and you can correct it before saving. What you add is ready to replay straight away; there is no separate approval step. On a dataset's page each case has a **Replay** switch to include or exclude it, **Edit output** in its row menu, and clicking a case opens the run it came from. Cases added by an agent or a capture rule wait with Replay off until someone turns them on. The Recorded cases tab and the review screen are gone. [Try it in KAOP](https://kaop.komodor.com/quality-lab/datasets) **September 30, 2026** ### Improved: A replay compares two answers, not three A dataset replay now shows the **Recorded answer** beside the **Replay answer**, next to the evaluator's verdict. The middle "Original recorded" column is gone: approving a recording makes its recorded output the answer a replay is judged against, so the two columns always showed the same thing. [Try it in KAOP](https://kaop.komodor.com/quality-lab/datasets) **September 29, 2026** ### Fixed: A tag you type is saved even if you don't press Enter Tag fields used to keep only the tags you had turned into chips with Enter. Text still sitting in the box when you clicked Save was dropped, so **Edit tags** on a dataset could report "Dataset tags updated" while saving nothing. Now any text left in the box becomes a tag as soon as you click away, on every tag field: dataset and record tags, capture rules, scenarios and integrations. [Try it in KAOP](https://kaop.komodor.com/quality-lab/datasets) ### Fixed: Pick tools in the candidate wizard again The **Tool selection** picker on a candidate's Change step said "No tools in the catalog yet" even when your MCP servers advertised tools, because the catalog request was rejected. It now loads your tools, and if the catalog ever cannot be loaded the step says so and offers a retry instead of showing an empty list. [Try it in KAOP](https://kaop.komodor.com/candidates) **September 28, 2026** ### New: Tag datasets and filter the Datasets list by tag A dataset can now carry its own `key=value` tags (`team=payments`, `env=prod`). Add them in **New dataset**, or change them any time with **Edit tags** on the dataset page. The Datasets list shows each dataset's tags, and its tag chips filter the list: pick several and only datasets carrying every one of them stay. Dataset tags are for finding datasets — which records a replay runs is still chosen by the records' own tags. [Try it in KAOP](https://kaop.komodor.com/quality-lab/datasets) ### Improved: Set tool permissions before you deploy The **Tool permissions** card on the Add-Agent wizard's Permissions step used to tell you to come back after the deploy: a permission attaches to an agent, and an agent deployed from the catalog only gets its id at the very end. So the one screen that asks which tools an agent may run on its own, and which ones need a person to say yes first, was the one screen you could not answer there. You can answer it now. Author the policy while you are building the agent, and it is applied with the deploy — the agent's first run is already governed by it. The last step says **Done** rather than Save, because nothing is written until the deploy goes through. The tool picker works too. It offers the tools this agent will actually be able to reach, read from the integration scope you chose one step earlier — not the whole account's catalog, which would let you write a rule over a tool the agent has no route to: armed, matching nothing, and silent about why. [Try it in KAOP](https://kaop.komodor.com/fleet/new) ### Improved: Scope a catalog agent while you deploy it Deploying an agent from the catalog used to skip the **Permissions** step: the roles it holds and the secrets it may fetch were something you set afterwards, from the agent's Permissions tab in Fleet, once it was already running. The step is on that rail now, and what you choose there rides the deploy itself — there is no window in which the agent exists unscoped. A handful of workers do not have the step, and would not use it. The workflow orchestrator is scoped by the roster its **Workers** step defines, and each specialist it delegates to carries its own roles. The Golden Scenarios workers, the base evaluator and the memorizer are invoked by a pipeline or by the platform rather than by you, so their scope is not yours to set. Every other catalog agent gets the step, and the tool-permission card on it, exactly as a hand-built one does. [Try it in KAOP](https://kaop.komodor.com/fleet/new) ### Improved: Replaying a dataset has its own permission Replaying a dataset now needs the **golden\_scenario.run** permission as well as permission to invoke the agent it replays against. Developers, admins, owners and automation hold it by default; members and operators, who can read datasets, no longer see **Replay now** or **Replay by tags** enabled. A custom role that replayed datasets before needs `golden_scenario.run` added. [Try it in KAOP](https://kaop.komodor.com/quality-lab/datasets) ### Fixed: Replays show their evaluator, filter by candidate or live, and stay out of bulk capture A dataset replay now records and shows the evaluator's model and prompt version on its page, next to whether it ran on the live agent or a candidate, and a dataset's Replays list filters by All, Live or Candidate. Capture to dataset from History no longer picks up dataset replays as new cases. [Try it in KAOP](https://kaop.komodor.com/quality-lab/datasets) **September 27, 2026** ### New: Standing capture rules automatically add runs to a dataset Quality Lab → Datasets now has a **Capture rules** tab. A rule watches an agent's succeeded runs and, on a match, captures them into a dataset automatically — tagged, sampled at whatever rate you choose, and always left awaiting approval so nothing ships without a person reviewing it. Pause, edit or delete a rule at any time. [Try it in KAOP](https://kaop.komodor.com/quality-lab/datasets?datasets-tab=capture-rules) ### New: Review awaiting-approval cases one at a time Quality Lab → Datasets now has a review screen that walks through awaiting-approval cases one at a time, scoped to a single dataset or the whole case library. Each case shows its input, tool calls and results, and expected output, with **Approve** (with an optional note) and **Skip** actions and a running "N remaining" count. [Try it in KAOP](https://kaop.komodor.com/quality-lab/datasets/review) ### New: Rerun history shows candidate vs live, and which evaluator judged it A dataset rerun's history now says whether it ran on a Candidate or the agent's live version, and records the judge's model plus its prompt version. Rerun history can be filtered by `agent_variant=live` or `candidate`. [Try it in KAOP](https://kaop.komodor.com/quality-lab/datasets) ### Improved: One verdict per rerun — pass, failed or inconclusive Every dataset rerun now shows a single **Verdict**: pass, failed, or inconclusive. A replay gap (a tool call with no recorded answer) always means inconclusive, even if the judge still reached a same/different opinion — that opinion stays visible, it just no longer counts as the verdict. A contaminated run (a live call happened) keeps its own distinct label rather than being folded in. [Try it in KAOP](https://kaop.komodor.com/quality-lab/datasets) ### New: See which org and repositories a GitHub connection reaches Open **View details** on a GitHub App connection in Integrations to see the organization it is installed on, whether it covers all repositories or a selected set, and the repositories themselves — searchable, with a link to manage the installation on GitHub. The dialog also shows who connected the integration. The list is read live from GitHub and cached for ten minutes; **Refresh** fetches it again on demand. [Try it in KAOP](https://kaop.komodor.com/integrations) ### Improved: Funnel drill-down rows open the incident Clicking a terminal stage of the alerts funnel lists the incidents that ended there. Each of those rows is now a link to the incident's own page, so the drawer is a way in rather than a dead end. The link closes the drawer as it navigates; open it in a new tab or from the keyboard and the drawer stays where it was. Alert rows are unchanged — an alert has no page of its own. [Try it in KAOP](https://kaop.komodor.com/incidents/workflows) ### New: Edit a recorded case's expected outcome, with full history A recorded case's expected outcome or recorded tool responses can now be replaced without losing the original. Editing creates a new version that starts **awaiting approval** — the original recording and every replacement stay visible in the case's decision history, with who made each change, when, and why. A run that already used the old version keeps using it; only future runs see the edit. [Try it in KAOP](https://kaop.komodor.com/quality-lab/datasets?datasets-tab=cases) ### New: Delete a dataset Quality Lab → Datasets can now delete a dataset, from the trash icon on its row or **Delete dataset** on the dataset page. Its records are kept — they stay under Recorded cases and in any other dataset — and every finished replay keeps its results. Delete is unavailable while a replay of the dataset is queued or running, or when the dataset is declared by a cluster. [Try it in KAOP](https://kaop.komodor.com/quality-lab/datasets) ### Fixed: Dataset reruns no longer skew an agent's run history and metrics A dataset replay run used to show up in its agent's own run history and count toward its success rate, run counts, activity, and cost panels — making a batch of regression reruns look like a spike of real traffic. Replays are now excluded from an agent's history and metrics by default, while staying fully reachable from the dataset's own rerun history and by direct link. [Try it in KAOP](https://kaop.komodor.com/fleet) ### Improved: Dataset replays work for kubectl agents, with a minimum coverage per dataset Replaying a dataset of a Bash/kubectl agent now answers a kubectl command from the recording even when the agent spells it differently, and the agent's skills load as in a normal run. Each dataset has a **Minimum coverage** (default 80%): a replay with a few unanswered calls is judged by the evaluator as long as enough of its calls came from the recording. The replay page shows the coverage and, on each call, how the recording answered it. [Try it in KAOP](https://kaop.komodor.com/quality-lab/datasets) ### Improved: See a dataset's health at a glance — including which agents fed it A dataset's counts (total, approved, awaiting approval), when a case was last added, and when it last ran are now computed by the server and available on every surface — UI, API and MCP. The dataset list and detail pages also show **which agents** supplied its recordings. [Try it in KAOP](https://kaop.komodor.com/quality-lab/datasets) ### Fixed: An agent's Documentation link now opens its documentation The **Documentation** link on a catalog agent's card, and on a deployed agent's detail panel, used to point at a private Komodor repository — so it opened a GitHub 404 for everyone outside Komodor. Each of the fourteen catalog agents that has a published page now links straight to its own section of the Komodor docs, and the few with no page yet simply show no link rather than a broken one. [Try it in KAOP](https://kaop.komodor.com/marketplace) ### New: Capture a filtered list of runs to a dataset in one action The run History page can now capture many completed runs into recorded cases at once, from whatever filter you already have applied (agent, status, text, time range). Already-captured runs are skipped, only succeeded runs are eligible, and you can tag every captured case and drop them straight into a dataset in the same action. [Try it in KAOP](https://kaop.komodor.com/runs) ### Improved: The alerts funnel prices each stage On a workflow's Overview, the Investigation and Remediation headers now read as two lines: the count, then `Total $12.40 · P95 latency 9m`. The number is what the agent runs that stage's steps dispatched spent, over every run behind the window's incidents — investigating is priced under Investigation and remediating under Remediation, so the two add up to the runs' whole spend. A stage nothing priced shows no dollar figure rather than \$0. The header is now the one place a P95 is drawn; the line under a bar keeps its share and hint, and the node's own P95 moves to its tooltip. Ingest and Correlation keep the count alone: nothing before an incident opens runs an agent, and both happen inside the webhook request, so neither has a spend or a latency worth a line. [Try it in KAOP](https://kaop.komodor.com/incidents/workflows) **September 25, 2026** ### New: Use your own agent as an evaluator Turn on **Evaluator** when you add an agent and it appears in Quality Lab as an evaluator you can put on an eval rule. A rule either grades against its evaluator's default criteria, shown on the criteria step, or against criteria you define; criterion weights are now sliders that always add up to 100. The platform checks every evaluator's answer against the expected shape and asks it to fix a malformed one before the score is recorded. [Try it in KAOP](https://kaop.komodor.com/quality-lab/evaluators) ### Fixed: Replays are judged on the agent's answer Your Base Evaluator now reads the rerun's own answer when it judges a dataset replay. Before, it was shown the replay's bookkeeping instead, so a replay that reached the approved conclusion could still be marked different and flagged as a regression. [Try it in KAOP](https://kaop.komodor.com/quality-lab/datasets) **September 24, 2026** ### Improved: Tag dataset records and take them out of a dataset Each record in a dataset now shows its tags, which you can edit in place, and has a **Remove from dataset** action that takes it out of that dataset without deleting it. On a run that has not succeeded, **Add to dataset** now stays visible but disabled, and says why the run cannot be captured. [Try it in KAOP](https://kaop.komodor.com/quality-lab/datasets) ### New: See every recorded case in one place, and clear the ones awaiting approval Quality Lab → Datasets now has a **Recorded cases** tab: every case captured from a run, with its approval status, tags and the datasets it sits in. Filter to **Awaiting approval** to see how many are waiting and approve them right from the list. A replay now shows Expected, Before (the original run's output) and After side by side, and recordings no longer clutter the Golden Scenarios library. [Try it in KAOP](https://kaop.komodor.com/quality-lab/datasets?datasets-tab=cases) ### New: Define weighted criteria on an eval rule An eval rule can now carry **custom criteria**, each with a name, description, and weight, plus pass and warn thresholds. Every criterion becomes its own score you can trend; beside them you get a weighted overall and a Pass / Warn / Fail badge. Need to steer how a criterion is judged? Add it to the rule's own instructions. Only evaluators that accept custom criteria appear in the picker when criteria are on. [Try it in KAOP](https://kaop.komodor.com/quality-lab/evaluators) ### Improved: Datasets hold any agent's runs, and you pick the agent to replay against A dataset is no longer tied to one agent. **Add to dataset** on a finished run now offers every dataset. A run already in one dataset can be added to another, and it stays the same record, so approving or re-tagging it shows in both. On a dataset page, **Add case** brings in a record captured from any agent. **Remove from dataset** takes a record out of that dataset only: it is not deleted, and it stays in every other dataset that holds it. **Replay now** asks which agent to replay against, with the dataset's default agent picked for you, and each replay shows the agent it ran against. A dataset no longer needs an agent when you create it. [Try it in KAOP](https://kaop.komodor.com/quality-lab/datasets) ### Improved: Only people approve dataset records, and capturing and approving are their own permissions Approving a dataset record, or removing its approval, is now a person's decision only: the assistant and other agents can capture runs and organize datasets, but they can no longer approve an outcome. Capturing a run, approving an outcome and managing datasets are separate permissions, so a role can be given one without the others. Replaying a dataset needs permission to invoke its agent, not a dataset permission. [Try it in KAOP](https://kaop.komodor.com/quality-lab/datasets) ### New: key=value tags on cases, and a faster way to build a dataset A case's tags are now `key=value` pairs (`env=prod`, not a bare word) — edit them from any dataset that holds the case, via the new **Edit tags** action on each record row. A dataset's header gains **Add by tag**, which pulls in every live case carrying a pair you give it, and a new **Tag coverage** section shows how many cases and *approved* cases each tag has, with an **Untagged** row alongside — click a tag to filter the records below to it. **Replay by tags** narrows a whole-dataset replay to cases that carry (or don't carry) given pairs, with a live preview of how many approved cases will run before you start it. [Try it in KAOP](https://kaop.komodor.com/quality-lab/datasets) **September 23, 2026** ### New: Ask Slack what it can do, and who answers you here Type the AgentOps slash command on its own, or with `help`, and it lists what it can do rather than guessing what you meant. Type it with `here` and it tells you, privately, which agent answers you in that conversation and what is stopping you if nothing does: an allowlist you are not on, an agent that is switched off, a Slack account you have not connected yet, or no routing rule for that conversation at all. Until now each of those looked identical from Slack, which was silence. Both work before you have connected your Slack account, which is when people are most lost, and both answer only you. [Try it in KAOP](https://kaop.komodor.com/channels) ### New: See whether a replay reached the approved outcome Every completed dataset replay is now judged same or different against the outcome you approved by your Base Evaluator, which checks the conclusion rather than the wording, and a different outcome is flagged as a regression. The replay page shows the evaluator's reasoning beside the verdict, and the approved expected outcome, the original recorded output and the rerun's output side by side, next to both tool transcripts. The Datasets page shows whether your evaluator is live. [Try it in KAOP](https://kaop.komodor.com/quality-lab/datasets) ### Improved: See whether a replay got more expensive or slower Every dataset replay now shows its cost and duration next to the original recorded run, with the change in dollars, time and percent. Any increase is flagged, so a version that answers the same way but costs more or takes longer stands out, without changing the replay's outcome. [Try it in KAOP](https://kaop.komodor.com/quality-lab/datasets) ### Improved: Skills can carry many files, and every agent can use them A skill is no longer just its `SKILL.md`. Add references, scripts and assets to it, and your agents get all of them, in the layout their framework expects. Claude Code and ADK agents load the skill with their own built-in skill support and open the file they need when a task calls for it, including files the skill's instructions never link to. A Claude Code agent can also run the skill's scripts. [Try it in KAOP](https://kaop.komodor.com/skills) ### Improved: Edit one file of a skill from the co-pilot or the API The chat co-pilot can now read a skill's files and add, replace or remove one at a time, and the API has the same with `PUT` and `DELETE /api/v1/skills/{skill_id}/files/{path}`. Each edit publishes a new version that keeps the body and every other file, so nothing is resent. The agent now sees every file too: the `SKILL.md` it reads ends with a list of the skill's files by path, so a file the body never links is still found. [Try it in KAOP](https://kaop.komodor.com/skills) ### Improved: Delete a dataset record without losing its replays Each record in a dataset now has a **Delete** action. The record leaves every dataset and can no longer be replayed, but its source run and every finished replay stay, marked **Deleted**, and the dataset's replay totals do not change. A record, or a dataset, with a replay still queued or running cannot be deleted until that replay finishes. [Try it in KAOP](https://kaop.komodor.com/quality-lab/datasets) ### Improved: Dataset records show who captured them, and what the run cost Each dataset record now shows who captured it, plus the original run's cost and duration. Only a run that succeeded can be added to a dataset, so failed and cancelled runs no longer offer **Add to dataset**. [Try it in KAOP](https://kaop.komodor.com/quality-lab/datasets) ### Improved: Approving a dataset record keeps its recorded output, and approval can be removed Approving a dataset record now always makes the run's original output the expected outcome, and records who approved it, when, and why. An approved record has a **Remove approval** action that sends it back to awaiting approval: future replays skip it, and replays that already ran keep their results. A record can no longer be edited, archived or published without review. [Try it in KAOP](https://kaop.komodor.com/quality-lab/datasets) **September 22, 2026** ### Improved: Pick the tools before a server connects, and find them in a long list Adding an MCP server used to publish every tool it advertised, and the Configure step let you save without looking. A server now starts publishing nothing and will not let you continue until you say what an agent may call — the disabled button tells you why on hover. Long tool lists are navigable now. MCP tools fold into Read-only and Writes groups, OpenAPI operations into GET, POST, PATCH and DELETE, each collapsed with a running `12/40 selected` count and its own select-all. An upstream advertising hundreds of tools fits on one screen instead of scrolling past the destructive ones. A server that answers but advertises no tools at all is now refused with that reason, rather than saving as something an agent can reach and never use. [Try it in KAOP](https://kaop.komodor.com/integrations) ### Fixed: Autoscaling a self-hosted agent now adds a replica per queued run Turning on KEDA autoscaling for a self-hosted agent used to start you at a queue depth of five per replica, so six runs had to pile up before a second replica appeared. An agent takes one run at a time, so those five were waiting on capacity that could never exist. The field now starts at one, matching the Helm chart, and one queued run brings up one replica. Agents already deployed keep whatever value you set; change it on the agent's scaling settings if you want the new default. [Try it in KAOP](https://kaop.komodor.com/fleet) ### Fixed: Insights now counts the same 30 days everywhere on the page The **Run status** card and the **Run trend** chart on Insights were counting a different set of runs from the 30-day tiles above them — the newest 100 runs of all time, rather than the last 30 days. On a workspace with more than 100 runs the card could not show the window at all, and on an older workspace it counted runs from outside it, so the card's own numbers did not add up to the total beside them. Both now read the same 30-day window as the rest of the page, and the card's outcomes add up to the run total it reports. Cancelled and queued runs, which were being dropped entirely, are included. [Try it in KAOP](https://kaop.komodor.com/insights) ### Fixed: Correcting your LLM gateway's URL now reaches the agents already using it Editing a "Managed by me" gateway's base URL changed the provider but not the agents bound to it. Each agent holds its own copy of the address, taken when it was bound, so workers kept dialling the old endpoint while the Providers page, the agent's own page and the generated values.yaml all showed the new one. The only way out was to move the agent to another provider and back. The edit is now carried to every agent already on that gateway, matched on the address each one actually holds — so an account running several gateways only moves the agents on the gateway that changed. The next run picks it up; no redeploy needed. [Try it in KAOP](https://kaop.komodor.com/providers) ### New: See exactly what a run and a deployed agent executed with A run's **Configuration** tab now shows the resolved config version, worker build, and every credential bound to it by reference — kind, source, version, and which keys were delivered or denied. A deployed agent's **Used secrets** card carries the same detail, and its **Versions** tab adds a **Deploy history** card showing who created, edited, redeployed, or archived it, and when, straight from the audit log. [Try it in KAOP](https://kaop.komodor.com/fleet) ### Fixed: Editing a catalog agent now produces a values.yaml helm accepts Editing an agent deployed from the catalog produced a values.yaml with an empty `agent.instructions`, and the chart requires that field, so the generated `helm upgrade` failed with `agent.instructions is required` before it applied anything. The wizard had stopped short of the file for these agents; once it started producing one, the file itself was unusable. A catalog worker's prompt ships inside its image and is never written into your values, so the file now carries the same standing-in line every catalog release already has, with a comment saying so. Nothing about the agent changes and no prompt is exposed. [Try it in KAOP](https://kaop.komodor.com/agents) ### Improved: Author a whole skill folder in the editor, not just SKILL.md A skill is a folder — `SKILL.md` plus the `references/`, `assets/` and `scripts/` files its body links to — and the editor now writes all of it. Each file is a tab beside the body, a script can be marked executable, and the agent gets the folder back on disk exactly as you laid it out. The SKILL.md tab also has a Preview toggle, so a body reads as rendered markdown instead of raw source. Editing a body no longer touches the folder: publishing content on its own keeps every file the skill already had, whether the publish came from this editor, the chat co-pilot, the API or a GitOps `Skill` resource, which can now declare the whole folder too. [Try it in KAOP](https://kaop.komodor.com/skills) ### Improved: The alerts funnel reads as one picture On a workflow's Overview, the Ingest bar now spans the whole chart. Everything that arrives is ingested — folding a duplicate or dropping an alert on a filter is a decision taken afterwards — so that bar is the 100% the rest of the funnel is read against, however much of the volume the noise exits take. A provider's "resolved" notice is no longer counted as an alert or drawn as a Recovery signal stage. It closes an incident whose firing alert was already counted at Ingest, so counting it again made the ingested total and every share of it too high. The incident it closed is charted under Auto remediation instead: it resolved without anyone acting on it, whatever the run had applied. Each stage's number sits centred over its own stage and stays inside it, instead of running into the column beside it, and Remediation now carries how long a plan waited on a person alongside the count. The Show values table is gone: its numbers were already on the graph. The two readings the chart cannot draw — a stage nothing reached this window, and a caveat on a number that is known to be biased — are printed under the legend instead. [Try it in KAOP](https://kaop.komodor.com/incidents/workflows) ### Fixed: A workflow that finished is no longer charted as Failed Failed on the alerts funnel now means one thing: a run that broke. It is read from the workflow's own step records — the run failed outright, or a step in it failed or timed out — instead of from the incident's status, which carries the same word for a run that broke and for one that did not. Three populations move out of Failed as a result. A run that walked every step of its graph and stopped because nothing was left to do is charted where its steps got to, not as a breakage. So is a run an operator cancelled. And an incident whose workflow never ran at all — every incident on an account with the engine switched off — now lands under Triaged directly, which says why the engine never ran, rather than under a failure that never happened. A run still in flight stays under Still running even if one of its steps threw and is being retried. [Try it in KAOP](https://kaop.komodor.com/incidents/workflows) **September 20, 2026** ### New: Approve a paused action, and stop it asking next time When an agent pauses on a governed action and you approve it, you can now approve **and stop asking** — one click that releases this call and edits the agent's action policy so this tool stops pausing in future, the way Claude Code's "Yes, and don't ask again" works. It allows the tool for any arguments — the agent stops asking about that tool from then on. It is honest about what it can do. Because a stricter rule always wins, simply allowing a call does nothing while a broader rule still asks about it — so the control edits the rule that asks, and when it can't (a pattern it can't cleanly narrow, or a rule that refuses the call outright) it tells you so rather than writing a rule that changes nothing. Editing the standing policy needs the `action_permission.manage` permission, so an approver who may answer but not change the rules still gets the plain Approve, and every rule change is recorded in the audit trail. The affordance shows up wherever you answer — the run, the Actions inbox, and in chat — behind the `actions_approvals` flag. [Try it in KAOP](https://kaop.komodor.com/actions) **September 18, 2026** ### New: Watch an agent work in Slack, and find one without leaving Ask an agent something in Slack and the thread shows what it is doing while you wait, step by step, with a link to the full run. A run parked on an approval says it is waiting on a person, with the deadline in your own timezone, so a working agent no longer looks the same as a stuck one. If you do not know which agent to ask, `/agentops agents` lists the ones you are allowed to run, and `/agentops agents kubernetes` searches them by name. Each result carries the handle to type, and asking costs you nothing: the list is private to you and starts no run. Agents also read less of your channel than before. Each channel carries a history window, a week by default, and an agent cannot read further back than that however it asks. [Try it in KAOP](https://kaop.komodor.com/channels) ### New: Turn any finished run into a replayable dataset record Open a finished run and choose **Add to dataset**: the input, every tool call and the output are recorded. Approve the record under Quality Lab → Datasets, then **Replay now** to run the agent against the recording. Each replay says whether it was clean, inconclusive (a call had no recorded data) or contaminated (a live call slipped through), and shows Expected, Before and After side by side. [Try it in KAOP](https://kaop.komodor.com/quality-lab/datasets) **September 17, 2026** ### Fixed: An agent's Versions tab lists real changes, not every image rebuild An agent got a new version every time its worker image was rebuilt, because the build commit was stamped onto each of the agent's skills and counted as part of the agent's identity. The version history filled up with entries whose diff was empty, and the genuine changes were buried among them — one code reviewer showed 60 versions covering 7 actual edits, the most recent of which was five weeks older than the newest entry claimed. Rebuilding an unchanged agent no longer creates a version, and a version that really does carry no change to the definition now says so instead of opening an empty diff. [Try it in KAOP](https://kaop.komodor.com/agents) ### Fixed: The right-sizing policy picker explains an account with no policies yet Picking a right-sizing policy for a workload used to open onto a blank list on an account that hadn't created one yet, with no explanation and no way out. The picker now disables itself and shows "No policies yet" with a "Create your first policy" link straight to the policy drawer, so you're never staring at an empty dropdown wondering what's missing. [Try it in KAOP](https://kaop.komodor.com/k8s-cost/right-sizing) ### Improved: One webhook endpoint can feed several agents, and every one of them fires An endpoint used to serve exactly one destination, and dispatch used to stop at the first one it found — so an endpoint pointed at an agent silently starved the incident workflow listening on the same URL: the request returned 200, the agent ran, and no incident was ever opened. Endpoints now attach to as many agents and workflows as you need, and a single inbound request reaches all of them, incident workflows included. You pick the endpoint from the agent's own trigger step rather than routing it when you create it, so the endpoint wizard ends at Test & mapping; the Endpoints table's new "Used by" column names everything listening on each one. Unhooking an agent detaches it and leaves the endpoint's URL and token working for everything else. [Try it in KAOP](https://kaop.komodor.com/integrations?section=webhooks) ### Improved: The co-pilot now lives in the top-right header The **Co-pilot** button has moved from the floating bottom-right corner into the global header, as the first control in the top-right cluster — before Search, Demo and Notifications. It no longer sits on top of a wizard's own buttons, so nothing behind it is lost. Opening it still slides the co-pilot in beside the page and resizes the content to make room, and it stays aware of the page and step you're on. On a narrow window the button collapses to an icon with a tooltip. [Try it in KAOP](https://kaop.komodor.com/agents/new) ### Improved: Awaiting approval now follows the run, not just the agent A run paused on a governed action used to read "Awaiting approval" only in the Actions inbox or on the run's own page — everywhere else it looked like it was quietly working. The badge now follows the run wherever it's listed: Overview's recent runs, the History ledger, an agent's own run history, and an incident's orchestrator and specialist runs. When a specialist or orchestrator is paused on a governed tool call, the incident's Workflow runs row now shows "Needs your answer" and the incident increments both the "Needs your answer" and "Needs your attention" facet counts. The incident header badge and the Incident feed tab count update the same way, and the specialist row in the agent team panel marks which agent is waiting. The "Needs your answer" help text now names this as a third kind of pause alongside a remediation choice and a guardrail hold. [Try it in KAOP](https://kaop.komodor.com/runs) **September 16, 2026** ### Fixed: The workflows list counts every incident, not just the open ones The Incidents module's workflows table showed each workflow's **open** incident count under a column headed plainly "Incidents", so a workflow whose incidents had all been resolved reported zero — on the same row as the resolution time and the spend those incidents produced, and next to an Overview that reported them. The column now counts every incident triggered in the selected window, matching the Overview's Volume, and names the open subset beneath it when there is one. [Try it in KAOP](https://kaop.komodor.com/modules/incidents/workflows) ### Fixed: A workflow's field-mapping override is saved, and the Review step says so The Field Mapping section of the workflow wizard's Trigger step confirmed an override in three places — the "1 field overridden" badge, the line naming what decides the field, and the filter's own field list — while the value reached no request and no column. Finishing the wizard produced a workflow that still read every field the way its endpoint does: wrong titles, and a dedup key that silently splits or merges incidents. Re-opening it showed the section back at "Inherits the endpoint's mapping" with the box empty. An override now rides the same save every other field on the step does, on create and on edit alike, and a field nobody touched still follows the endpoint rather than being restated. The Review step carries its own Field mapping card listing each overridden path, and the Dedupe card names a dedup key this workflow overrode instead of reporting one it inherited. [Try it in KAOP](https://kaop.komodor.com/modules/incidents/workflows) ### Improved: Tooltips now stand out from the page they float over A tooltip's surface used to be the same colour as the card underneath it — literally the same design token — so it was told apart only by a thin border and a shadow. In dark mode that meant dark text panels on dark cards; in light mode, white on white. Tooltips now render on their own surface, dark in light mode and light in dark mode, so the content reads as an overlay at a glance. The help-icon tooltips on the cost cards already looked this way; every other tooltip in the product now matches them. [Try it in KAOP](https://kaop.komodor.com/agents) ### Fixed: The orchestrator is not its own specialist An incident's **Agent team** card named the same agent twice — once as ORCHESTRATOR and once as SPECIALIST — so a workflow with no specialist bound at all read as a two-agent team. The cause was one level down: every step of a workflow dispatches to the same step-orchestrator agent, and only the first of those runs is recorded as the orchestrator, so each later step was filed under a specialist role while being the orchestrator itself. The card now lists only agents other than the orchestrator, and the **Investigation summary** counts the remainder in runs rather than in agents — *3 (2 orchestrator runs + 1 specialist run)* instead of *3 (1 orchestrator + 2 specialists)*, with a side that has no runs left out entirely. Every step run is still reachable from the step breakdown, and still counted. [Try it in KAOP](https://kaop.komodor.com/modules/incidents) ### Fixed: Steps say when a specialist never ran A workflow step's status reported on its orchestrator run alone. So a step whose bound specialist could not be dispatched — the agent offline, its run failing with *no worker to serve run* — completed **green**, with no failure reason, while the investigation behind it had never reached the cluster. The failure existed only on a sub-run one level down, and the incident's own specialist list named the dead run without a status, so it read as a specialist that took part. The step row now carries a **1 specialist failed** marker without being expanded, and expanding it names the agent and the error its run recorded. The Agent team card annotates a specialist that failed or was cancelled with its outcome instead of listing it like any other. The step itself is still reported honestly as completed — its own run did succeed — but an operator being asked to authorize a change built on that investigation is now told what it could not see. [Try it in KAOP](https://kaop.komodor.com/modules/incidents) ### Fixed: The investigation time limit now says when your account cannot change it Changing the orchestrated step time budget away from **900** in the incident workflow wizard failed the whole activation, with **"This workflow has no workflow definition yet … so there is nothing to set orchestrated\_step\_max\_duration\_seconds on."** The limit is stored on the workflow definition, and an account whose workflow engine is disabled — or that has no step-orchestrator agent deployed — never gets one stamped, so the value had nowhere to land. The field was offered anyway, and every save in between was rejected without saying so. The field is now read-only on those accounts and states which of the two reasons applies, the same way the workflow template picker already did. This covers both the current wizard's **Investigation team** step and the new wizard's **Investigation time limit**. Leaving it at the default always worked and still does. A rejected auto-save also no longer passes in silence: the wizard keeps a banner up saying your last change was not saved and what the API said, at the step where it happened rather than three steps later at **Activate**. Navigation is still never blocked. [Try it in KAOP](https://kaop.komodor.com/incidents/workflows/new) ### Fixed: Set up notifications without leaving the wizard Picking where a run or an incident workflow reports back used to be a dead end on an account with no notification sink yet: the step said "Nothing here yet" and offered nothing to click. It now explains what a sink is for and opens the Add-notification-sink wizard right there — Slack or a signed webhook — so you can finish the setup in one pass. ### Fixed: The incident timeline reads in your own timezone An incident's Timeline card printed each entry's timestamp exactly as the control plane stores it — `2026-09-16T10:38:29.777697+00:00`. Every reading of "when did this alert land, and how long did triage take" started with converting UTC in your head. Those timestamps now render on your own clock, as `Sep 16, 1:38:29 PM`. Seconds are kept, because a live incident routinely logs two entries a second apart, and hovering one still reveals how long ago it was. [Try it in KAOP](https://kaop.komodor.com/modules/incidents) ### Fixed: The alerts funnel stops calling a completed workflow "Failed A workflow that runs to the end and applies nothing marks its incident **Needs attention**, so a person reads the findings and decides what to do. The funnel treated that status as a failure and charted the incident under **Failed** — telling an operator their workflow had broken on a run that had done exactly what its template asked of it, investigated the alert and reported that the step meant to apply a change could not act. The chart now reads the run's own status alongside the incident's. A run that **completed** is placed at the stage its step runs earned, which for this population is **No remediation**, reached through its RCA verdict like any other finished run; the terminal's hint counts them, so "1 closed by hand · 2 completed without an applied change" says which incidents are behind the bar. Failed keeps the incidents it was written for: one that errored, and one that needs attention because its run stopped with nothing left to retry. Those incidents are still waiting on a person — the Needs human tile on the overview counts them, and the bar drills through to every one. [Try it in KAOP](https://kaop.komodor.com/modules/incidents) ### Fixed: The alerts funnel charts incidents whose workflow graph never ran The funnel's stages are read from a workflow run's step runs, and an incident triaged by the orchestrator directly has none. Those incidents were counted at Investigated and placed nowhere to the right of it — so an account whose workflow graph does not run saw an empty right half and a headline of "0 of N remediated", however much work its agents had actually done. That is not only old data: an account still on the legacy dispatch produces these today. Every stage those incidents can answer for themselves is now drawn. Status places a failed or in-flight one, the incident's own root cause places the RCA verdict, and a close recorded outside any run lands under No remediation with the hint saying which. What none of them settles ends at **Triaged directly**, reached straight from Investigated rather than through an RCA verdict, with a caveat saying no remediation stage can be proven either way and a hint naming why the graph never ran. Every bar is clickable through to the incidents behind it. [Try it in KAOP](https://kaop.komodor.com/modules/incidents) ### Fixed: The endpoint tester says what actually happened to a request Testing an endpoint told you a request had arrived, then ended with "Captured — no target is wired yet, so nothing ran." On an endpoint feeding an incident workflow that was never true: the workflow attaches to the endpoint and opens an incident rather than starting a run, so the panel's one question — is there a run? — always answered no, even for an alert that had just created an incident. Each captured request now reports what every incident workflow on the endpoint did with it: routed, filtered out, parse failed, throttled, ignored or errored, with the workflow's name, the reason, and a link to the incident it opened. **Filtered out** is the one worth having — it names the condition that rejected the alert, so a filter that is quietly dropping everything no longer looks exactly like a quiet week. The same detail now appears in the endpoint's History drawer. "Nothing ran" is still there for the endpoint it was written for: one with no target and nothing consuming it. [Try it in KAOP](https://kaop.komodor.com/integrations?section=webhooks) ### Fixed: Moving a self-hosted agent onto your own LLM gateway now takes effect Editing a self-hosted agent onto a "Managed by me" provider recorded the choice but could leave Komodor's gateway credentials still bound to it. The agent kept inferring through Komodor — billing to Komodor's budget and not to your gateway — while the wizard, the agent's detail page and the generated values.yaml all reported the new provider. Saving again did not help: the selection already matched, so nothing was re-applied. The save now reconciles against what the agent actually holds rather than against what was last recorded, so an agent left in that state is repaired by the next edit. An agent deployed from the catalog can be moved between providers too, which it previously could not be at all — its Edit wizard refused to produce a values.yaml and its provider was fixed at whatever it was deployed with. Deploying a catalog worker onto your own gateway works too. It previously accepted the choice and bound Komodor's credential anyway, with no field to enter your gateway key in. The Permissions step no longer lists the gateway credentials AgentOps manages for you (`LLM_PROXY_TOKEN` and friends). They are decided by the Model step, and a checkbox that disagreed with it could only break the agent. [Try it in KAOP](https://kaop.komodor.com/agents) ### Fixed: Edit keeps the in-cluster MCP server you picked at create A self-hosted catalog agent deployed with **"Use your own in-cluster MCP server"** lost that choice the moment you re-opened **Edit**: the toggle came back off, the URL and Kubernetes Secret fields were empty, and the regenerated `values.yaml` quietly dropped the `agent.mcpServers` override — so applying it sent the worker back to the MCP server baked into its image. The create wizard now records the URL and the Secret name/key with the agent, Edit re-opens on them, and the Review step lists **In-cluster MCP server** among the changes waiting for your `helm upgrade` instead of claiming there are none. A catalog agent that *was* configured here also no longer opens under the "recovered from the running agent" warning. [Try it in KAOP](https://kaop.komodor.com/fleet) ### Fixed: Deploy from catalog is a grid you can search On the incident workflow wizard's Investigate & remediate step, "Deploy from catalog" listed the catalog one full-height card at a time in a narrow modal — fourteen specialists, about two of them on screen, and no way to type a name. The dialog is wider now, lays the workers out in two columns, and clamps each card's description so a screenful is six workers rather than two. A search box filters by name, category or what the worker does, within the role scope rather than out of it, and the footer keeps naming the worker you picked even once a search has scrolled it out of view. [Try it in KAOP](https://kaop.komodor.com/incidents/workflows/new) ### Fixed: Cancel no longer leaves a draft workflow behind The incident workflow wizard saves a draft the moment you continue past the first step, so the rest of the wizard has something to write to. Cancel then navigated away without mentioning it, and the draft stayed — a catch-all workflow nobody chose to create, sitting in the same list you read to judge what the account is running. Cancel now says what it is holding and asks. **Discard draft** deletes it; **Keep draft** leaves it in the workflows list to finish later. Cancelling before the first Next still creates nothing, and cancelling out of an *edit* never offers to delete the workflow you opened. [Try it in KAOP](https://kaop.komodor.com/modules/incidents) ### Fixed: Budgets now stop runs mid-flight, and blocked runs explain themselves A `block new runs` budget is meant to halt an agent the moment its window spend crosses the cap — even for a run already in progress. That mid-run enforcement had stopped firing, so a long run could sail past the cap; it now terminates in-flight runs again. When a run is blocked or terminated by a budget, the run page now says so plainly — "Stopped by budget, not a run failure" — with the cap, the spend so far, and an **Increase budget** button that opens the edit dialog for that exact window, with a note on why you landed there. [Try it in KAOP](https://kaop.komodor.com/cost/agent-budgets) ### Improved: An agent's tool permissions are one screen, starting with what it does by default Writing an action permission meant filling in a dialog on the Actions page — which knows about no particular agent — and the one question that decides the most had no field at all: what does this agent do about a tool call nothing covers. Now it is the first step. Every agent gets a **default** — let everything run, ask a person first, or refuse everything — and rules become overrides on top of it, grouped by what happens rather than listed one by one. A third step says where an `ask` is delivered: a run started from Slack is still answered in its own thread, and everything else can now land in a channel you choose instead of waiting unannounced in the Actions inbox. You can try a named call against the whole thing before saving it, arguments included. **How long a question waits is now set in one place** — on the agent, in that third step, anywhere from 5 minutes to 12 hours, defaulting to an hour. The per-rule wait and the account-wide setting are both gone: three places could answer "how long does this wait" and no single screen could, so a number you set in one could be quietly overruled by another. Choosing **ask** now says what it costs, wherever you choose it: the agent stops on the call and holds its only slot until somebody answers, and runs queued behind it start failing after about an hour. The Actions page no longer offers **New permission** — a rule is one line of an agent's posture, so it is written on the agent, under Fleet → the agent → Permissions. Rules already there stay editable. [Try it in KAOP](https://kaop.komodor.com/actions) ### Improved: AgentOps is now KAOP The product you use to build, run, and operate your agents is now called **KAOP**. You'll see the new name across the app, in emails, and throughout the docs. Nothing about how it works changes — your agents, runs, integrations, and API keys are exactly as you left them, and existing links keep working. **September 15, 2026** ### Fixed: Workflows say when no step graph will run The incident workflow engine is not enabled on every account. Where it is off, incidents are triaged by the orchestrator directly — which is intended — but the product said nothing about it. The wizard drew the full step graph, including a `Select remediation` card badged *Awaits operator*; the Remediation step said only that a different template could not be saved; the workflow's Metadata panel described a remediation mode where "an operator selects before anything is applied"; and an incident's Steps card showed an empty list that read as a graph yet to start. All four now say the same true thing: no workflow graph runs for this account, why, and the consequence — no step runs, and no operator approves anything before it is applied. The step cards are marked **Will not run** rather than drawn as a graph that executes, and the template you pick is described as recorded for when the engine is turned on. [Try it in KAOP](https://kaop.komodor.com/modules/incidents) ### Fixed: The workflow wizard's Source step opens on the essentials Step 2 of the incident workflow wizard no longer greets you with two expanded advanced sections. **Field Mapping** and **Trigger Filtering** now start collapsed, so the step opens on the endpoint picker. Each one still unfolds on its own when the workflow already carries something to show — a stored filter or a mapping override — and stays where you put it once you touch it. [Try it in KAOP](https://kaop.komodor.com/modules/incidents/workflows/new) ### Improved: The workflow wizard no longer assigns specialists that are not running The specialist grid let you assign an agent that is not running, marked only by a line saying it would not run until it reconnected — so a workflow could be created with a specialist that silently never did anything. Those tiles now sort below the agents that are up and refuse the click, naming what has to happen first: reconnect, finish deploying, or become healthy. A workflow you are editing keeps the specialist it is already bound to, so you can still clear or change it. The review step's last two bands — the one-line summary of what the workflow runs, and the flow of what happens to an alert — now carry titles like the cards above them. [Try it in KAOP](https://kaop.komodor.com/incidents/workflows/new) ### Fixed: Role holder counts only count active holders The **Holders** column on Settings → Roles counted every grant ever issued for a role, including ones held by disabled service accounts or members. On accounts with many disabled principals this could overstate a role's real holders several times over — most visibly for `admin`. The count now reflects who can actually use the role today: it excludes disabled principals and any grant that has expired. [Try it in KAOP](https://kaop.komodor.com/settings/roles) ### Fixed: Two right-sizing policies that would not save now save Editing a right-sizing policy could fail in two ways that had nothing to do with what you were trying to change. Picking "Only when pod is created/restarted" and then switching back to "Apply Immediately" left a hidden restart setting behind, and saving the policy was rejected with `ALLOW_RESTART_NOT_APPLICABLE` — an option that only applies to the on-creation protocol being sent alongside the immediate one. The restart setting is now sent only when it applies. Separately, a policy with priority `0` — which the default policy has — reported "Priority is required" and blocked the step, even though `0` is a real priority and the API accepts it. Priority now accepts `0`, and an empty field says it is required while an out-of-range one says so instead. [Try it in KAOP](https://kaop.komodor.com/k8s-cost/settings) ### New: Create and find right-sizing policies from the Right-Sizing page The Right-Sizing by Workload header now carries the two policy actions it was missing. "Add Policy" opens the full policy wizard in place — the same create flow Settings → Right-Sizing Policies uses, without leaving the table you were reading. "View Policies" takes you to that settings tab when you want the whole list. Add Policy requires the `k8s_cost.policy.manage` capability and is hidden without it. View Policies is always available, since the policy list itself is readable by anyone who can open the page. [Try it in KAOP](https://kaop.komodor.com/k8s-cost/right-sizing) ### Fixed: A resolved incident's banner says what actually happened The resolved status banner on an incident's run drilldown always read "Closed — either manually or automatically on Datadog recovery." — even for a person's manual remediation, or a run that stopped at Investigate and applied nothing. The banner now reads the incident's own resolution record: who or what closed it, and what was (or wasn't) applied. [Try it in KAOP](https://kaop.komodor.com/modules/incidents) ### Fixed: The Remediation agent is no longer offered as an investigation specialist The investigation specialists picker offered the catalog's Remediation agent and counted it among the agents that can investigate — but the draft then refused to save, with a validation error naming an internal value you had never typed. The only way forward was to work out which of your picks was at fault and untick it. That agent is not an investigator: it reads a finished investigation and proposes a fix, and it is chosen on the remediation step instead. The picker now says so where it is listed, rather than offering a selection the save would reject. An agent of your own that happens to be named "Remediation" is still yours to assign as an investigator, and now saves. [Try it in KAOP](https://kaop.komodor.com/incidents/workflows/new) ### Fixed: Pausing a workflow says so, and the status updates straight away Pausing or activating a workflow from the Incidents overview table left the row showing the status it had before the click — the badge still read **Active** and the button still offered **Pause** — until the page was reloaded. The action had already succeeded; only the table was out of date, which made a successful click look like a no-op and invited a second one. The overview now refreshes with the rest of the surface when a workflow's status changes, and every Activate/Pause control across the Incidents screens confirms the result with a short message. [Try it in KAOP](https://kaop.komodor.com/modules/incidents) ### Fixed: A newly connected cluster says it is collecting data, not that its agent is out of date A cluster connected through **Add cluster** appeared under **Unavailable clusters** in the K8s Cost cluster picker with the tooltip "Agent update required" — on an agent installed minutes earlier, often the newest one in the account. Nothing was wrong with it, and there was no update to install. Cost data is written hourly, so a cluster in its first hours has no rows yet. That was being read as the same condition as an agent too old to report any, and the two shared one message. A live cluster still waiting for its first rows now says "Collecting data - Cluster will be available in less than 24 hours", which is what is actually happening; it becomes selectable once it has a day of history. A cluster that has been connected for longer and still has no data, or whose agent has stopped reporting, still says the agent needs attention — there, the message is true. [Try it in KAOP](https://kaop.komodor.com/k8s-cost/settings?tab=clusters) ### New: Ask a specific agent by name in Slack Slack messages used to reach whichever agent a routing rule pointed at, so sending one question to the right specialist meant editing standing configuration that changes who answers for everyone in the channel. You can now name the agent you want by typing its handle to the bot, and that agent answers instead. It applies to that one message only and changes no configuration, so the next message is routed the way it always was. Get the name wrong and the bot says so and stops, rather than quietly passing your question to a different agent. Naming an agent you are not allowed to run reads the same way, so a reply never reveals that an agent you cannot see exists. Answers can now also carry the answering agent's own name and icon rather than the app's, which makes a thread with several agents in it readable. A specialist the assistant hands work to posts its own message in your thread instead of arriving folded into the assistant's summary. Workspaces connected before this shipped keep the single app identity until an admin reinstalls the Slack app and grants the new permission. [Try it in KAOP](https://kaop.komodor.com/integrations) ### Fixed: Memory spaces say when nothing is reviewing them A memory a run saves is a draft until the space's memory worker reviews it, and a draft is invisible to search. A space with no memory worker therefore keeps collecting drafts that no agent can ever retrieve — memory looks switched on and returns nothing. Manage spaces said so only in a line of grey helper text, styled exactly like the two neutral hints beside it, and only while the picker sat on *Find by role*. That state now draws a warning that names the consequence: every memory written here stays an unreviewed draft, so agents keep saving and never retrieve anything. It appears whenever the space has no effective worker — including when someone picked an agent that no longer resolves — and it says what to do: pick an agent, or give one the `role=memory-worker` label. Nothing is lost in the meantime; drafts already waiting are reviewed as soon as a worker exists. [Try it in KAOP](https://kaop.komodor.com/memory) ### Fixed: Added by" now shows a name, not an internal id An integration created with a service account's API key showed its raw internal id (`prin_...`) in the Integrations table's "Added by" column instead of a readable name — while one created by a person correctly showed their email. That column now resolves a service account to its display name, and falls back to a generic "Automated" label rather than an opaque id when the account can't be resolved. [Try it in KAOP](https://kaop.komodor.com/integrations) ### Fixed: The Insights Failures breakdown now adds up to its total The Failures card on the Insights page shows a headline total with a per-category breakdown underneath — but the breakdown only ever showed the top 3 categories, so whenever a 4th category had failures, the numbers shown no longer summed to the headline. The breakdown still leads with the top categories by count, but now folds anything beyond that into a `+N more` remainder, so what you see always reconciles with the total. If the line is too long to fit the card, hover it to see the full breakdown. [Try it in KAOP](https://kaop.komodor.com/insights) ### Fixed: An incident workflow can no longer go live with nobody to investigate Creating an incident workflow with no investigation specialist used to be allowed with only a warning. Every incident it opened was a no-op — the orchestrator had nobody to delegate to, so no evidence was gathered and no root cause was established, yet the incident still spent tokens and closed as Resolved. Assigning at least one specialist is now required: the Investigate & remediate step holds until you pick one, and activation is refused over the API as well. [Try it in KAOP](https://kaop.komodor.com/incidents/workflows) ### Fixed: New incident workflow stops toasting "Connect Slack first. Clicking **Next** on the Name step of a new incident workflow no longer raises a red "Request failed / Connect Slack first." toast when your account has no Slack integration. Slack not being connected is an expected state here — the Endpoint step already says so inline and lets you paste a channel ID — so it is no longer reported as a failed request. The same applies to the Slack channel pickers in notification sinks. A Slack integration that genuinely fails still surfaces an error as before. [Try it in KAOP](https://kaop.komodor.com/incidents/workflows/new) ### Improved: Know when a policy's clusters cannot right-size immediately "Apply Immediately" resizes running pods in place, which Kubernetes only supports from 1.33 with the in-place pod vertical scaling feature gate enabled and a recent enough Komodor agent. A policy scoped to clusters that do not meet that bar still saves, but its changes quietly wait for the next pod restart instead. The When to Apply step now says so. Pick "Apply Immediately" while any cluster in the policy's scope cannot honour it, and the step flags that those workloads will wait for a pod restart instead — so what you are reading matches what the policy will actually do. The warning is advisory — it never blocks saving the policy. [Try it in KAOP](https://kaop.komodor.com/k8s-cost/settings) ### Fixed: Deploy from catalog offers only what the picker will accept On the incident workflow wizard's Investigate & remediate step, "Deploy from catalog" opened the same unfiltered worker catalog whichever picker you clicked it from. From the orchestrator picker you could deploy a plain specialist — which that picker then refuses to list — so the dead end only showed up after the deploy had finished. The dialog now scopes itself to the picker that opened it: orchestrators from the orchestrator picker, everything an orchestrator can delegate to from the specialist pool. A banner names the scope and counts what it hid, and "Show all agents" drops it if you want the full catalog back. [Try it in KAOP](https://kaop.komodor.com/incidents/workflows/new) ### Fixed: The config operator's install commands work on a clean cluster The Install dialog for a configuration cluster showed `kubectl create secret` before `helm install --create-namespace`, so copying the commands in order failed on a cluster that had never seen the `agentops` namespace: the Secret step ran first and had nowhere to land. The dialog now leads with `kubectl create namespace agentops`, and its title names what it installs — "Install the config operator into …" — rather than just the cluster. [Try it in KAOP](https://kaop.komodor.com/settings/configuration-clusters) ### Improved: You can un-pick an integration or group in the agent wizard The Integrations step lets you scope an agent to a single integration or to a group of them. Once you had picked one, there was no way to un-pick it — every option in the list selects something, and none of them means "none". Both pickers now carry a clear button. It empties the field back to its placeholder so you can choose again, and it takes the tool selection with it so no tick from the previous server is carried over. It does not unscope the agent: the kind of scope you chose stays chosen, waiting for a new pick. [Try it in KAOP](https://kaop.komodor.com/agents/new) ### Fixed: Back returns you to the workflow feed you came from Opening an incident from one workflow's feed and clicking **Back** could drop you on the general incident feed instead of the workflow you were working in — losing the scope, and the filters and search you had set on it. Back now returns to the exact feed the incident was opened from, and names it, so a shared link, a reload, or an incident opened in a new tab all land back where you started. [Try it in KAOP](https://kaop.komodor.com/modules/incidents) ### Fixed: Connecting AWS from the agent wizard offers both ways in AWS can be connected two ways: a static key pair, or a cross-account IAM role you create by launching a stack in your own console, which stores no long-lived keys anywhere. The Integrations page has always asked which you want. The agent wizard never did — it rendered the key form and nothing else, so the keyless route was unreachable without leaving the wizard. "Connect new" on a provider that offers a choice now asks the same question the Integrations page asks, on that provider, and brings you back to the step you left with the new connection selected. Providers with a single way in are unchanged. [Try it in KAOP](https://kaop.komodor.com/agents/new) ### Fixed: Activate is disabled on a workflow that is not ready yet An incident workflow draft used to offer an enabled **Activate** button even when it could not possibly activate — the wizard saves the draft as soon as you name it, so abandoning it before picking an investigation team left a workflow whose only feedback was an "Activate failed" toast after the click. **Activate** is now disabled on a workflow the control plane would refuse, and hovering it says which step is missing — an orchestrator agent, or an endpoint / Slack trigger to route alerts in. This applies everywhere the action appears: the workflows list, the workflow page, and the "Workflow not activated" empty state on its feed. [Try it in KAOP](https://kaop.komodor.com/modules/incidents/workflows) ### Fixed: Connecting an integration from the agent wizard asks for the right fields The wizard's inline "Connect new" form now renders each provider's fields the way the provider describes them, matching the Integrations page. A field with a fixed set of values — Datadog's site, an AWS region — is a picker rather than a box you had to spell the value into, defaults are filled in, and the guidance the catalog writes for a field appears beside it. Fields that only apply to the authentication method you picked are hidden, and no longer submitted. Saving is no longer blocked on one of those hidden fields, which could leave the button inert with nothing on screen left to fill in. [Try it in KAOP](https://kaop.komodor.com/agents/new) **September 14, 2026** ### Improved: A webhook sink URL says why it was refused A notification sink's destination URL has always been checked for private, loopback and link-local addresses before it is saved — but the form only asked that it start with `https://`, so anything else came back as a save error after the fact. The field now applies the same rules the server does, as you type: a non-HTTPS scheme, an internal hostname such as `api.default.svc.cluster.local`, and a private or loopback address each say what is wrong in the field itself. The server still has the last word, because it also checks the address the hostname resolves to at delivery time. [Try it in KAOP](https://kaop.komodor.com/integrations?section=notification-sinks) ### Fixed: Switch between knowledge bases The Knowledge page always showed the most recently created knowledge base and had no way to reach any other. An account that had more than one — for example a base declared through cluster IaC alongside the account's own base — could permanently lose access to every older base and its documents the moment a newer one appeared. The page now defaults to the account's original base rather than whichever is newest, and a selector appears next to **Add page** whenever there is more than one to choose from. An account with a single knowledge base sees no change. [Try it in KAOP](https://kaop.komodor.com/knowledge) ### Fixed: Skills count badges match your search The "N skills · N agents · N tags" badges on the Skills page always showed the full catalog, even while a search or kind filter had narrowed the list beside them — so searching for something that matched nothing still showed the total count next to "No skills match this search." The badges now count the filtered results, so they always agree with the list. [Try it in KAOP](https://kaop.komodor.com/skills) ### Improved: Search for an agent in the incident workflow wizard Choosing the orchestrator and the investigation specialists for an incident workflow meant reading the whole fleet — a specialist grid with every eligible agent in it, and an orchestrator dropdown with no filter. With no way to narrow either one, finding an agent by name came down to the browser's own Ctrl+F. Both now filter as you type, and match on the agent id as well as the display name, so an agent you know only as `datadog-alert-triage` is one query away. A query that matches nothing says so rather than leaving you with an empty panel, and the "N assigned" count and the fleet-coverage line keep counting the fleet, not whatever the search left on screen. [Try it in KAOP](https://kaop.komodor.com/incidents/workflows/new) ### Improved: The workflow named on a run page now links to that workflow The incident run page names the workflow that produced the run, but the name was plain text — to read the workflow's overview you had to open **Workflows** and find it by name yourself. The name is now a link straight to that workflow's overview, so a run drills up to the workflow that ran it in one click. [Try it in KAOP](https://kaop.komodor.com/modules/incidents) ### New: Register a private Azure AI Foundry as your own gateway The **"Managed by me"** path — your own LLM gateway that your self-hosted agents dial directly, with no inference traffic through Komodor — now offers a gateway kind. Alongside the OpenAI-shaped kinds (LiteLLM, Hosted vLLM, Ollama), you can register an **Azure AI Foundry** resource: point it at `https://.services.ai.azure.com` and Komodor lists the deployments over Foundry's own route. This is the **private** Foundry path, and it is deliberately distinct from "Bring your own Azure AI Foundry" under **Add a new key**. The Managed-by-me one stays inside your network: your workers reach the resource directly and Komodor is never in the inference path — only discovery, verify and spend travel the control-plane channel (through an Outpost if the resource is private). The Add-a-new-key one is public and routes inference through Komodor's shared gateway. The kind picker names the difference so you register the right one. [Try it in KAOP](https://kaop.komodor.com/providers) ### Fixed: K8s Cost Overview's Pod Placement cards no longer flash 0% while loading The "Pod placement savings" card and the "Savings from Pod Placement" figure on the "Potential Savings" card briefly showed 0% while their data was still loading, before jumping to the real value. They now show a loading state until the real figure is ready. [Try it in KAOP](https://kaop.komodor.com/k8s-cost/overview) ### Improved: Connections and tools are one step, and picking the tools picks the connection Setting up a catalog agent asked for its integration twice — once as a connection, once as an MCP server — and the two could disagree with nothing on screen saying which one the agent would use. They are now a single **Integrations** step. For an agent built around one provider, picking its server or its integration group is the whole answer: the connection comes from the pick and is shown back to you, so there is no second field to keep in sync. The pickers list only what that agent can actually use, and anything else stays visible with the reason it cannot be chosen. Where an integration is optional, every scope stays available and the agent runs without it. Activate no longer lets a half-finished scope through. A scope whose kind you chose but whose server you did not used to deploy as no scope at all; it now says so and takes you back to the step. [Try it in KAOP](https://kaop.komodor.com/agents/new) ### Improved: Registering your own LLM gateway is easier to get right Adding a "Managed by me" gateway now guides you past the two places it used to trip. When the gateway is reached through an Outpost, the wizard checks the host against that Outpost's allowlist before you submit — so a host the relay would refuse is caught up front, with a note to add it and re-install the Outpost, instead of failing only once it tries to list models. Picking which models to expose is no longer a wall of pre-selected chips: the discovered models come as a searchable, checkable list with Select all and Deselect all, so paring hundreds down to the few your agents use is a couple of clicks. And once a gateway is created you can step Back to review it, or Edit to change the endpoint or key. [Try it in KAOP](https://kaop.komodor.com/providers) ### Fixed: GitHub can be connected from the agent wizard again Creating a GitHub agent walks you through an **Integrations** step that asks you to connect GitHub. That step only ever offered one way to do it — type a name, press **Save connection** — and GitHub does not work that way: it is connected by installing its app, so every attempt came back with *"GitHub uses an app installation; start the app installation flow instead"* and no way to start that flow without abandoning the wizard. The step now offers the flow the provider actually uses. GitHub opens its app installation in a new tab; a provider that signs in with OAuth opens its authorization instead. Finish in that tab and the step picks the new connection up on its own and selects it, so the wizard carries on where you left it. Where a provider has not been configured on your deployment, the step says so up front rather than failing at the last press. [Try it in KAOP](https://kaop.komodor.com/agents/new) ### New: Bring your own Azure AI Foundry Azure AI Foundry now sits alongside Azure OpenAI in the provider catalogue, so an account can register its own Foundry endpoint and point agents at the models deployed there. It is a separate provider from Azure OpenAI because Foundry is a different API, and the difference matters for Claude: a Claude deployment on Foundry speaks Anthropic's Messages API but authenticates the Azure way. Picking **Azure AI Foundry** routes it correctly, while a non-Claude model registered under the same key takes Foundry's OpenAI-compatible path — so one key covers a tenant's whole Foundry catalogue. Add the key and the endpoint (`https://.services.ai.azure.com`) and the wizard lists the deployments that key can reach. An endpoint behind your own API gateway may publish no catalogue at all; the model id can always be typed by hand. The API version field is optional and only widens that listing. [Try it in KAOP](https://kaop.komodor.com/providers) **September 13, 2026** ### Improved: Every Slack message the bot accepts now gets an answer A `/agentops` question is answered privately to whoever asked it, rather than posted where they cannot see it, and one typed in a direct message now reaches your DM rules. An incident raised by mentioning the bot reports back into the thread it came from, both when it concludes and while it is waiting on someone to pick a remediation. Handing work to an incident pipeline or a workflow leaves a reply and a ✅ or ❌ instead of a message stuck wearing 👀. A DM conversation keeps its context across follow-ups, and a job stranded with no worker is reported in minutes rather than after an hour of silence. ### Fixed: Renaming a key you added a while ago works too Renaming a key without re-entering its secret only worked for keys added recently. An older one asked for the key again and refused to save without it. Any key can now be renamed on its own. [Try it in KAOP](https://kaop.komodor.com/providers) ### Improved: Every key on Providers shows what it is costing Each key row on Providers now carries the spend, calls and tokens of all the models underneath it, so you can read what a key is costing without opening it. A key whose models the gateway has not metered in the current window stays blank rather than claiming zero. [Try it in KAOP](https://kaop.komodor.com/providers) ### Improved: The alerts funnel now follows your workflow's steps "Where the alerts went" on a workflow's Overview tab now reads how far each incident actually got: Investigated, RCA found or not found, then Remediation plan, Auto remediation or No remediation, with Still running and Failed as their own ends. Every stage shows its share and a P95 duration, and the Remediation plan stage shows how long it waited on a person. The left half is unchanged in substance: alerts ingested, folded as duplicates, filtered out, recovery signals, grouped into incidents. The chart changes unit once, at "Grouped into incidents", and says so on the node — 7,280 alerts became 286 incidents, not 7,280 of anything else. Placement comes from the workflow's own record — a step run for a human decision means a person was asked; an applied action in the run's ledger means the agents changed something — never from how the incident was eventually closed. A manual close or a provider recovery is now a line of detail under the stage the run reached, not a stage of its own. The five tiles above the chart are all read from the funnel itself: **Incidents created** is now **Incidents investigated** and counts the distinct incidents behind the window's grouped alerts, **RCA success** is **Root cause found**, and **Remediation coverage** is **Remediated**, a count split by plan and automatic. Clicking an incident-unit stage lists the incidents behind it. [Try it in KAOP](https://kaop.komodor.com/modules/incidents) **September 11, 2026** ### Improved: Slack pings you when the answer lands, not before Asking an agent in Slack no longer gets you an instant "..." placeholder that is quietly edited minutes later — Slack never notifies on an edit, so the answer used to arrive in silence. The bot now posts a single message, the moment it has something to say, and reports progress on your own message instead: 👀 received, ⏳ running, ✅ answered, ❌ failed. ### Improved: Your keys are the rows on Providers, and they can be renamed Providers now groups the table by the key that serves each model, so one key backing four models is one row you can open rather than four rows repeating its name. Each key row carries the name you gave it, its identifier underneath, and how many models it serves; a search narrows the table to the keys that match and opens them. The actions that act on a key (add models, rotate, delete) now live on the key's own row, and the ones that act on a model stay on the model. You can also rename a key without re-pasting it. Adding a provider no longer asks you to invent an identifier either: give the key a name if you want one, and we generate the identifier. Across the screen the same thing is now called a key everywhere, rather than a credential in one place and a key in the next. [Try it in KAOP](https://kaop.komodor.com/providers) ### Fixed: A model you just added becomes selectable in the wizard on its own Registering a model leaves it unverified for a moment, and the agent wizard used to need a browser reload before it would offer it. It now keeps checking while something is waiting on verification, and refreshes the list when you come back to the tab, so a model that becomes ready while the wizard is open can be picked there. A model that is not selectable yet also says why in the dropdown, with the same explanation the Providers page gives, rather than appearing as a greyed-out name with no reason attached. [Try it in KAOP](https://kaop.komodor.com/agents/new) ### New: Connect a Kubernetes cluster to K8s Cost without leaving the page **Add cluster** connects a new cluster in three steps: name it, run the Helm command it gives you, and watch the agent report in. The command carries the installation key already filled in, with a copy button, so nothing has to be pasted together by hand. It sits beside the cluster selector on Settings → Configured Clusters, and on every K8s Cost tab that has nothing to show yet. Asking the co-pilot to connect a cluster opens the same three steps. The name is checked as you type, against the same rules Komodor's own installer enforces, so a name it would reject is caught before anything is created. A name that is already connected is flagged too — as a warning rather than a block, since reconnecting after a key rotation or a reinstall is a real thing to want. The last step is the useful one. Creating a cluster mints a key but tells you nothing about whether anyone actually ran the command, so the window keeps checking and confirms the moment your cluster first reports in — including while you are away in a terminal. If it is taking longer than usual it says so rather than spinning silently. The agent installs in cost mode: metrics and live cost data only. It does not read pod logs or exec into containers. [Try it in KAOP](https://kaop.komodor.com/k8s-cost/settings?tab=clusters) **September 10, 2026** ### Fixed: Incidents created now agrees with the tiles beside it A workflow's **Overview** tab could state **0** under **Incidents created** while **RCA success** right beside it read "221 of 269 incidents". The tile was counting **alerts** — the alerts funnel's `opened` node, which counts an alert row and so counts a re-fire twice — while its two neighbours divided by **incidents**. It now reads the same incident count they do, so the three tiles share a denominator and the row can no longer contradict itself. Expect this number to change on workflows where alerts re-fire: it was previously inflated by every re-fire, and it is now the count of incidents actually created in the window. **Run statistics** in the same tab is now five tiles rather than five bare rows — Duration, Runs, Tokens, Cost and Errors — each with a line saying what its number means (for example, Cost is "Run cost in window", not all-time spend). A non-zero **Errors** count stays red. The section also carries **one** name: it previously sat under a "Run statistics" heading with a card inside titled "Investigation summary". [Try it in KAOP](https://kaop.komodor.com/modules/incidents/workflows) ### Fixed: Uploading a page again replaces it Re-uploading a corrected file used to add a second page at the same path, leaving the old text in the knowledge base and retrievable — so an agent could cite the version you had just replaced. Now the page is replaced in place: it keeps its identity and history, and only the new text is retrievable. Before the upload runs, the dialog names any pages that are about to be replaced, so a batch of files can't quietly overwrite something you couldn't see. [Try it in KAOP](https://kaop.komodor.com/knowledge) ### Improved: Supply your "Managed by me" gateway key in the helm command instead When you pick a **"Managed by me"** model in the add-agent wizard, the Model step now offers a checkbox — **"I'll provide the API key myself in the helm command"**. Tick it and the **Gateway API key** field is disabled and no longer required, and the generated `helm install` command omits the key entirely, leaving you to add your own `--set-string llm.anthropicApiKey.value=…`. It's for teams who would rather not paste an inference key into the browser at all: the wizard never sees it, and it reaches your cluster only through the command you run yourself. [Try it in KAOP](https://kaop.komodor.com/fleet) ### New: Scope which knowledge pages each agent can read Any agent that could search the knowledge base could read all of it. Now a page carries tags, an agent carries a `kb-tags` label, and the two are matched at search time: an agent labelled `kb-tags: payments` retrieves the payments pages and not the security ones. Tag a page from its Details panel in Knowledge; set the label from the agent's Metadata tab, where a picker offers the tags already in use and tells you how many pages the agent would reach. Two rules are worth knowing before you rely on it. **An agent with no `kb-tags` label still reads everything**, so nothing changes for the agents you have until you scope one deliberately. And **a page with no tags stays readable by every agent** — a page nobody has classified is not hidden, so the Details panel says so on the page itself rather than leaving an empty row. Tags are read on each search, so retagging a page takes effect on the agent's next question with no re-indexing. [Try it in KAOP](https://kaop.komodor.com/knowledge) ### Improved: Scope an agent to an integration in one step, not two The agent wizard's **MCP tools** step used to ask twice. First a Connection question — *no MCP servers*, or *connected integrations* — and only then, nested underneath it, the scope that actually mattered. The first question added a click without adding a decision. It now opens straight on the scope: **Single integration** or **Integration group**. Choosing neither is how you say an agent needs no tools, and the step says so plainly rather than leaving you to infer it. Two things the dropdown now tells you before you commit. Each option is labelled with what it is — an **Integration** authenticated by a connection you set up, or a **Custom integration**, a raw MCP server registered by hand. And an integration whose tools the gateway has not discovered yet is listed but cannot be selected, because binding it would grant the agent nothing. **Outpost routing** moved into a collapsed **Advanced** section at the bottom. Its current value stays on the header, so a routed agent still says where its traffic goes without being opened — and an agent already bound to an Outpost opens with the section expanded. On a Komodor-cloud agent the control is now visible and disabled with the reason, instead of missing. [Try it in KAOP](https://kaop.komodor.com/agents/new) ### Improved: Schedules now read in plain English Fleet → Schedules leads each row with what the schedule actually does — "Every 15 minutes", "Mon–Fri at 07:00", "Day 1 of every month at 09:00" — with the cron expression and its timezone underneath for anyone who reads cron directly. The schedule drawer says the same thing in its header. An expression that cannot be stated exactly still shows itself rather than a rough paraphrase: a day-of-month and a weekday together, for instance, are OR'd by cron, and a description that is only nearly right is one you have no way to check. [Try it in KAOP](https://kaop.komodor.com/agents?tab=schedules) ### Fixed: A looped run's steps now read in the order they ran On a run whose workflow looped back — **Verify** did not pass, so the graph returned to **Investigate** — the **Workflow steps** card numbered its rows `1 2 3 4 5 1 2 3 6` and showed **Investigate** twice with nothing but a small badge to tell the two apart. The list was in execution order, but the number was the step's position in the *workflow*, so a repeat read as a sorting glitch rather than as what it was: the run genuinely doing that step a second time. Rows are now numbered by the order they actually ran, `1..N`, and each row states its place in the workflow beside its name — `step 2/6 · attempt 2`. Where the run went back, the card says so, on its own line: **Looped back to Investigate**. No run data changed, and nothing was collapsed — every step execution is still its own row. Only what the card claims about them is different. [Try it in KAOP](https://kaop.komodor.com/modules/incidents/feed) ### New: Route a single-server agent through an Outpost, not just a group A self-hosted agent scoped to one MCP server can now be routed through an Outpost in its own cluster, the same way a group-scoped agent already could. Pick one on the agent wizard's MCP step and its tool calls go straight to that Outpost's local listener instead of AgentOps. As with group scope, that trade is real: tool calls made this way never reach AgentOps, so no guardrail, policy, or audit applies to them, and the listener authenticates no caller — a NetworkPolicy in your own cluster is what decides who may use it. [Try it in KAOP](https://kaop.komodor.com/agents) ### New: Reach a private "Managed by me" gateway through an Outpost A **"Managed by me"** gateway that lives inside your own network used to have to be publicly reachable for Komodor to list its models, verify it or read its spend. Now the Add-provider form and a gateway's Edit dialog carry an **Outpost** picker: bind the gateway to one of your Outposts and Komodor reaches it for that metadata over the Outpost's tunnel instead of a direct dial. Leave it on **None** for a gateway that is reachable directly. This changes only Komodor's **metadata** path — your self-hosted workers still dial the gateway directly on their own network with their own key, exactly as before, so nothing about inference or where your credential lives changes. The picker is available anywhere the customer-gateway tier is, and the chat co-pilot can prep the field too. [Try it in KAOP](https://kaop.komodor.com/providers) ### Fixed: An incident no longer reports Agent offline for an agent that was never going to run it An incident workflow names an orchestrator agent, and on workflows the platform runs step by step that agent is not what executes them — each step runs on the platform's own step orchestrator. The incident was still checking the named agent's heartbeat, so a workflow that was running perfectly well showed an **Agent offline** badge, and its timeline an entry saying the investigation was queued until that agent reconnected. Neither was true: nothing was waiting on it. Those workflows no longer read the named agent's presence, and an incident that picked up the badge before its workflow moved onto the platform engine drops it on its next alert. Workflows that do dispatch to the agent you named — accounts not yet on the platform engine — are unchanged and still tell you when it is offline. [Try it in KAOP](https://kaop.komodor.com/incidents) ### Improved: "New workflow" now sits in the same place on every Incidents screen The Incidents Overview offered **New workflow** as a small action tucked into the right-hand end of the tab row, while the Incident feed offered it as the filled primary button beside the page title. Same action, two different buttons in two different places — so the one you had just used was never where you next looked for it. Every Incidents screen now puts it in one place: the primary button at the top right, beside the page title. The tab row carries tabs only. **The Workflows list stopped offering it twice.** That page rendered both copies at once — the title-row primary and the tab-row action — which read as two different buttons rather than one action. It now shows the single primary. Creating a workflow is still one click from ⌘K, and the workflow and run drilldowns keep their own actions instead of a create button they never needed. [Try it in KAOP](https://kaop.komodor.com/modules/incidents) ### Fixed: Runs keep the model they actually ran on, not the "Synthetic" placeholder When an agent's last turn failed, the CLI reported that turn under a placeholder name rather than a real model, and the run's whole token bill was reattributed to it. `` then appeared in the Fleet Analytics model mix as if it were a model your fleet had chosen to run, taking one of the eight slots. Runs recorded from now on keep the model they actually spent their tokens on, and the failed turn's tokens are still counted against it; a run whose only reported model was the placeholder now lands in `(unattributed)` instead, so growth there is the fix working. Runs recorded before this fix may still show the placeholder until they age out of the window. [Try it in KAOP](https://kaop.komodor.com/agents?tab=analytics) ### Improved: Your knowledge base is already there **Knowledge** used to open on a setup gate — "Knowledge base not set up", and a **Set up knowledge base** button you had to press once before the page did anything. Every account now has its knowledge base from the start, named after the account, so the page opens straight into the wiki and there is nothing to set up. Nothing changes for an account that already had one: it keeps the knowledge base it has, under the name it was created with. The **Set up knowledge base** button is gone either way. [Try it in KAOP](https://kaop.komodor.com/knowledge) ### New: See every cluster reporting into K8s Cost, and how recently each one checked in K8s Cost → Settings has a new **Configured Clusters** tab listing the Komodor agents reporting into your account — the agent version each one runs, when it was connected, and how long ago it last sent a heartbeat. Sort on any column, hide the ones you don't need, narrow to specific clusters, and page through at 10, 25 or 50 rows. Agents that have stopped reporting are hidden by default, since a dormant cluster is rarely what you came to look at. Turn on **Show inactive agents** to bring them in; every row carries a status dot, so a cluster that has gone quiet is visible at a glance rather than something you infer from its heartbeat. A cluster that has never reported its version shows a dash, not a blank — it means the agent has not said, not that it is running nothing. Restarting an agent from this table is not available yet. [Try it in KAOP](https://kaop.komodor.com/k8s-cost/settings?tab=clusters) ### Improved: Click a configured cluster to see everything its agent reports Rows in **K8s Cost → Settings → Configured Clusters** now open a details panel. Alongside what the table already showed, it carries the agent's id, the Helm chart version it was installed from, and the Kubernetes version it reports running on — each with a copy button, so an agent id goes straight into a ticket or a `kubectl` command without retyping a uuid. The timestamps copy the full instant rather than the shortened form on screen, since "Aug 12, 2026" pasted into a query loses the time. Anything the agent has never reported shows a dash and offers no copy button, rather than looking like an empty value. Below the details, a **JSON** section shows the agent's full reported configuration — the Helm values it is running with, useful when a cluster is behaving differently from its neighbours and you want to see what actually differs. Credential-shaped values are replaced with a marker before they leave the control plane. [Try it in KAOP](https://kaop.komodor.com/k8s-cost/settings?tab=clusters) ### Fixed: An incident's counters now see a failed specialist An incident whose specialist agent failed reported a clean run: **Errors 0**, **Errors %** at 0, a **Runs** count that left the specialist out, and a **Specialist runs** list naming only the step orchestrator. The runs an orchestrator delegates to were never linked to the incident at all, so nothing that counts them could see them — including any alerting built on those numbers. They are linked now, through the delegation lineage the platform records rather than anything the agent reports about itself. On an affected incident expect **Runs** to rise, **Errors** to show the failure in red, and the failed specialist to appear in the run list by name. **Errors %** on the incident feed and the incident page now reads 100% when any run of the investigation errored, not only when the incident's own status is *Errored* — a hard specialist failure on an otherwise-progressing incident is no longer invisible there. One line was removed: the **Runs** tile on a workflow's **Overview** tab no longer shows an "N orchestrators + M specialists" breakdown. That split subtracted the workflow's *configured* specialist count from a run total covering **every incident in the selected window**, so the number it printed was not a count of anything. The Runs total itself, and the other four tiles, are unchanged. [Try it in KAOP](https://kaop.komodor.com/modules/incidents) ### New: Connect Grafana through an MCP server you already run in-cluster The Grafana integration now asks which shape it is. **Cloud**, the default, is what it always was: a service-account token and your Grafana URL, for Grafana Cloud or a self-hosted Grafana. **In-cluster MCP server** records an MCP server you already run inside your cluster — its cluster-internal URL and, optionally, the name and key of a Kubernetes Secret that holds its bearer token. The token itself never leaves your cluster; only the reference is stored, and the control plane is never in the MCP data path. Because a cluster-internal address cannot be reached from the control plane, an in-cluster connection reports as *unverified* rather than healthy — the worker proves it at runtime. Existing Grafana connections are unchanged and stay in cloud mode. [Try it in KAOP](https://kaop.komodor.com/integrations) ### Improved: Fleet's Triggers tab is now Schedules, and you can filter and edit from the list The tab is named for what it actually holds — recurring **schedules**. Inbound webhook endpoints never lived here; they are under Integrations → Endpoints. Alongside the rename it picks up the facet rail the Agents tab already had: filter by **Enabled** / **Disabled** and by any label on the agent a schedule fires, plus a search box that matches the agent, the cron expression and the schedule ID. The old `?tab=triggers` links still work. Each row now shows the **timezone** under its cron expression, so a 9am schedule can no longer be misread as 9am UTC, and carries **edit** and **delete** buttons — with a confirmation step before anything is removed. A schedule declared in an agent's `agent-spec.yaml` says so on the row and explains, on hover, why its controls are refusing instead of leaving them dead. The detail drawer drops the diagram that restated the cron expression and shows the **input payload** each run is invoked with in its place, keeps its edit and delete together, and marks run rows as the links they always were. [Try it in KAOP](https://kaop.komodor.com/agents?tab=schedules) ### Fixed: Dropdowns show the detail beside each option again Several dropdowns carry a second piece of information beside the option's name — the servers an integration group contains, what a policy targets, what a saved selection resolves to. On any dropdown attached to a full-width field, that detail was being cut off: the list rendered narrower than the field above it and everything past its right edge disappeared. The most visible casualty was the agent wizard's **Integration group** picker, which is where you decide what an agent may call. It listed group names alone, so answering "which MCP servers does this group actually grant?" meant selecting a group to find out, one at a time. It now reads `ops-tools — kubernetes, datadog` in the list itself, and a group with no servers says so before you pick it. A dropdown is now never narrower than the field it belongs to, so the same detail is back in the integration, policy, history and incident pickers that were losing it too. ### Fixed: Disconnecting an integration no longer reports a failure that did not happen Disconnecting an integration — and deleting a credential, an MCP server, a memory space, a notification sink or a golden suite — showed a success message and an error message at the same time. The page was re-reading the row it had just deleted, getting the expected "not found" back, and reporting it as a failure. The delete itself always worked; only the alarming second message was wrong, and it is gone. [Try it in KAOP](https://kaop.komodor.com/integrations) ### Improved: Every cron field now reads back what it will actually do The plain-English reading that Fleet → Schedules got has spread to every place a cron is written or shown. As you type into a schedule field — in the agent wizard, the Schedules dialog, the workflow builder or a maintenance window — the hint under it stops describing the *format* and starts describing your expression: "Every 15 minutes", "Mon–Fri at 07:00", "Day 1 of every month at 09:00". So a mistake is visible before you save it, not after it fires. The confirmation you get after deploying an agent says the same thing, as do the workflow canvas's trigger node and the maintenance wizard's review step. An expression that cannot be stated exactly still shows itself instead of a rough paraphrase. ### New: Agents deployed from the catalog can now be edited An agent you deployed from the catalog used to be frozen at creation. The two things most likely to be wrong on the first attempt — which integration connection it uses, and which model it runs on — could only be changed by deleting the agent and starting again, which lost its history and its triggers. Those agents now open in the same edit wizard a self-built agent uses, for exactly the fields the deploy asked you for: the integration connection per provider, the model, the MCP group or server and its tools, the display name, labels, and how many copies run. A connection change takes effect on the agent's next run; the rest arrives with its redeploy. Everything the catalog image defines — its instructions, skills and tool surface — stays fixed, and now says so where you would look for it, with the reason, instead of the field quietly disappearing. Komodor-cloud and self-hosted catalog agents are both editable, for the same set of fields. [Try it in KAOP](https://kaop.komodor.com/agents) ### Improved: An agent's metadata now shows which LLM provider it uses The **Metadata** tab on an agent's page now has a **Provider** field beside **Model**, so the model name no longer hides which gateway actually serves it. It always shows one of **Komodor managed gateway**, **Bring your own**, or **Managed by me** — for every agent, including hosted and catalog ones, falling back to the managed gateway when an agent declared no provider of its own. [Try it in KAOP](https://kaop.komodor.com/fleet) **September 9, 2026** ### Improved: Workflow wizard's "+ Create new endpoint" now opens the real endpoint popover The New incident workflow wizard's Endpoint step used to create an endpoint through its own stripped-down inline form. "+ Create new endpoint" now opens the same endpoint popover the Endpoints page and the chat co-pilot use — every source, auth, and mapping option is available, not just a name and a sender. On success the new endpoint is selected in the wizard automatically, and nothing else you had already entered is touched. [Try it in KAOP](https://kaop.komodor.com/modules/incidents/workflows/new) ### Improved: A workflow row opens from anywhere on the row, not just its name On the Incidents Overview and Workflows tables, only the workflow's name was clickable — the volume, resolution, error-rate and spend cells you were actually reading were dead space, so drilling into a workflow meant travelling back to the first column. The whole row is now the click target, and it shows a pointer on hover so you can see that before you try it. The row's own controls still belong to themselves: the Activate/Pause button, the Actions kebab and the copy-the-agent-id affordance do their own job and nothing else — clicking one does not also open the workflow. The name is still a real link, so it keeps its keyboard focus, Enter to open, and open-in-new-tab. [Try it in KAOP](https://kaop.komodor.com/modules/incidents) ### New: A workflow now tells you when it was last updated The **Record** group on a workflow's **Steps & config** tab now shows **Last updated** beside Created and Last incident — so "did this workflow change before or after the alert I'm looking at?" is answerable without guessing. It reports the last write to the workflow for **any** reason, and says so: activating or pausing a workflow moves it too, not only a config edit. A workflow nothing has touched since it was created reads **Never updated**, which is deliberately different from a workflow whose timestamp we could not read. [Try it in KAOP](https://kaop.komodor.com/modules/incidents/workflows) ### Improved: Sort the incident workflow tables by any column The Incidents Overview and Workflows tables arrived in one fixed order, so finding the workflow that spent the most, errored the most, or has not fired in weeks meant reading every row. Each column header is now a sort control: click it to sort, click again to reverse. Metrics, money and dates open on the end you are usually looking for — the biggest number, the most recent date — and text columns open A-Z. The active column and its direction are marked in the header. Sorting is a view, not a setting: the tables still open in the order the platform returns them, and a workflow with no data yet stays at the bottom in both directions rather than pretending to be the cheapest or the fastest. [Try it in KAOP](https://kaop.komodor.com/modules/incidents) ### Improved: Shadow A/B results now sit on the version they judged A shadow experiment compares two versions of one agent, and each side is its own generation. But production keeps shipping while a candidate stays pinned, so the same candidate ends up judged against several production versions — and until now every one of those verdicts was folded into a single row. On one staging agent that blended three separate experiments into one, and the two baselines it was actually measured against disagreed on whether the candidate used *more* tokens or fewer. The **Versions** tab on an agent now marks a generation that ran as a **shadow candidate**, carries its verdict on the row, and expands into one result per production baseline — each with its own win distribution, score spread, and cost/token/latency deltas. Baselines are summary lines you open one at a time, so a candidate measured against ten production versions stays readable. A generation that served only as the baseline links across to the candidate instead. Quality Lab's **A/B Test Results** table splits the same way. Verdict counts still fold across baselines — "Production wins 6/7" — but scores and costs never do, because averaging verdicts from different baselines states something no evaluator said. **One number changes.** Per-generation invocation counts were attributed by timestamp, which broke whenever two generations were live at once — exactly what a shadow is. They now follow the worker that actually served each run, so a candidate that served no production traffic correctly reads zero instead of borrowing its baseline's runs. Expect existing counts on agents with shadow history to shift. [Try it in KAOP](https://kaop.komodor.com/agents) ### New: Shadow comparisons now show why an arm won, not just that it did A shadow experiment could tell you Production or Shadow came out ahead and by how much, and nothing else — the evaluator's reasoning was recorded on every verdict and never rendered. Clicking any comparison in an experiment's sample list now opens its full report: both arms' weighted totals with the winner, a one-line summary of what separated them, each rubric's contribution to the total, and the evaluator's point-by-point comparison behind every score. Below that, what each arm did better — and **what both missed**, the gap neither arm covered, which is the one thing no score can tell you. Rubrics are now weighted rather than counted, so an arm that wins the dimension that matters most is no longer outvoted by two narrow losses on lighter ones. Comparisons graded before this show their scores and say plainly which parts an older evaluator did not record. [Try it in KAOP](https://kaop.komodor.com/quality-lab/agents) ### Improved: A workflow run page now offers one way out, and it says where that goes Opening a workflow run used to hand you the module's whole tab bar and a **New workflow** button — so a page you opened to read one incident also offered to switch sections and start something new. The run page now carries a single back link at the top left and nothing else. **The link names where you came from.** Arrive from the Incident feed and it reads **Incident feed**; arrive from one workflow's **Workflow runs** tab and it reads **Workflow runs**. Either way it returns you to that exact list with your filters still applied. A run opened from a bookmark or a pasted URL has no origin to name, so it points at the unfiltered Incident feed and says so. [Try it in KAOP](https://kaop.komodor.com/modules/incidents/feed) ### Improved: The run page headline now spans the page, with findings beside the summary The incident run page put every section inside one two-column grid, so the headline — title, severity, status, workflow and trigger time — was squeezed into two-thirds of the width with an empty rail beside it, and **Findings** was read through a narrow column. The headline card now spans the full page width, and directly under it **Findings** and the **Investigation summary** sit side by side in one row. Findings gets the wide column, so an agent-written root cause takes far fewer screens to read. The panel that asks you for a decision and the errored / needs-review banners that carry **Retry** stay full width above that row, and everything below it — the steps panel, deduped alerts, the collapsed alert payload, the Agent team card, **Mark resolved** and the timeline — keeps the order it already had. On a narrow window the row stacks, and a run that never started shows its findings full width with no empty summary card beside them. [Try it in KAOP](https://kaop.komodor.com/modules/incidents) ### Improved: A run waiting on your decision now keeps it docked at the bottom of the page When a workflow run stops on something a person has to answer, the decision used to sit inline in the middle of the page — so on a long run it scrolled out of sight exactly when you went looking for the findings you needed in order to answer it. It is now docked to the bottom of the page instead: it stays put however far you scroll, and you can collapse it to a single line while you read. Everything above it stays fully readable and scrollable — the dock sits at the bottom of the content, it does not float over it. Both the workflow run page and the incident detail page use it, and nothing about the decision itself changed: the same frozen options, the same steps and undo instructions, the same "selecting nothing applies nothing". [Try it in KAOP](https://kaop.komodor.com/modules/incidents) ### Improved: Field mapping now shows what reads each field, and tests a path before you save it The four field-mapping boxes on an endpoint — and their per-workflow overrides — used to sit empty, which read as four blanks waiting to be filled when an empty box actually means "let the sender decide". Each one now shows what does decide the field and is locked until you press Override, with the button that hands it back named for what takes over: Use default on the endpoint, Follow endpoint on a workflow. What a path may look like is now stated rather than implied: dot-separated keys that can nest, like `details.incident.title`, with your sender's own example in each box. Paths that could never resolve — a pasted `or` list, brackets, quotes — are caught before the step will advance, instead of being saved and quietly doing nothing on every delivery. And the mapping can finally be tried before it is trusted. The endpoint's Test & mapping step carries a payload you can edit, seeded from the last request it received, and shows exactly what your paths find in it — including when a dedup key resolved to nothing and the sender's default answered instead. Nothing is sent and no incident is opened. [Try it in KAOP](https://kaop.komodor.com/integrations) ### Improved: Incident Slack messages carry three times more findings An incident's Slack notification used to cut its findings at 2,800 characters, which trimmed roughly half of them mid-report. The budget is now 8,400, so about nine in ten incidents arrive whole — and the ones still long enough to be cut are trimmed on a whitespace boundary and marked, as before. Slack allows 40,000 characters in a message, so there is room left over. [Try it in KAOP](https://kaop.komodor.com/modules/incidents) ### Improved: Triaging and Triaged now read Investigating and Investigated "Triage" was jargon that appeared in exactly one place — the status badge. Everything around it already said *investigation*: the help text, the detail banner, the timeline entry, even the agent doing the work. The badge now matches. **Triaging** reads **Investigating**, and **Triaged** reads **Investigated**, on the incident tile, the Status filter rail, the detail page and the notification event list. **Investigated does not mean "waiting for you."** It covers every stage after the investigation finishes, so an incident reads Investigated while the workflow is proposing a remediation, applying one, verifying it, or writing the postmortem — as well as when it is genuinely sitting there. The status tooltip used to claim it was "awaiting operator sign-off to resolve", which was simply untrue for an incident mid-remediation. It now says what the status actually covers. If you want the incidents that really are blocked on a person, that is the Status rail's needs-a-person filter and the count on the Incident feed tab — not this status. Filters and API values are unchanged: `status=triaging` and `status=triaged` still work, so saved filters and scripts keep working. [Try it in KAOP](https://kaop.komodor.com/modules/incidents/feed) ### Improved: The incident status formerly shown as Errored now reads Error The status an incident lands in when its workflow stops on a failure used to render as **Errored** on the incident tile, the Status filter rail, the detail banner and the notification-event list. It now reads **Error**, matching the noun-shaped vocabulary the rest of the status set uses. Nothing about the state itself changed: the same incidents are in it, **Retry** still resumes the workflow from the step that did not finish, and the value the API accepts and returns (`status=errored`) is unchanged — so a saved filter, a webhook subscription or a script keyed on it keeps working. [Try it in KAOP](https://kaop.komodor.com/modules/incidents/feed) ### Improved: The wait marker now says who owes the answer An incident parked on a person used to carry a **Waiting for response** marker. That wording said neither *who* owed the response nor that the row was waiting on you rather than on the alert provider, and it sat next to **Needs review** — a different state, wanting a different action — with nothing to tell the two apart. The marker now reads **Needs your answer**, on the incident feed, every per-workflow feed and the incident's own page. The three things a person can be asked for are now one name each: **Error** (press Retry), **Needs review** (the workflow finished with nowhere left to go — read the findings and close it out), and **Needs your answer** (the workflow asked you a question and is paused on it). The marker's tooltip also stops claiming the question is always a remediation choice: since a guardrail hold parks a run the same way, it can equally be an approval the workflow is waiting on. Nothing about the underlying state changed, and the marker still layers on the status — a parked incident still reads **Triaged** while it waits. [Try it in KAOP](https://kaop.komodor.com/modules/incidents/feed) ### Improved: The incident feed reads as a tile list, keeping every column it had The **Incident feed** is no longer a table. Each incident is now a divided tile — provider mark, severity, title, and a muted line carrying the module, the workflow, when it started and its orchestrator run. Nothing was dropped. **Duration**, **Errors** and **Cost** keep fixed, right-aligned slots that still line up down the list, each labelled so a number reads on its own, and **Status** keeps an aligned position of its own beside them — together with the badges for an incident waiting on a person, one whose agent is offline, and one that needs attention. The per-workflow **Workflow runs** tab gets the same row. [Try it in KAOP](https://kaop.komodor.com/modules/incidents/feed) ### New: Spot incidents that are stuck, right from the incident list An incident that cannot move without a person — errored, out of retryable steps, or parked on a remediation decision — now carries an amber marker on the row itself, distinct from a running one, and a new **Stuck** filter in the Status rail narrows the list to exactly those rows in one click. **Stuck** is a different question from the feed tab's own badge: the tab counts the whole backlog that will eventually need a person, including incidents still being investigated; this filter is only what is blocked right now. The two numbers are allowed to differ. The marker clears the moment you act (resolve, retry, or answer the parked decision) — no reload needed, on the module feed and every per-workflow feed. [Try it in KAOP](https://kaop.komodor.com/modules/incidents/feed) ### New: The Incident feed tab shows how many incidents need a person The **Incident feed** tab in the Incidents module now carries a count of the incidents waiting on a human — open, triaging, errored, needs-review, or parked on a decision — reading "3 incidents need you" on hover and to a screen reader. It is visible from every page of the module, so you can see the backlog without leaving the workflow you are reading. It is a needs-a-person count, **not** an unread count, and the difference is worth knowing: the number drops when *any* operator resolves an incident, not when you look at it. Each incident is counted once even when it is both errored and awaiting a decision. The badge is absent while the count is loading, if the read fails, or when nothing needs a person — so a number on the tab always means there is genuinely something waiting. [Try it in KAOP](https://kaop.komodor.com/modules/incidents/feed) ### Fixed: The Incident feed count now says what it counts — and counts the right rows The number on the **Incident feed** tab used to read "3 incidents need you", and it was counting more than that claimed. Incidents still being investigated — `Open` and `Triaging` — were included, even though the workflow is running and nobody is waiting on a person. The badge was a backlog figure wearing a needs-a-person label. It now counts exactly what is blocked on someone: incidents in **Error**, in **Needs review**, or carrying a **Needs your answer** marker. It reads **"3 incidents need your attention"**, so the number says who is being asked and what for. **Expect the number to get smaller.** That is the point — the old one was too high. If you want the rows behind it, the Status rail's filter (renamed from **Stuck** to **Needs your attention**) now selects exactly the same set, so the badge and the filter can no longer disagree about what needs a person. The badge is still absent while the count is loading, failed, or genuinely zero, so a rendered badge always means a real, positive number. [Try it in KAOP](https://kaop.komodor.com/modules/incidents/feed) ### Fixed: Fleet status badges no longer spill into the next column An agent carrying two status badges — Online plus Disabled, or Deploy Failed plus Disabled — laid them side by side and overflowed the Status column, rendering on top of Last run. Each badge takes its own line now, so the pair stays inside its column. Rows with a single badge are unchanged. [Try it in KAOP](https://kaop.komodor.com/agents) ### New: Edit a knowledge page in place A wiki page can now be corrected without deleting and re-uploading it. Open a page, hit Edit, and change its title or Markdown body — the page keeps its identity, so the citations agents have already made and its retrieval count both survive the edit. Saving a body change re-indexes the page so agents retrieve the corrected text; renaming a page skips re-indexing entirely. [Try it in KAOP](https://kaop.komodor.com/knowledge) ### Fixed: Connection status now shows before the install steps for self-hosted agents Activating a self-hosted agent used to bury the "waiting for the first heartbeat" checklist below the worker token and install commands, so you had to scroll to see whether the agent had actually connected. It now appears right away, above the copy-paste steps — matching where it already showed up when importing an existing agent. [Try it in KAOP](https://kaop.komodor.com/agents/new) ### Fixed: Coding agent connect commands now paste cleanly into zsh The `claude mcp add` and `codex mcp add` commands on the Coding Agents page carry a `?account=` query string that pins the connection to your current workspace. Pasted unquoted into zsh — the default shell on macOS — the `?` was read as a filename wildcard and the command aborted with `zsh: no matches found` before it ever reached the CLI. The URLs in every generated command are now quoted, as is the `` placeholder in the Codex snippet, whose angle brackets were being read as a file redirection. The commands are unchanged for anyone who was already running them in bash, and the quoting is safe to paste into PowerShell and cmd as well. [Try it in KAOP](https://kaop.komodor.com/settings/coding-agents) ### Improved: New agents can chat by default The Chat capability toggle on the Agent instructions step now starts on instead of off, so a newly created agent shows up in Chat right away. You can still switch it off before creating the agent if you want it to run without a chat surface. [Try it in KAOP](https://kaop.komodor.com/agents/new) ### Improved: Activating a self-hosted agent now reads as two clear steps The Activate step used to keep its "generate a worker token" framing — and a live button — after the token already existed, so the connection status and the install commands read as parts of token generation, and pressing the button again quietly replaced a token your cluster might already be running. Creating the agent is now its own phase, and once it exists the step leads with the connection status, followed by the install steps. A separate "Rotate token" action covers the case the shown-once token gets lost: it issues a new one without re-creating the agent's triggers or re-applying its roles and secrets. [Try it in KAOP](https://kaop.komodor.com/agents/new) ### Fixed: A bad New Relic key is now refused when you connect Connecting New Relic accepted any API key. The connection went green, said it was connected, and the key was never exercised — the failure only appeared later, when discovering MCP tools returned "401 Unauthorized" and pointed at a connection that still looked healthy. Connecting now calls a read-only New Relic tool with the key you supplied and refuses the connection if New Relic rejects it, so a wrong or expired key fails on the screen where you can still fix it. A key that works is unaffected. [Try it in KAOP](https://kaop.komodor.com/integrations) **September 8, 2026** ### Improved: Workflows now list in one deliberate order across the Incidents module The Workflows table and the Overview comparison table used to sort differently — one by creation date, the other alphabetically — so the same workflows appeared in two orders on two screens of one module. Both now share a single order: active and paused workflows first, most recently fired at the top, then the drafts by when you last edited them. A workflow that has never run sorts to the bottom of its group instead of landing wherever an empty timestamp happened to fall, and the order is applied on the server, so it holds across pages. [Try it in KAOP](https://kaop.komodor.com/modules/incidents/workflows) ### Fixed: Creating an endpoint inside the workflow wizard now shows its URL and token An endpoint created with "+ Create new endpoint" on the New incident workflow wizard's Endpoint step now reveals the endpoint URL, the one-time token and a ready-to-run curl command, exactly as the Endpoints page does. The token was previously issued and never shown, leaving a live endpoint no sender could authenticate against; the reveal now stays put while you move between wizard steps and clears only when you dismiss it. [Try it in KAOP](https://kaop.komodor.com/incidents/workflows/new) ### Improved: A workflow row now offers one primary action plus a 3-dot menu Every **Incidents module** workflow row — on both the Workflows list and the Overview comparison table — now shows exactly one primary action chosen by status (**Activate** on a draft or paused workflow, **Pause** on an active one), with Edit, Duplicate and Delete behind a 3-dot menu. **Delete** is last and marked destructive, so it is no longer one mis-click from Duplicate, and it still asks for confirmation by name. An action the workflow's current status forbids is now shown disabled with the reason instead of silently disappearing. [Try it in KAOP](https://kaop.komodor.com/modules/incidents/workflows) ### Improved: The workflow Overview gets the same six-option window picker Each incident workflow's **Overview** tab now offers the same six windows as the module Overview — 1h, 3h, 24h, 7d, 14d, and 30d, defaulting to 24h instead of 7d. Both the run-statistics panel and the alerts funnel honour the selection, including the three sub-day options. The workflow Overview keeps its own window control, independent of the module Overview's — picking a window here never changes what the module Overview shows, and vice versa. [Try it in KAOP](https://kaop.komodor.com/modules/incidents) ### New: A workflow now tells you whether it is healthy, not only whether it is on Each incident workflow's **Overview** tab now carries a health verdict on the alerts-funnel card, beside — not instead of — the Draft / Active / Paused badge. That badge says what someone configured; the verdict says what the runs since then actually did. It reads **Pipeline healthy**, **Pipeline degraded**, or **Pipeline unhealthy** from the share of the window's runs that errored — under 5%, 5% to 20%, and 20% or more — and states the arithmetic in words beside the pill: *"3 of 24 runs errored (12%) in the last 7 days."* The denominator is named on purpose: this is a rate over the workflow's own **runs**, which is a different question from the module Overview's incident-level "% of errors". Two cases are withheld rather than guessed. Under ten runs in the window it reads **Not enough data** — with that few, a single failure would decide the verdict on its own. A workflow that has never been activated reads **Not activated**, since it has dispatched nothing to judge. The verdict follows whichever window the picker holds, and it is a description of what happened, not an SLO. [Try it in KAOP](https://kaop.komodor.com/modules/incidents) ### Improved: Combine several conditions in one Slack routing rule A routing rule can now match on a list of conditions rather than one idea. Pick **all of** or **any of**, then combine a mention, a channel message, a direct message, a keyword, or a named sender, and scope the whole thing to particular Slack channels. The routing list reads the rule back in words, so a rule says "mention or direct message" rather than showing a type name. The bot is also quieter. Once an agent has answered in a channel thread, further messages there are treated as conversation between people, so a team can talk through an answer without the bot replying to every line. Mention it again to bring it back, and direct messages are unaffected. Three things got safer alongside it. A channel has exactly one fallback rule, it is shown last because it is only ever tried after everything else, and switching which rule holds it now takes one step instead of two, so a failure halfway can no longer leave a channel with no fallback at all. A new rule starts out needing a condition, so saving one without thinking no longer creates a rule that quietly matches every message. And a freshly connected workspace gets one starter rule instead of two, which still answers a mention or a DM and nothing else. [Try it in KAOP](https://kaop.komodor.com/channels) ### Fixed: A run's token count now adds up The Tokens tile on a run's page showed a total alongside an "in" figure that did not account for it — on a prompt-caching agent the caption described under 1% of the number above it, because it counted only fresh input and left out the tokens served from and written to the prompt cache. The caption now sums every input the run consumed, so it reconciles with the total beside it. Cost was always priced from the full picture and is unchanged. [Try it in KAOP](https://kaop.komodor.com/history) ### Improved: An incident run now opens on the findings, not on raw JSON The incident run page used to make you scroll past the step accordion and the full alert payload before you reached what the investigation actually concluded. **Findings** now leads the page, directly under the header and the banners you may need to act on. The steps panel follows it, and the **Alert payload** sits last, collapsed until you ask for it. The right rail was reordered to match: the investigation summary, then an **Agent team** card that names the orchestrator and each specialist by role — replacing the unlabelled list of run links — then **Mark resolved**, with the timeline closing the page. [Try it in KAOP](https://kaop.komodor.com/modules/incidents) ### New: See why a workload can't be right-sized The Right-Sizing table now shows an info icon next to a workload's Automated toggle when it can't be right-sized yet — hover it for the reason (managed by an HPA, too new, not seen recently, or a QoS guardrail) and, where applicable, a shortcut to update the policy. [Try it in KAOP](https://kaop.komodor.com/k8s-cost/right-sizing) ### New: A Cost & Optimization drawer for each Right-Sizing workload Clicking a row in the Right-Sizing table now opens a drawer with that workload's cost summary, a 30-day cost trend, and a per-container breakdown — searchable, with a 7/30-day time window and expandable rows showing CPU/Memory usage graphs. A "Right Sizing automation" card lets you add the workload to its resolved policy or jump to Settings to review policies. [Try it in KAOP](https://kaop.komodor.com/k8s-cost/right-sizing) ### Fixed: Quality Lab names a deleted agent instead of its id A shadow-comparison verdict deliberately outlives the agent it graded, but the Agent column and an evaluator's own row used to fall back to the opaque `agt_…` id once that agent was gone. Both now read **Deleted agent**, with the id still one hover away for anyone who needs it. [Try it in KAOP](https://kaop.komodor.com/quality-lab/agents) ### New: See what your agents' memories are about, and filter by it The Memory page has a Labels tab: the dimensions you declared for a space (deployment, cluster, env, team) and the keys agents invented while writing, each with a line saying what it holds and its values with the number of memories carrying them. Click a value to see exactly those memories, or a label on any memory to filter the list by it. A search now ranks a memory about your environment above a near-identical one about somewhere else. **Agents fill these in as they work, and they write that line too.** An agent saving a memory says what it is about and can say what a new key holds, so the next writer reuses the key you already have instead of inventing a synonym beside it. A key an agent invented can be promoted to a dimension of its own, keeping its key and its values. **Stop tracking a dimension and nothing is lost.** The values stay, the memories keep them, and they still count towards ranking. The key stops being one of the dimensions agents are asked to fill in, and stays on the tab marked **Not tracked**, with the date it stopped. [Try it in KAOP](https://kaop.komodor.com/memory?tab=labels) ### Improved: Incidents module workflows table gets aggregate columns; the feed is now a table The **Incidents module Workflows** page now shows Status, Agents, Incidents (open), Duration (P95), Errors %, Cost, Created, and Last Seen for every workflow, scoped by a Window picker and a Workflow selector — with row actions moved into a 3-dot menu. The **Feed** is now a table with Title, Status, Started, Duration, Errors %, and Cost, replacing the card list. [Try it in KAOP](https://kaop.komodor.com/modules/incidents/workflows) ### Improved: A quieter filter sidebar on the Incidents feed The **Incidents feed**'s filter sidebar was a three-group rail with icons on some rows and not others, which made it read as unfinished and buried the counts you actually scan. It is now a narrow two-group rail — **Status** and **Priority** — with no icons and the count sitting plainly beside each label. Filtering by workflow moved out of the rail to a **workflow picker** next to the search box, where it belongs: unlike the two filters above it, it narrows the incidents we fetch rather than the ones already on screen. A workflow's own **Feed** tab now shows exactly the same sidebar as the module-wide feed. [Try it in KAOP](https://kaop.komodor.com/modules/incidents/feed) ### Fixed: Filter the Incidents feed for Errored and Needs review The **Incidents feed**'s Status filter listed only four of the six statuses an incident can hold, so an errored incident — or one that ran to completion with nothing left to retry — could only be found by scrolling the unfiltered list. **Errored** and **Needs review** now appear on the rail with their counts, on both the module feed and a workflow's own Feed tab, in lifecycle order alongside the statuses that were already there. [Try it in KAOP](https://kaop.komodor.com/modules/incidents/feed) ### Improved: An incident waiting on your decision now says so in the feed An incident parked on a human decision used to be indistinguishable from one still being worked: it read **Triaged** like any other, and nothing in the feed said a person was the thing it was waiting for. Incident rows now carry a **Waiting for response** marker — on every incident feed and detail page — and the Status rail gained a facet that filters straight to them, so the incidents that need you are one click away instead of something you find by opening rows. The marker sits **alongside** the status rather than replacing it, because both facts are true at once: a parked incident really is triaged *and* waiting. It is a wait marker layered on the status, not a status of its own — which is why such an incident is still counted under **Triaged** too. The incident detail page says the same thing in words, under the status it already explained. One scope note: the marker tracks a pending remediation *decision*. A run held on a guardrail approval is a different kind of wait, and this marker does not claim to cover it. [Try it in KAOP](https://kaop.komodor.com/modules/incidents/feed) ### Fixed: Incident findings always read as a report An investigation whose alert asked the agent to answer in JSON used to land on the incident's **Findings** card as the raw payload — a wall of `{"status": …, "answer": "…\n…"}` with every line break escaped, even though the root cause was written inside it. The answer is now unwrapped back into markdown, so headings, tables and evidence render the way they do for every other run. The card also stops printing *"No root cause identified yet"* above a narrative that plainly states one — that line now appears only when there really is nothing else to show. [Try it in KAOP](https://kaop.komodor.com/modules/incidents) ### Fixed: The incident feed row tells you which workflow triggered it, and which run it dispatched Every row in the **Incidents** feed now carries a meta line under its title naming the module, the workflow that triggered the incident, and the orchestrator run it dispatched — the run id links straight to that run. An incident whose alert arrived with no severity is now tagged **Unknown** instead of showing a blank tag. [Try it in KAOP](https://kaop.komodor.com/modules/incidents/feed) ### Improved: Fleet shows why a hosted agent never came up A hosted agent whose pod is stuck — `CrashLoopBackOff`, `ImagePullBackOff` and friends — no longer sits on **Deploying** with nothing to say. Its status chip now reads the Kubernetes reason itself, and the agent's panel shows the kubelet message together with the last lines of the crashed container's log, so a bad image, a missing credential or a broken entrypoint is visible without `kubectl`. The status clears on its own once the pod is healthy again; editing, archiving and rolling back stay available throughout. [Try it in KAOP](https://kaop.komodor.com/agents) ### New: Findings now closes with what was applied to the incident — or why nothing was The **Findings** card on an incident run ended at *Next steps*, which is advice — it never said what the workflow actually did. It now closes with a **Resolution** section stating the fact: the action that was applied and the step that applied it, or, far more often, why nothing was. A run that errored names the step it stopped at, a workflow that only proposes a remediation says an operator has not selected one yet, and a run still in flight says it is not resolved yet. The sentence is derived from the run's terminal step and the workflow's remediation mode rather than written by an agent, so it cannot report a fix that never happened. [Try it in KAOP](https://kaop.komodor.com/modules/incidents) ### Improved: Findings on a run leads with the root cause instead of everything at once The **Findings** card on an incident run used to render every section expanded, so an agent-authored investigation could run several screens tall and push the rest of the run page out of view. It now opens on the root cause alone, height-capped to a few lines so a long one cannot defeat it, with **Show all findings** revealing the key evidence and next steps underneath. Everything behind the toggle is narrative — nothing you need to act on is hidden. [Try it in KAOP](https://kaop.komodor.com/modules/incidents) ### Improved: A remediation decision now shows the procedure, the verification and how to undo it A parked remediation decision used to show you an option's label and a single shell command. It now shows what the workflow actually froze behind the question: the finding that justifies asking at all, and per option the numbered procedure, the one check that says it worked, its cost or capacity impact, and the steps that reverse it. Where there is no undo, the option says which kind of no it is: an action recorded as irreversible reads as the fact somebody established, and an action nobody described reversing reads as missing information you should go and check. Neither is ever left as a blank section — before approving a production change, an absence you can see beats one you have to infer, and a gap nobody filled in should never look like a judgement somebody made. There is also an "Other" answer now: describe the action you would rather have reviewed, and nothing is applied. Everything shown is the copy taken when the question was asked, so a decision you answer a week later is the one you were originally offered. [Try it in KAOP](https://kaop.komodor.com/attention) ### Fixed: Run token and cost figures no longer double-count A run's recorded tokens and cost are now accurate in two cases that previously inflated them. A run whose worker was killed before it reported a result is requeued and executed again. Every span in that replay is new, so the replayed turn's usage used to fold into the run's total a second time — a re-executed run could report several times the tokens and cost it actually spent. Usage is now counted once per provider message, so a replay adds nothing it already counted. Separately, the per-turn usage an agent reports is now cumulative on every field. Mixed semantics previously let one run record a full conversation's worth of cache reads beside a single turn's input and output tokens, which made its estimated cost far larger than what it spent. Both apply to runs executed by agents on SDK 0.3.17 or later; figures already recorded are left as they were. [Try it in KAOP](https://kaop.komodor.com/quality-lab/agents) **September 7, 2026** ### Improved: The workflow wizard gives remediation its own step The New incident workflow wizard's steps are now Name → Endpoint → Investigation team → Remediation → Review & activate. Picking the workflow template, choosing who remediates, and subscribing a notification sink used to be bundled into "Team & steps" alongside the orchestrator and investigate/verify specialists; they now live on their own Remediation step, so investigation staffing and remediation/delivery decisions no longer compete for the same screen. [Try it in KAOP](https://kaop.komodor.com/incidents/workflows/new) ### Fixed: The workflow template picker only offers what it can actually save The **New incident workflow** wizard offered a choice of workflow templates to every account, then rejected two of the three at save in environments where only one of them can run. The template a workflow runs is recorded on its workflow definition, and in an environment where nothing creates one, every workflow ends up on the operator-approved template no matter what was picked. The picker now asks the account whether a choice can be recorded at all. Where it cannot, it shows the one template the workflow will actually run and says why — either the workflow engine is not enabled for the account, or no step-orchestrator agent is deployed for it — instead of rendering buttons that fail on save. Where a choice can be recorded, nothing changes: the full list stays selectable. An existing workflow keeps showing its own template, including one this account could not create today. [Try it in KAOP](https://kaop.komodor.com/incidents/workflows/new) ### Improved: The Steps tab now shows what Ingest and Group produce The Steps tab's "Before the workflow" group (Alert trigger, Dedup, Filtering) previously showed only a summary of each stage's configuration. Alert trigger and Dedup now also list the fields they hand to the rest of the pipeline — `alerts`, `started_at`, `title`, `service`/`environment` for Alert trigger, and `incident`, `merged_alerts`, `description`, `severity` for Dedup — in the same Outputs chip list a real workflow step shows. Filtering declares none, since it only decides whether an alert is routed at all. [Try it in KAOP](https://kaop.komodor.com/incidents/workflows) ### New: The workflow Overview now shows five headline stats Each incident workflow's **Overview** tab now shows five headline numbers above the alerts funnel: alerts ingested, noise suppressed (with the count behind the percentage), incidents created, RCA success, and remediation coverage. RCA success and remediation coverage are new: they read the run-statistics window already loaded for the tab, and each tile explains what it counts. [Try it in KAOP](https://kaop.komodor.com/modules/incidents) ### Improved: A workflow's Incidents feed now has search and a Priority filter The per-workflow **Incidents feed tab** now matches the module Incidents feed: a search box that matches incident title and alert summary server-side, and the Severity facet relabeled **Priority** (same field, same highest-first order). Every result stays scoped to the one workflow. [Try it in KAOP](https://kaop.komodor.com/modules/incidents) ### Improved: Edit a workflow step's agents from the Steps tab Every orchestrated step on a workflow's **Steps tab** now has an "Edit specialists" link, in both List and Graph view. It deep-links straight to the wizard's Team & steps page with the right specialist pool already open, instead of leaving you to find the wizard and the matching step yourself. Investigate and Verify — and, on a human-selected workflow, Propose and Apply remediation — share one binding, so the tab now says "Also feeds: …" next to the link whenever changing one step's agents changes another step too. [Try it in KAOP](https://kaop.komodor.com/modules/incidents/workflows) ### Fixed: Review no longer reports a remediation agent a workflow will never call The **Orchestrated loop** template has no remediation step — the orchestrator remediates inside Investigate and asks an operator from there. But if you picked a remediation agent first and then switched to that template, the pick stayed behind: the wizard's last screen listed it as the workflow's remediation, and the workflow was saved carrying a binding its graph could never dispatch. Switching to a template with no remediation step now drops the pick, and **Review & activate** reports remediation as handled by the orchestrator inside Investigate rather than naming an agent. Switching between two templates that both have a remediation step keeps your selection, as before. [Try it in KAOP](https://kaop.komodor.com/incidents/workflows/new) ### Fixed: New Relic connections can pick their region New Relic serves its MCP endpoint from three regional hosts, and a User API key issued in one region is rejected by the other two. The connection always dialed the US host, so an EU or JP account could paste a valid key, watch the connection go green, and then get a 401 from every tool call — a failure that reads as a bad key rather than as the wrong endpoint. Connecting New Relic now offers a **New Relic region** field alongside the API key. Existing connections are untouched and stay on the US endpoint, which is where they already were. [Try it in KAOP](https://kaop.komodor.com/integrations) ### Improved: Loki setup asks which auth you use Connecting Loki now starts by asking how your Loki is fronted — basic auth, a bearer token, or nothing — and shows only the fields that shape needs, instead of offering all three and rejecting the combinations that do not go together. Basic auth is the default, since that is how Grafana Cloud Logs works. Loki has no MCP server we host, so the next step has always been registering your own `loki-mcp`. That form now opens on the SSE transport when you reach it from a Loki connection — `loki-mcp` speaks the legacy SSE transport, and picking HTTP fails in a way that looks like an unreachable server rather than a wrong setting. The credential picker also says what binding actually injects, including the tenant header, and notes that `loki-mcp` queries the tenant in its own `LOKI_ORG_ID`, so a mismatch there is something you can see rather than discover through empty results. [Try it in KAOP](https://kaop.komodor.com/integrations) ### Improved: Incidents Overview adds 1h/3h/24h windows, and defaults to 24h The **Incidents module Overview**'s window picker now offers **1h, 3h, and 24h** alongside the existing 7d/14d/30d, and defaults to **24h** instead of 7d. The volume, resolution-time, automation-rate, and spend trends bucket **hourly** for a sub-day window and daily otherwise. [Try it in KAOP](https://kaop.komodor.com/modules/incidents) ### Improved: The Incidents feed's Severity filter is now Priority The **Incidents module feed** filters incidents by the same field as before — no new data, no schema change — but the facet in the sidebar is now labelled **Priority** instead of **Severity**, sorted highest-first. The underlying field is still `severity` on the URL and the API. [Try it in KAOP](https://kaop.komodor.com/modules/incidents) ### New: Search the Incidents feed by title or alert summary The **Incidents module feed** now has a search box. It matches an incident's title or its alert summary, filters server-side (so results stay correct once the list pages), is debounced as you type, and is reflected in the URL so a search is shareable and survives a refresh. An empty result now says which term didn't match, instead of the generic "no incidents in this view" message. [Try it in KAOP](https://kaop.komodor.com/modules/incidents/feed) ### Fixed: Creating an incident workflow no longer fails partway through Creating a new incident workflow could fail on a later step with two error toasts, both saying the remediation mode (now the workflow template) "is chosen when the workflow is created and cannot be changed." The wizard creates the pipeline as soon as you leave the Name step, before the template picker is even shown — every later autosave was then re-sending the template on every forward navigation, which the backend correctly refused once the pipeline had nothing to record it against. The wizard now only sends the template when you actually change it, so walking through the rest of the wizard no longer restates a value you never touched. A single failed save also now raises one error toast instead of two. [Try it in KAOP](https://kaop.komodor.com/incidents/workflows/new) ### Improved: Every incident now says which module it came from Incident rows in the module feed and the incident drilldown header now name the **module** that owns the incident's workflow, alongside the workflow and the orchestrator run — the full Module → Workflow → Run trail. The module is read from the workflow's own definition, so it stays correct without a copy stored on the incident, and it is what tells two modules' incidents apart once more than one module ships workflows. [Try it in KAOP](https://kaop.komodor.com/modules/incidents/feed) ### Improved: Incident KPI tiles now show the prior period and a delta The Incidents module Overview showed a scalar and a sparkline per KPI, with no way to tell how a number moved without eyeballing the trend line. Every tile — Volume, P95 resolution time, % automated, and Spend — now also shows the prior period's value and a signed delta, coloured by whether the move was good or bad for that KPI (or left neutral where a rising or falling count isn't itself good or bad, like Volume). [Try it in KAOP](https://kaop.komodor.com/modules/incidents) ### Fixed: An imported agent can notify a sink when its runs finish Importing an agent you already run offered no way to be told when its runs finish. The Triggers step's second section — **When the run finishes** — was hidden for an import, on the reasoning that an import deploys nothing and so has no runs to hear about. That reasoning was wrong. Deploying is not what produces a run. AgentOps dispatches an imported agent's runs, records them, and emits their lifecycle events exactly as it does for an agent it deployed itself — which is the same basis on which this wizard has always let you give an imported agent a **schedule**. A trigger fires, a run finishes, and a subscribed sink hears about it. Offering one half of that pair and hiding the other was the bug. The picker is now on the Triggers step for every kind of agent, and the sink you choose is applied when you import, alongside the roles, secrets and triggers. **Two smaller things on the same screen.** The **Import agent** button now states everything blocking it in one message rather than stacking one per blocker, and repeats it when you hover the disabled button. And a **Not connecting?** section lists what to check when your agent has not appeared yet — including the mismatch between `AGENTOPS_AGENT_ID` and the id the token was minted for, which is refused as an authorization error and so reads like a bad token when it is not. It is deliberately not a timed warning. Wiring up an agent in your own repository and pipeline can take days, and a page that called that broken after five minutes would be wrong nearly every time it said so. [Try it in KAOP](https://kaop.komodor.com/agents/new) ### Improved: Google Cloud now covers fourteen services, with mutating tools hidden by default Connecting Google Cloud now offers fourteen of Google's hosted MCP servers rather than three: Cloud Logging, Monitoring, Error Reporting, Trace, Service Health, Quotas, Compute Engine, GKE, Cloud Run, Billing, Recommender, Asset Inventory, Network Management, and Policy Troubleshooter. One service-account key covers all of them, and you pick which to register when you connect. The setup commands now grant `roles/mcp.toolUser` alongside `roles/viewer` — without it Google refuses every tool call while the connection still tests green, which previously looked like a broken integration rather than a missing role. Two services need one extra grant each, and the wizard says so: GKE wants `roles/container.clusterViewer`, and Billing wants `roles/billing.viewer` on the billing account. Servers registered through the wizard also hide the mutating tools these endpoints expose — creating, deleting or patching instances, clusters and services — so agents see the read-only surface and nothing that could change your project. You can widen that on the server itself if you want it. [Try it in KAOP](https://kaop.komodor.com/integrations) ### Fixed: Gateway-routed run costs were priced off the wrong model A run's worker reports its own client-computed cost alongside its token counts, and the control plane trusted that figure once it had a nonzero cost, even when it came from a stale local price table with no entry for the model actually used — most visibly on Sonnet 5, priced at Sonnet 4.6's rate and shown roughly 1.5x too high. Run cost is now priced from the account's own gateway rates whenever the reported figure's provenance is the client's own guess rather than a real billing identity. A genuinely provider-billed cost is still never second-guessed. Existing runs correct themselves the next time they are viewed — no migration needed. [Try it in KAOP](https://kaop.komodor.com/runs) ### Improved: Incident Findings now show root cause, key evidence, and next steps The drilldown's Findings card used to render whatever markdown the investigating agent wrote, in whatever structure it chose — sometimes a root cause up front, sometimes buried, sometimes not named at all. It now reads the investigation's own structured output and shows named sections: **Root cause**, **Key evidence**, and **Next steps** (all bulleted lists). Older incidents whose investigation recorded only a free-form narrative still show that narrative under **What happened**. An incident whose investigation produced no root cause says so explicitly instead of showing an empty section. [Try it in KAOP](https://kaop.komodor.com/incidents) ### New: See which alerts merged into an incident, not just how many The incident drilldown showed a "fired N times" count and nothing else, so there was no way to see which fires actually merged into an incident that refired. A refired incident now shows a Deduped Alerts card listing each merged fire with its timestamp. An incident that never refired shows no card. [Try it in KAOP](https://kaop.komodor.com/modules/incidents/feed) ### Improved: Create a webhook endpoint without leaving the workflow wizard The New incident workflow wizard's Endpoint step now creates an endpoint in place: name it, pick which sender it is preset for, press Create, and the new endpoint is selected for the workflow. Previously the only way out of "no endpoints yet" was a trip to the Endpoints page, which meant abandoning a half-filled wizard and starting over. Everything already entered — the workflow name, the field mapping, the filters — survives the create, and a create that fails says so inline and leaves the draft untouched. [Try it in KAOP](https://kaop.komodor.com/incidents/workflows/new) ### Fixed: Autoscaling a self-hosted agent now works, and keeps its tool access Turning on KEDA autoscaling for a self-hosted agent could stop it calling any of its tools, reporting `401 Unauthorized` from the control plane and ending every run as "orchestration incomplete". Autoscaling gave every replica of the agent the same worker id, and each replica's start-up replaced the one tool credential that id had — so whichever pod started first kept running with a credential that no longer worked, and only a restart cleared it. Each start-up now takes its own credential instead of replacing its siblings', so replicas no longer cut each other off. The same shared id also broke the queue-depth reading KEDA scales on, which answered `403` and left the agent stuck at its minimum replica count — that now resolves the agent from the worker's own credential, so scaling actually happens. Upgrade the chart to pick both up. Nothing to change in your values, and credentials already issued keep working. [Try it in KAOP](https://kaop.komodor.com/agents) ### New: Attach an agent to one MCP server, and pick its tools The **MCP tools** step in the agent wizard now offers **one server with the tools you choose**, alongside the integration group it has always offered. Pick **Connected integrations**, then **One server**: choose a server, tick the tools this agent may call, and that is the agent's whole tool surface. Scoping an agent to a single server used to mean creating an integration group with one member — shared, named registry state you then maintained for as long as the agent existed, with its tool list fixed for every agent bound to it. The per-server scope is the agent's own: no group record, and the tools are chosen per agent. Ticking every tool records the tools that exist **today**, so a tool added to that server later is not granted automatically. The group option is unchanged and still the right choice when several agents should share one governed tool set — its panel now says why its tool list is read-only. The first option has been renamed **No MCP servers**, which is what it does on a fresh deploy. Available on both Komodor-cloud and self-hosted agents. Two fewer ways to over-grant: the wizard will not let you save a server with no tools picked, and a tool the server itself withholds cannot be ticked — the step names the layer that withheld it. [Try it in KAOP](https://kaop.komodor.com/agents/new) ### Improved: Pick the time window on an agent's Observability tab The Observability tab on an agent's detail page was locked to a fixed 30-day view. You can now switch it to 7, 30, or 90 days, and the comparison follows your pick instead of always comparing against a fixed 30-day average — so a change from earlier this week shows up right away instead of getting averaged out. [Try it in KAOP](https://kaop.komodor.com/agents) **September 6, 2026** ### New: See shadow A/B verdicts in Quality Lab, not just in Postgres Shadow deploy (dual-deploy A/B) runs a candidate worker beside production under the same agent, with an evaluator grading the two arms head-to-head. Until now the only way to see a verdict was to query Postgres by hand. The **Agent Score Results** tab now has a **Shadow Comparisons** section listing every agent that's been A/B'd, alongside the existing run-level scorecard. Open an agent to see comparison volume over time and an **A/B Test Results** table — production vs. shadow, per rubric, with the individual graded pairs and their un-blinded scores one click away. A low sample size (under 5 comparisons) is flagged rather than shown with false confidence. [Try it in KAOP](https://kaop.komodor.com/quality-lab/agents) ### Fixed: A run's Output tab now renders like its Transcript The **Output** tab on a run — the final answer, separate from the turn-by-turn Transcript — used a thinner markdown renderer than the Transcript did. Tables and inline code (`like this`) came out flat and hard to scan on Output even when the exact same text looked properly typeset one tab over, because both were rendering identical source text through two different, drifted implementations. Output now renders through the same Streamdown-backed component the Transcript already uses — syntax-highlighted code blocks, styled tables, the same typography — so a run's answer looks the same wherever you read it. [Try it in KAOP](https://kaop.komodor.com/runs) ### Fixed: Run activity chart no longer flattens the days before your agent existed An agent's Run activity chart used to draw a flat line of zeros for every day before the agent was created, making a brand-new agent look like a dormant one that just woke up. Those days now show as a gap instead, and the week-over-week change on each tile compares only the days the agent could actually have run. [Try it in KAOP](https://kaop.komodor.com/agents) ### New: Preview a right-sizing policy right from the Right-Sizing table The Right-Sizing table's Policy cell now has a "Preview" link on every policy option — hover an option in the dropdown to see it, in place of its priority number. Preview opens a richer view of that policy: the workload it would apply to (cluster, namespace, workload, kind), the policy's full configuration including its right-sizing guardrails, an "Edit policy" shortcut straight into the policy editor, and a "Manually Apply Policy" action to apply it to this workload on the spot. Edit policy requires the `k8s_cost.policy.manage` capability; without it, the button stays disabled with a tooltip explaining why. [Try it in KAOP](https://kaop.komodor.com/k8s-cost/right-sizing) ### Improved: The incident workflow wizard now reads its templates from the account The **New incident workflow** wizard's "Team & steps" step no longer hardcodes which workflow templates exist or which one is offered by default. It now calls the account's own `available-templates` listing and renders the picker, the step graph, and each step's name and description straight from that response — so an account that only has the operator-approved template sees exactly one option, and Komodor's own account sees all three, with the create default always preselected correctly rather than assumed by position. An existing workflow whose template the account can no longer create — one made before an entitlement changed — still shows its own template rather than an empty picker. The `remediation_mode` field this wizard sent is renamed to `template_key` across the create/update API and the incident pipeline detail response, since it names a workflow *template*, not a runtime mode. [Try it in KAOP](https://kaop.komodor.com/incidents/workflows/new) ### Fixed: Incident Slack notifications are no longer cut mid-sentence An **Incident triaged** notification posted to Slack could stop partway through a word — in one report, mid-way through a run id — leaving a code span open so the rest of the line rendered as raw markup, and giving no sign that anything had been dropped. A truncated triage summary read exactly like a complete one that simply had little to say. Incident summaries now get roughly five times more room before anything is trimmed, so most reports arrive whole. When one genuinely is too long for Slack, it is cut at a word boundary, any open code block is closed, and the message ends with **…(truncated)** — with **View incident** still there for the full report. The same fix covers the resolved, failed, and needs-review notifications, which shared the behaviour. [Try it in KAOP](https://kaop.komodor.com/incidents) ### New: Set your Enterprise Discount Program rates K8s Cost → Settings has a new Discount Settings tab. Enter your AWS, Azure, and GCP Enterprise Discount Program rates so cost estimates across Allocation, Right-Sizing, and Pod Placement reflect your actual pricing. The tab also lets you set your account's Pricing Fallback configuration — the default $/CPU and $/GiB rates Komodor uses when real pricing data isn't available. [Try it in KAOP](https://kaop.komodor.com/k8s-cost/settings) ### New: Connect Loki from the Integrations page Loki is now in the integrations catalog in its own right, so a Loki you run yourself — or a Grafana Cloud Logs endpoint — reaches your agents without a Grafana in front of it. Give it the Loki API base (not your Grafana URL), and the tenant if you run multi-tenant. Authentication is whichever shape your deployment uses: a basic-auth username and password, which is how Grafana Cloud Logs works — the username is your numeric Loki instance ID — or a bearer token, or neither, because a Loki running `auth_enabled: false` accepts an address alone. Testing the connection lists label names on your tenant, so a wrong address or a missing tenant is caught at connect time rather than on an agent's first query. Point an agent at your own Loki MCP server and bind the connection to it, and AgentOps injects the credential and tenant on every call; agents that run their own MCP server receive them as `LOKI_URL`, `LOKI_ORG_ID`, `LOKI_USERNAME` and `LOKI_PASSWORD`/`LOKI_TOKEN`. [Try it in KAOP](https://kaop.komodor.com/integrations) ### Improved: The alerts funnel now lives on the workflow it describes The **alerts funnel** has moved off the Incidents module Overview and onto a workflow's own **Overview** tab, where it is scoped to that one workflow. Open any workflow and the funnel sits between its configuration and its run statistics: every alert that reached this workflow, out to what became of it. It was on the module Overview because that is where it was built, not because that is where the question gets asked. "How much noise is this workflow filtering out, and is it filtering the right things?" is a question about one workflow, and it was being asked next to four KPI tiles about all of them — with a scope selector between the two doing the reconciling. One window control now drives the whole tab, so the funnel and the run statistics below it always cover the same days. The module Overview keeps its window and workflow-scope controls for the KPI tiles, and everything the funnel could already do — the drill-down behind each terminal, the values table, the conservation guarantee — is unchanged. [Try it in KAOP](https://kaop.komodor.com/modules/incidents/workflows) **September 4, 2026** ### Improved: The routing list names its targets, and a pipeline's trigger stays put A channel's routing rules now read as names rather than ids: an agent, workflow or incident pipeline is shown by its own name, falling back to the id only when it cannot be resolved. A channel's status reads as a label ("Paused", "Revoked") instead of a raw value, and `error` and `revoked` are no longer missing from the frontend's own list of statuses. A route that belongs to an incident pipeline is now shown read-only: editing, reordering, disabling, defaulting or deleting it here would have silently broken the pipeline's Slack trigger, so those controls are gone and the pipeline owns the route. And a mention or DM that matches no route gets a short reply saying so, instead of silence. [Try it in KAOP](https://kaop.komodor.com/channels) **September 3, 2026** ### New: The workflow Steps tab now shows what happens before your workflow runs Every incident workflow's Steps tab now opens with a "Before the workflow" group — three read-only cards for Alert trigger, Dedup, and Filtering, summarizing which endpoint feeds the workflow, the dedup-key strategy in force, and the filter conditions and rate limit every alert has to pass before an investigation ever starts. Each card links straight to the wizard step where that setting is edited. A workflow that hasn't been provisioned with a step graph yet used to render a blank page. It now shows this same prelude plus a clear "No steps yet" message for the workflow section, so a pre-backfill workflow's configuration is never invisible. [Try it in KAOP](https://kaop.komodor.com/modules/incidents/workflows) ### New: A run's own alert trigger, dedup and filtering now show on its timeline An incident's run detail page now opens with the same "Before the workflow" group as the workflow's Steps tab — Alert trigger, Dedup, and Filtering — but carrying *this incident's own* facts instead of the workflow's current configuration: which alert arrived, the dedup key it resolved to, and every fire that merged into it. A dedup key derived as `random` now says so plainly: the payload identified nothing, so this incident can never merge with a re-fire of the same alert — a re-fire opens a second incident instead. That used to be invisible on the run itself. A legacy incident with no workflow run — whose step breakdown is empty — used to render nothing at all here. It now shows this same prelude, with no step list beneath it. [Try it in KAOP](https://kaop.komodor.com/modules/incidents/feed) ### Improved: The run timeline's alert prelude now lives inside the Steps card The "Before the workflow" group — Alert trigger, Dedup, Filtering — used to sit above the Steps card as three separate always-expanded cards. It's now the first group inside the same Steps card, in the same collapsible row grammar as a workflow step: an icon in place of a step number, a dashed "Context" chip in place of a status chip, and no duration. A thin group header still marks it as pipeline context rather than a workflow step, with "Workflow steps" labelling the real steps below it. [Try it in KAOP](https://kaop.komodor.com/modules/incidents/feed) ### Improved: Run cost now comes from the gateway that bills it A run's estimated cost is now read from your LLM gateway's own per-model rates — the same table that meters the request — so the figure on a run and the figure the gateway bills you cannot drift apart. Cache reads and cache writes price at each model's own published cache rate rather than one blanket multiplier. This replaces a price list that shipped inside AgentOps and had to be updated by hand. That list had gone stale: Opus runs were estimated roughly 3x too high and Haiku 4.5 runs roughly 4x too low. Because a run's cost is recalculated when you open it, your existing runs show the corrected figure straight away — expect historical Opus costs to drop and Haiku costs to rise. A model your gateway does not serve now shows "Cost unavailable" rather than a number we cannot trace to anything that billed it, and the Cost tile says which model it could not price. Runs that never go through the gateway — an agent you host with your own API key, for instance — are in that group. A cost the worker reported itself is unaffected and still shown as reported. [Try it in KAOP](https://kaop.komodor.com/runs) ### New: See per-cluster pod placement savings, and turn it on right from the table The Pod Placement page now has two tables below the resource allocation chart: Potential Savings lists clusters that could save CPU and memory by turning pod placement on, and Active Savings lists clusters where it's already running. Sort any column, page through the results, and flip the "Pod Placement" switch on a row to enable or disable it for that cluster — turning it off asks for confirmation first. Requires the `k8s_cost.policy.manage` capability to toggle; without it, the switch stays disabled with a tooltip explaining why. [Try it in KAOP](https://kaop.komodor.com/k8s-cost/pod-placement) ### Improved: Pin an agent to any skill version, and preview a rollback before you run it On a skill's edit page, the **Currently attached** panel now lets you pin each agent to **any** published version — not just *latest* or the one it was already pinned to. Pick `v2` for one agent while another tracks `latest`; publishing a new version never moves a pinned agent until you repin it. Rollback now shows you what you're about to restore. Instead of a bare confirmation, **Rollback** opens a review with a **diff against the current version** (toggle to full content) so you can read the exact SKILL.md you'd republish before committing to it. [Try it in KAOP](https://kaop.komodor.com/skills) ### New: Incident workflows can ask you mid-investigation A new incident workflow mode, **Orchestrated loop**, runs three steps — Investigate, Verify, Postmortem — and does the remediation *inside* Investigate rather than as steps of its own. The orchestrator investigates, delegates to your specialists, and pauses to ask you whenever the next decision is yours: which fix to apply, whether to apply one at all. You answer in Attention or on the incident, and the same investigation carries straight on with your answer instead of starting over. It can ask as many times as the work needs, and the time you spend deciding is no longer charged against the step's time budget. Verification is mandatory on this mode, with one exception that matters: if nothing was actually applied — you declined every option, say — there is nothing to verify, so the workflow says so rather than spending a retry rediscovering it. That distinction comes from a record the workflow keeps of every change an agent makes, written before the change is attempted, so "this step succeeded" and "this step fixed the incident" are no longer the same claim. A failed verification re-investigates exactly once, and an incident that ends unresolved still gets written up. [Try it in KAOP](https://kaop.komodor.com/incidents/workflows) ### Improved: An endpoint now shows how its payloads are read — and what a test request maps to Integrations → Endpoints has a **Test & mapping** step. Send a test request (or wait for a real one) and the captured headers and body now come with what that request *becomes*: dedup key, status, title, `SEV-1…SEV-5` severity, summary. The rehearsal runs server-side through the same extraction a real delivery uses, so it cannot disagree with what a live alert would produce, and it opens no incident. Beside it, in a collapsible section, the endpoint's own **field mapping** — the mapping every incident workflow on that endpoint starts from. Each field arrives prefilled with the provider's default path (with a link to that provider's webhook-payload reference), so what you see is what gets stored rather than a suggestion, and switching provider rewrites only the fields you haven't edited. Clearing a field hands that one decision back to the provider. The prefilled path is one that actually resolves, which is not always the path the docs quote: a Grafana payload stores `fingerprint`, not `alerts[].fingerprint`. And when a configured path resolves to nothing, the mapped fields now say so instead of quietly falling back to the default. The create wizard also has a **Back** button now. The endpoint is created when you leave the Authentication step, so going back there saves your changes rather than creating a second endpoint. [Try it in KAOP](https://kaop.komodor.com/integrations?section=webhooks) ### New: See where your alert volume actually goes The Incidents module Overview now leads with an **alerts funnel**: every alert that reached AgentOps, out to what became of it. Filtered out by a workflow's rules, dropped because nothing was listening, rate limited, folded into an incident that already existed, or opened as new work — and for the work that opened, whether it was fixed with no human, needed someone in the loop, was closed by hand, or resolved when the provider recovered. Until now none of that was answerable. An alert a workflow's filters declined, or that arrived at a paused workflow, left no record anywhere — so "how much of our alert noise are we actually filtering out?" had no answer, and neither did "is a workflow quietly paused?". Every arrival is now recorded with the reason it ended where it did. The numbers conserve: each terminal's count sums exactly to the total that arrived, so a percentage read off one node means what it says. Click any terminal to see the alerts behind it, with the filter condition or rate-limit counter that put them there. The funnel shares the Overview's existing window and workflow controls, and the four KPI tiles now sit below it. Two things the funnel is deliberate about: it counts only alerts that **reached** AgentOps, never a share of everything your monitoring fired; and the "Fixed, no human" figure carries its known over-reporting caveat, because a guardrail approval hold is not visible to the engine today. [Try it in KAOP](https://kaop.komodor.com/modules/incidents) ### Improved: Connecting a provider mid-wizard now says where its values come from Deploying an agent from the catalog lets you connect a provider it needs without leaving the wizard — but that inline form asked for Azure's four opaque GUIDs with nothing on the screen saying where any of them are found, while the Integrations page had said so since the start. The wizard's **Connect new** form now opens with the same **Before you start** block: the exact command to run, a button to copy it, a line saying which output value goes in which field, and a link to the provider's setup documentation. As on the Integrations page it comes from the provider, so it appears only where a provider has something to say. [Try it in KAOP](https://kaop.komodor.com/agents/new) **September 2, 2026** ### Fixed: The Steps tab now names a human-selected workflow's real graph A human-selected incident workflow runs six steps — Investigate, Propose remediation, **Select remediation** (the decision an operator has to answer), Apply remediation, Verify, Postmortem. The Steps tab's `GET /incident-pipelines/{id}/steps` endpoint used to describe every workflow as the four-step automatic graph regardless of which one it actually ran, so a human-selected workflow's Steps tab named a "Remediate" step it didn't have and never showed Select remediation at all. The endpoint now reads the workflow's own instantiated definition — the same read the Team & steps graph view already used — so the Steps tab lists exactly the steps a workflow runs, in order, with the agents actually bound to each one. [Try it in KAOP](https://kaop.komodor.com/incidents/workflows) ### Fixed: A human decision now sits where it ran on the run's Steps card On a human-selected incident workflow, **Select remediation** is the step where a person picks what gets applied. On the run drilldown's **Steps** card it was the one step you could not place. It sorted *after* Apply remediation, Verify and Postmortem, with a blank duration — because a decision step dispatches no agent, so nothing ever recorded when it started, and the card orders by start time. An operator reading the page could not tell that the decision had run at all, let alone when, or how long it had waited for an answer. The moment the question goes live is now recorded as the step's start. **Select remediation** sits in execution order between **Propose remediation** and **Apply remediation**, and shows how long it was open — whether it was answered or expired unanswered. Each row is also numbered by its position in the workflow graph rather than its place in the list, so the number matches the one on **Team & steps** even on a run that looped back to Investigate. [Try it in KAOP](https://kaop.komodor.com/modules/incidents) ### Fixed: A run now shows the guardrail that cleaned its input A run whose **only** guardrail activity happened at the very start — a rule redacting something out of the payload before the run was allowed to begin — showed no **Guardrails** tab at all. The decision was recorded and enforced correctly; the run page just could not find it, because that check happens *before* the run exists and so is stamped a fraction of a second earlier than the run's own start time. The tab now appears for those runs, and the row says what it is: **"Before the run started"**, marked *on the payload this run started from*. That is a stronger statement than the page can make about most other decisions — this run began from the cleaned payload, rather than merely overlapping with a rule firing nearby. Nothing changed about enforcement, and nothing changed for runs no guardrail touched — they still have no Guardrails tab. [Try it in KAOP](https://kaop.komodor.com/runs) ### New: Assign a policy and toggle automation right from the Right-Sizing table The Right-Sizing table now has "Policy" and "Automated" columns, pinned to the right so they stay reachable without scrolling. Pick a policy to manually pin a workload to it, or switch it back to "Default" to fall back to whichever policy would otherwise apply. Toggle "Automated" to turn a workload's right-sizing recommendations on or off — turning it off asks for confirmation first. Automating a workload requires a policy to already be assigned; the switch stays off until one is. [Try it in KAOP](https://kaop.komodor.com/k8s-cost/right-sizing) ### Fixed: Resource Requests charts open on the last 7 days, not 14 The Resource Requests chart on the Right-Sizing and Pod Placement pages was opening on a 14-day window. Every other cost page opens on the last 7 days by default — the chart now matches. [Try it in KAOP](https://kaop.komodor.com/k8s-cost/right-sizing) ### Improved: Reconnecting Slack brings your channel and routing back Reconnecting a Slack workspace used to start you over: you got a new channel with default routing, and any rules you had customized stayed behind on the old one. Reconnecting now picks the original channel back up, so your routing rules come back with it. This applies whether you disconnected the integration yourself or the Slack app was uninstalled and reinstalled. If a rule pointed at an agent that has since been deleted, that rule cannot be restored — the channel comes back with default routing instead of routing nowhere. If the app was uninstalled from Slack, reconnecting also simply works again. It previously reported "this workspace is already connected to AgentOps" and left you with no working channel and no way forward. Workspaces disconnected more than 30 days ago reconnect fresh, with default routing. The older channel and its rules are still kept, not deleted. [Try it in KAOP](https://kaop.komodor.com/integrations) ### New: Pod Placement now shows a Resource Allocation Optimization chart The Pod Placement tab's summary cards now sit above a "Resource Allocation Optimization" chart — current vs. optimal CPU and memory capacity across your selected clusters, plus how much of the gap between them is already being saved through automation and how much is still on the table. Switch between CPU and Memory, and between 7/14/30-day windows, the same way the Right-Sizing page's own chart works. [Try it in KAOP](https://kaop.komodor.com/k8s-cost/pod-placement) ### Fixed: Operator-approved incidents keep their root cause in the postmortem On a pipeline where an operator picks the remediation, the write-up at the end of the incident lost everything the investigation had established: the postmortem's root cause and follow-ups read **Not recorded**, even though the investigation step had found both. The moment a person answered the question, the incident's context stopped travelling with the run. Answering a decision now carries that context forward, so an operator-approved incident gets the same postmortem an automatic one does. A verification that fails and sends the run back to investigate also re-enters with the original alert instead of starting blind. [Try it in KAOP](https://kaop.komodor.com/modules/incidents) ### Improved: See what memory is learning, and from how much of the work The Memory page now opens on an Activity tab that answers three questions at a glance: how many of the last seven days' sessions taught an agent something, what became of every memory the worker reviewed, and which memories agents actually read while working. Each chart follows the space picker. Below them, the loop reads as one list instead of a Running and a History tab, with anything still running at the top. A run's digest and its review are one row (memorizing, reviewing, then what the review decided), the running step is highlighted, and the list refreshes itself every minute while you watch it. Search, the memory list and the space bindings moved to their own tabs, and every tab is a shareable link. [Try it in KAOP](https://kaop.komodor.com/memory) ### New: Keep an agent's tool traffic inside your own network An agent you run in the same cluster as an Outpost can now be pointed at that Outpost directly, so its tool calls reach your private MCP servers without leaving your network at all. Pick the Outpost on the agent wizard's **MCP tools** step and the values file it generates points there instead of at AgentOps. Self-hosted agents only — an Outpost's listener is a Service inside your cluster, and a Komodor-cloud agent cannot reach it. **These tool calls are not governed, and that is worth deciding on purpose.** They never reach our gateway, so guardrails, policies and audit do not apply to them, and the Outpost's local listener authenticates no caller: anything in that cluster which can reach it gets every tool in the groups that Outpost serves. A NetworkPolicy is what decides who may, and the chart now refuses to enable the listener until you have either let it create one or told it you manage your own. Guardrails you author at the tool gates say so when you have an Outpost that can serve a colocated agent. Two smaller changes came with it. A group that mixes servers behind the Outpost with servers reached any other way now works from a colocated agent, which previously saw only part of it. And an Outpost running an older configuration than we have rendered for it now shows the upgrade command on its own page, instead of only a badge. [Try it in KAOP](https://kaop.komodor.com/outposts) ### Fixed: Updating a hosted agent now always takes effect Saving a change to a hosted agent — a new prompt, a different image — could be accepted and then quietly never applied, leaving the agent serving its old configuration with nothing reporting a problem. Updates now roll out however many agents your workspace runs, and an agent that ends up unable to authenticate recovers on its own instead of sitting offline. One thing to expect: an update briefly takes the agent offline while it finishes any in-flight runs, rather than handing straight over to the new version. [Try it in KAOP](https://kaop.komodor.com/fleet) ### Improved: The guardrail form asks what a rule does before asking what it matches Writing a guardrail used to mean one screen holding every question at once — the name, where the rule stands, what it does on a match, and then a card per way-this-can-happen with its own tool globs and conditions underneath. **That is now two steps.** *What it does* is four short answers, each a name or a pick from a list. *What it matches* is the open-ended part, on its own. Nothing moved except where it is asked. The rule still reads the way it always did — **on** a gate, **do** an action, **for every call that matches** one or more criteria — and *Add another way this can happen* still adds a second criterion as an either/or. **A new starting point on the empty state: "No payment card numbers reaching a model."** It arrives already pointed at what your agents send to a model, already set to redact rather than refuse, and already carrying the condition — so a rule that keeps card numbers out of your prompts is a name and a save. Two smaller things while we were in here. The operator that picks a class of value now reads **"contains PII/secret"** rather than "contains a", which sat next to plain "contains" and read like a variation on it. And a **Create** button that cannot be pressed yet now says which step is unfinished, instead of leaving you to find it. [Try it in KAOP](https://kaop.komodor.com/guardrails/new) ### Improved: Guardrail activity links straight to the agent and the run The recent-activity list under **Govern → Guardrails** told you a rule had fired and then left you to find the rest yourself. Now the **agent** name is a link to that agent, with its icon beside it, and the **timestamp** is a link to the run the decision was made in. The columns read in the order the question is usually asked: **when · agent · outcome · rule · tool**. The agent moved next to the timestamp because "which of my agents is this happening to" is what groups everything else, and the tool name — the longest value in the row — now ends it. **A timestamp that is not a link means there is no run to open**, not that something is missing. A rule that checks a trigger before a run has started has no run to point at, and neither does a check on the way to a model when the agent's own framework does not tell us which run it was for. [Try it in KAOP](https://kaop.komodor.com/integrations?section=guardrails) ### Fixed: A failed incident step now tells you why it failed Expanding a failed step in an incident's run drilldown showed a red **Failed** badge and, when the step produced no output, nothing else — you had to reach for pod logs to find out what went wrong. The control plane had recorded the cause all along and returned it on every step; the drilldown just never displayed it. A `failed` or `timed_out` step now shows that reason in its expander, above the run link and the output. A step that timed out also reads **Timed out** instead of the raw `timed_out`, in the same red as a failure rather than the grey it shared with a step that hadn't started yet. [Try it in KAOP](https://kaop.komodor.com/modules/incidents) ### Improved: Credential detectors now catch every GitHub, Slack and Anthropic token type The **credential** detectors you can pick in a guardrail condition covered the common form of each credential and missed the rest. They now cover the family: * **GitHub token** — was personal access tokens only. Now also **App installation tokens** (`ghs_`, `ghu_`), OAuth (`gho_`) and refresh (`ghr_`). If your agents authenticate to GitHub through an app rather than a PAT — the usual case — this is the form that was slipping through. * **Slack token** — was bot and user tokens. Now also app-level (`xapp-`), configuration tokens (which mint other tokens), session credentials, and the legacy formats. * **Anthropic API key** — now also **admin keys** (`sk-ant-admin01-`), which administer the whole organisation. * **Google Cloud API key** — now also Google's newer `AQ.Ab8RN6…` format. * **OpenAI API key** — now also the short-segment project and service-account form. **If you already have a rule armed on one of these, it will start catching more.** That is the intended direction: a detector that misses a leaking credential fails at the one job it has. Nothing that matched before stops matching. Our credential patterns now track [betterleaks](https://github.com/betterleaks/betterleaks) rather than gitleaks, which its author has declared feature-complete and which points there itself. We take the patterns as data — never their option to verify a secret by sending it to the vendor, which in a redaction path would leak the exact value we were asked to contain. [Try it in KAOP](https://kaop.komodor.com/guardrails/new) ### Improved: Author a new skill without leaving the agent wizard The Agent instructions step of the Create agent wizard now has a **Create skill** button beside the attach picker. When you have no authored skills yet — or just want a new one — you can author it inline instead of leaving the wizard for the Skills page and losing your draft. It opens the same create-skill dialog the Skills page uses, and the skill you create is attached to the agent's draft at latest the moment you save — so it is on the agent from its first run. [Try it in KAOP](https://kaop.komodor.com/agents/new) **September 1, 2026** ### Fixed: New workflows no longer start with remediation mode locked Starting a brand-new incident workflow and clicking Next past the Name step used to silently create the backend workflow record right there — locking Remediation Mode to "Automatic" on the Team & steps step before you had a chance to choose. The workflow is now created only once you leave Team & steps, when the mode you actually picked is known, so both "Automatic" and "Human-selected" stay selectable until then. [Try it in KAOP](https://kaop.komodor.com/incidents/workflows/new) ### Fixed: A tab left open across a deploy recovers instead of showing "Unexpected error Each release ships its own set of content-hashed JavaScript chunks, so a tab that loaded the app before a deploy still points at chunk URLs the new release no longer serves. Opening any page from that tab was meant to reload quietly and pick up the new build. Instead it flashed a generic **Unexpected error** card reading `Cannot read properties of undefined` for the whole time the reload took, and the card's own button did nothing while it was up. The recovery listener was suppressing the browser's module-loading error rather than letting it reach the code that reloads, which left the page holding a module that had never actually loaded. Navigation from a stale tab now waits on the reload it already triggered, and anything that survives that reload gets a **Stale UI after a deploy** card with a working **Hard reload**. ### Fixed: Setting up Slack MCP now asks for the access it actually needs Slack's hosted MCP server searches and reads **as you**, not as the AgentOps bot — it accepts your personal Slack token and refuses the bot's. Connecting Slack asked for that separately, through a checkbox worded *"Also connect my user account"*, left unticked by default. So the requirement was met by accident or not at all. When it was not, nothing said so until tool discovery, which failed with *"the provider rejected the stored credentials… add or replace the connection with a valid key"* — advice that pointed at a credential that was working perfectly everywhere else Slack was used. Three things changed: * **The MCP setup wizard states the requirement** instead of offering it, and requests your access as part of the step. You still approve exactly what it grants — on Slack's own authorization screen, which is where that decision was always really made. * **A connection that cannot drive the endpoint is named before you bind it.** One authorized for the bot alone shows as **Bot only** in the connection list and picker, with a one-click reconnect that adds your access; registering a server against it is blocked rather than left to fail later. * **The error tells the truth.** A refusal of this kind now reads "this connection was authorized for the bot only… reconnect the integration and include your user access" rather than blaming the credential. Connecting Slack from the Integrations page for channels, notifications or messaging is unchanged — the bot token is all any of those use, and your personal access is still opt-in there. [Try it in KAOP](https://kaop.komodor.com/integrations) ### Fixed: The Right-Sizing table shows every row, and the savings chart's colors are fixed The Right-Sizing workload table was capped at a handful of visible rows with its own scrollbar, even when you asked to see more per page. It now shows every row you've asked for, and only the page scrolls. Row-size choices are also back to the standard 10/25/50 you'd expect. The Resource Requests chart's "Active Savings" and "Additional Potential Savings" bands had their colors swapped, so the chart under-represented how much was actually already saved — fixed. [Try it in KAOP](https://kaop.komodor.com/k8s-cost/right-sizing) ### New: Filter Right-Sizing recommendations by when a workload was last seen The Right-Sizing table now has a "Last seen" filter, so you can focus on workloads that are actually still running instead of ones a cluster hasn't reported on in weeks. Pick a window from the last 2 to 30 days and the table narrows to workloads seen at least once in that range. [Try it in KAOP](https://kaop.komodor.com/k8s-cost/right-sizing) ### Improved: Incident workflow steps get more time before they time out — and it's now yours to set Every orchestrated step of an incident workflow — Investigate, Verify, and the remediation step(s) — shared one hardcoded 300-second ceiling. Real investigations routinely used most of that budget, and some remediation steps timed out on their first attempt, only completing on a retry. Whether a step landed depended on retry luck more than on the work being tractable. The default is now **900 seconds**, based on measured durations from real incidents. It's also configurable per workflow: set **Orchestrated step time budget** in the wizard's Team & steps step (or leave it blank for the new default), and — unlike Remediation mode — it stays editable after the workflow is created, so a ceiling that turns out too tight doesn't require rebuilding the workflow. [Try it in KAOP](https://kaop.komodor.com/incidents/workflows) ### New: Turn any OpenAPI document into agent tools Add an integration and you can now point AgentOps at an OpenAPI document instead of registering an MCP server. Paste its URL or upload the file, pick the operations you want, and each one becomes a tool your agents can call, with the same groups, policies and credentials every other gateway server uses. Operations you leave off are not exposed. This was previously limited to accounts it had been enabled for. [Try it in KAOP](https://kaop.komodor.com/integrations) ### New: Guardrail conditions can name what to look for, instead of a pattern A condition can now say **"contains PII/secret"** and pick from a list — a payment card number, a US Social Security number, an IBAN, an email address, or one of eight credential types (AWS, Google Cloud, GitHub, Slack, Anthropic, OpenAI, JWT, PEM private key). No regular expression to write, and none to get subtly wrong. **The reason to use it instead of a pattern is the checksum, not the typing.** A regex for sixteen digits also matches order ids, invoice numbers and truncated timestamps — and a rule that refuses legitimate work all day is a rule you end up switching off. `Payment card number` verifies the card's own check digit, so `1234-5678-9012-3456` is not one and `4111-1111-1111-1111` is. It works **everywhere a condition works**: on a tool call's argument, on what your agent sends to a model (messages or system prompt), on what a tool returned, on what a model answered, and on a trigger payload before a run exists. Pair it with **"Redact the value"** to clean the value out and let the work continue, or with **"Refuse the call"** to stop it. **Two things worth knowing before you test one.** `US Social Security number` matches the separated form — `456-78-9123` — and deliberately not a bare `456789123`, which is indistinguishable from an order number. It also does not match `123-45-6789` or `078-05-1120`: those are published example SSNs, skipped so that the fixtures and tutorials in your own repositories do not trip the rule. Anything not on the list is still a pattern condition you write and own. Names, addresses and phone numbers have no reliable pattern — state those as a **policy for a model to judge** instead. [Try it in KAOP](https://kaop.komodor.com/guardrails/new) ### Fixed: Editing an incident workflow now shows the graph it actually runs An incident workflow's **Remediation mode** decides which step graph it runs — `Automatic` goes Investigate → Remediate → Verify → Postmortem, while `Human-selected` splits the middle into Propose remediation → Select remediation → Apply remediation so an operator approves the option before anything is applied. Opening an existing workflow for edit showed **Automatic** for all of them. The mode is fixed when the workflow is created and the graph is built from it, so there was nothing wrong with the workflow itself — but the wizard could not read the mode back, so it fell back to the default and drew the four automatic steps. A human-selected workflow was described as automatic, and its human decision step was invisible on the very screen you would go to in order to check for it. The mode is now part of what the API reports for a workflow, so **Team & steps** shows the mode the workflow was created with — still locked, because it cannot be changed after create — above the step graph that mode really runs. [Try it in KAOP](https://kaop.komodor.com/incidents/workflows) ### Fixed: Disconnecting a Slack integration now releases the workspace Disconnecting a Slack integration deleted the connection but left its channel behind — still listed as active, still holding the workspace it no longer had a credential for. Nothing said so, and the leftover row was not obviously anyone's. That row is what a later connection attempt collided with, so reconnecting the same workspace, from this account or another one, came back as "already connected to a different AgentOps account" with nothing visible to disconnect. Disconnecting now revokes the channel and releases the workspace, so the next connect works. The channel is kept rather than deleted, so nothing about your setup is thrown away when you disconnect. [Try it in KAOP](https://kaop.komodor.com/integrations) ### Fixed: Deleting an incident workflow no longer erases its incidents without asking Deleting an incident workflow also deletes every incident it produced and the agent runs recorded against them — and it used to do that silently. Now the delete is refused until you confirm: the workflow you are removing tells you exactly how many incidents and incident runs would go with it, and the delete only proceeds once you tick the box that says so. A workflow that has no incidents still deletes in one click, and pausing a workflow remains the reversible way to stop it without touching the history it already recorded. [Try it in KAOP](https://kaop.komodor.com/modules/incidents/workflows) ### Improved: The connect form now says where its values come from Connecting Azure meant filling four fields that all want an opaque GUID, plus a secret, with nothing on the screen saying where any of them are found. The one `az` command that emits every one of them was in the docs, and the link to those docs was not rendered anywhere. The connect dialog now opens with a **Before you start** block: the exact command to run, a button to copy it, a line saying which output value goes in which field, and a link to the provider's setup documentation. The mapping matters more than it sounds — Azure's command prints three of its four values, and the fourth is an argument you passed in, so it appears in no output at all. Fold the block away once you have read it. For Azure the block also states what the access it asks for actually grants — Reader can read every resource's configuration and tags, your full network topology, all Log Analytics data, and Key Vault secret names, but never secret values, and it cannot write or run anything. That is the part worth reading before you approve a service principal, and it was previously only discoverable by reading Microsoft's role definitions. The block comes from the provider, not from the form, so it appears only where a provider has something to say. A token you copy off a settings page still gets the form it had before. [Try it in KAOP](https://kaop.komodor.com/settings/integrations) ### New: Connect Dynatrace from the Integrations page Dynatrace is now in the integrations catalog, so problems, DQL queries, logs, metrics and Davis AI reach your agents with no custom setup. Paste a platform token (`dt0s16.…`) and your environment URL — exactly `https://.apps.dynatrace.com`, not the classic `.live.` address — and AgentOps derives the hosted MCP endpoint for that environment itself. The classic API token (`dt0c01.…`) is an optional second field, only needed for the classic environment API v2. Agents that run their own Dynatrace MCP server instead receive the credential as `DT_PLATFORM_TOKEN`, `DT_API_TOKEN` and `DT_ENVIRONMENT`. Dynatrace SaaS only — Managed serves no hosted MCP endpoint. [Try it in KAOP](https://kaop.komodor.com/integrations) ### Fixed: A workflow that ran out of retries is no longer reported as errored An incident workflow can stop because a step genuinely failed, or because it ran every step to completion and simply had nowhere left to route — an exhausted retry cap, or an output shape no transition matched. Both used to show up on the incident as **Errored**, with a message telling you to retry a run whose retries were exactly what ran out. The two are now told apart. A step that failed still reads **Errored**, with a Retry button — retrying re-attempts it and might succeed. A run that ran out of retries now reads **Needs review**: no Retry button (retrying would only repeat finished work), and copy that says there is nothing left to retry — read the findings and resolve it manually, or update the workflow's template. [Try it in KAOP](https://kaop.komodor.com/incidents) **August 31, 2026** ### Fixed: Editing a self-hosted agent no longer breaks its Helm upgrade Upgrading a self-hosted agent after editing it could fail while Helm rendered the chart (`nil pointer evaluating interface {}.enabled`), if the release had been installed from an older chart version. The chart now tolerates the values of any release it supports, so the command the Edit wizard gives you works again — no change needed on your side beyond pulling the current chart. Your worker token and LLM key are carried forward as before. [Try it in KAOP](https://kaop.komodor.com/agents) ### New: Redact a value out of a tool call, instead of refusing the call **"Redact the value"** now works at **Tool call** — the first of the two outbound gates, and the first place a rule changes what your agent *sends* rather than what it reads. The tool runs, with every value your conditions match replaced by `[redacted]` in its arguments. This is the one to reach for when a credential, a card number or an internal hostname keeps ending up in a ticket body, a search query or a webhook payload. Refusing those calls stops the work; cleaning them lets the work happen without the value leaving. **One difference from the inbound gates, and it is worth knowing before you pick it.** When we redact something an agent *reads*, it sees `[redacted]` and can reason about it. When we redact something it *sends*, it is not told — so it goes on to interpret whatever the tool made of a value it never sent. That is the right trade when keeping the value out of the tool matters most, and the wrong one when the agent has to act on the answer. **"Refuse the call"** tells it plainly; this does not. **A redacting tool-call rule needs a condition naming the argument.** Tool patterns on their own describe an *action* rather than a value, so there would be nothing to remove — that rule is a "Refuse the call", and the form says so instead of saving something that could only ever refuse. The rules you already know are unchanged. All-or-nothing: if a matched value cannot be removed, the call is refused instead. Two redacting rules both apply. A rule that refuses still beats both — and so does one that holds for approval, so nobody is ever asked to approve a call whose arguments we quietly rewrote first. That leaves **Model request** as the last gate without it. [Try it in KAOP](https://kaop.komodor.com/guardrails/new) ### New: Redact a value out of the prompt — the last gate gets it **"Redact the value"** now works at **Model request**, which was the last of the five gates without it. The request reaches the model with every value your conditions match replaced by `[redacted]` — in the messages, in the system prompt, or both, depending on what your conditions point at. This is the gate that can promise a value never went to a model provider at all. If a card number, a customer identifier or an internal hostname keeps arriving in a prompt, refusing those requests stops the agent working; cleaning them lets it work without the value leaving your account. **The outbound trade applies here too, and it bites differently.** When we redact something an agent *reads*, it sees `[redacted]` and can reason about it. When we redact something it *sends*, it is not told. At **Tool call** that means a tool acting on a value the agent did not send — the tool either works or it errors. At **Model request** it means a model answering a question with a hole in it, and a model will answer confidently either way. The hole shapes the answer and nothing in the exchange says so. Reach for **"Refuse the call"** when the agent has to act on what comes back. **Redacting the system prompt is the sharper version of that.** The system prompt is *your* text, not the conversation's — so a rule pointed at it removes part of what the agent was told to do, rather than part of what it was told about. Worth pointing at `messages` unless you mean the instructions. Everything else is as it is at the other four gates. All-or-nothing: if a matched value cannot be removed, the request is refused rather than sent half-cleaned. Two redacting rules both apply, and each gets its own line in the audit trail. A rule that refuses beats both, and so does one that holds for approval — nobody is asked to approve a request whose prompt we quietly rewrote first. `redact` now stands at all five gates. [Try it in KAOP](https://kaop.komodor.com/guardrails/new) ### Fixed: Reconnect now repairs an integration instead of failing When an integration's key or token stopped working, the **Reconnect** button on it could not actually repair it: submitting new credentials came back with "an integration named X already exists", blaming the connection's name for what was a credential problem. The only thing that did work was disconnecting and setting the connection up again from scratch. Reconnect now replaces the credential on the connection you are already looking at. The name, and anything bound to it, stay as they were — so agents using that integration keep working, rather than quietly losing access the way they did when the connection had to be rebuilt. A key the provider rejects is refused before it is saved, which means a failed attempt leaves the old credential in place instead of replacing a working one with a broken one. This covers every integration you connect with a key or token, including AWS, Datadog and Grafana Cloud — useful whenever a secret expires or you rotate one on a schedule. [Try it in KAOP](https://kaop.komodor.com/settings/integrations) ### Fixed: A Slack workspace connects once, and reconnecting repairs it Connecting a Slack workspace your account already had produced a second connection rather than an error. Both were marked active and both held a live bot token, so replies could go out through one while messages came in on the other — and the second attempt failed outright after the damage was done, leaving behind a channel that could never receive an event. An account now holds one connection per Slack workspace. Re-running the connect flow for a workspace that is already there does the useful thing instead: if its credential has been rejected — the app was removed from Slack, the token was revoked — the fresh authorization replaces the dead one in place, and the connection starts working again without you having to find it first. If it is healthy, the attempt is refused and says which workspace it already belongs to. A provider that cannot be reached counts as neither, so a network blip can never be read as a broken credential. Connections are also named after the workspace itself now. A row reads "Acme Corp" rather than "Slack 2", so which is which no longer depends on remembering the order they were added in — though a name you set yourself is left alone. The same guard covers re-installing a GitHub App into an organization it is already installed in, which had the same duplicate behaviour. [Try it in KAOP](https://kaop.komodor.com/settings/integrations) ### Improved: Search and per-integration grouping in the integration-group picker The **Add group** dialog for integration groups got three improvements: * The member-server checklist now has a search box, so picking a few servers out of a long list no longer means scrolling to find them. * The Allow/Deny tool list now scrolls correctly with a mouse wheel or trackpad — it previously only responded to keyboard navigation. * Tools in the Allow/Deny list are now grouped under the integration they belong to, each with its own "select all" checkbox, instead of one flat alphabetical list with no indication of which server offered which tool. * Each selected rule now names the integration it matches, and hovering it shows how many tools it covers on each one. A wildcard rule like `*delete*` reports its real reach across every member instead of reading as an unknown tool. [Try it in KAOP](https://kaop.komodor.com/integrations?section=groups) ### Fixed: Incidents stop toasting "Not found Opening the incidents inbox or an incident no longer raises a "Request failed / Not found" toast every 15 seconds. The page polls an inbox that is only present once module workflows are enabled for your account; where they are not, the affordances that need it are simply absent instead of erroring. A genuine failure still surfaces as before. Retry on an incident whose workflow engine is switched off now says so and leaves the incident where it was, instead of reporting success and quietly waiting on a step nothing would run. [Try it in KAOP](https://kaop.komodor.com/incidents) ### Fixed: Incidents and runs get their names An incident whose alert carried no title of its own now gets a short readable name instead of "Untitled alert". A brief LLM summary of the alert payload is written over the placeholder in the background, as `{workflow} — {five words}`, so the inbox is scannable at a glance. Standalone agent runs get the same treatment on Fleet and History. This was built to work that way and never did: the model it calls was configured in no environment, so every attempt failed closed and quietly kept the placeholder. It now runs where the incident webhook and the run pipeline actually execute, and a failure is logged loudly instead of silently. An alert that **does** name itself keeps that name. The summary only replaces the placeholder, never a title your monitor already wrote — a Datadog or AlertManager alert reads exactly as it did before, and costs no model call. On the incident page, **Orchestrator run** and **Specialist runs** now show the agent's name — Klaudia Investigator, rather than `run_60b72c82c53865e17da3244b`. The run id is still there on hover, and still the link. [Try it in KAOP](https://kaop.komodor.com/incidents) ### New: See which notification sinks fired for each incident Incident workflows and individual incidents now show a **Notifications** strip that tells you exactly which sinks were notified — and what happened when they were. On the **workflow overview tab**, the strip lists every notification sink subscribed to that pipeline. On an **incident detail page**, it shows which sinks fired for that specific incident, the event type (Triaged, Resolved, Errored), each delivery's outcome and timestamp, and the error body when a delivery failed or was retried. Sinks that were skipped — because the sink was disabled or the incident event type was not in the sink's filter — appear with a dash and a tooltip explaining why nothing was sent. [Try it in KAOP](https://kaop.komodor.com/incidents) ### Fixed — "Hold for approval" on a model request let the request through **"Hold it for approval"** could be set on a **Model request** rule through the API or IaC, and when it fired, the request went to the model anyway. Nothing said so. The rule appeared armed, its decision was recorded, and the traffic it was written to gate passed. The cause: an approval works because the boundary comes back to ask a second time. At **Tool call** it does — the call is refused naming an approval, and the agent's retry finds it granted. The model-request boundary has no second ask; it refuses a request outright or sends it. So there was never anything to resolve a hold there, and the proxy — which checked "is this a refusal" rather than "is this something I know how to carry out" — treated the unfamiliar verdict as permission. **Both halves are fixed.** The model boundary now refuses anything it does not recognise, rather than passing it, so a verdict from a newer control plane can never again read as approval. And `hold` is no longer accepted on a model-request rule at all — the API answers 422, so nobody arms an approval that cannot be answered. **If you had such a rule**, it has been rewritten to **"Refuse the call"** — which is what it has been doing since the boundary fix, and what its author most likely wanted. Nothing was deleted. If you wanted the value kept out of the prompt rather than the request stopped, **"Redact the value"** now works at that gate. "Hold it for approval" is unchanged everywhere else, and still the right tool at **Tool call** inside a workflow step. [Try it in KAOP](https://kaop.komodor.com/guardrails) ### Improved: Setting up a provider's MCP server walks one step at a time Connecting a provider's hosted MCP server used to show all of its steps at once, with the ones you could not act on yet greyed out. It now walks through **Connect account**, **Expose tools** and **Route** — the same shape as adding your own MCP server or importing an OpenAPI document. The tool list also reads what a server actually publishes. A server set to publish nothing, or everything except a named tool, used to show every tool ticked — and ticking anything would quietly republish the lot. Those are now shown as what they are, and a selection written as patterns is left alone rather than offered as checkboxes that cannot express it. **Publish every tool** is a switch on that step now. Left on, a tool the provider adds to its server later is published too; turn it off to pick from the list. [Try it in KAOP](https://kaop.komodor.com/integrations) ### Improved: An agent-start rule now refuses a webhook at the door A rule at **Agent start** is the only one that can say a value was never kept, rather than that we did not pass it on. That was true of the run, and not quite true of everything around it: an incident pipeline wrote the alert body, a review workflow wrote four fields off the pull request, a workflow webhook queued the whole payload and answered **202** before any rule was asked, and an endpoint capturing samples stored the body whether or not anything ran. The rule then refused the run, next to a copy of what it refused. Those four are now asked at the edge, on the body as it arrived. A refusal answers the sender **403** naming the rule, and nothing derived from the payload is written — no incident, no queued trigger, no sample. A redacting rule cleans the sample as well as the run, so the two no longer disagree about what arrived. A body that is not JSON is read too, since otherwise the way past every condition was to send one. **One thing to know when you write the rule: leave it untargeted if the point is storage.** A rule that names agents cannot be applied at the edge, because at the edge we do not yet know which agent will run — an incident fans out per pipeline and a review per bound reviewer, and picking one would refuse the others' traffic. Those rules are enforced a step later, when the run is created, which is after the ingress row exists. An untargeted rule has no such ambiguity and is asked at both. [Try it in KAOP](https://kaop.komodor.com/guardrails) ### Fixed: The K8s Cost Overview page's Cost Trend chart now shows real dates Every bar on the Cost Trend chart was labeled with the same date, making the 30-day trend unreadable. The chart now shows the correct date under each day's spend. [Try it in KAOP](https://kaop.komodor.com/k8s-cost/overview) ### New: Connect an Azure subscription Azure now sits in the connection catalog beside AWS: connect a subscription with a Microsoft Entra service principal, and agents you bind it to can read that subscription — Resource Graph, Monitor metrics, Log Analytics, and AKS cluster metadata. The three IDs are checked before anything is stored, so the two mistakes that used to be invisible are caught in the form rather than becoming a connection you would have to delete and rebuild: the Object ID pasted where the Application (client) ID belongs, and the tenant ID pasted into the Subscription field. Pressing **Test** names which of the four values to fix, and an expired client secret says so outright. Setup and what a subscription-scoped Reader role exposes are in the Azure integration reference. [Try it in KAOP](https://kaop.komodor.com/settings/integrations) ### Improved: Six kinds of config now take a name that is unique in your account Review workflows, notification sinks, eval flows, golden scenarios, golden suites and triggers each had a name that was free to repeat. Nothing needed one to be a key until something tried to apply the same configuration twice and could not tell which row it meant. Names in each of those six are now unique per account. Existing duplicates were renamed once, in place: the oldest row of each set keeps the name it had, and every other gets a numbered suffix — `Nightly`, `Nightly (2)`, `Nightly (3)` — skipping any suffix already in use. Nothing is addressed by id, so no link or webhook URL moved. A create or rename onto a name already taken now comes back as a plain conflict naming the name, instead of an internal error. One place *does* resolve a trigger by name: an incident pipeline's `triggerRef`. Where an account had two triggers of one type sharing a name, such a ref matched whichever was created most recently; after the rename the surviving `Nightly` is the **oldest** one, so the ref settles on that row instead. Only an account that actually had duplicate trigger names is affected, and the ref is unambiguous from here on. **Triggers are the one with a wrinkle worth reading.** The name is unique per account *per trigger type*, so a schedule and a webhook may both be called `nightly`. Triggers your worker declares in its `agent-spec.yaml` are outside the rule entirely — the heartbeat owns those names and may repeat them — and a trigger with no name at all is unaffected. **One behaviour genuinely changes.** A trigger name declared in an agent's manifest at create time is stored as if you had made it yourself, so it *is* covered: two agents in one account can no longer each declare a trigger named `nightly`. The second agent is still created; its trigger is reported as failed with the conflict named, and it succeeds once one of the two is renamed. Both trigger lists also take `include_agent_spec=false` now, which narrows them to exactly the rows the new rule covers. The default is unchanged, so nothing you read today looks different. ### Improved: A catalog agent can state its scope before you name it Some agents reach less far than their name suggests, and the wizard used to let you find that out after the install. A catalog agent can now carry a **Notice** on the first step of the Add agent wizard — the scope stated where you pick the name, with a checkbox you tick before **Next** will move. The first one is **Kubernetes RCA**: a read-only investigator for the single cluster it is deployed in, with Klaudia the recommendation for multi-cluster RCA, and a suggestion to postfix the agent name with the cluster name — because you deploy one per cluster, and the name is how you tell their runs apart. **Only an agent that needs one gets one.** A notice on every card would train everyone to tick without reading, which would cost the agent that genuinely needs it its only defence. Every other catalog agent's first step is unchanged. The co-pilot cannot tick it for you either. Ask it to fill the wizard and it quotes the notice back and waits for your answer — the acknowledgement is yours to give. [Try it in KAOP](https://kaop.komodor.com/agents/new) ### New: Get notified when agents are created, deployed, or go unreachable Notification sinks now support **agent lifecycle events** — six new event types you can subscribe to from the sink wizard: * `agent.created` — fires when a new agent is registered in your account * `agent.updated` — fires when an agent's configuration changes * `agent.archived` — fires when an agent is archived * `deployment.succeeded` / `deployment.failed` — fires when a hosted-agent deployment completes or fails * `agent.unreachable` — fires when an agent goes offline `agent.created` is **account-wide**: because the agent doesn't exist yet when you configure the sink, the subscription step automatically adds an account-wide agent subscription and explains why it can't be scoped to a specific agent. All other agent lifecycle events can be scoped to specific agents or set to receive events from all agents. [Try it in KAOP](https://kaop.komodor.com/integrations?section=notification-sinks) ### Improved: Add an integration from where you noticed you needed one Two places used to stop with nothing you could act on when a workspace had no integrations yet. Building an agent, the **MCP tools** step offered to create an integration group — and the dialog it opened had nothing to put in one. It now asks for the integration first, because a group is assembled out of integrations, and only asks about the group once there is something to group. The **Add integration group** dialog says the same thing in its member list, and offers the way out rather than the sentence "Register a server first." Both open the integration catalog over the page you were on. The agent you were part-way through building is still there when you come back — and if the flow you picked routed what it connected into a group on its way out, that group is already selected. [Try it in KAOP](https://kaop.komodor.com/integrations) **August 30, 2026** ### Improved: A detached credential goes back, and two more fields can be declared An MCP server's credential is now put back when it is detached outside your manifests. It was the one binding the operator pushed only when the manifest itself changed, so detaching it here left the resource declared and diverged at once, under a green condition. The next resync re-binds it, with no edit to the custom resource — and a server already bound to what it declares is still never re-written, so a cluster where nothing changed still costs no writes. An incident workflow can now name its endpoint by name. `spec.triggerRef` takes the endpoint's name where `spec.triggerId` took an opaque `trg_…` id: readable in review, still right after the endpoint is recreated, and portable to a second workspace. The id form keeps working for an endpoint nobody named. And an MCP server on an OAuth client-credentials grant can now declare `spec.oauthClientSecretRef`. That was refused before, because AgentOps served no way to replace a stored client secret — so a manifest could set it once and never again. Rotating the Kubernetes Secret now reaches the server. [Try it in KAOP](https://kaop.komodor.com/settings/configuration-clusters) ### Fixed: Scheduled runs now reach the shadow arm An agent whose real traffic is a schedule — a nightly report, a recurring sweep — now produces comparison pairs while a shadow is live, the same as every other way a run starts. Each cron fire creates the production run and its shadow together, linked by one comparison id. Before, a cron-fired run was the one path that never paired. The experiment looked completely healthy from the outside — the candidate read online, Fleet showed the production and shadow split — and the candidate claimed nothing, forever, with nothing anywhere explaining the empty comparison. Two limits are worth knowing. Scheduled runs pair from the database rather than the application, so the fleet-wide fan-out kill switch does not reach them: turning it off halts pairing everywhere else and leaves schedules pairing. And a paired fire counts twice toward an agent's queue backlog, which can scale the production deployment up for work only the shadow may claim. [Try it in KAOP](https://kaop.komodor.com/agents?tab=schedules) ### New: Remove a configuration cluster from its own row A cluster you added to try the config-plane operator can now be deleted from the Configuration clusters list. Until now the row offered only Install and Pause/Resume, so removing one meant calling the API by hand. The confirmation names what deleting actually costs, because two thirds of it are easy to guess wrong. The resources the cluster declares are **not** deleted — they stop being managed and become editable here, and the dialog says how many. Its credential is revoked, so an operator still running in that cluster stops reconciling rather than carrying on quietly. And unlike a pause, nothing reverses it: the ownership rows are released in the same transaction as the delete, and the released count survives only in the audit record. Rotating a cluster's credential is still not a row action. It invalidates the credential the running operator is using, and nothing in this view can carry the new one to the cluster. [Try it in KAOP](https://kaop.komodor.com/settings/configuration-clusters) ### New: Redact what the model answered, instead of withholding the whole reply **"Redact the value"** now works at **Model response**, the last of the three inbound gates. Pick it and the agent reads the model's answer with every value your conditions match replaced by `[redacted]`, instead of an error where the answer should have been. At this gate the difference matters more than anywhere else, because blocking here is the most expensive refusal we have: the completion already exists and is already billed, so withholding it spends the tokens and delivers nothing. Redacting spends the same tokens and delivers the answer, minus the part your rule was written about. A long analysis that mentions one internal hostname is still a useful analysis without it. **The rules are the ones you already know.** All-or-nothing: if a matched value cannot be removed, the answer is withheld instead and the decision record says the redaction is what degraded. Two redacting rules both apply. A rule that withholds still beats both. And it takes conditions rather than AI judgment, because removing a value means knowing where it is. Streaming is unchanged — a streamed answer is still refused rather than inspected, because there is no whole completion to read. Redaction does not change that: a rule that cannot see the text cannot clean it. That leaves the two **outbound** gates — the tool call and the model request. They are a different problem rather than a smaller one: those change what your agent *asked for*, so they owe it a way to know its own request was rewritten before we can offer this there honestly. [Try it in KAOP](https://kaop.komodor.com/guardrails/new) ### New: Redact what a tool returns, instead of withholding the whole result **"Redact the value"** now works at **Tool response** as well as Agent start. Pick it as the action and the agent reads the tool's answer with every value your conditions match replaced by `[redacted]`, instead of getting an error where the answer should have been. This is the softer sibling of blocking at this gate, and at this gate the difference is worth a lot. The tool has already run either way — its effects stand — so refusing the result buys nothing except the agent not seeing the content. Redacting keeps the part of the answer the agent actually needed. A file listing that happens to contain one private key is still a useful file listing without it. It reaches **structured content too**, not only text blocks. A tool answering in JSON is not a way around a rule about what your agents may read, and you should not have to know which of your tools do that. **The rules that make it safe are the same ones you already know.** It is all-or-nothing: if a matched value cannot be removed — including when removing it would break the result's own JSON — the result is withheld instead, and the decision record says the redaction is what degraded. Two redacting rules both apply rather than competing. A rule that withholds still beats both. And it takes conditions rather than AI judgment, because removing a value means knowing where it is. Still to come at the remaining three gates — the tool call, the model request, and the model answer. [Try it in KAOP](https://kaop.komodor.com/guardrails/new) ### New: Redact a value instead of refusing the whole run A guardrail at **Agent start** can now do something other than let the payload through or refuse it. Pick **"Redact the value"** as the action and every value your conditions match is replaced with `[redacted]` — then the run starts, on the cleaned payload. Dropping an entire order because one field held a card number is worse for the customer than removing the field, and this is the action that says so. Because the rewrite happens before the run's row is written, the original value **never enters your storage**. This is the same property that makes Agent start the only gate that can promise it, and the reason redaction is offered here first: the other four gates stand on traffic passing through us, so the most they could offer is removing a value from what we forward. **It is all-or-nothing, and it fails towards refusing.** If a value your rule matched cannot be removed — because the excision would break the payload's own structure — the run is refused instead of started. A value we found and could not remove is never let through dressed as a success. So a redacting rule can still stop a run, and the decision record says that is what happened. **Two redacting rules both apply.** They are not in competition: two rules removing two different values is not a contest between them, so what comes out is everything either of them found. A rule that refuses still beats both. Redaction takes **conditions, not AI judgment**. Removing a value means knowing *where* it is, and a judged policy answers whether something was violated, not where — so the wizard offers the action only on rules built from conditions, and says so rather than quietly doing something else. It is offered only at the gates that can carry out a rewrite; pick another and the wizard explains why instead of the option disappearing. Unlike **"Allow it, and record it"**, redaction is visible to the agent: what it reads has already been changed, so hiding that would leave it reasoning about a question it did not ask. It is told which rule acted and how much came out — never what the removed value held. [Try it in KAOP](https://kaop.komodor.com/guardrails/new) ### New: Reach an OpenAPI API that only your own network can resolve An OpenAPI integration used to be refused the moment the API it calls was not reachable over public HTTPS, which left every internal service out. The Connect step of the import wizard now offers an **Outpost**: pick the relay already running in the network the API lives in, and the integration is reached through it. With an Outpost picked, the **API base URL** stops being optional and may be a private or plaintext `http` address, because the relay is the only thing that has to reach it. The document is unchanged in that respect: AgentOps still fetches it over public HTTPS, or you upload it. Every call the generated tools make then travels that relay's tunnel instead of leaving the gateway's own network, and the rule the relay is given is derived from the base URL you named, not from wherever the document is served. Binding one re-renders that Outpost's configuration, so it reads **Upgrade pending** until its operator re-applies the install values, or **Config unknown** if the relay is not running yet. Clearing the Outpost puts the API back on the gateway's own egress, so the base URL has to become a public HTTPS address, or be blanked, in the same edit. The wizard says so before it lets you save. [Try it in KAOP](https://kaop.komodor.com/integrations/openapi/new) ### Improved: One bring-your-own tile instead of two **Add integration** used to show a separate tile for registering your own MCP server and for importing an OpenAPI document, reading like two different products rather than two ways into the same one. They're now a single **Custom integration** tile — its **Add** button opens a small chooser naming both paths, then hands off to the exact same wizard as before. [Try it in KAOP](https://kaop.komodor.com/integrations) ### Improved: Connecting an MCP server is a guided setup now Registering your own MCP server used to be one long form: thirteen fields at once, including two boxes asking for glob patterns like `get_*` to decide which tools the gateway would publish. It now walks through **Connect**, **Configure** and **Review**, the same shape as importing an OpenAPI document. Configure lists the tools the server actually advertises, so you tick the ones you want instead of describing them. A tool you leave out has no tool at all — an agent never sees it. The pattern boxes are still there for the cases that need them, and for a server that cannot be read until it exists. Review now asks which **group** the server joins. A group is the endpoint an agent connects to, so a server in no group reaches nobody — previously you had to know that and go to the Groups tab afterwards. It names what the group already reaches and how many agents gain the tools you just exposed. For a server that signs in through the gateway, the tools stay unreadable until it is saved and authorized. That step now happens in place: finish the sign-in and the wizard offers the tool list rather than closing. **Static headers** are name and value rows on both this wizard and the OpenAPI one, instead of a JSON object you had to get right by hand. [Try it in KAOP](https://kaop.komodor.com/integrations) ### Fixed the MCP group dialog stretching past the screen Adding or editing an integration group with a lot of MCP servers used to stretch the whole dialog downward, pushing Name, Allow/Deny, and Create hundreds of pixels off screen. The member checklist now scrolls within its own bounded box, so the rest of the dialog stays in view no matter how many servers there are. [Try it in KAOP](https://kaop.komodor.com/integrations) ### Fixed cost reporting for Kimi and other non-Anthropic models Runs on Kimi (and any other model reached through litellm rather than Anthropic directly) were showing roughly 8x their real cost. The run cost calculator was trusting a number the Claude Agent SDK invents for models it doesn't recognize, instead of pricing them from real token counts at their actual published rate. Cost and spend numbers for these runs are now accurate going forward. [Try it in KAOP](https://kaop.komodor.com/insights) ### Fixed: Identical alerts now merge into one incident A plain webhook that sends the same alert twice now updates one incident — refire count, timeline entry, and a `merged_alerts` list of every fire that landed on it — instead of opening a second incident. Providers that name their own alert cycle (Datadog, Grafana, AlertManager, PagerDuty, OpsGenie) always merged correctly. A `generic` endpoint that sent no `dedup_key` or `fingerprint` did not: the ingest gave each fire a random key, and a random key can never match, so every POST opened a fresh incident with a refire count of zero. An alert that put its name at the top level rather than under `labels` hit this even though it looked perfectly ordinary. Such a payload now gets a key derived from what the alert is and what it is about — `alertname`, `service`, `environment`, `namespace`, `host` and their siblings — deliberately ignoring anything that changes between fires, so a repeat hashes to the same key. Sending your own `dedup_key` or `fingerprint` still wins outright. If a payload carries nothing identifying at all, every fire still opens its own incident. That is now stated rather than silent: the incident reads `dedup_key_source: random` and opens with a **Dedupe key unavailable** timeline entry naming the fix. [Try it in KAOP](https://kaop.komodor.com/modules/incidents) ### Fixed: Human-selected remediation mode now activates Choosing "human-selected" remediation mode on the Team & steps step of a new incident workflow now takes effect — Save Draft and Activate both used to fail with "Remediation mode is chosen when the workflow is created and cannot be changed to 'human-selected'", because the workflow was silently created as `automatic` three steps earlier, before you ever reached the mode picker. The workflow is now created once the mode (and the rest of the team) is actually chosen, so the pick you make is the one that ships. [Try it in KAOP](https://kaop.komodor.com/incidents/workflows/new) ### New: End a hosted shadow from the agent's page A hosted agent running a shadow now has **Remove shadow** on its page, the same one-click teardown an agent you deploy yourself already had. It removes the shadow's release, its worker token and its GitOps directory; the main agent keeps running on its own token, with its config untouched. Before, hosted was the one arm with no teardown in the product. Fleet showed the experiment running — the replica count split into production and shadow — but the shadow has no row of its own to act on, so ending one meant an API call. The two paths that did exist in the UI were far too blunt: deleting or archiving the main agent sweeps the shadow away with it, and ends the agent too. The confirmation says what actually happens here. Ending a shadow you deploy yourself tells you your candidate is still yours to remove; for a hosted shadow AgentOps removes the release itself, so that sentence would send you after a cluster you do not have. One thing worth knowing if you scripted around this: the API teardown, `DELETE /api/v1/hosted-agents/{customer}/{agent_id}-shadow`, used to refuse with *"archive the agent before deleting it"*. That guard is there to stop a live agent being decommissioned by accident, and it was never meant for a shadow — a shadow's teardown leaves the agent serving. It no longer applies to the shadow arm, so the call now succeeds directly, from the UI and from chat alike. [Try it in KAOP](https://kaop.komodor.com/agents) ### Improved: The attention banner collapses when a lot needs fixing An account with more than a handful of failing credentials used to see all of them listed at once above the Integrations table, pushing the table itself below the fold. The banner now shows the four most important failures — widest blast radius first — with a **Show N more** toggle for the rest. It can also be dismissed with the **×** in its corner. Dismissing only hides the failures you've already seen — a credential that fails afterward reopens the banner rather than staying silently tucked behind an old acknowledgement. [Try it in KAOP](https://kaop.komodor.com/integrations) ### Fixed: An incident now says when its agent is offline An alert routed at a workflow whose orchestrator agent had no worker running was accepted and then did nothing — the incident sat at Open with no error, no run, and a timeline holding only "Alert received". Such an incident now carries an **Agent offline** badge in the feed and a timeline entry explaining that the investigation is queued until the agent reconnects, and a workflow bound to an agent that is not running shows that agent as Offline on the workflow detail page. When the bound agent no longer exists at all, the incident is marked Errored with the reason instead of waiting indefinitely. [Try it in KAOP](https://kaop.komodor.com/modules/incidents) ### Fixed: A workflow's run duration reads as a duration again The Run statistics panel on an incident workflow now shows the mean duration of the runs in the window — `1m 30s`, or an em dash when no run recorded both ends. Before, that row read `NaNs` at every window. The panel asked the API for a field name it had stopped sending, got nothing back, and formatted the nothing. Every other number on the card — runs, tokens, cost, errors — was correct the whole time, which made the one broken row easy to read as a real measurement of something. Durations elsewhere on the card are hardened the same way now: a value that is missing or is not a number renders as an em dash rather than as arithmetic on it. [Try it in KAOP](https://kaop.komodor.com/modules/incidents) **August 29, 2026** ### New: See the incidents one workflow produced An incident workflow now has a Feed tab beside Overview and Steps, listing only the incidents that workflow created — severity, status, who it is waiting on, and the investigation run, in the same rows as the incident inbox. Answering "what has this workflow actually caught?" no longer means going back to the inbox and reading the workflow column row by row. [Try it in KAOP](https://kaop.komodor.com/incidents/workflows) ### Improved: Every server sits with the integration it belongs to The separate **Servers** section is gone. Integrations already listed every server — the ones sharing an integration's credential grouped under it, the rest as rows of their own — so the two screens were showing you the same records and you had to know which one to be on to do anything. Now the row is where the work happens. Open the **⋮** menu on any server for **Manage tools**, **Edit**, **Delete**, and, on servers that sign themselves in, **Login** — the same forms as before, reached from the row you were already looking at instead of a second screen. **Test** stays where it was. Two things the row now tells you without being asked. A server built from an OpenAPI document is marked **OpenAPI**, so it is obvious which ones are edited through the spec wizard rather than the server form. And a server reaching its upstream through an Outpost names that Outpost — flagged **not routing** when the Outpost has been switched off, which until now looked healthy from every angle while refusing every call made through it. **Reload gateway** moved to the top of the page, beside Add integration, because it always applied to the whole gateway rather than to one server. Old links to the Servers section still work — they land on Integrations. [Try it in KAOP](https://kaop.komodor.com/integrations) **August 28, 2026** ### New: Import an OpenAPI document by uploading it An OpenAPI integration no longer needs its document published at a public HTTPS URL. The Connect step now asks where the document comes from, and **Upload the document** takes the file itself: a spec generated into a build artifact, exported from an internal portal, or simply handed to you. Everything after that is the same. The same operation picker, the same tools, the same groups, policies and credentials. An uploaded document is stored with the integration, so opening the editor lists every operation it declares without fetching anything, and you can widen the selection later. Choose a file again to replace it. Refresh, which re-reads a document from its URL, reports that there is nothing to re-read for an uploaded one and points you at re-uploading instead. One thing the chat co-pilot cannot do for you: it can choose the upload source, but it has no way to attach a file, so it will ask you to pick it. [Try it in KAOP](https://kaop.komodor.com/integrations/openapi/new) ### Fixed: Shadow an agent you deploy yourself Deploying a shadow of an agent you run yourself — your own image, your own infrastructure, connected through the SDK — now hands you what that actually takes: a worker token for the shadow, and the environment to start your candidate build beside the one already running. Before, it handed you a `values.yaml` and a `helm install` for our generic agent-base image. Running that started a *different agent* under your agent's id. Both arms completed, both showed up as a comparison pair, and the shadow side was never your candidate — a comparison that looked right and was not. Only agents whose configuration AgentOps itself renders still get the chart command, because for those it is the right answer. The hand-off is also explicit that the candidate must run on the shadow's own token. Deploying it with a copy of the main agent's token is the quiet failure: it registers as an ordinary production replica and starts answering real callers, and nothing stops it. Your existing deployment needs no change. **Remove shadow** on the agent's page ends the experiment and revokes the shadow's token — the one thing that was previously impossible without an API call. Your candidate deployment is still yours to remove. [Try it in KAOP](https://kaop.komodor.com/agents) ### New: Pick which AgentOps tools your agent sees An agent connecting to the AgentOps MCP endpoint received all 47 tools whether it used one of them or forty. Add `?toolset=` to the URL and it sees only the groups it asked for — `?toolset=slack_bot,knowledge` serves eleven tools instead of forty-seven, leaving the rest of the context window for the actual work. Name specific tools with `?tools=`, combine both to union them, and send no parameter at all to keep the full set exactly as before. `whoami` always answers, so a narrowly scoped agent can still tell you which workspace it is in. Slack bot tools moved onto the same endpoint as the `slack_bot` group. Anything pointed at the old `/slack-bot/mcp` address should now use `/internal/mcp?toolset=slack_bot`. ### New: Guardrails on the payload that starts a run, before the run exists A guardrail can now stand at the **entry**. Pick **Agent start** as the gate on the first step and the rule is checked on the payload that would start a run — a webhook body, a schedule's input, whatever an API caller sent — using the same conditions and operators every other gate already uses. With this the wizard covers five gates: what arrives, a tool call and what it returned, a request to a model and what the model answered. This is the only gate that can promise a value **never entered your storage**. The other four stand on traffic passing through us and refuse to pass it on; this one is asked as the run is created, before its row is written. So "no card numbers in anything that arrives" is a rule you can actually keep, rather than one that catches the number after it has been saved. The card is the conditions and nothing else — there is no key to name. A payload's keys belong to whoever sent it: a Datadog webhook, a GitHub event and a schedule agree on nothing, so a rule naming one key would mean something different per source. The conditions read the whole payload, nested values included. **Refusing means refusing.** The run never starts, no run appears in your history, and the sender gets an error naming the rule that refused it. A webhook that retries has its retry refused the same way — the rule is deterministic and the second delivery is the first payload. Ask-me-first is not offered here, and for the opposite reason to the two inbound gates: there, the call already happened and there is nothing left to approve; here nothing has happened at all, so an approval would have no call to name. If you are not sure what a rule would catch, arm it as **"Allow it, and record it"** first — every trigger it matches is recorded and nothing is refused, so you can read the matches before the rule changes anything. [Try it in KAOP](https://kaop.komodor.com/guardrails/new) **August 27, 2026** ### Improved: A closed co-pilot stays closed Closing the wizard co-pilot is now remembered. Reload the page, or open a different wizard, and the panel stays shut until you bring it back. Until now the choice lasted only as long as the page: every wizard reopened with the co-pilot expanded, so anyone who prefers the form on its own had to close it again each time. The **Co-pilot** button that brings it back has moved to the bottom-right corner of the window, where a floating assistant is usually found. It used to sit in the top-right of the wizard, on top of that wizard's own header. [Try it in KAOP](https://kaop.komodor.com/agents/new) ### New: See which resources a cluster declares, and when it last synced Anything declared from a Kubernetes cluster — a guardrail, an MCP server, group or policy, a credential, a workflow, an incident workflow or a knowledge base — now carries a Cluster-managed badge, so you find out before you try to edit it rather than from the error afterwards. The badge names the cluster and where in it the declaration lives, and where the edit really would be refused the row's own switch and delete button are disabled with the reason. It also distinguishes a cluster that is enforcing its manifests from one that is paused — there your edit sticks until the cluster resumes and reasserts what it declares. Adding a cluster now hands you its credential together with the two commands that consume it — the Secret to create and the Helm install, with your URL already in them, and every row has an Install action that shows those commands again later. Settings now has a Configuration clusters page listing every cluster running the config-plane operator: whether it is enforcing, its namespace and operator version, how many resources it declares, and when it last reported in. "Never" there is the first thing to check on an install that looks dead. [Try it in KAOP](https://kaop.komodor.com/settings/configuration-clusters) ### Improved: The review history puts the newest pass on top A review's history listed its passes oldest first, so on a PR that has been pushed to a dozen times the pass you came to look at — the one running now, or the one that just finished — was at the bottom of a list that grows with every push. Newest first now, with the `latest` marker on the top row where it belongs. [Try it in KAOP](https://kaop.komodor.com/code-review) ### Improved: Notifications are one group The Integrations rail groups what belongs together. **Webhooks** and **Channels** now sit under Notifications — everything that carries events out of AgentOps — while Integrations, MCP servers and Integration groups sit under Agent tools, which is everything an agent calls. "Notification sinks" is now just **Webhooks**. The old name described the row in the database rather than the thing you set up, and every one of them is a signed POST to an HTTPS endpoint. **Channel routing is gone from the rail.** It was a read-only view of routes you can already see — and edit — on a channel's own page, so it is one place now instead of two that disagreed about which was authoritative. **Inbound endpoints** left the rail too. They receive events rather than send them, so they never belonged beside the outbound half; they are reachable from Webhooks and from the incident pipeline wizard while they find a permanent home closer to the agents they trigger. [Try it in KAOP](https://kaop.komodor.com/integrations) ### New: Guardrails on what a model answers, before your agent reads it A guardrail can now stand on the model's **answer**. Pick **Model response** as the gate on the first step and the rule matches on the text a model produced — so "never tell my agent to drop a database" is one rule with one condition, using the same operators every other gate already uses. With this the wizard covers all four gates: a tool call and what it returned, a request to a model and what the model answered. This is the gate whose damage lands on a **person** rather than on a system. A model can answer with advice that harms whoever acts on it, or raise something your organisation has decided a model must not raise at all — and neither of those appears in the prompt you sent or in anything a tool returned, so no other gate can see it. That is also why deterministic conditions fit it well: a command template or a keyword is something you can name, explain, and get the same answer from twice. The card is the simplest of the four — the conditions and nothing else. There is no tool filter here, because an answer has nowhere to have come from but the model. Say what it does and does not do, because "refuse" means something weaker here: the answer has already been generated and billed by the time the rule is asked. What the rule withholds is the text, and the agent gets an error naming the rule instead of the content. Ask-me-first is not offered at this gate, for the same reason it is not offered on a tool result — there is nothing left to approve. **One thing to know before you arm one: a streaming answer is refused outright.** A streamed reply never exists as one whole text, so there is nothing for a condition to read — and rather than let it past unchecked, the request is refused before any tokens are spent, with an error naming your rule. If your agents ask for streaming answers, arm the rule as "Allow it, and record it" first: a recording rule never causes that refusal, so you can see what it would catch before it changes anything. [Try it in KAOP](https://kaop.komodor.com/guardrails/new) ### Fixed: Closing the co-pilot no longer leaves a button on top of the wizard Closing the wizard co-pilot left its **Co-pilot** reopen button floating in the top-right corner of the page, on top of whatever the wizard's own header had there. The button now takes its own space above the form and lines up with the content, so nothing it covers is lost behind it. [Try it in KAOP](https://kaop.komodor.com/add-agent) ### Improved: Import an OpenAPI document from the catalog **Add integration** now offers *Custom integration (OpenAPI)* beside the bring-your-own MCP server. Point AgentOps at an OpenAPI document and its operations become tools your agents can call — no MCP server of your own required. The wizard is the one that shipped last week. What changed is where you find it: importing a document was reachable only from the Servers panel's Add menu, which is somewhere you go to manage servers rather than to connect something new. [Try it in KAOP](https://kaop.komodor.com/integrations) ### Improved: Change which keys an agent receives, after the credential exists **A multi-key credential's per-agent key scope is now editable from the Access panel.** Open Access on a credential, and every bound agent shows what it actually receives — `All keys`, or `2 of 3 keys` — with an **Edit keys** control to narrow or widen it. Granting access to a new agent can carry a restriction from the start. Until now the checkboxes existed only in the create wizard. A scope set there was permanent in practice: the Access panel showed no restriction and had no way to change one, so the only route to a different key set was to remove the agent and grant it again — which handed over *every* key. That mattered most right after replacing a value. When a replace removes keys, AgentOps prunes them from every binding scoped to them, and deletes a binding left with none — deliberately, so an agent scoped to a key that no longer exists does not silently inherit access to new ones. The notice telling you to re-grant that agent now points at a panel that can re-create the narrow scope, and it names the agents rather than printing their ids. One quieter change with the same shape: re-granting a credential from anywhere else in the product — the agent wizard's permissions step, an agent's secrets card — no longer clears a key restriction. Widening a binding to every key is now something a screen has to ask for explicitly. Also: the Edit dialog says whether a credential holds one value or a named key set, the new-credential wizard reports an over-limit or malformed key set at the field instead of failing on submit, and a multi-key credential is no longer offered where only a single string can work — as an LLM key, or as an MCP server's bearer token. [Try it in KAOP](https://kaop.komodor.com/credentials) ### New: Answer what is waiting on you **Attention** is one list of everything in the workspace that has stopped and needs a person, with the act on the row rather than three screens away: * **A held tool call.** A guardrail armed to hold refuses a call and registers it for approval. The row shows the tool *and its arguments* — the whole point of an approval — with Approve and Deny. A held call made inside a workflow step resumes on its own once you approve it. * **A remediation selection.** When a workflow parks on a choice, the options it froze when it asked are the options you pick from, "take no action" always among them. Answering advances the run down that branch immediately. * **A run that gave up.** Anything that stopped and needs review, linked back to the incident it belongs to so you can retry it there. Incident rows and the incident page now carry *Awaiting approval*, *Awaiting your selection* and *Needs attention* badges, and the header bell counts all three rather than just approvals. [Try it in KAOP](https://kaop.komodor.com/attention) ### Improved: Creating an agent takes two fewer steps Scaling and Notify me are no longer steps of their own. KEDA autoscaling is now set where you choose to self-host, and the notification sink is now the second half of the Triggers step — so the step count no longer changes under you when you switch between self-hosted and Komodor cloud. The last step is called Activate rather than Review: it drops the configuration JSON and the permissions recap that only restated what you had already filled in, leaving the one action that actually creates the agent. Capabilities now sit next to the instructions that decide what the agent does with them. [Try it in KAOP](https://kaop.komodor.com/agents/new) **August 26, 2026** ### New: Jump from an incident straight to the alert that started it An incident that came in from Datadog or PagerDuty now shows an **Open in Datadog** / **Open in PagerDuty** button on its detail page, linking to the monitor or incident on the provider's side. The link is read from the alert payload itself — for Datadog, make sure your webhook template includes `$LINK`. When the payload carries no link, no button is shown. [Try it in KAOP](https://kaop.komodor.com/incidents) ### Improved: Model-request guardrail decisions now show on the run A run's **Guardrails** tab used to list only the decisions made about its tool calls. Decisions made about what the agent sent to a model were recorded in the audit trail and shown nowhere near the run they belonged to, so the first place anyone looks when a run behaves oddly was the one place that did not mention them. They now appear alongside the tool-call ones. They come with a **during this run** mark, and the distinction is worth knowing. A tool-call decision is made inside a run-scoped session, so it names the run it stopped. A model-request decision is made in the LLM path, where the request is built by the agent's own harness — nothing there can attach a run to it, so the tab matches it by the agent it names and the time it landed. That supports "this fired while your run was open" and not "this stopped your run", and the mark is which of the two you are reading. A decision that names a *different* run is left out rather than shown unmarked. The audit list gained the filters behind it: you can narrow it to one agent, and to a time span, without writing a query by hand. [Try it in KAOP](https://kaop.komodor.com/runs) ### Improved: MCP setup picks the group for you to choose, and saves as you go The last step of MCP setup used to be a blank box you typed a group name into. It is now a picker over the groups you already have — each one listing the servers it holds — with an explicit option to create a new one. Choosing an existing group names the agents already on it, so you can see who gains the tools you just exposed before you commit. Typing a name that is already taken, or one the group endpoint could not carry, is now said out loud instead of failing on save. Exposing tools no longer needs a save button: ticking a tool writes straight through, and the step says so. Unticking every tool now means every tool is withheld, which is what it looked like it meant. Finishing is one button. **Done** saves the group and closes; if the servers have no group yet it says so — with a **Close anyway** next to it, since an ungrouped server is registered and safe, just not reachable by any agent until you group it. [Try it in KAOP](https://kaop.komodor.com/integrations) ### New: Integrations show the servers they feed Integrations is one list now. A connection and the MCP servers that authenticate with it sit together — a Datadog connection feeding five servers is one row you expand, instead of one row under Connections and five unrelated rows under MCP servers. Custom servers you registered yourself are rows in the same list rather than a separate surface. Every row says who is using it. "Used by" resolves through the groups your servers belong to to the agents scoped to those groups, and names them rather than counting them. It is also honest about the case that used to be invisible: an agent with no MCP group can reach every server on the gateway, so a row the whole fleet can call now says so instead of reading as unused. A failing credential is called out above the list, with what broke, when, and how many servers are serving unauthenticated because of it. Reconnect appears on the row while it is actually failing, and nowhere else. The catalog moved behind Add integration — a searchable panel by category, with the servers you run yourself first. An account with nothing connected still sees it inline, since there is nothing else on the page to look at yet. [Try it in KAOP](https://kaop.komodor.com/integrations) ### New: Guardrails on what your agents send to a model A guardrail can now stand in front of a model request, not only a tool call. Pick where the rule stands on the first step: **Tool call** checks a tool before it runs, **Model request** checks what the agent sends to a model — the prompt, before the model sees it. A rule at the new gate matches on the messages or the system prompt, with the same operators tool-call rules already use, so keeping an internal codename or an account number out of a prompt is one condition rather than a code change. The form also authors the two criterion shapes it previously could only display. **Model judgment** states a policy in plain language — "no customer identifiers leave the account" — and names which of your models answers it, for the worry no pattern can describe; you choose how long it may hold the call open before an unanswered policy counts as unreachable. A rule carrying either shape now opens for editing instead of refusing, which is the more useful half of the change: until now a rule written through the API could be read in the console and not changed there. One thing the new gate does not get is the replay. The backtest works by re-reading the tool calls your agents already made, and nothing records what they sent to a model, so there is no history to test a prompt rule against — the review step says so rather than showing an empty result. Arm such a rule as "Allow it, and record it" to watch what it would catch on live traffic, then switch it to refuse. [Try it in KAOP](https://kaop.komodor.com/guardrails/new) ### New: Guardrails on what a tool returns, before your agent reads it A guardrail can now stand on the way **back**. Pick **Tool response** as the gate on the first step and the rule matches on the text a tool returned — so "no private keys in anything a tool returns" is one rule with one condition, using the same operators tool-call rules already use. It reads every content block a tool sends and its structured JSON answer too, so a server replying in JSON instead of text does not slip past. The fields are the other way round at this gate, which is the point of having it. What comes back is what the rule is about, and the tool picker is demoted to a filter: **Only from these tools**, where leaving it empty means any tool. That is the reverse of a tool-call rule, where naming no tool covers nothing. Say what it does and does not do, because "refuse" means something weaker here: the tool has already run by the time this gate is asked, so its effects stand. What the rule withholds is the result, and the agent gets an error naming the rule instead of the content. Ask-me-first is not offered at this gate for the same reason — there is nothing left to approve. If a result arrives in a shape the gate cannot read at all, it is withheld rather than passed. There is no replay here either: nothing stores what your tools returned, and that is deliberate — a tool result is not kept in the audit trail either, so no decision can hand it back to you. Arm the rule as "Allow it, and record it" to watch what it would catch on live traffic, then switch it to refuse. [Try it in KAOP](https://kaop.komodor.com/guardrails/new) ### New: Connect any REST API from its OpenAPI document Point AgentOps at a service's OpenAPI document, tick the endpoints worth exposing, and those endpoints become tools your agents can attach. Nobody has to write or host an MCP server for that API first, so connecting an internal service or a vendor's API stops being a development task. Add integration now offers **Custom integration (OpenAPI)** alongside the MCP server option. The wizard reads the document, lists everything it declares, and lets you filter and pick — reads start ticked, anything that writes starts off. Credentials bind exactly as they do for an MCP server, and the generated tools appear in the catalog under the integration's name. Come back to the same wizard any time to change the selection, add a note to an operation, or re-point the credential. Refreshing re-reads the document and tells you which endpoints the upstream has stopped offering. [Try it in KAOP](https://kaop.komodor.com/integrations) ### Improved: The agent wizard shows what an MCP group actually grants Picking an MCP group for an agent used to tell you the group's name and nothing about what the agent could do with it. The step now lists the group's tools per member server, and marks the ones the agent will not get — separating a tool the server itself denies from one a group rule withholds, because they are fixed in different places. Each server folds away, and a server with a long tool list scrolls in place instead of pushing the rest of the wizard off screen. [Try it in KAOP](https://kaop.komodor.com/add-agent) ### Improved: A tool pattern that matches nothing now shows you what would Writing a tool pattern by hand, the wizard already warned you when it matched none of your tools — but not what your tools actually look like, which is the thing you need to fix the typo. It now names a few, one per namespace rather than the first handful in catalog order, so a catalog full of `k8s_*` tools cannot hide the fact that the prefix you meant was `datadog_`. [Try it in KAOP](https://kaop.komodor.com/guardrails/new) **August 25, 2026** ### Fixed: Tables and menus sit straight Providers fits on screen again. A status pill like "Verification failed" used to wrap inside its own border, and the verified-at column carried a full timestamp that took two lines, so every row was half again as tall as it needed to be and the table ran past the bottom of the window. The pill stays on one line, the timestamp is the compact form the rest of our tables use — the full value is still there on hover — and nine models now fit where six did. Every row menu in the console had its icons pressed flush against their labels. They no longer do, and the menus follow the same spacing, cursor and label styling as the dropdowns beside them. Fleet's "Last change" no longer reads as a dash for an agent you just created. Until the agent's definition changes for the first time, the column shows when it was created, which is the only change there has been. [Try it in KAOP](https://kaop.komodor.com/providers) ### Fixed: Rows and filters look clickable Hovering a review row or a sidebar filter on the Code Review inbox showed a plain arrow, as if the row were inert. Every inbox built from the same parts — Incidents and the Quality Lab included — now shows a pointer over its rows, its filters, and the collapsible filter groups. [Try it in KAOP](https://kaop.komodor.com/code-review) ### New: Get notified when your agent's runs finish The agent creation and edit wizards now include a **Notify me** step. Choose any notification sink from your account and it will receive run lifecycle events — run started, run succeeded, run failed — for every run that agent starts. The step shows a searchable dropdown of all your configured sinks with their kind (Webhook, Slack, MS Teams), and once you select one it lists exactly which events that sink is subscribed to so there are no surprises. The step is optional: skipping it leaves the agent with no sink attached. [Try it in KAOP](https://kaop.komodor.com/marketplace) ### Fixed: The MCP group you pick now sticks to the agent Build an agent from scratch, pick an MCP group, and the group is now recorded on the agent itself. It always reached the worker — the group is in the values.yaml you download — but the control plane's own record of it stayed empty, so Fleet showed no group and nothing that reads an agent's tool group could see one. The only screen that showed it correctly was the Edit wizard, which was reading back its own notes. Changing the group from the Edit wizard now takes effect too, instead of being remembered by the wizard and nowhere else. Setting an agent to use every gateway MCP server clears the group as you would expect. Picking a group that no longer exists is now refused when you deploy, with the group named, rather than accepted and quietly ignored. [Try it in KAOP](https://kaop.komodor.com/agents) ### Fixed: An imported Claude Agent SDK worker now shows its trace, not just its cost Importing an existing Claude Agent SDK agent through the wizard produced a worker that registered, connected, and reported real usage and cost — but its runs' Trace Timeline stayed empty. No tool calls, no message content, nothing to show what the agent actually did, even on a run that succeeded. The generated wrapper now merges AgentOps' hooks into the agent's own `ClaudeAgentOptions` before calling `query()`, the same two-line change the "importing an existing agent" skill now documents. Nothing about the agent's own loop, prompt, or options changes — the hooks only observe. A freshly imported Claude Agent SDK agent's tool calls and message content now show up in its Trace Timeline like any other agent's. [Try it in KAOP](https://kaop.komodor.com/marketplace) ### Fixed: A slow deploy is no longer a failed one An agent that took a few minutes longer than usual to roll out would report "Deploy timed out — the agent did not come online within the expected window" and keep reporting it forever, even while the agent was up and serving runs. The deploy really had missed its window; what was wrong is that nothing ever revisited the verdict once the agent arrived. Two changes. The window a rollout is given now covers the whole chain — the build, the sync, the image pull and the pod starting — rather than expiring while the pod is still being pulled. And when a deploy does time out, it is no longer the last word: as soon as the agent comes online, the deploy settles as done on its own and the banner clears. A deploy that failed for a reason the agent cannot answer — the change never merged, its credentials were never provisioned — still reads as failed, which is what you want it to say. [Try it in KAOP](https://kaop.komodor.com/fleet) ### Improved: A guardrail reads in one glance The guardrails list gave its widest column to a sentence describing each rule, which is fine for one rule and unreadable for a page of them. A row now answers the question in three words instead: the gate it stands at, the kind of matcher it uses, and whether a match is refused or only recorded. The patterns and the description are still there, on the rule itself. Guardrails also shows how many you have in the Govern menu, like every other section of the catalog does. [Try it in KAOP](https://kaop.komodor.com/guardrails) **August 24, 2026** ### New: Outposts say whether they still run the config AgentOps rendered An Outpost relay reports a digest of the configuration document it loaded, but nothing compared it against anything: drift was recorded on every connect and detected by nobody. An Outpost could be enrolled, connected, and quietly serving traffic through an allowlist nobody in AgentOps had written. Every Outpost now carries a **config verdict** in the list and on its detail page, beside its enrolment and connection badges. AgentOps records the digest of each configuration it renders install values for, and compares it against what the relay says it is running. The verdict separates the two cases that matter, because they call for opposite responses. **Upgrade pending** means the relay runs a configuration AgentOps did render, just not the newest one — ordinary after any change on our side, and fixed by running the upgrade command on the Install tab. **Config differs** means it runs a configuration AgentOps never rendered: edited in the cluster, or overridden by a chart value. That one is worth a look. Two more verdicts exist so the badge never overstates what it knows. **Config mixed** appears when replicas of one relay disagree with each other, which is normal mid-rollout; a fleet part-way through an upgrade no longer reports as though every replica agreed. **Config unknown** means there is nothing to compare yet — nothing live is reporting, or install values have never been rendered. Turning the local listener on does not move the verdict. It changes how the relay is deployed, not what it may route or reach, and a badge that cried drift over it would be one you learned to ignore. [Try it in KAOP](https://kaop.komodor.com/outposts) ### New: Incident and run names are short summaries now New workflow incidents are named `{workflow} — {short summary}` instead of the raw monitor title from Datadog or AlertManager, so the inbox is scannable. Agent runs in Fleet and History get the same kind of five-word summary as soon as the run is queued. Chat sessions are unchanged — they already use your first message. Existing rows are not rewritten. [Try it in KAOP](https://kaop.komodor.com/incidents) ### Improved: Connection checks now work for Datadog, GitHub, AWS and more When connections started reporting whether their credentials actually work, two providers could never report a pass — including Datadog, the most connected provider. The check was talking to their tool servers without opening a session first, so those servers refused it and the connection showed a problem that was ours, not yours. GitHub had a second version of the same fault: the check called a tool that server does not offer. Both are fixed, and the check now covers AWS, Buildkite and Grafana Cloud too. Between them and Slack and Google Workspace, every provider anyone has connected can now say whether its credentials are accepted — where before, most connections had never been checked at all. Each of these was verified against a real account, twice: once with working credentials to confirm it passes, and once with deliberately broken ones to confirm it fails. That second check is the one that matters, and it changed two of them. Buildkite's obvious endpoint needed a permission many tokens do not have, so a working connection would have been reported as broken; GitHub's needed an endpoint that suits an app rather than a person. Providers nobody has connected yet still say they cannot be verified, which is the honest answer rather than a green tick nobody earned. [Try it in KAOP](https://kaop.komodor.com/integrations?section=connections) ### Improved: Agents are named, not numbered Screens that referred to an agent by its internal id — `agt_9f3c2d1e8b4a` — now use the agent's name. The Workflow details drawer was the worst of them: the orchestrator, every specialist and the remediation agent were all listed as ids. Guardrail decisions, a golden run's evaluator, the agents bound to a credential and the run history on Cost and Overview have all changed too. The ids have not gone away, because they are what you paste into an API call or a support thread. Hover any agent name to see its id and its handle, with a button to copy the id. Two agents can share a display name, since that name comes from the worker rather than from you. Where that happens on one screen, both are shown with their handle so you can tell them apart. [Try it in KAOP](https://kaop.komodor.com/incidents/workflows) ### Fixed: The agent edit dialog no longer offers to rename an agent Editing an agent used to show Name and Description boxes you could type into. Nothing you typed was ever saved — the dialog reported success and the old values came back a few seconds later. Those two come from the agent itself, published by its worker, and they are refreshed every time the agent checks in. The dialog now shows them as read-only and says where they come from. To change them, update the agent's card where the agent is defined and redeploy it. Turning automated triggers on and off, which is the part that always worked, is unchanged. [Try it in KAOP](https://kaop.komodor.com/agents) **August 23, 2026** ### Fixed: Pull requests are reviewed whatever branch they target A pull request opened against another topic branch rather than the default one — the middle of a stack, `master → branch-1 → branch-2` — was silently skipped: no review, no run, no commit status, and no comment on the pull request. A review workflow only accepted pull requests targeting a single branch, and when its **base branch filter** was left empty it fell back to the repository's default branch instead of accepting every branch. An empty filter now means what the wizard always said it did: every base branch is reviewed. Naming a branch in the filter still restricts reviews to pull requests targeting exactly that branch, and a filter that was set can now be cleared again. Re-pointing a pull request at another branch also triggers a fresh review. GitHub does that automatically the moment the branch under a stacked pull request merges, and the review that follows describes the changes the pull request actually contains now rather than the ones it carried against its old base. [Try it in KAOP](https://kaop.komodor.com/code-review) ### Fixed: A connected MCP server's tools reach your agents right away Connecting an MCP server and attaching its credential is now all it takes — its tools show up on the gateway for every agent within seconds, with nothing to press and no wait for a restart. Previously a server could pass its connection test, list every tool in **Manage**, and still be invisible to your agents. **Manage → Refresh gateway tools** republishes one server's tools on demand, for when you have changed something on the provider's side and want the gateway to pick it up immediately. [Try it in KAOP](https://kaop.komodor.com/integrations) ### Fixed: Incident findings are no longer cut off mid-sentence The Findings card on an incident used to stop around two thousand characters, often in the middle of a heading or a sentence, because we stored only a short excerpt of the orchestrator's write-up. The page now keeps the full report. Slack still posts a short preview with a link back to the investigation. [Try it in KAOP](https://kaop.komodor.com/incidents) **August 20, 2026** ### New: Filter which alerts start an investigation, and see what happened to the rest An incident workflow used to investigate every alert its endpoint received. Narrowing that was a Datadog-side monitor change, or a second endpoint — and if an alert never became an incident, there was nothing to look at to find out why. The workflow's **Endpoint** step now has a filter: rows of `severity`, `title`, `status`, `labels.` or `raw.` compared with is / is not / contains / excludes / at least / is any of, and so on. All of them have to hold, and no conditions still means every alert. **contains** is the one that matters for real payloads. Senders routinely bury the value you want inside a human-readable string rather than giving it a field — a Datadog webhook on the stock template arrives with its priority in the title as `[P2] ...`, and a PagerDuty orchestration event carries the alert name in `event.data.incident.summary`. `raw.title contains [P2]` and `raw.event.data.incident.summary contains [FIRING:1]` are now expressible; before, the closest you could get was pasting a whole per-incident string that would match exactly one alert. Severity comparisons work across the ways providers spell it. `severity at least SEV-2` ranks correctly whether the payload says `P2`, `critical` or `high`, and `severity is SEV-4` now matches PagerDuty's `Sev-4` and Datadog's `P4` — the value shown in the test panel is the value that matches. Every workflow also gained an **Events** feed showing each alert its endpoint delivered and what the workflow did with it — routed (with a link to the incident), filtered out (naming the condition that rejected it), throttled, or deliberately ignored. Integrations → Endpoints → History shows the same thing from the endpoint's side, for every workflow listening on it. An alert that was dropped on purpose no longer looks the same as a webhook that never arrived. If one endpoint feeds workflows that read its payloads differently, a workflow can now override the field mapping — the dedup key, title, severity and transition paths — and preview it against a real captured event before activating, so a wrong path shows up as a visibly empty title instead of silently mis-parsed incidents. All of it is available to the co-pilot and to `aops workflows filter set|test`, using the same ` ` phrasing. [Try it in KAOP](https://kaop.komodor.com/incidents) ### Improved: The create wizard now waits for your agent, instead of telling you to go and look Creating an agent used to end on a promise: "your agent will appear in Fleet once it reports its first heartbeat". True, but unactionable — and indistinguishable from a deploy that quietly failed. The last step now **watches for that heartbeat** and tells you which of the two happened. It fills in on its own as the agent comes up, and once it connects it shows you the card the agent published about itself — its name, model, version, capabilities and skills — so you can see it started with the configuration you deployed. You can leave the page and come back; the agent is already registered either way. If nothing arrives after a few minutes you get a short list of things worth checking, not a spinner and not a failure: an agent that has not connected yet is an install you have not finished, and it waits as long as you need. Komodor-deployed agents no longer jump you to the agent page the instant the deploy is accepted — they stay put and report progress, so you can watch the thing actually start before moving on. For agents you run yourself, the two artifacts now arrive in the order you have to use them: **values.yaml first, then the `helm install` command that reads it**. The command references that file, so a screen that offered it first was asking you to run something that could only fail. [Try it in KAOP](https://kaop.komodor.com/agents/new) ### Improved: Test an MCP connection before the setup wizard registers it Setting up an MCP server from a connected account used to mean registering it first and finding out afterwards whether the address was right and the credentials worked. If they didn't, the wizard had already created the server and had to go back and delete it. There is now a **Test connection** button beside Register. It reads what the address advertises before anything is created, and it says which question it actually answered: with an integration connection selected the test sends that connection's credentials, so a clean result means those credentials were accepted — not merely that something responded. Where several servers are registered in one click, each endpoint is tested and reported on its own line. A result disappears the moment you edit the URL, switch service or change connection, so what you are reading always describes what is on screen. Testing is advisory, never a gate: a server reachable only through an Outpost cannot be probed before it exists, and those still need to be registerable. Signing in to a provider got two fixes on the way through. The login window is now opened by your click rather than a moment later, which is what browsers' pop-up blockers were stopping. And once you finish signing in, the connection is re-tested and the result shown in place — with a little tolerance for a token that has just been written and has not settled yet, which used to surface as a freshly authorized server appearing broken. [Try it in KAOP](https://kaop.komodor.com/integrations) ### Fixed: The Providers page tells you when a provider refused Adding models to a provider used to look the same whether the provider had nothing to offer or had rejected the key outright, and the page would show a "Known models" badge over an empty list. It now states the provider's own reason where the list would have been, and keeps the hand-typed model id available. Model ids are also offered exactly as the provider spells them, so ids that carry their own provider prefix (`groq/compound`) register and verify instead of failing with a not-found that looked like a permissions problem. Models you tick are tied to the credential they came from, so switching credentials starts the selection over rather than saving the previous provider's picks. [Try it in KAOP](https://kaop.komodor.com/providers) ### Fixed: A server whose credential was deleted now says so, instead of "permission denied Disconnecting an integration destroys its credential, but an MCP server bound to that connection kept its binding. The server's status then read **permission denied** — the wording for a credential the upstream refused — which sent people looking for a missing scope on a credential that no longer existed. Those servers now report **credential missing**, and testing the connection says plainly that the credential is gone and that the fix is to reconnect the integration or pick another credential under Authentication. The same wording appears whether the status came from a connection test, a startup check or live agent traffic, so the row and the detail under it agree. Choosing a credential that has since been disconnected while adding a server is now caught the same way, before the server is saved. [Try it in KAOP](https://kaop.komodor.com/integrations) ### New: Match a tool argument with a regular expression A guardrail's **Only when** conditions could compare an argument four ways: equals, is one of, contains, and a glob. That covers a value you can name, and misses the rules people kept asking for — *any* destructive statement, *any* role outside our own account — because those are shapes, not values. The workaround was `contains`, one condition per verb, and a rule broader than it read. Conditions now take a fifth operator, **matches regex**. The pattern is found anywhere in the argument, so `(?i)\b(drop|truncate)\s+table\b` is one condition where four `contains` rules used to be an approximation. It is case-sensitive unless the pattern starts with `(?i)`, and unanchored unless you write `^…$` — the same expression you would test anywhere else. Back it with **Try it** before arming it, which replays the rule over real traffic and is worth more here than anywhere: a regex is the one condition you cannot check by reading it twice. Two guarantees behind it. A pattern that does not parse is refused when you save it, not quietly stored as a rule that never fires. And a pattern is given a fixed budget of time on each call, so a badly-behaved expression cannot slow down the agents the rule was written to protect. [Try it in KAOP](https://kaop.komodor.com/guardrails) ### Fixed: Disconnecting an integration now asks first — and says what it takes down **Disconnect** on a connection used to be a single click with no way back. It destroys the stored credential and removes the connection outright, and the MCP servers authenticating with that connection did not stop working in any way you could see: the gateway carried on connecting to them without credentials, so the first sign was a tool call failing somewhere else with what looked like a permissions problem. Disconnect now opens a confirmation that names the MCP servers bound to that connection, says they will keep running but unauthenticated, and states plainly that the credential is destroyed and that reconnecting means authorizing the provider again. Cancel leaves everything exactly as it was. The dialog also says what it cannot see — Slack channel routing and agent credential bindings using the same connection are not listed yet — so the list it does show can be trusted as far as it goes. [Try it in KAOP](https://kaop.komodor.com/integrations?section=connections) ### Fixed: Connections now tell you whether the credentials actually work A connection went green the moment you saved a credential, whether or not the provider would accept it. Paste a Datadog API key with a typo in it and the row read "active" — and pressing "Test connection" agreed, because the test only checked that the fields were filled in on our side. Nothing had ever tried the credential against the provider. Now something does. Where we can verify a provider's credentials, we do: a key the provider rejects is refused at the point you save it, with the provider's own reason, instead of becoming a connection that looks fine and fails later. Testing an existing connection makes a real call too, and the result is stored on the row rather than vanishing with the toast. Connections show this next to the existing status, because the two answer different questions — one is whether the connection was set up, the other is whether its credential works today. Where we cannot verify a provider yet, the row says so plainly rather than showing a green tick nobody earned. That is deliberate: telling you a credential is unproven is honest, and telling you it works when nothing checked is how this went wrong in the first place. More providers will move from unproven to verified as we confirm each one against a live account. [Try it in KAOP](https://kaop.komodor.com/integrations?section=connections) **August 19, 2026** ### New: A run now shows which guardrails it tripped Finding out why a guardrail refused a tool call meant going to Settings → Audit, filtering to the guardrail category, and matching rows to a run by their timestamps — which is guesswork as soon as two runs overlap. Guardrail decisions are now recorded against the run that caused them, and a run with any decisions gains a **Guardrails** tab showing what was refused or recorded, which rule decided it, and which tool it was about. The count is on the tab, so you can see whether a run tripped anything without opening it. Nothing changes about what guardrails do — this is the trail catching up to the enforcement. [Try it in KAOP](https://kaop.komodor.com/runs) ### Improved: Upgrading an Outpost no longer discards your own Helm values An Outpost's install panel now gives you a separate upgrade command with its own values document, and the upgrade merges into the release instead of replacing it. Settings AgentOps does not render — a CA bundle, the local listener, resources, image pull secrets — now survive an upgrade rather than being dropped, which previously took the relay off the air whenever the lost value was the CA in front of AgentOps. Needs Helm 3.14 or newer. [Try it in KAOP](https://kaop.komodor.com/outposts) ### Improved: Connected MCP servers now report their own health, and recover on their own A connected server's status used to describe the last time someone pressed "Test connection". A token that expired the next day, a scope someone revoked, an upstream that went down — none of it showed until the next manual test. Real traffic now keeps that status honest. When an agent's tool call is refused because the credential no longer works, or the server cannot be reached, the server says so on its own; when a later call succeeds, it clears itself, so a server that recovers needs nothing from you. Each row shows when its status was last observed and what observed it, so a green tick you can trust is distinguishable from one nobody has confirmed in a week. Two things this fixes along the way: the status badge now appears for every connected server rather than only ones using OAuth sign-in, and a server whose only safely callable tool is something like a version endpoint no longer reports a clean bill of health — calling it proved the server was up, never that your credentials work, and it now says so and points you at setting a validation tool. [Try it in KAOP](https://kaop.komodor.com/integrations) ### New: Grafana Investigator can connect to Grafana Cloud's own hosted MCP server Until now, a Komodor-cloud deploy of Grafana Investigator could only reach Grafana through its bundled MCP tools — reliable, but not a fit if you'd rather point at Grafana Cloud's own hosted MCP endpoint instead. Deploying to Komodor cloud now offers an **MCP tools** step where you can bind a gateway-connected MCP server — register Grafana Cloud's hosted server once, then pick it here. Self-hosted deploys are unchanged: the bundled Grafana MCP and the "use your own in-cluster MCP" option still work exactly as before. [Try it in KAOP](https://kaop.komodor.com/marketplace) ### New: See your whole fleet at once on the new Analytics tab Fleet answered "what do I have" but not "how is it doing". **Analytics** now leads the tabs, ahead of Agents, Triggers and Workflows: outcomes and reliability, cost, mean latency and versions deployed across the window, then runs, cost and latency plotted day by day, a worst-error-rate and a slowest-agents board ranking five agents each (with the run count beside every error rate, so you can see how much evidence a rate rests on), and the model mix behind it all as a donut whose legend names every slice and what it cost. A Today / Last 7 days / Last 30 days selector sits in the Activity tier's header and governs that tier. Above all of that is Live status, which is about right now rather than the window and says so: fleet composition (online, offline, draft), replica capacity, queued work and its oldest item, channel reachability and the SDK versions your workers are running. Every composition bucket links straight into the agents list already filtered, so a count you want to act on is one click from the agents behind it. The agents list gained a matching **Activity (30d)** column: runs and cost over error rate and latency for each agent, sortable by any of the four from the column's header menu. Narrow the fleet with the label, status or replica filters and Analytics follows, so the numbers describe the same agents the rail has selected. [Try it in KAOP](https://kaop.komodor.com/agents?tab=analytics) **August 18, 2026** ### Improved: Run the Remediation agent in your own cluster **Remediation can now be deployed self-hosted**, not only to Komodor cloud. Pick it from the worker catalog and the deploy step offers both homes; nothing else about the agent changes. It is the same agent either way. Remediation is plan-only — it reads a finished investigation and returns a decision, and it has no shell, no cluster access and no credentials for the systems it reasons about — so a self-hosted deploy is not a limited version of the hosted one. What moves is where its prompt runs and where its model credential lives: your cluster instead of ours. The install is the usual one: AgentOps mints the worker token, renders a `values.yaml` for the `agentops-agent-base` chart, and gives you the `helm install` to run. No cluster RBAC is granted — this agent reads nothing from Kubernetes. [Try it in KAOP](https://kaop.komodor.com/agents) ### Improved: Testing an MCP connection now proves the credentials actually work Testing a connected MCP server used to check only that we could reach it and list its tools. A token could pass that check and still be refused the first time an agent used it — because it was valid but missing a scope or a permission. The test now runs a second check: it invokes one read-only tool for real and reports whether that succeeded. The two results are shown separately, so a server that is reachable but not yet permitted to do anything is obvious at a glance, and a permissions problem no longer looks like a connection problem. The tool used is picked automatically, and you can name a specific one per server when the automatic choice is not the one you want. [Try it in KAOP](https://kaop.komodor.com/integrations) ### New: Give an agent more than one trigger The Triggers step asked one question — schedule, webhook, or neither — and took one answer. An agent that should run every morning **and** on an inbound event had to be created with one of those, then finished off by hand somewhere else. The step now holds a list. **+ Add trigger** appends a card, each card has its own type and fields, and they are created together with the agent in a single call. Deploy an agent with two schedules and a webhook and all three exist the moment it appears in Fleet — each reported individually, so a schedule that was created never reads the same as one that wasn't. Two limits are the agent's, not the form's, and the form now says so up front rather than letting you find out at deploy: an agent may hold **one webhook**, so that option greys out in the other cards once one has it, and **Manual / on-demand** means *no* automated trigger, so it cannot sit alongside others. Remove the card holding either and it becomes available again immediately — nothing here is one-way. There is always at least one card, because zero triggers is what Manual already says. Editing an agent opens the step on the triggers it actually has, which is the part that used to go wrong. Switching a schedule to a webhook once left the schedule firing and quietly added the webhook beside it; the wizard warned you and pointed at a cleanup screen. Now removing a trigger removes it, changing its type replaces it, and both wait for **Save** — the step tells you what is about to be deleted before anything is. Triggers an agent declares in its own `agent-spec.yaml` show up read-only, since that file owns them and the next heartbeat would overwrite an edit made here. [Try it in KAOP](https://kaop.komodor.com/marketplace) ### Fixed: Deploying an agent no longer looks like it did nothing Pressing **Deploy to Komodor cloud** on the wizard's Review step took you to your new agent — except when it didn't. Anything that went wrong after the agent had already been created was swallowed, and all you saw was the configuration summary quietly disappearing from a page that otherwise looked untouched, Deploy button and all. It read as "nothing happened", so the natural next move was to press it again. The agent's existence is now recorded on screen the instant it is created, before anything else is attempted. You still land on the agent's page as before; if you don't, you are left looking at **Agent deployed** and a link to it, rather than at a Review step that appears never to have been used. The Deploy button cannot come back either way, so a retry can no longer deploy a second agent. [Try it in KAOP](https://kaop.komodor.com/marketplace) ### New: Build a code reviewer without writing a worker Adding a custom reviewer meant writing and hosting an SDK worker. That is still the right answer for a reviewer that needs its own GitHub plumbing or incremental re-review — but it was the *only* answer, even for a reviewer whose whole job was "read this diff against our conventions and say what you find". An agent built in the **Add agent** wizard can now declare **Code review** beside **Chat**. It runs the shared agent image, so its instructions are the only thing you write: give it an MCP group with GitHub, tell it what to look for, and it appears under **Reviewers** with a **Custom** badge, ready to add to any workflow next to the built-in reviewer. Worth knowing: the toggle makes the agent *selectable*, and nothing more. How good the review is comes entirely from its instructions, and a reviewer that returns no structured verdict shows up as a run link with no findings breakdown rather than an error. Both are also declarable over the API and the agent-management MCP, so an agent onboarding another agent can set them too. [Try it in KAOP](https://kaop.komodor.com/code-review/reviewers) ### Fixed: A label typed in capitals now works Labels are matched by exact equality — everywhere they are used, a filter, an access grant, or a workflow looking for a particular agent compares the label character for character. A label saved as `ROLE: Scenario-Runner` therefore matched nothing that looked for `role: scenario-runner`, and nothing said so: the save succeeded, and the Labels list shows keys in capitals regardless, so a broken label looked exactly like a working one. Labels are now lowercased when they're saved, keys and values both. Type `ROLE` and you get `role` — no error, no correction to make by hand — and the label meets the grants and filters written against it. Existing agent labels have been converted, so one that never matched before starts matching now. The visible change is in values: an agent labelled `owner: AgentOps` now reads `owner: agentops`. Keys look the same as they always did, since the list displays them in capitals either way. If two labels on the same agent differed only in case, they were always two separate labels, and they now collapse into one. [Try it in KAOP](https://kaop.komodor.com/fleet) **August 17, 2026** ### New: Try a guardrail before you arm it A guardrail had one outcome: refuse the call. That left one question unanswered — *is this pattern actually right?* — and only one way to find out, which was to arm it on live traffic and see whose run broke. A guardrail can now be set to **allow the call and record it** instead. Same rule, same matching, no refusal: the call goes through untouched and shows up under **Recent decisions** with the agent and tool that triggered it. Read a day of that, then switch the same rule to refusing once you can see what it catches. It is also the honest way to run a rule you only want to *watch* — a scope boundary you would like reported, not enforced. Two things worth knowing. A call covered by both a recording rule and a refusing one is still refused, so adding a watcher can never quietly disarm a control you already rely on. And a recorded call is invisible to the agent — told it had been flagged, an agent would start working around whatever it was, and the rule would stop measuring the thing you armed it to measure. [Try it in KAOP](https://kaop.komodor.com/guardrails) ### New: See what a guardrail would have stopped, before you turn it on The form could already tell you which of your tools a pattern matches. It could not tell you the thing you actually want to know: **would this rule have refused work my agents were really doing?** So the only way to find out was to arm the rule and wait. The review step now has a **Check the last 7 days** button. It replays your unsaved rule — the same matching the gateway uses, argument conditions included — against the tool calls your agents have already made, and answers with the count and the agents and tools behind it: *"would have refused 3 of 240 tool calls in the last 7 days"*. It only counts calls a guardrail could ever have decided, so a worker's own shell commands don't pad the total, and it never shows you a call's arguments — those can carry credentials, and an authoring screen is no place for them. [Try it in KAOP](https://kaop.komodor.com/guardrails) ### Improved: See which agents have a shadow running, without opening them A shadow deploy runs every request **twice** — the candidate version alongside the live one — for as long as it stays up. Until now the only way to notice one was already running was to open that agent's Instances tab. The Fleet list now shows it directly: an agent with a live shadow splits its replica count into **N production · M shadow** right in the row, and a new **Shadow active** filter in the left rail narrows the whole fleet down to agents currently running one. Works the same way for both self-hosted and cloud-hosted shadows. [Try it in KAOP](https://kaop.komodor.com/agents) ### New: Scope an agent's roles while you build it Every agent has always started with the built-in `agent` role — the full set of control-plane and MCP capabilities a worker needs. Narrowing that meant creating the agent first, then finding it on the Access page and editing its roles separately, with a live agent running on the default set in between. The Agent Builder wizard now has a **Permissions** step, right before Review, on every flow — scratch, catalog, self-hosted and cloud. Leave it alone and the agent gets the same built-in role it always has. Choose **Custom** and pick a narrower set of roles instead, and the agent is scoped to exactly that from the moment it's created — no gap where it runs wide open. The same step also lets you bind the credentials the agent will need, so an operator scoping access and an operator wiring up secrets aren't two separate trips through Settings. Custom roles **replace** the built-in set rather than add to it — the built-in role already covers everything a worker needs, so adding to it wouldn't narrow anything. Assigning roles is an admin-level action; the step only appears for an operator who holds it. [Try it in KAOP](https://kaop.komodor.com/agents/new) ### Improved: Catalog agents get a Model step and a Scaling step Deploying a catalog worker (AWS Investigator, Grafana Investigator, and the rest) now walks the same shape of wizard as building an agent from scratch. Any catalog worker with its own model credential gets a real **Model** step to pick Opus, Sonnet, or Haiku — previously no catalog worker ever offered one. Installing a catalog worker into your own cluster now offers the same **Scaling** step (KEDA-based autoscaling) a self-hosted custom agent already had. The AWS Investigator's IRSA option (letting a self-hosted pod inherit AWS access from its own IAM role, with no credentials stored in AgentOps) is now shown alongside the normal connection picker rather than replacing it, and is clearly disabled — not hidden — when you've chosen to deploy to Komodor cloud instead of your own cluster, since IRSA only works self-hosted. The Model step now also picks up the same provider/model picker the from-scratch wizard uses, and is the one place you set how the agent authenticates — no more seeing the same credential asked for twice, once on Model and again on Integrations. "Bring your own key" and "use an existing secret" are hidden for catalog workers, since a pinned catalog image can only authenticate through a named credential reference. Integrations now comes before MCP tools in the step order, matching the order the rest of the wizard reasons about a worker's setup. The Integrations step no longer shows up empty for a worker that has nothing to configure there — the Orchestrator, which authenticates entirely on the Model step and reaches other workers directly rather than through a bound connection, now skips straight from Model to Review (or Workers, if you've picked which agents it may delegate to). It also no longer offers a wizard-created schedule or webhook, since it's meant to be invoked over chat or ad-hoc runs instead. Grafana Investigator can now run on Komodor cloud too, not just in your own cluster — pick "Where it runs" like any other dual-capable worker. "Use your own in-cluster MCP server" stays disabled (with an explanation) when you choose Komodor cloud, since there's no cluster of yours for it to reach from there. Grafana, Datadog, and Kubernetes RCA Investigators can now be chatted with directly, joining the Orchestrator and Klaudia Investigator — and now so can the AWS Investigator (chat support lives in each worker's own image, not a wizard setting). The Card step's Capabilities section (Chat, Code review) is scratch-agent-only: a catalog worker's capabilities are baked into its pinned image, not chosen in this wizard, so the whole card is hidden when deploying a catalog worker rather than showing a toggle with nothing behind it. [Try it in KAOP](https://kaop.komodor.com/marketplace) ### Fixed: A tool call that failed no longer reads as still running A run transcript closed a tool call only when its result arrived, and a call that **failed** never recorded one — so it kept a spinner and read as still running, in a run that had long since finished. The Trace tab said `error` at the same time. The first place this showed up was a guardrail refusing a write: the agent narrated the refusal in plain words, and the tool row above it claimed the call was in flight. A failed call now records its result like any other, carrying the error, and the row shows **Failed** with the reason. And because a result can also go missing for reasons nothing can record — a worker killed mid-call — an unpaired call in a finished run now reads **No result** instead of spinning: not a success, not a failure, just the honest statement that nothing came back. [Try it in KAOP](https://kaop.komodor.com/runs) **August 16, 2026** ### New: Try a change on a cloud agent without changing it **Shadow deploy now works for agents we host for you**, not just self-hosted ones. Edit a cloud agent as usual, and on the last step choose **Deploy as a shadow** instead of updating it: AgentOps deploys your edited version *beside* the live one and sends it a copy of every run — while callers keep getting only the live agent's answer. Nothing about the running agent changes. It keeps its own prompt, its own worker token and its own traffic; the shadow gets a fresh token and its own deployment, both provisioned for you. There is no command to run. Each request then produces a **comparison pair** — the candidate's run badged **Shadow** next to production's, linked both ways, with a side-by-side view in the run console. A shadow may differ from the live agent in its prompt, model, skills, image or replica count — the things that live in its own config. It shares the live agent's credentials, tool group and triggers, so those stay as they are. When you have seen enough: apply the change for real by editing the agent again and choosing **Update the main agent**, then ask the assistant to remove the shadow (or call `DELETE /api/v1/hosted-agents/{customer}/{agent_id}-shadow`). Archiving the main agent pauses both. [Try it in KAOP](https://kaop.komodor.com/agents) ### Improved: Storing a key you already have no longer makes a second copy **AgentOps now recognises a secret it already holds, whatever you call it.** Adding a credential whose value your account already stores no longer creates a duplicate — you get the existing credential back, and nothing is written. That used to be impossible to catch. The only check was on the *name*, so the same API key stored as `datadog-prod` and `dd-key` looked like two unrelated secrets. Rotating one of them quietly left the other live and still handed to whichever agents were bound to it. Two things you will notice. Adding a value that is already stored says so, and names the credential that holds it — which may not be the name you typed, so that is the one to pick in the agent wizard. And a name clash now means something sharper: the name is taken by a credential holding a **different** value, which is worth a look rather than a shrug. Local credential discovery gets the same treatment: running it a second time creates nothing, even when a teammate already stored that key under another name. Deliberately unchanged: replacing a credential's value still stores exactly what you give it, since rotating a key is not the same as merging two credentials that agents depend on separately. [Try it in KAOP](https://kaop.komodor.com/credentials) ### New: Guardrails — stop an agent from taking an action, before it takes it An agent that can call a tool can call it in production. Until now the only ways to prevent that were to withhold the tool from the agent entirely, or to find out afterwards from the audit trail. **Guardrails** are the rule in between: name some tools, optionally name which agents and under which arguments, and a matching call is refused at the gateway *before* it runs. The agent is told which rule refused it, by name, so it can escalate to a human instead of retrying blindly. The rule most people want is not "never delete pods" — it's "not **in production**". So a guardrail can condition on the call's arguments: `k8s_delete_pod` stays available for staging while the production call is refused. Conditions compare on JSON text, so `0` and `"0"` mean the same thing to a rule, and a call that omits the argument entirely is not covered. Authoring is a picker, not a syntax. **Guardrails** lists your real tools, grouped by server, so nothing has to be typed from memory — and if you do write a glob by hand, it tells you how many of your tools it currently matches. A pattern that matches nothing is the mistake worth catching: it produces a rule that looks active and protects nothing. Or describe the rule and let the co-pilot draft it. The form sits beside a conversation: say *"stop my agents deleting anything in production"* and it fills in the tools, the targeting and the conditions from your account's real catalog, for you to check and save. It never submits — the rule exists when you press the button. Targeting one particular agent is a pick from a list too, by the name you gave it. The rule stores the internal id underneath, because that is what the gateway checks a call against — but nobody should have to know that, or type it. Every block is audited — which guardrail, which tool, which agent — and the page shows recent blocks, so "did my rule actually fire?" has an answer on the same screen where you wrote it. Managing guardrails needs the `guardrail.manage` capability; it is an admin verb, because a guardrail is a control over other people's agents. [Try it in KAOP](https://kaop.komodor.com/guardrails) ### Improved: Editing a self-hosted agent now tells you what each change will actually do The Edit wizard could tell you a field wasn't editable, but not much else. It now explains itself at each step, and the final screen says what your edit will do before you commit to it. **The Review step lists your changes, grouped by when they take effect.** Some of what this wizard saves is live the moment you continue — a schedule or webhook is a Komodor-side resource and starts firing straight away. The rest only reaches your agent when you run the generated `helm upgrade` in your own cluster. The button now reads **Save and generate command**, because that is the step that writes; the command screen after it is just the part we cannot run for you. **Switching a trigger to Manual does not delete it.** This wizard only creates and updates triggers, so an existing schedule keeps firing after you switch away from it. The step now says so, and points you at Automations to remove it. **Agents built from your own image are recognised more reliably.** An agent whose `agent-spec.yaml` doesn't set a version was previously reported as one we couldn't classify, so the wizard declined to say anything about it — even though its repository and source path were right there on its card. Most agents don't set a version, so this affected a lot of them. They're now identified from where their spec came from. **Instructions you can't edit are still shown.** When your agent's prompt lives in a baked `agent-spec.yaml`, the field used to be locked *and* empty. It now shows the instructions the running agent is reporting, so you can see what it's actually doing while editing it stays a repository change. Smaller things: the MCP step's controls are genuinely disabled when they can't be changed rather than only looking that way; the agent id field no longer shows a validation error for an identifier you can't edit; and the two overlapping "we don't know much about this agent" notices are now one. [Try it in KAOP](https://kaop.komodor.com/agents) ### Fixed: Editing an agent no longer drops the MCP servers you installed yourself An agent you created over the API — rather than through the wizard — could be edited here in two ways that quietly did the wrong thing. **Your own MCP servers survive an edit now.** If you picked the **group** MCP mode, the generated `helm upgrade` wrote the AgentOps gateway into `agent.mcpServers` — the same list your original `helm install` used for your own servers. Helm replaces lists rather than merging them, so that upgrade removed every server you had configured there, and nothing in the wizard said so. AgentOps never sees those servers' definitions. Their transport, URLs, headers and environment stay in your cluster deliberately — a free-form environment block would otherwise copy your secrets into our database — so the wizard cannot list them back to you, and it should never have written a list it could not see. It now writes only its own entry, under `agent.managedMcpServers`, and the chart merges the two. Your list is left alone. This needs the **agentops-agent-base** chart at 0.1.5 or newer, which the generated command already points at. **A bring-your-own Anthropic key can no longer be skipped.** Opening a BYO agent showed an empty key field with **Next** still enabled, so you could walk through to Review and produce an upgrade command carrying no key. The field is empty because the key goes straight into your cluster's Secret and AgentOps keeps no copy of it — it was never lost. The wizard now says exactly that, waits for you to type it in, and puts what you typed into the command instead of discarding it. An agent that never recorded how it authenticates is left alone rather than being asked for a key it may not use. [Try it in KAOP](https://kaop.komodor.com/agents) ### New: Create an agent and its trigger in one call **`POST /api/v1/agents` now takes `triggers` and `autoscaling`**, so an agent can be created fully configured without a wizard and without follow-up calls. Scripts and the assistant reach the same field set — the MCP `manage_agent` create tool shares it. ```jsonc theme={null} { "agentId": "release-bot", "instructions": "You cut releases and announce them in #releases.", "credentialRef": "LLM_PROXY_TOKEN", "triggers": [{ "type": "schedule", "name": "nightly", "cron": "0 9 * * 1" }], "autoscaling": { "keda": { "enabled": true, "maxReplicaCount": 5 } } } ``` **Name a schedule and re-posting edits it.** The name is that schedule's identity on the agent, so sending the same create again with a different cron updates it instead of adding a second one — and an identical retry changes nothing rather than firing your agent twice a tick. **A trigger that fails is never reported as made.** The agent is created before its triggers are, so if one fails you get **207** instead of 200, with the real error and the endpoint that creates just that trigger. Retry the trigger, not the create — re-running the create rotates the agent's worker token and would invalidate a `values.yaml` you have already applied. A webhook returns its URL path, header and one-time token in the same response, so it is usable immediately. `webhookType: "github"` is supported too, given a signing secret. **Autoscaling here is what you declared, not what your cluster is doing.** It is recorded on the agent and written into the Helm values you install, but AgentOps cannot see your cluster — whether KEDA is installed, or whether it is actually scaling anything. In the Add-Agent wizard, the **Triggers** step now tells you when you don't have permission to create one, instead of accepting a schedule the deploy would go on to refuse. [Try it in KAOP](https://kaop.komodor.com/agents) ### Fixed: An archived draft agent stays archived Archiving a draft agent now actually retires it everywhere. Because a draft never sends a heartbeat, and an agent's status is derived from its last heartbeat, an archived draft kept reporting itself as a draft forever — so four screens went on treating it as unfinished work long after it had been deployed and archived. Most visibly, the Add-Agent wizard offered it back as **"You have an unfinished agent"**, and its **Continue setup** button really did deploy it a second time from a configuration that had already shipped. The Fleet's Archived tab linked those rows straight back into the same wizard, and an archived draft's own page told its operator to go and deploy it. Now the offer only ever names a draft you can genuinely still finish, archived rows in the Fleet open their agent page like every other archived agent, and that page says plainly that the draft was archived — with no setup link to follow. Unarchive it from the Fleet if you do want to pick it back up. [Try it in KAOP](https://kaop.komodor.com/marketplace) ### Fixed: A chat waiting on a busy agent now says so, instead of looking stuck Asking an agent a question while it was already working on another run left the chat spinning with no explanation, and after five minutes it gave up with what looked like a network error. The run was fine the whole time — it was queued, waiting for the agent to free up, and nothing said so. **The wait is now visible.** While your question is queued you see "Waiting for a free worker", with how many runs are ahead of it when there are any — counting down as they finish. The notice clears the moment a worker picks your question up, and the reply streams in as normal. **And waiting your turn no longer fails the turn.** The timeout that ended those questions exists to catch a run that has genuinely stalled, but it was counting the queue wait against the same budget — so a healthy question that simply had to wait was reported as failed. Being picked up by a worker now resets that budget, so the time your question spent queued is no longer held against the answer. One case is still open: a question that waits more than five minutes for its turn is still given up on, so if the run ahead of yours takes that long you will still see an error. [Try it in KAOP](https://kaop.komodor.com/chat) **August 14, 2026** ### Fixed: The Edit wizard now tells you which changes can actually reach your agent Two agents can both be self-hosted and behave oppositely. One built from the generic **agentops-agent-base** image takes its instructions, model, capabilities and MCP servers from the config the chart writes — so editing them here and running the generated `helm upgrade` genuinely changes how the agent behaves. One built from your own image carries its own `agent-spec.yaml` and never reads that config at all. The wizard used to present both the same way. Editing the prompt of an agent in the second group produced a success message, a values.yaml, a helm command, and no change in behaviour whatsoever — plus a **Redeploy pending** badge in Fleet that nothing could ever clear, because the running agent was never going to move toward what you saved. Now the wizard works out which case your agent is in and says so: * Fields your image owns are shown **read-only**, with the reason and a pointer to the repository whose `agent-spec.yaml` actually defines them. Change them there and rebuild the image. * What still applies from here stays editable — the display name, triggers, replicas and scaling all reach your agent regardless of how it was built. * Fleet distinguishes **Redeploy pending** (you have changes to deploy, and the badge clears once you do) from **Spec mismatch** (your image disagrees with the saved config, and no redeploy will reconcile it). * Values your worker will ignore are no longer recorded as if you had chosen them, so the badge that could never be cleared can no longer appear. When your agent hasn't reported enough for us to tell which case it is, nothing is locked and no promise is made either way — you get a note saying so rather than a guess. [Try it in KAOP](https://kaop.komodor.com/agents) ### Fixed: Model lists come from the provider, not from a built-in list Adding models to a provider you configured earlier used to show a short built-in list for that provider kind, because the key you had saved could not be read back to ask with. The list looked like an answer, but it was the same handful of ids for everyone — including, for Bedrock, models AWS has since retired. AgentOps now asks the provider itself, using the key you already gave it: Bedrock (foundation models and inference profiles), Gemini, Cohere, Azure deployments, and every OpenAI-compatible provider answer with what your key actually serves. When a list still cannot be confirmed — a provider with no list to ask for, or a credential saved before this change — the page says so instead of passing the built-in list off as live. Saving that credential's key once makes it live. [Try it in KAOP](https://kaop.komodor.com/providers) ### New: Jump to any page with ⌘K Press **⌘K** (or **Ctrl+K**) anywhere in the app to open the command palette: type a few letters, hit Enter, and you're on the page — Members, API keys, Quality Lab evaluators, an Integrations section — without reaching for the sidebar. It searches everything you can navigate to, including sub-pages and tabs, understands aliases like *token* for API keys or *slack* for channels, and keeps your five most recent jumps one keystroke away. There's also a **Search…** button in the header if you prefer the mouse. **August 13, 2026** ### Fixed: A trigger set in the Add-Agent wizard is now created when you deploy The Triggers step used to collect your cron and drop it: the agent deployed, the wizard reported success, and no schedule existed until you reopened the agent and added it again. Now the trigger is created together with the agent, for cloud and self-hosted deploys alike — a schedule fires from day one, and a token webhook is created with its endpoint URL and one-time inbound token shown right on the review step. If the agent deploys but its trigger can't be created, the wizard says exactly that — the agent exists, the trigger doesn't, and why — with a one-click retry. No more silent success. One deliberate exception: a GitHub-type webhook needs a signing secret the wizard doesn't collect, so instead of creating an endpoint that would reject every delivery, the wizard points you to **Integrations → Endpoints** to finish it with the secret after deploy. [Try it in KAOP](https://kaop.komodor.com/marketplace) ### Improved: Reuse a model credential you already stored, whatever you named it Deploying a catalog worker no longer asks you to paste a key the account is already holding. The Credentials step used to look for a credential named exactly what the worker expected, so a key you had stored as `anthropic-prod` read as **Missing** and the only way forward was to paste it again under the expected name — leaving two copies of the same secret to rotate. Now that step offers the credentials your account already has, and picking one answers the requirement: no second copy, and no renaming. Adding a brand-new credential is still there, presented as the last resort rather than the only option. If you don't have permission to list the account's credentials, the step says so instead of showing you an empty list with no explanation. [Try it in KAOP](https://kaop.komodor.com/marketplace) ### Improved: Search for a credential or timezone instead of scrolling and typing Creating an agent no longer means hunting through an unsorted credential list — the model step's credential picker now has a search box and lists credentials alphabetically with their descriptions. The Triggers step's timezone field is a searchable picker too, rather than a text box you had to spell an IANA name into by hand: it shows every zone with its current UTC offset, puts your own zone and UTC at the top, and finds a zone by offset (`+05:30`) or by its modern name as well as its older one. Both changes also apply when you edit an existing agent, and the same timezone picker now backs the schedule forms in Fleet. [Try it in KAOP](https://kaop.komodor.com/marketplace) ### Fixed: History search now looks at your whole workspace, not just the newest page Searching History used to filter only the 100 most recent sessions the page had already loaded, so a run or chat older than that came back as "No matching history" — for something that was plainly still there. Search now runs against the whole workspace, and matches on the run or session id, its title or prompt, and the name of the agent behind it, so an agent's name finds its work however far back it goes. History is also honest about the page now: when the list is showing as much as it can, it says so instead of quietly stopping, and the tiles above it (Runs, Active, Run cost) report the last 30 days across the workspace rather than a tally of the rows on screen. [Try it in KAOP](https://kaop.komodor.com/runs) ### Fixed: Editing an agent no longer creates a GitHub webhook that rejects every delivery Choosing **Webhook → GitHub** in either agent edit wizard used to create the endpoint immediately, without a signing secret. GitHub deliveries are verified against that secret, so every one of them was rejected — while the trigger row existed and Fleet showed a configured webhook. The only visible symptom was an agent that never ran, which reads as a broken agent rather than a webhook that was never valid. The edit wizards now behave like the Add-Agent wizard: a GitHub webhook isn't created there at all, and the step says why. Create it under **Integrations → Endpoints** instead, which collects the signing secret and routes the endpoint to your agent. Everything else about the edit saves normally, and static-token webhooks are unaffected. If you already have one of these, you can spot it: **Integrations → Endpoints** now marks a GitHub endpoint with no signing secret with a **No signing secret** badge. Fix it in place — edit the endpoint and add the secret. [Try it in KAOP](https://kaop.komodor.com/integrations) ### Improved: A searchable agent picker for eval rules Setting up an eval rule used to mean scanning an unlabeled wall of pills for the agent you wanted. The Targets step now has a searchable, alphabetically-sorted list instead — type a few letters to find an agent, and the count of what you've picked stays visible with a one-click Clear. The step also drops the "Run filter" control: every rule now evaluates successful runs, which is what every existing rule already did. If the agent you picked has never had a successful run, the review step says so before you activate the rule. [Try it in KAOP](https://kaop.komodor.com/quality-lab/evaluators) **August 12, 2026** ### Fixed: Create agent always starts fresh — and your draft is still there if you want it Clicking **Add agent** now always opens a clean wizard, so you can start a second agent, or switch between building from scratch and deploying from the catalog, without the previous attempt following you in. Your unfinished work is not thrown away: it is offered as a "Continue where you left off" card on the start screen, naming the agent and the step it reached, with a Discard beside it. And if you simply reload the page mid-wizard, you land straight back where you were — no card, no click. [Try it in KAOP](https://kaop.komodor.com/marketplace) ### Improved: Create agent asks where it runs up front, and only shows the steps that apply The agent wizard now asks where the agent will run at **step two**, right after you name it — your own cluster, or Komodor cloud. Previously the question waited until the very last screen, which meant you could work through **Scaling** and set up KEDA autoscaling for an agent you then chose to have Komodor run, where those settings do not apply and were quietly dropped. Answering up front shapes the rest of the wizard: pick Komodor cloud and the Scaling step is not there at all, because there is nothing for you to scale. Pick self-hosted and it is, as before. The Review screen no longer asks a second time — it opens straight on the deploy path you chose. Catalog workers get the same question when they can run in both places, and skip it when their image only runs in one. Drafts already in progress are unaffected: an unfinished agent resumes on the step you left it on, still set to run in your own cluster. [Try it in KAOP](https://kaop.komodor.com/marketplace) **August 11, 2026** ### Fixed: Change which specialists an orchestrator and an incident workflow use An orchestrator's delegatable workers are now shown on its page in Fleet, and you can change them there — previously the list was chosen at deploy time and could never be seen or edited again, so an orchestrator could not gain a new specialist after launch. Incident workflows follow the same rule as code review workflows: pause one and it becomes editable, so you can add a specialist to a workflow that has already gone live. [Try it in KAOP](https://kaop.komodor.com/fleet) **August 10, 2026** ### Improved: See the impact and cost of your code review program Code Review now opens with a 7-, 30-, or 90-day view of review volume, open findings, spend, webhook coverage, and performance by repository and reviewer. Compare the typical first pass with incremental re-reviews, and hover the engineer-time estimate to see exactly how it was calculated. [Try it in KAOP](https://kaop.komodor.com/code-review/overview) **August 8, 2026** ### New: The AgentOps Orchestrator is now available in the agent gallery Deploy the AgentOps Orchestrator straight from the agent gallery — a lead agent that, given a problem, decides which specialists in your fleet should handle it, delegates a scoped sub-task to each, and synthesizes one combined answer. Chat with it directly or trigger it as a run. [Try it in KAOP](https://kaop.komodor.com/marketplace) ### New: The Code Review Platform is now available to every account Point a reviewer agent at your repositories and every new pull request gets an automatic review — no waitlist. Set up a review workflow, watch reviews land in the inbox as GitHub pull requests open and update, and see each reviewer's verdict alongside the PR. [Try it in KAOP](https://kaop.komodor.com/code-review) **August 6, 2026** ### New: Test a new worker version in shadow, next to production **Dual-deploy (A/B)** is here for self-hosted agents: deploy a candidate version of a worker next to the serving one, and every request runs on **both** — while callers keep getting only the production answer. The candidate can never slow down, break, or change what a user hears. Turning it on is one environment variable on the candidate deployment — same agent id and worker token, plus `AGENTOPS_VARIANT_ROLE=secondary`. No changes to the serving version. Each request then produces a **comparison pair** you can spot everywhere: * **History** — the candidate's run is tinted amber and badged **Shadow**, production's is badged **Production**, and each row links its sibling. A `pair_…` id correlates the two. * **Run console** — an amber band compares both arms side by side: status, duration, cost, and output. * **Agent page** — the Instances tab marks the candidate replica, and the Versions tab splits traffic into `production · shadow` instead of inflating the count. For a local end-to-end loop, `aops shadow` spins up two demo workers (`aops shadow start --agent demo-ab`), fires one message at both (`aops shadow trigger`), and verifies the pair ran on both versions (`aops shadow validate`). [Try it in KAOP](https://kaop.komodor.com/runs) ### Improved: Choose which agent remediates an incident workflow The **Remediation** step of the incident workflow wizard is now a choice rather than a fixed panel. Pick any remediation agent you have deployed, or **Investigate only** to stop at the conclusion and hand off to a person. The investigation itself is unchanged either way — it still produces the same evidence-backed findings. A workflow binds at most one remediation agent, because the orchestrator hands the findings to exactly one worker once triage converges. The step lists every remediation agent in your fleet, offline ones badged with their status — your pick is the only remediation worker the workflow's runs can reach, so an unreachable one means remediation does not run, never that another agent stands in for it. An existing workflow whose agent has left the fleet says so instead of failing quietly. [Try it in KAOP](https://kaop.komodor.com/n) **August 5, 2026** ### New: Describe what you want and the form fills itself Every multi-step form in the app — new agent, credential, integration provider, evaluation flow, code-review and incident workflows, incident pipeline, maintenance agent — now opens with a co-pilot beside it. Tell it what you are trying to set up in plain language and it fills the fields in, walking you to the step it changed, so you review a filled form instead of starting from an empty one. It never submits and never touches a secret field: you type the secret yourself and you press create, so nothing exists until you say so. [Try it in KAOP](https://kaop.komodor.com/credentials/new) **August 4, 2026** ### Improved: Every Slack answer names the runs behind it An AgentOps reply in Slack now ends with a footer naming each agent that contributed to the answer, with a link to its run. When the assistant delegated the question to a specialist, the footer names the specialist and the assistant that routed it — so a summarized answer is one click from the full output it was built from. The footer is written by the control plane, not by the agent, so it stays accurate even when the reply is a paraphrase. [Try it in KAOP](https://kaop.komodor.com/runs) ### New: Run the Kubernetes RCA agent inside your own cluster **Kubernetes RCA** now installs into your own Kubernetes cluster. Pick it from the marketplace and the wizard hands you a worker token and a ready-made `helm install` — the agent runs on your side, investigates the cluster it is installed in, and reports findings back to AgentOps. Nothing about your cluster leaves it except the finished analysis. This is the only way to deploy it, by design: the agent's evidence is read-only `kubectl` against the cluster it runs in, so a Komodor-cloud copy would faithfully investigate the wrong cluster. The generated values grant read-only access — a binding to Kubernetes' own `view` role plus the cluster-scoped reads it omits, such as nodes and storage classes. Secrets and pod `exec` are deliberately **not** granted; both are documented opt-ins you enable yourself if you want them. Investigating clusters connected through a Komodor agent instead? **Kubernetes RCA (Actions)** still runs on Komodor cloud and needs nothing installed. [Try it in KAOP](https://kaop.komodor.com/marketplace) ### Improved: Continue an investigation from a finished run Beneath a finished run's output there is now a **Continue this investigation** composer. Type your own follow-up, choose who answers, and press **Continue** — a chat session opens with your question already asked against that run. It replaces the old **Review in chat** button, which could only ask one fixed question of one fixed agent. Now the run's own agent answers when it supports chat; otherwise the AgentOps Assistant steps in and the composer tells you why. Leave the box empty and it simply asks what to look at next, rather than assuming something went wrong. You can also hand the investigation to **Claude Code** or **Codex**, or copy the prompt for any other coding agent. Those handoffs carry only the run's id and link — never its output, logs or traces — so nothing sensitive ends up in a desktop URL. The coding agent reads the run through AgentOps MCP using your own permissions. [Try it in KAOP](https://kaop.komodor.com/runs) ### Improved: Filter one agent's run history by status An agent's Run history tab now has the same status filter the global run ledger has, so you can pull up just that agent's failures — or just what is queued right now — without paging through everything it has ever done. The choice lives in the URL, so a refresh keeps it and a link you paste to someone else opens on the same filtered view. [Try it in KAOP](https://kaop.komodor.com/agents) ### New: See a month of an agent's runs at a glance An agent's Run history tab now carries a 30-day density strip in its header: one thin bar per day, stacked by what those runs did — blue for running, yellow for queued, red for failed, green for succeeded, grey for cancelled. Every colour is drawn on the same pixels-per-run scale, so a segment's height is a count you can compare between days, and a day with nothing at all is a flat grey tick rather than a gap. Run activity, the metrics-and-trend dashboard, has moved to a tab of its own, so the history tab is just the runs. [Try it in KAOP](https://kaop.komodor.com/agents) **August 3, 2026** ### New: Ask the AgentOps assistant to review a finished run A finished run now has a **Review in chat** button. Press it and the AgentOps assistant opens with the question already asked — it reads the run's output, logs and trace and tells you whether anything actually went wrong, and what to do about it. It is offered on every completed run, not only failed ones, because a run that could not do its work still finishes as *succeeded* — an agent that found no tools bound, an integration that was never authenticated, or a bad credential all report success. If the run did what it was asked, the assistant says so. [Try it in KAOP](https://kaop.komodor.com/runs) ### Fixed: Pausing a review workflow now actually lets you edit it Editing a review workflow that had ever been activated used to fail with "Only draft workflows can be updated. Pause the workflow first." — advice that could not be followed, because pausing left the workflow `paused` rather than back in `draft`, so the same error came back every time. A `paused` workflow is now editable, which makes pausing the real unlock it always claimed to be. An `active` workflow still cannot be edited underneath a live webhook, and the detail page now hides its Edit button until you pause, instead of offering an action that was going to be rejected on save. [Try it in KAOP](https://kaop.komodor.com/code-review/workflows) ### New: See how an agent's volume, spend and reliability moved An agent's **Run activity** now charts a metric over time instead of showing only today's totals: invocations, spend, cost per invocation, error rate, and the days it changed version. Pick a tile to plot it, and each tile carries its week-over-week move — so "it costs more this week" and "it started failing on Tuesday" are things you can see rather than infer. Version changes are marked on the chart, which is usually the first place to look when a line bends. [Try it in KAOP](https://kaop.komodor.com/agents) ### Improved: See agent outcomes, latency, cost, and changes together Agent run history now opens with successful investigations and errors stacked by day, with bar or trend views grouped by day or version. Cost shows total and per-invocation spend, latency is tracked from completed runs, and selecting a change bar opens that exact revision's field-level diff in Versions. [Try it in KAOP](https://kaop.komodor.com/agents) **August 1, 2026** ### Improved: Check connectivity now reaches the agent itself **Check connectivity** now sends a request down the agent's own connection and waits for the agent to answer, showing you what it said. Previously it started a run that was answered without ever reaching your agent's code — so it could report success for a worker that could not actually do anything. It also tells you *which* thing is wrong: nothing connected, not responding, or connected but reporting a problem of its own. [Try it in KAOP](https://kaop.komodor.com/agents) ### New: Agents can now cite your knowledge base Agents you build from scratch, and the investigators in the Komodor catalog, can now search your knowledge base while they work — the same runbooks, service docs and postmortems you uploaded on the Knowledge page. An answer that leans on one comes back with the document title and section it came from, so you can check the source instead of taking the agent's word for it. There is nothing to turn on: an agent picks this up the next time it runs on a current image. [Try it in KAOP](https://kaop.komodor.com/knowledge) **July 31, 2026** ### Improved: A review now shows every pass it made, not just the latest A pull request that gets pushed to repeatedly is reviewed repeatedly, and until now the review page only showed where things stand right now — the newest run per reviewer. A new Review history section lists every pass the review has dispatched, newest first: which revision it looked at, what it concluded, and a link to the run. Passes that produced no verdict are in the list too, labelled by why — a run that failed reads differently from one that finished but returned output the platform could not parse, and previously both simply left a gap where a verdict should have been. Selecting a revision in the history switches the reviewer cards above to that revision, so the two always agree about what is on screen. [Try it in KAOP](https://kaop.komodor.com/code-review) ### New: Connect Google Workspace and pick which products agents can use Google Workspace is now in the integrations catalog. One connection covers Drive, Docs, Sheets, Slides, Calendar, Chat, Contacts, and Gmail — you choose which of them to include when you connect, and only the scopes those products need are requested on the consent screen. Drive is selected by default; Gmail is never preselected, since it needs a restricted scope. Each selected product is registered as its own MCP server behind the same credential, so agents get exactly the Workspace tools you approved. You can come back later and add a product to an existing connection — the consent screen only asks for the new scopes. "Test connection" now makes a real read-only call to Google rather than just checking that a token is stored, so a revoked token or a Workspace API that was never enabled for your Google Cloud project shows up as a failure with the reason. [Try it in KAOP](https://kaop.komodor.com/integrations) **July 30, 2026** ### Improved: Run transcripts now read like the agent's actual session A run's transcript now shows what the agent said between its tool calls, in the order it happened — reasoning, then the tools that reasoning led to, then the next step. Previously the agent's narration was collected up and dumped as a single block of text at the end of the run, so a long run looked like a wall of bare tool calls followed by one paragraph that mentioned all of them at once. The run's Output card is unchanged: it still shows the final answer, and while a run is in flight it previews what the agent is working on right now rather than everything it has said so far. [Try it in KAOP](https://kaop.komodor.com/runs) ### Improved: See every model you can call — and what it costs — in one table Providers has moved under Connect and is now a single table of models rather than a tab per tier and a card per credential. Each row shows the provider, who manages it (Komodor or one of your own credentials), the upstream model behind it, its spend, its requests and tokens, and whether it is working — so searching and filtering covers the models Komodor manages and your own at the same time, with your gateway budget above it. Adding a provider now ends on a verify step where you can send a model a real prompt and read its reply, and Baseten and Nebius Token Factory join the list of providers you can bring. [Try it in KAOP](https://kaop.komodor.com/providers) ### Improved: Reviewers now pick up where they left off when you push a fix A reviewer used to start from a blank sheet on every push, so findings could appear and disappear between passes, ones you had already declined came back, and a fix was never acknowledged. A re-review now carries the reviewer's own previous findings: it works out which ones your new commits fixed, keeps the rest open unchanged, and reviews the new commits in full — widening beyond them whenever the change warrants it. Each review reports the ledger — what is new, what is still open, what you resolved — and a fixed finding gets answered on its own comment thread instead of only in a fresh summary you have to cross-reference. A push that changes no content at all, such as an amend or a rebase in place, no longer produces a second review of identical code. Hitting Re-review still gives you a full pass from scratch. [Try it in KAOP](https://kaop.komodor.com/code-review) **July 29, 2026** ### Improved: Configure a whole Slack routing rule from the agent spec A `slack_channel` trigger in `agent-spec.yaml` now carries the whole routing rule, not just the channel list. Pick what the agent answers with `rule` — `mention` (the default), `channel` for every human message, `keyword` for messages containing a term, or `private` for direct messages — add `keywords`, opt into other apps' posts with `allow_bots: true`, and order the route against your other rules with `priority`. Declaring it in the spec is enough; no follow-up step on the channel page. [Try it in KAOP](https://kaop.komodor.com/channels) ### New: Pick one of your own providers when you create an agent The new-agent wizard no longer offers a fixed Anthropic model list. Alongside the models Komodor manages, it now shows the provider deployments you registered on Providers, under your own names, with the provider kind beside each one. A deployment you cannot call yet is still listed but not selectable, and says why — it is still verifying, or your account routes straight to Anthropic instead of through your providers. The model you choose is checked again when the agent is created, so a deployment that stops being reachable is refused at creation rather than at the agent's first run. [Try it in KAOP](https://kaop.komodor.com/agents/new) **July 28, 2026** ### Improved: Page through the reviews inbox instead of scrolling all of it The Reviews inbox now shows 10 reviews per page instead of every review at once, so the page stays a readable length as your review history grows. Pick 25, 50, or 100 per page if you'd rather see more, and your page and page size stay in the URL — so a refresh keeps your place and a copied link opens on the same page. Narrowing the list with a status/repo filter or the search box takes you back to the first page so you never land on an empty one. [Try it in KAOP](https://kaop.komodor.com/code-review) ### Improved: Connect AWS Bedrock with a Bedrock API key Adding a Bedrock model no longer forces you to paste an IAM access key pair: a single AWS Bedrock API key now works on its own, with the access key pair, session token and runtime endpoint left optional beside it. Each field explains what it is for, and Providers tells you up front that a Bedrock credential needs either the API key or a complete access key pair before it will save. [Try it in KAOP](https://kaop.komodor.com/providers) **July 27, 2026** ### Fixed: Open a review workflow to see its status, not the setup wizard Clicking a review workflow used to relaunch the setup wizard, landing on the last step every time. It now opens a read-only detail page showing status, repositories, webhook health, and reviewers at a glance — with an explicit Edit button when you actually want to change something. [Try it in KAOP](https://kaop.komodor.com/code-review/workflows) **July 26, 2026** ### Improved: Search reviews, track revisions, and see reviewer cost Find a review instantly by searching a PR number, title, author, or repo. When a PR gets pushed to again, switch between revisions on the review page to see what each reviewer said about that specific commit, and a "Merged" badge now shows alongside merged PRs so you don't mistake them for a plain close. Reviewer cards on the Reviewers tab also show 30-day spend, and review detail now renders full-width for easier reading. [Try it in KAOP](https://kaop.komodor.com/code-review) **July 25, 2026** ### Improved: Filter the fleet by team, tier, or any other label Fleet now lists your agents as a scannable table — name, status, last run, last change, and instances — with the filter sidebar every other platform page uses. Each label key you use (team, tier, env, role, …) becomes its own multi-select category ordered by how many agents carry it, so you can narrow to `team: sre or team: platform` and `tier: production` in two clicks and share the resulting URL. Credentials filters the same way. [Try it in KAOP](https://kaop.komodor.com/agents) **July 24, 2026** ### New: Quality Lab — evaluate agent runs with judge panels Quality Lab is now an evals platform. Set up an **eval rule** — pick which agents' runs to evaluate, how often to sample them, and a **panel of judges** (any agent with the grader capability) — and every matching run is graded automatically. Browse the verdicts in the **Evals** inbox, open a run to read each judge's score and reasoning side by side, and see the **Judges** roster and a single quality **score per agent** on the Agents tab. Human star ratings sit right alongside the judges' verdicts. [Try it in KAOP](https://kaop.komodor.com/quality-lab) **July 22, 2026** ### New: Uninstall managed agents from the Fleet Agents you deployed from the catalog now show a **Managed** badge in the Fleet, and you can tear one down without leaving the product. Archive the agent, then use **Uninstall** on its detail page — this decommissions the deployment and revokes its worker token. No more agents lingering in the deployment after you archive them. [Try it in KAOP](https://kaop.komodor.com/agents) ### New: Deploy catalog workers to your own infrastructure Catalog workers like AWS Investigator can now be deployed self-hosted — on your own Kubernetes cluster instead of Komodor cloud. The marketplace review step generates a worker token and a ready-to-use Helm install command with your credentials. Choose between bringing your own Anthropic API key or using the managed LLM proxy. [Try it in KAOP](https://kaop.komodor.com/marketplace) ### New: Pause, resume, and delete code review workflows from the list Each review workflow now has inline actions on the Review workflows list. Pause an active workflow to stop it from reviewing new pull requests, resume a paused or draft one to provision its webhooks again, and delete a workflow you no longer need. Delete only appears once a workflow is paused (or still a draft), so an active workflow can't be removed out from under in-flight reviews. [Try it in KAOP](https://kaop.komodor.com/code-review/workflows) **July 20, 2026** ### New: Automate incident response with alert-triggered workflows Create an incident pipeline that wires any alert source — Datadog monitors, AlertManager, or a generic webhook — through routing rules to an orchestrator and its specialist team. Each firing alert becomes an incident, gets deduplicated and severity-mapped, and dispatches an investigation automatically. [Try it in KAOP](https://kaop.komodor.com/incidents) ### New: Prove a new agent works the moment it's deployed Every agent now ships with two built-in self-check commands. Use **Check connectivity** for an instant, model-free confirmation that a freshly deployed agent is reachable, or **Sample run** for a real run through the model — so you can prove a new agent works in one click, right after deploying it. [Try it in KAOP](https://kaop.komodor.com/agents) **July 19, 2026** ### New: See what shipped recently, right here AgentOps now has a What's New page — this one — with everything recently shipped, newest first. Watch for the badge on "What's new" in the sidebar: it counts the updates you haven't seen yet and clears when you visit. [Try it in KAOP](https://kaop.komodor.com/whats-new) ### New: Knowledge base is now a real wiki The Knowledge page now reads like a wiki: pages render as formatted Markdown with working links between documents, a navigation tree grouped by document type, and a Raw view of the original file. Drop several files at once anywhere on the page to upload them in one batch — OKF frontmatter (title, type, tags, owner, source) is picked up automatically, and you can watch indexing progress live. [Try it in KAOP](https://kaop.komodor.com/knowledge) **July 16, 2026** ### New: Cancel runs that are queued or in flight You can now cancel a run from History or the run detail page — whether it is still queued, claimed, or already executing. Cancellation interrupts the worker itself rather than just marking the row, so a runaway run stops doing work (and spending tokens) the moment you cancel it. [Try it in KAOP](https://kaop.komodor.com/runs) ### Improved: Search and filter the audit log The audit log now supports free-text search across actors and actions, plus a status filter, with server-side sortable columns. Finding who did what — and whether it succeeded — no longer means paging through the whole ledger. [Try it in KAOP](https://kaop.komodor.com/settings/audit) # Limits & quotas Source: https://docs.komodor.com/reference/limits-quotas The caps, sizes, timeouts, and page limits KAOP enforces — what each one is, and what happens when you reach it. The Komodor Agentic Operation Platform (KAOP) enforces the limits below and returns a specific status at each one. Some are firm; several are adjustable per account, and where a limit is a default you can be given more of, it says so. > [!NOTE] > > **There is no rate limiting.** No endpoint throttles, and nothing in KAOP returns `429`. If a `429` > reaches you it came from an upstream provider one of your agents called. Quota exhaustion is always > `409`; an oversized payload is always `413`. ## Agents | Limit | Value | Notes | | - | - | - | | Active agents per account | **60** | Adjustable per account. | The cap is enforced when an agent is created, so you get an immediate, explicit error rather than discovering the limit later: ``` account agent limit reached (maximum 60); archive or delete an agent before creating a new one ``` returned as **`409`**. Three things soften it: * **Archived agents do not count.** Archiving frees a slot without deleting the agent's history, so the way to make room is to archive rather than delete. * **Reusing an existing agent does not consume another slot.** Deploying again under a name that already exists reuses that agent — it rotates its token and redeploys, and the active-agent cap does not block it. Authentication, permissions, and validation still apply. * **The check is at create time only.** Nothing in the live registration or presence path is capped, so an agent at the boundary keeps running normally. The cap applies to every creation route: creating an agent, minting a worker token, and creating a draft. ## Runs Runs have no count limit. What they have are timeouts, and each one exists to stop a different way a run can hang. | Limit | Default | What it does | | - | - | - | | Lease window | **60 seconds** | How long a claim on a queued run is held before it can be re-claimed. A run that sits past it unclaimed is what **Stuck unclaimed** reports. | | Presence window | **90 seconds** | Nothing heard from a worker for this long and its agent reads offline. | | Maximum queue wait | **1 hour** | A run nothing can serve is failed rather than queued forever. Adjustable per account. | | Queue ceiling for batch work | **12 hours** | Schedule-driven and agent-to-agent runs get a longer ceiling when a healthy worker is simply busy. Adjustable per account. | | Offline-agent grace | **120 seconds** | How long a run waits before failing fast when its agent is already known to be offline, a draft, or failed to deploy. Adjustable per account. | | Absolute run wall clock | **24 hours** | A claimed or running run is failed at this point regardless. Adjustable per account. | > [!WARNING] > > The wall clock is a backstop, not a design target. An agent that legitimately needs hours should > report progress rather than run silently — a run that produces nothing for hours is > indistinguishable from a hang, to you and to the platform. ### Per-worker concurrency A worker runs **one** run at a time by default. Raise it with `AGENTOPS_MAX_CONCURRENT` on the worker. ```bash theme={null} AGENTOPS_MAX_CONCURRENT=4 ``` A value of zero or less is clamped to one rather than rejected — zero does not mean unlimited. There is no account-wide concurrent-run cap; capacity is however many workers you run. An SDK-based worker also stops a single run after **100** model calls, which bounds a reasoning loop that will not terminate. ## Knowledge base | Limit | Value | Response at the limit | | - | - | - | | Document size | **10 MB** | `413` — "Document exceeds 10 MB limit" | | Files per upload batch | **20** | `413` — "Batch exceeds 20-file limit" | | Documents per knowledge base | No cap | — | Ingested formats are PDF, Word (DOCX), Markdown, and plain text. Semantic search additionally requires an embedding provider configured for your deployment; until one is, nothing can be indexed or retrieved — see [Troubleshooting](/reference/troubleshooting). ## Payload sizes | What | Limit | | - | - | | Run artifact content | **6 MB** (`413`) | | Worker snapshot payload | **6 MB** | | Evidence batch shipped by a worker | **768 KB** per batch | | Inbound endpoint sample body captured for inspection | **8 KB** per sample, **50** samples retained | ## MCP Gateway and OpenAPI surfaces | What | Limit | Configurable | | - | - | - | | Operations parsed from one OpenAPI document | **1,500** | Yes, by environment variable | | Operations you may enable from one document | **200** | Yes, by environment variable | | OpenAPI document fetched over the network | **5 MB** | Yes | | OpenAPI document stored | **2 MB** | Yes | | Generated tool definitions per document | **8 MB** | Yes | | Response returned from one OpenAPI-backed tool call | **1 MB** | Yes | | Credential vends per run | **5** | No | Tool output returned to an agent is truncated rather than rejected: long text is cut at a fixed length, collections are capped, and nesting is bounded — so an enormous response degrades into a usable one instead of failing the run. ## Credentials The limits are on a credential's shape, not on how many you may have: | Limit | Value | | - | - | | Keys in one multi-key credential | **64** | | Key name length | **128** characters | | Value length | **10 KB** per key | | Total credential size | **64 KB** | There is no cap on the number of credentials in an account. ## Memory | Limit | Value | | - | - | | Memory title | **200** characters | | Memory body | **4,000** characters | | Memories created in one batch | **200** | | Archived memory retention | **90** days | | Page size | **200** maximum; 100 by default | | Maximum offset when paging | **10,000** | | Statistics window | **90** days | ## Triggers and schedules A cron expression is validated for syntax when you save it, and rejected with `422` and the expression quoted back if it is invalid. A cron-and-timezone pair with no single equivalent in UTC — a daylight-saving edge case — is rejected the same way. > [!NOTE] > > **No minimum interval is enforced**, so a once-a-minute schedule is accepted. One minute is the > practical floor: schedules are driven at minute granularity. ## API keys and sessions | Limit | Value | | - | - | | API key expiry | **Optional and caller-supplied.** No default expiry and no maximum lifetime — a key created without one never expires. | | API keys per account | No cap | | Browser session lifetime | **24 hours** | An expiry must be an ISO 8601 timestamp, must carry a timezone offset, and must be in the future; each of those is a `400` with an explicit message. > [!WARNING] > > A key with no expiry never expires. Set one deliberately on anything long-lived, and see > [API keys](/security-and-governance/identity-and-access/api-keys) for rotation. ## Reading and paging | Endpoint family | Default page size | Maximum | | - | - | - | | Runs | 100 | **500** | | Chat sessions | 100 | **500** | | Quality and evaluations | 50–200 | **500** | | MCP Gateway listings | 100 | **500** | | Notification sinks | 50 | **500** | | Audit log | 100 | **200** | | Memory | 100 | **200** | Two related bounds: a free-text search parameter is capped at **200** characters, and a bulk lookup by run id accepts up to **500** ids. ### Analysis windows Every endpoint that takes a `window_days` parameter — fleet activity, spend, quality, insights, per-agent observability — accepts **1 to 365** days and defaults to **30**. ```bash theme={null} GET /api/v1/runs/activity/account?window_days=90 ``` The authorization decision log is the exception: it is queryable over a window of up to **720 hours**, which is 30 days. ## Workflows | Limit | Value | | - | - | | Maximum duration of a step waiting on a human decision | **30 days** | | Condition nesting depth in a workflow definition | **32** | | Failure-reason text stored on a step | **2,000** characters | ## Asking for more Several limits above are adjustable per account — the agent cap and the run timeouts in particular. If one is blocking you, work out which figure you are actually hitting first (a `409` on agent creation is the cap; a `413` is a payload; a `422` is validation), then see [Support](/reference/support). ## Next steps > **[Troubleshooting](/reference/troubleshooting)** > > What each status code means when you hit one. > **[Support](/reference/support)** > > Requesting an adjustment, and what to include. > **[Agent spend attribution](/manage-your-agents/optimize/agent-spend-attribution)** > > Budgets — the one limit you set yourself. # Support Source: https://docs.komodor.com/reference/support How to get help with KAOP — who to contact, what to include in a report, and the resources that resolve common issues. Contact your Komodor team for help with troubleshooting, account access, or configuration questions about the Komodor Agentic Operation Platform (KAOP). When reporting an issue, share what happened, when it occurred, and any relevant error messages. If a run was created, include its run id to help us investigate. Share whatever details you have — you do not need to complete every item below before contacting us. ## Contact support Reach the Komodor team whichever way suits the question: > **Chat** > > The chat launcher at the bottom of this page opens a conversation with Support Engineering. > Best for a quick question, or when you are mid-incident. > **[Open a ticket](https://help.komodor.com/hc/en-us/requests/new)** > > The support request form. Best when you have logs or screenshots to attach, or the detail > below to hand. > **[Email](mailto:support@komodor.com)** > > `support@komodor.com` reaches the same queue as the form, and is the one to use if you would > rather just write. > **Your Komodor contact** > > The person or channel that onboarded your team, if you have one. You do not need to find them > to get help. Any of these is the right route for: * Troubleshooting an issue you have run into. * Account, access, and membership questions. * Account-level data export or deletion requests. * Enabling a module that is off for your account. * Raising an adjustable limit. * Configuring an embedding provider so knowledge search can index and retrieve. * Deployment questions about where agents and the control plane run. ## Explore troubleshooting resources These resources may help you resolve common issues. If you need assistance, we are here to help. > **[Troubleshooting](/reference/troubleshooting)** > > Offline agents, `401` responses, empty knowledge search, triggers that do not fire, failing > runs. > **[Limits & quotas](/reference/limits-quotas)** > > Whether you have hit a cap. If agent creation returns a `409` error, check the response message > and the Limits & quotas page for guidance. > **[The run's own evidence](/manage-your-agents/run/runs-evidence)** > > A failing run records what it received, which tools it called, and how each call ended. > [!TIP] > > For fleet-shaped questions — "what is failing lately", "which agent is burning the most budget" — > ask the built-in assistant in **Chat**. It answers against your own account, so you get your > numbers rather than a general explanation. ## What to include in a report Sharing a few details helps our team understand the issue and investigate. Include any of the following details you have available. You can still contact us if you cannot find them. ![Seven details to include in a support report, and where to find each. 1. Run id, if a run was created: the run's detail page, and the monospace meta line on any card that references it; a run id resolves to the full evidence trail for whatever went wrong, so it is worth including whenever there is one. 2. Agent id: the agent's detail panel in Fleet. 3. Account id: Settings then Account, shown as acct\_… alongside your account name and slug. 4. Timestamp and timezone: when you saw the behavior; runs are timestamped, so this narrows the search immediately. 5. What you expected, and what happened: describe what you expected to happen and what you observed instead. 6. The exact error: copy the message verbatim, including any status code; KAOP shows failure reasons as reported rather than rewriting them, so the raw text is meaningful. 7. Whether it is reproducible: once, intermittently, or every time — and if every time, the smallest sequence that reproduces it.](https://mintcdn.com/komodor-test/rTYNyhAk9hRWRFnt/figures/fig-support-what-to-include.png?fit=max\&auto=format\&n=rTYNyhAk9hRWRFnt\&q=85\&s=194b1a06389132000b19828703ac68d6) > [!WARNING] > > Do not paste credential values, API keys, or worker tokens into a report. KAOP never needs the > secret to investigate — the credential's name and the agents bound to it are enough. If you believe > a secret has been exposed, rotate it first and say so in the report. ## Common support scenarios > **A run produced the wrong answer** > > Send the run id and say what the right answer would have been. A wrong answer is a quality > question, not an availability one — grading it under > [Evaluations](/manage-your-agents/optimize/evaluations) captures the judgment durably and gives > the report something concrete to point at. > **An agent will not come online** > > The offline-agent section of [Troubleshooting](/reference/troubleshooting) covers the common > causes: the worker process, its outbound access to the control plane, its token, and its agent > id. If none of those explain it, send the agent id and the worker's own startup logs. > **A tool or integration call keeps failing** > > Send the run id and the name of the integration. The run's evidence records the failing tool > span with the upstream error, which usually identifies whether the problem is the credential, > the permission, or the upstream system. > **Cost looks wrong** > > Say which figure looks wrong and over what window. Spend is built from what each run reports, so > the answer is often that some runs carry no usage metadata — see > [Agent spend attribution](/manage-your-agents/optimize/agent-spend-attribution) for how a run's > cost is classified. > **A permission decision surprised you** > > Send the actor, the action, and the resource. Access decisions are recorded, so a denial that > looks wrong can be traced to the rule that produced it — see > [Audit log](/security-and-governance/controls/audit-log). ## Request configuration changes Some things are configuration rather than defects, and they are handled the same way — through any of the routes above: * **A higher limit.** Several caps are adjustable per account. For a limit increase, include the limit you would like adjusted and your use case, if known. See [Limits & quotas](/reference/limits-quotas) for the enforced caps. * **A module turned on.** Module enablement is per account; see [Modules → Overview](/modules-and-workflows/overview) for what is available. * **An embedding provider.** Semantic search over your knowledge base needs one configured for your deployment. ## Next steps > **[Troubleshooting](/reference/troubleshooting)** > > The common failures, with the checks that resolve them. > **[Limits & quotas](/reference/limits-quotas)** > > The enforced caps and what happens at each one. > **[What's new](/reference/changelog)** > > Whether the behavior you are seeing changed recently. > **[Runs & evidence](/manage-your-agents/run/runs-evidence)** > > How to read the trail a run leaves behind. # Troubleshooting Source: https://docs.komodor.com/reference/troubleshooting Find help with agent connectivity, authentication, credentials, integrations, triggers, and runs. Use the sections below to find the issue you're experiencing in the Komodor Agentic Operation Platform (KAOP) and work through the recommended checks. Run details, error messages, and agent health information can help you identify where to start. > [!TIP] > > For recurring failures, check [Insights](/manage-your-agents/optimize/insights) first. It groups > related failures and includes the reported failure reason, which may help you identify a common > cause. ## An agent appears offline **What you might see:** an agent appears as **Offline** in **Fleet**, or remains in draft after you start its worker. An agent's status reflects its worker's connection to KAOP. The control plane checks the connection every **30 seconds** and marks the agent offline after **90 seconds** without a response. A draft agent becomes live when its worker first connects. Work through these checks: **1. Check that the worker is running** Review the worker's startup logs for errors. If startup fails before the worker connects, the agent may remain in draft. **2. Check connectivity to KAOP** Confirm that `AGENTOPS_URL` points to your account's host and that the worker can make outbound HTTPS connections. Workers do not require inbound connections. **3. Check the worker token** The worker authenticates using `Authorization: Bearer `. If the token was rotated or revoked, update the worker with a valid token. You can generate one from the agent's **Permissions** tab under **Worker token**, then restart the worker. **4. Check the agent ID, if configured** If the worker sends an `X-Agent-Id` header, it must match the agent associated with the token. You can also omit this optional header and let the token identify the agent. **5. Review the last heartbeat** Open the agent's **Metadata** tab and check **Last heartbeat**. This can help you determine whether the connection stopped recently or the worker has been disconnected for longer. > [!NOTE] > > A revoked token and an agent ID mismatch both return `401`. If these checks do not resolve the > issue, contact [Support](/reference/support) for help identifying the cause. ### An agent is online but does not receive work An online status confirms that the worker is responding to connectivity checks. [Fleet health](/manage-your-agents/optimize/fleet-health) provides additional information about whether it can receive work. | Fleet health | What it means | What to check | | - | - | - | | **Online, 0 live replicas** | The agent is connected, but no live replica is available to serve production work. | Check the production deployment. This can occur when a candidate deployment is running but the production deployment is unavailable. | | **Heartbeat-online, channel-unreachable** | The worker is responding, but its work delivery channel is disconnected. | Restart the worker and check Fleet health again. | ## API or webhook requests return 401 or 403 **What you might see:** a request returns an authentication or permission error. Start by identifying which type of credential the request requires. | Request type | Required credential | What to check | | - | - | - | | **API client** | `Authorization: Bearer ` | Confirm the key is present, correctly formatted, and valid. The associated member or service account must also be active. | | **Worker** | `Authorization: Bearer ` and, optionally, `X-Agent-Id` | Check the worker token and agent ID using the offline-agent steps above. | | **Webhook sender** | The endpoint's configured credential; the default header is `X-Webhook-Token` | Confirm the token or signature matches the endpoint's configuration. If the token was rotated, update the sender with the new value. | Webhook endpoints use their configured authentication method. An API key does not replace the endpoint's webhook credential. * GitHub-type endpoints expect `X-Hub-Signature-256`. * Bearer-type endpoints expect `Authorization`. * If the configured header is absent, KAOP also accepts `Authorization: Bearer ` as a fallback for senders that cannot use custom headers. > [!WARNING] > > Also check whether the webhook's signing credential is active. A deactivated credential fails > verification even if the token value has not changed. A `403` response can also indicate that the caller lacks the required permission or that the account is suspended. ## A run reports a missing credential **What you might see:** a run fails, or a tool is unavailable, with an error that names a credential. Start with the error message in the run details. It may identify the missing credential and the configuration needed. For example, a Datadog error may name `DD_API_KEY` and `DD_APP_KEY`. Then check the following: **1. Confirm the credential is bound to the agent** Credentials are assigned to individual agents. A credential configured for your account still needs to be bound to the agent that uses it. **2. Check that the credential is available** A deleted or empty credential can cause the connection's health badge to report a missing credential. **3. Check whether the integration needs user access** Some MCP servers require authorization for your user as well as the application. If the error requests user access, reconnect the integration with that access and test it again. **4. Check the execution context** Credentials are provided during a run. A tool or search helper called outside a run may be unable to request them. > [!NOTE] > > Credential values are not displayed in run evidence or agent logs. To troubleshoot, use the > credential's binding, status, and connection test results. ## An integration or MCP server is failing **What you might see:** a tool call fails, or a connection displays a warning badge. Check both the connection's status and its most recent test result: * **Lifecycle status** shows whether the connection is pending, active, in error, revoked, or disabled. * **Health badge** shows the result of the most recent credential check. An active connection may still need attention if its latest check failed. | Health badge | What it means | | - | - | | **credential verified** | The credential worked when checked within the last day. | | **verified earlier** | The credential worked, but the check is more than a day old. | | **not checked** | The credential has not been verified. | | **permission denied** | Authentication or authorization failed. | | **unreachable** | The provider could not be reached. | | **timed out** | The provider did not respond in time. | | **provider returned an error** | The provider was reached but returned an error. | | **no usable credential** | No credential is available for authentication. | | **cannot be verified** | An automated verification check is not available for this provider. | | **not working** | The check failed without a more specific reason. | | **unproven** | The check was skipped. | > [!WARNING] > > Connection checks do not run continuously in the background. Select **Test** to get a current > result. After you reconnect an integration or update its secret, the previous result is cleared. Test the connection again to verify the updated configuration. ## Slack does not connect the way you expect **What you might see:** a channel on your own Slack app shows a connection error, or the **Connect Slack** dialog does not offer the app or the use you want. | What you see | What to do | | - | - | | A channel on your own Slack app shows a connection error | On the channel's page, choose **Replace tokens** and paste fresh tokens. See [Replace tokens](/manage-your-agents/run/slack-own-app#replace-tokens). | | The dialog offers KAOP's Slack app only | Komodor has not turned your own Slack app on for your account. Contact your Komodor representative to turn it on. | | The Slack MCP is not offered for KAOP's Slack app | KAOP's app serves chat. Connect [your own Slack app](/manage-your-agents/run/slack-own-app) for the Slack MCP. | ## A trigger does not start a run **What you might see:** a schedule or endpoint is configured, but the expected run does not appear. Start with the guidance for your trigger type: > **Schedules** > > * **Is the trigger enabled?** A schedule is only registered while it is enabled and has a cron > expression. Disabling it unregisters it without deleting it. > * **Is the agent enabled?** A disabled agent does not fire triggers. It still answers manual and > chat invocations, which is why a disabled agent can look healthy while its schedule is silent. > * **Is the cron expression what you meant?** An invalid expression is rejected at write time with > the expression quoted back at you. So is a cron-and-timezone pair with no single UTC > equivalent, which is how daylight-saving edge cases surface. > **Inbound endpoints** > > * **Did the request pass verification?** A request that fails the endpoint's authentication is > rejected before any run is created. See **API or webhook requests return 401 or 403** above. > * **Is the target enabled?** A request aimed at a disabled agent or a disabled workflow is > refused with a conflict rather than queued. > * **Is the body the right shape?** For an endpoint, the **whole request body** is the agent's > input. The invoke and schedule APIs differ: they wrap the object under an `input` key. > * **Does the target exist?** A missing endpoint, trigger, agent, or workflow returns a > not-found response, which is worth distinguishing from an authentication failure. > **Channel messages** > > A channel rule fires on a mention by default, and messages posted by bots do not fire it unless > the rule explicitly allows bots. That is the usual reason an automated Slack post is ignored while > a person's identical message works. If a run was created but remains queued, continue to the next section. ## A run fails with a certificate error **What you might see:** a self-hosted agent is online, but its runs fail with **Agent does not trust a TLS certificate**, or with an error such as `SELF_SIGNED_CERT_IN_CHAIN` or `unable to get local issuer certificate`. Your model gateway logs show no requests from the agent. The agent checks every TLS certificate against the public certificate authorities (CAs) in its image. A certificate issued by your organization's own CA fails that check, so the connection closes before any request is sent. Common causes are a model gateway or MCP server on an internal hostname, and a TLS-inspecting proxy that re-signs outbound traffic. To have the agent trust your CA in addition to the public ones: **1. Export your CA certificate** Get the root CA certificate, and any intermediate certificates, in PEM format. Your network or security team can usually provide them. **2. Store the certificate in the agent's namespace** ```sh theme={null} kubectl -n create configmap internal-ca --from-file=ca.crt= ``` The agent trusts every PEM file in the ConfigMap. **3. Point the agent at the ConfigMap** ```sh theme={null} helm upgrade oci://ghcr.io/komodorio/charts/agentops-agent-base \ -n --reuse-values --set caBundle.existingConfigMap=internal-ca ``` For a new agent, add the setting to the values file before you run `helm install`: ```yaml theme={null} caBundle: existingConfigMap: internal-ca ``` If the agent already uses `caBundle.existingConfigMap` and runs still fail with this error, the ConfigMap is missing the CA that issued the certificate. Add that CA's root and intermediate certificates to the ConfigMap. > [!NOTE] > > The agent reads the certificates when its pods start. After you change the ConfigMap, restart the > agent's pods. Disabling certificate verification removes the protection from every connection the > agent makes; add your CA instead. ## A run is queued or fails before work begins **What you might see:** a run remains queued, or fails without showing agent activity. The failure category can help you choose your next step. | Category | What it means | Next step | | - | - | - | | **Agent run failed** | The agent started and encountered an error. | Review the run's spans and logs for the failing step. | | **Worker unreachable** | The run failed before any agent activity was recorded. | Check Fleet health and follow the offline-agent checks. | | **Tool call failed** | A tool call returned an error. | Check the integration's health and the tool's error message. | | **Stuck unclaimed** | The run remained queued beyond the time allowed for a worker to claim it. | Check whether a worker is online and available for the agent. | | **Worker offline** | Work is queued or claimed while the worker is offline. | Check Fleet health and restore worker connectivity or capacity as needed. | | **Unclassified failure** | The available evidence does not identify a specific failure category. | Review the reported error, spans, and logs. Contact Support if the cause remains unclear. | Runs also have time limits: * A queued run can fail if no worker becomes available within the allowed time. This may happen sooner when the target agent is already offline, in draft, or has failed to deploy. * A claimed run can fail when it reaches its execution time limit, including when a worker stops responding. ## Knowledge search returns no results **What you might see:** an agent's knowledge search returns no matches, or the Knowledge page reports that search is unavailable. Check the following: **1. Search availability** Semantic search requires an embedding provider. If the error says the embedding backend is not configured or cannot be reached, contact your Komodor representative for help. **2. Document indexing status** Newly uploaded documents become searchable after indexing finishes. If a document has an error status, re-index it. **3. Search filters** Filters for service, environment, document type, or sensitivity may exclude relevant documents. Try broadening the filters. **4. Knowledge base access** Confirm the documents are in a knowledge base available to the agent. Search is scoped to your account and the agent's configured knowledge bases. > [!NOTE] > > An empty result can also mean the search completed successfully but found no matching documents. > Check the response to distinguish this from search being unavailable. ## HTTP status codes Use this table to help interpret an API error. | Code | Meaning | | - | - | | `400` | A request contains an invalid value. For example, an API key expiration must be a future ISO 8601 timestamp with a timezone offset. | | `401` | Authentication failed. Check the credential required for the request type. | | `403` | The caller lacks permission, the account is suspended, or a webhook credential failed verification. | | `404` | The requested resource was not found, or the capability is not enabled for your account. | | `409` | The request conflicts with the current state or exceeds a quota, such as the maximum number of agents. See [Limits & quotas](/reference/limits-quotas). | | `413` | The payload exceeds the allowed size. This can apply to knowledge documents, batches, or run artifacts. | | `422` | Validation failed. Check values such as cron expressions or worker registration details. | | `503` | A dependency is temporarily unavailable. Retry the request. | > [!NOTE] > > KAOP's control plane does not return `429` rate-limit responses. If a run reports `429`, check the > response from the upstream provider used by the agent. ## Need more help? If the issue continues, contact [Support](/reference/support). Include the run ID and exact error message, if available, along with the checks you have already tried. ## Next steps > **[Limits & quotas](/reference/limits-quotas)** > > Review limits that may affect your request. > **[Runs & evidence](/manage-your-agents/run/runs-evidence)** > > Learn how to inspect run details, spans, and logs. > **[Support](/reference/support)** > > Find contact information and details to include in your request. # Agent isolation & tenancy Source: https://docs.komodor.com/security-and-governance/architecture-considerations/agent-isolation-tenancy How KAOP binds requests to one account and limits every agent to its own identity, permissions, and credentials. The Komodor Agentic Operation Platform (KAOP) uses the account as its tenancy boundary. Every authenticated request is associated with one account, and every resource the request addresses must belong to that account. Inside the account, each agent has its own identity, permissions, and credential bindings. ## One request, one account Authentication establishes both **who is acting** and **which account the identity belongs to**. KAOP uses that account context when it reads or changes agents, runs, credentials, knowledge bases, and other resources. Ownership is also carried into the records an agent creates: * A run gets its account from the agent being invoked, not from an account value supplied by the caller. * Events, evidence, and outputs inherit ownership from their parent run. * Resource names are resolved within the account. Two accounts can each have an agent named `triage` or a credential named `github` without sharing either resource. An agent therefore cannot widen its tenancy scope by changing a friendly name or adding an account identifier to a request. ## Isolation between agents in one account Sharing an account does not give agents the same access. KAOP evaluates each agent as a separate principal. | Control | What it limits | | - | - | | **Worker token** | Authenticates one agent in one account. | | **Roles and grants** | Limit the capabilities and resources available to that agent. | | **Credential bindings** | Deliver a stored credential only to agents explicitly allowed to use it. | | **Resource scopes** | Restrict grants by resource ID, pattern, or labels such as team and environment. | | **Gateway guardrails** | Inspect or block governed model and tool traffic that passes through a configured gateway. | Granting production access to one agent does not grant it to another. Use [Access explorer](/security-and-governance/identity-and-access/access-explorer) to inspect an agent's effective access before you run it against a sensitive environment. ## Revoking access Choose the control that matches what you need to stop: * **Revoke a worker token** to prevent that token from authenticating again. This does not stop the worker process itself. * **Remove a role or grant** to prevent future authorization for the affected capabilities. * **Remove a credential binding** to prevent future authorized delivery of that credential. * **Revoke the credential at its provider** when a value may already be loaded in a running process or exposed outside KAOP. Rotating a worker token requires replacing the deployed token before that worker can authenticate again. Removing a credential binding does not erase a value that a running process has already received. ## Design checklist * Give every agent only the roles and credentials required for its job. * Use label-scoped grants to separate teams and environments inside an account. * Keep production and non-production credentials separate. * Test a representative capability and resource in Access explorer. * Revoke credentials at the source as part of incident response. ## Next steps > **[Secrets & credential handling](/security-and-governance/architecture-considerations/secrets-credential-handling)** > > How an authorized credential reaches a worker or run. > **[Data handling & redaction](/security-and-governance/architecture-considerations/data-handling-redaction)** > > What agents can see and what KAOP masks automatically. > **[Agent identity](/security-and-governance/identity-and-access/agent-identity)** > > How worker tokens establish which agent is acting. > **[Network & egress control](/security-and-governance/architecture-considerations/network-egress-control)** > > How deployment choices affect inbound and outbound traffic. # Data handling & redaction Source: https://docs.komodor.com/security-and-governance/architecture-considerations/data-handling-redaction What agents can see, what KAOP masks automatically, and how to control what reaches models, tools, and run records. The Komodor Agentic Operation Platform (KAOP) records the inputs, outputs, and evidence that an agent emits so you can understand its work. Access controls decide what the agent can reach; credential masking and guardrails reduce what sensitive data appears in that record. These controls have different boundaries, so use them together. ## What an agent can see An agent can receive data from several sources, subject to its configuration and permissions: * Run input and conversation context * Tools, integrations, and knowledge bases available to the agent * Credentials bound to the agent * Prior state or history supplied to the run * Model and tool responses returned during the run KAOP does not copy every source record merely because an agent can read it. Data becomes part of a run record when the agent, SDK, model, or tool emits it as input, output, an event, evidence, or saved state. ## Automatic credential masking The KAOP SDK masks exact occurrences of credentials delivered to a run before reporting run output, events, diagnostics, and saved state. For example, if a tool echoes an API token in a response, the matching value is replaced with a redaction marker in the data emitted through the SDK. > [!WARNING] > > No rule is required for credential masking, but it is not a general data-loss-prevention system: > > * Encoded, hashed, split, truncated, or otherwise transformed values may no longer match the > original credential. > * Personal or business data that is not a delivered credential is not automatically masked by this > mechanism. > * Custom code that sends data outside the SDK reporting path is outside this protection. > * A value already delivered to a running process remains available to that process until its > lifecycle ends. > > Do not put credentials in prompts, logs, tool arguments, or saved state. Automatic masking is a > backstop for accidental disclosure, not permission to expose or forward a secret. ## Redaction with guardrails [Standards and guardrails](/security-and-governance/controls/standards-guardrails) can inspect, block, or redact governed traffic at configured model and tool gateways. KAOP includes detectors for common sensitive values such as payment card numbers, US Social Security numbers, email addresses, private keys, JWTs, and several cloud and SaaS credential formats. You can also define patterns for data specific to your organization. Guardrails apply only at the gateway stages you configure. A direct network call made by custom agent code does not pass through a KAOP gateway automatically. ## Reduce data before it leaves the source The strongest control is to avoid collecting unnecessary data: 1. Give the agent only the tools, credentials, and resources required for its task. 2. Filter and aggregate data close to the source. 3. Return findings and references instead of raw logs or full records. 4. Apply guardrails at model requests, model responses, tool calls, and tool responses where the associated gateway is used. 5. Review run evidence to confirm that the agent emits only what you expect. For example, prefer an output such as “three workloads use the affected image” with links to the workloads over copying the complete workload specifications into the run record. ## Management APIs do not reveal stored secrets Credential management surfaces can create, replace, disable, bind, and delete credentials, but do not return stored values. Plaintext is available only during authorized delivery to a worker or run. See [Credentials & secrets](/security-and-governance/identity-and-access/credentials-secrets) for the delivery modes and lifecycle controls. ## Next steps > **[Data retention, deletion & export](/security-and-governance/architecture-considerations/data-retention-deletion-export)** > > What happens to evidence after KAOP stores it. > **[Standards & Guardrails](/security-and-governance/controls/standards-guardrails)** > > Configure inspection, blocking, and redaction at governed gateways. > **[Secrets & credential handling](/security-and-governance/architecture-considerations/secrets-credential-handling)** > > Understand credential storage, binding, and delivery. > **[Network & egress control](/security-and-governance/architecture-considerations/network-egress-control)** > > Decide which traffic can leave the worker environment. > **[Runs & evidence](/manage-your-agents/run/runs-evidence)** > > Review the record created by an agent run. # Data retention, deletion & export Source: https://docs.komodor.com/security-and-governance/architecture-considerations/data-retention-deletion-export What KAOP keeps as operating history, what archive and deletion mean, and how to request an account-level export or deletion. The Komodor Agentic Operation Platform (KAOP) keeps the records needed to explain what an agent saw, what it did, and who changed its operating boundaries. Retention preserves that evidence for review; archiving, deleting a configuration object, and deleting an account are different operations with different effects. ## What is retained | Record | Why KAOP keeps it | | - | - | | **Runs and evidence** | To reconstruct an execution from its input through its tool and model calls to its output, status, and cost | | **Chats** | To preserve the conversation and the runs created from it | | **Audit entries** | To show governed changes, sensitive reads, rejected attempts, and the identity responsible | | **Approvals** | To show which proposed action was approved or rejected, by whom, and when | These records can contain customer data. Prompts, inputs, tool arguments and results, logs, messages, outputs, and artifacts are retained as the worker reports them. Stored secret values are masked before that evidence leaves the worker, but general personal or sensitive data is not automatically classified. See [Data handling & redaction](/security-and-governance/architecture-considerations/data-handling-redaction) before deciding what an agent may read or emit. Retention periods are not configured from the KAOP console. For the retention terms that apply to your account or deployment, contact your Komodor representative. ## Archive is not deletion Archiving an agent retires it from active use without erasing its history. Existing runs and evidence remain available so an earlier conclusion does not lose its context. You can review archived agents separately from the active fleet and restore one when needed. The same distinction applies to access. Revoking an API key, worker token, or credential prevents future use; it does not rewrite records that show how the identity or credential was used before it was revoked. > [!NOTE] > > An append-only audit log cannot be edited or selectively cleared from the console. Its entries > remain immutable while retained; expiry under the applicable retention policy and account-level > deletion are lifecycle operations, not edits to individual entries. ## Export account data KAOP does not expose a self-service, whole-account export in the console. To request one, contact your Komodor representative and include the account id from **Settings → Account**. State what you need exported and the reason or date range that defines the request; Komodor will confirm the available scope and delivery method. The runs API and OpenTelemetry export serve narrower operational use cases. They let you read run records or send new telemetry to your own backend, but they are not a substitute for an account-level data export. ## Delete data Delete individual configuration objects from their own product page or API where KAOP offers that action. Some objects are archived instead because their historical relationship to runs and audit records must remain understandable. Account-level deletion is not self-service. Request it through your Komodor representative and include the account id. Komodor will confirm the scope and next steps before the account and its tenant data are removed. > [!WARNING] > > Account deletion is irreversible. Request any export you need before asking for deletion, and do > not send credentials, API keys, worker tokens, or other secret values with the request. ## Next steps > **[Data handling & redaction](/security-and-governance/architecture-considerations/data-handling-redaction)** > > Which run fields can contain customer data and what is masked automatically. > **[Audit log](/security-and-governance/controls/audit-log)** > > The immutable record of governed actions while each entry is retained. > **[Runs & evidence](/manage-your-agents/run/runs-evidence)** > > Read the execution record stored for a run. > **[Support](/reference/support)** > > What to include when you contact Komodor. # MCP Gateway Source: https://docs.komodor.com/security-and-governance/architecture-considerations/mcp-gateway Expose your own tools to agents — an MCP server you run, or an OpenAPI document that becomes tools — governed by gateway policy. The MCP Gateway is the second of the two ways an agent reaches something external, and the one for tools that are yours. Point the Komodor Agentic Operation Platform (KAOP) at an MCP server you run, or at an OpenAPI document, and its operations become tools agents can call — with the gateway deciding which agent may call which. ## The two things you can register | You have | Register it as | What KAOP does | | - | - | - | | A server that speaks MCP | An **MCP server** | Connects to it, discovers its tools, and brokers them to agents | | A REST API with an OpenAPI document | An **OpenAPI document** | Turns the operations you select into tools — no MCP server of your own required | Both end up in the same place: a tool server in your **Integrations** list, groupable and governable the same way. The gateway dials outward to reach them; nothing needs to reach into KAOP. > [!NOTE] > > This is distinct from KAOP's *own* MCP server, which is an inbound endpoint your coding agent > connects to in order to operate KAOP. See [MCP server](/developer-tools/mcp-server) and > [Connect coding agents](/developer-tools/connect-coding-agents). The gateway is outbound; that one is > inbound. ## Register an MCP server From **Integrations**, press **Add integration**, choose the custom-integration tile, and pick **MCP server**. The wizard runs in three steps — **Connect**, **Configure**, **Review**. **1. Point it at the server** **Server URL** is the endpoint, and **Transport** is how to speak to it. **2. Choose how it authenticates** See [Authenticating to your server](#authenticating-to-your-server). **3. Decide which tools to publish** **Allow patterns** and **Deny patterns** are globs over tool names, one per line. An empty allow list means every discovered tool; a deny pattern wins over an allow. **4. Name it and put it in a group** **Name** becomes the namespace its tools are prefixed with — letters, digits, and hyphens only. **Group** is which integration group it belongs to. **5. Review and test** Save, then run the connection test. > [!WARNING] > > A server in no group reaches nobody. Add it to an > [integration group](/manage-your-agents/build/integration-groups) — that group's endpoint is what an > agent actually connects to. ### Transport | Transport | When | | - | - | | `http` | Streamable HTTP. The default, and what you want unless your server predates it | | `sse` | Server-sent events, for servers that only speak the older transport | The gateway proxies **remote** servers. There is no local-subprocess transport — a server has to be reachable over HTTP from the gateway. ### The other settings | Setting | What it does | | - | - | | **Timeout (seconds)** | How long a call to this server may take before it is abandoned | | **Connection-test tool** | Which tool the connection test invokes. Leave it automatic and KAOP picks a safe read | | **Static headers** | Headers always sent upstream. Use a reference, never a literal secret — see below | | **Tags** and labels | Metadata for organizing and for access control | | **Enabled** | Turn the server off without deleting it | ## Register an OpenAPI document Same entry point; choose **OpenAPI document**. The wizard is again **Connect**, **Configure**, **Review**. **1. Supply the document** Either a public HTTPS URL KAOP fetches, or upload the document itself. One or the other — not both. OpenAPI 3.0 and 3.1 are supported. **2. Override the base URL if you need to** By default KAOP calls the API at the base the document declares. **API base URL** overrides that — useful when one document describes several environments. **3. Select the operations to expose** KAOP lists every operation it discovered. Choose the ones that become tools. **4. Add notes where the API is not self-explanatory** A note on an operation is appended to its description, so it travels with the tool and the agent reads it when deciding whether to call. **5. Name it, review, and test** The name namespaces the tools, exactly as for an MCP server. > [!WARNING] > > The operation list is an explicit allowlist. There is no "expose everything" setting and no exclude > list — which means an operation added to your API *after* you imported it stays off until a person > enables it. That is deliberate: growing your API never silently widens what agents can do. Reconfiguring later — **Edit operations** on the server — is how you add or remove operations. Use **refresh** to re-read the document after your API changes; the tool names KAOP assigned stay stable so an agent's configuration does not break underneath it. > [!NOTE] > > Practical limits apply before you point KAOP at a large API: a document may declare up to > 1,500 operations, of which up to 200 may be enabled as tools, and the document itself is capped at > 5 MB. Select the operations an agent actually needs rather than everything the API offers. ## Authenticating to your server Four options, and the right one depends on what your server expects. **None** A public server or API with no credentials. Legitimate, and the wizard says so: *None — public server, no credentials*. **A stored credential** Bind a [credential](/security-and-governance/identity-and-access/credentials-secrets) or an existing integration connection to the server. KAOP injects it as a bearer token at request time. This is the option to prefer: the secret lives in the credential store, and rotating it there is the whole rotation. **Gateway OAuth** An interactive per-server sign-in. You save the server first, then authorize it — KAOP caches the resulting token and refreshes it automatically. **Static headers** For a server that wants a custom header rather than a bearer. Put a **reference** in the value — `${credential:my-api-key}` resolves from the credential store at connect time. > [!WARNING] > > Static headers are stored as configuration and are readable back through the admin API. A literal > secret in an authentication header is refused for exactly that reason. Use a `${credential:…}` > reference instead, and the value is resolved at connect time without ever being stored in the > header. > [!NOTE] > > For an OpenAPI-backed server the auth posture is deliberately strict: if the bound credential cannot > be resolved, the call fails rather than proceeding. Calling your REST API unauthenticated is worse > than not calling it. ## Gateway policy — what governs a call A tool being registered is not a tool being callable. Four layers apply, and they only ever narrow. ```mermaid theme={null} flowchart TB A["Agent calls a tool"] --> G["Guardrails
block, hold for approval, redact, or record only"] G --> P["Policies
tool allowlists and rules"] P --> S["Group allow / deny
can only narrow the server's own rules"] S --> V["Server allow / deny
which tools it publishes at all"] V --> U["Your server"] ``` | Layer | What it decides | Where you set it | | - | - | - | | **Server allow / deny** | Which of the server's tools exist as far as KAOP is concerned | On the server | | **Group allow / deny** | Which of a group's members' tools that group grants. It can only narrow — a tool the server denies stays denied | On the [integration group](/manage-your-agents/build/integration-groups) | | **Group binding** | Which group's endpoint an agent connects to, and therefore which tools it sees at all | On the agent | | **Policies** and **Guardrails** | Rules evaluated on every call — block it, redact a matched value, or record the match | **Integrations → Policies**, and [Standards & Guardrails](/security-and-governance/controls/standards-guardrails) | A guardrail is evaluated before a policy, so when both would refuse a call, the reason you read names the guardrail you wrote rather than a rule id. If policy evaluation cannot run, the call is refused — the gateway fails closed. ## Test and health A registered server carries the same two-part health story as a built-in integration, and the same rule applies: **skipped is never a pass**. | Badge | Meaning | | - | - | | **not checked yet** | Nothing has tried this server | | **healthy** | It answered recently | | **no recent activity** | It passed, but not recently enough to call current | | **unproven** | A check ran but could not conclude — most often because the server has no tool safe to probe | | A named failure | *unreachable*, *timed out*, *permission denied*, *sign-in required*, *credential missing*, *no tools*, and others | Two things a test can prove, and it tries both: that KAOP can **connect** and list the server's tools, and that the credential is **permitted** to actually do something. A server whose only safe tools need no credentials can only be checked for connectivity, which is what *unproven* means. Health is also updated by real agent traffic — the badge's tooltip says which observed it. Hover to see whether you are reading a connection test, a start-up check, or live traffic. > [!WARNING] > > Nothing tests your servers on a schedule. A credential revoked upstream can keep reading as healthy > until something touches the server, and a revocation is only reliably caught by a connection test or > a refresh. Test after any credential change on your side. ## Next steps > **[Integration groups](/manage-your-agents/build/integration-groups)** > > Required reading — a server in no group reaches nobody. > **[Credentials & secrets](/security-and-governance/identity-and-access/credentials-secrets)** > > Where the secrets your server needs should live. > **[Standards & Guardrails](/security-and-governance/controls/standards-guardrails)** > > Rules that block, hold, or redact a tool call. > **[Integrations overview](/manage-your-agents/build/integrations-overview)** > > The other path, and when to prefer it. # Network & egress control Source: https://docs.komodor.com/security-and-governance/architecture-considerations/network-egress-control A worker dials out and nothing dials in — and what an agent can reach on the way out is decided by its tool surface, not by its network. Two questions come up in every network review: what has to be open for this to work, and what can an agent reach once it is running. The first has a short answer — outbound access to the control plane, nothing inbound — and the second is decided by the agent's tool surface as much as by your network. ## The connection model A worker you run lives in your environment and is **outbound-only**: it dials the control plane over TLS, and nothing dials into it. There is no inbound listener to expose, no port to open toward your cluster, and no ingress rule to write for the platform's benefit. That has a practical consequence worth stating: an agent can work inside a private network that accepts no inbound traffic at all, because the connection is always established from your side. > [!NOTE] > > Inbound *events* — something in your estate telling an agent that a thing happened — arrive at an > endpoint on the platform side that your system posts to, under **Integrations → Endpoints**. Your > network still only makes outbound connections. ## What an agent can reach on the way out An agent's reach is decided by its tool surface as well as by its network, and the tool surface is the decisive one: an agent's capabilities are governed centrally, so a firewall rule alone does not narrow what it may do. The two work together — platform controls govern the supported tool paths, and network policy constrains the destinations the worker can reach at all. Two paths are governed — a built-in integration, or the [MCP Gateway](/security-and-governance/architecture-considerations/mcp-gateway) — and these are the controls over them: | Control | What it narrows | | - | - | | **Credential bindings** | Which agents can use which credentials at all. | | **Server groups** | Which upstream servers an agent's gateway traffic may reach, with allow and deny patterns over the tools inside them. | | **Per-request scope** | A single request can be narrowed to named servers and named tools, without changing any stored configuration. | | **Guardrails** | A rule at the tool-call gate can refuse or rewrite a specific call before it leaves. | > [!WARNING] > > These controls apply to traffic that goes through the platform's gateways. An agent that reaches an > external system by some other path — its own network calls, its own provider key — is outside them. > Routing tool access through the integration catalog or the MCP Gateway is what makes the governance > real rather than nominal. ## What crosses the wire Outbound from the worker: the run ledger and the evidence an agent produces — its inputs, tool calls, reasoning, outputs, and cost — with run-scoped secrets masked before they leave. See [Data handling & redaction](/security-and-governance/architecture-considerations/data-handling-redaction) for what that covers and what it does not. Inbound to the worker: work to do, and the credentials for the run doing it. Nothing else, and nothing unsolicited. ## Next steps > **[MCP Gateway](/security-and-governance/architecture-considerations/mcp-gateway)** > > The governed path for your own tools. > **[Data handling & redaction](/security-and-governance/architecture-considerations/data-handling-redaction)** > > What is in the traffic that does leave. > **[Agent isolation & tenancy](/security-and-governance/architecture-considerations/agent-isolation-tenancy)** > > The boundary the connection lands inside. > **[Standards & Guardrails](/security-and-governance/controls/standards-guardrails)** > > Refusing a call before it leaves. # Outposts Source: https://docs.komodor.com/security-and-governance/architecture-considerations/outposts A small relay you run inside your own network so agents can reach private services — outbound only, deny-by-default, with no inbound hole to open. Some of the systems your agents need are not reachable from the internet, and opening a way in is not on the table. An **Outpost** solves that: a small proxy you run inside your own network that dials out to the Komodor Agentic Operation Platform (KAOP) and carries calls back down that connection. ## It never accepts an inbound connection The Outpost opens a connection outward and holds it open. Calls travel down that existing connection, so from your firewall's point of view there is nothing to allow — it is the same outbound access your CI already has. No inbound rule, no exposed port, no address of yours to publish. ## Which path a call takes Three paths, and the one that applies depends on **where the agent runs**, not on a negotiation at call time. | Situation | Path | | - | - | | Agent hosted by Komodor, service private | agent → KAOP gateway → Outpost → your service | | Agent running alongside the Outpost, service private | agent → local Outpost → your service | | Service publicly reachable | agent → KAOP gateway → service, directly | The middle row is the one worth getting right: when the agent and the service are on the same network, the call should not leave that network. You choose per agent, on the deploy wizard's MCP step — the platform does not infer it, because the only location signal an agent gives is a name the process declares about itself, and guessing a routing decision from that would be worse than asking. > [!NOTE] > > The local listener that makes the middle row possible is off by default. You enable it in the > Outpost's own deployment values, which also sets the in-cluster network policy that governs which > agents may reach it. Until it is enabled, a colocated agent's calls take the first row — out to the > KAOP gateway and back down the tunnel — instead of staying on your network. ## It is built to refuse The Outpost holds network access its callers do not have, so it starts from deny. * **Deny-by-default allowlist**, on scheme, host, port, and an optional path prefix. An empty allowlist allows nothing. * **Host matching is exact, or one leading wildcard label**, compared label by label — never a substring or suffix. A suffix match on `internal.example.com` would also be satisfied by `internal.example.com.attacker.test`. * **A path prefix matches on segment boundaries**, so `/mcp` does not admit `/mcp-admin`. * **Addresses are checked after DNS resolves**, because the resolver is yours and a name can resolve anywhere. Loopback, link-local, multicast, and cloud metadata endpoints — IPv4 and IPv6 — are refused outright. * **A redirect off the allowlist is not followed.** The relay re-checks its own rules on every hop. * **Caller credentials are never passed upstream.** ## Every call is bounded Beyond the allowlist, each call the Outpost forwards runs inside fixed limits. * **Only ordinary HTTP methods are forwarded** — `GET`, `HEAD`, `POST`, `PUT`, `PATCH`, `DELETE`, and `OPTIONS`. `CONNECT` and any other verb are refused, so the tunnel cannot be turned into a raw TCP proxy. * **Request and response bodies are size-capped**, and the number of headers is capped. A body over the limit is rejected rather than streamed. * **Every call has a timeout**, and the number of concurrent calls per Outpost is capped. Calls beyond the cap are shed rather than queued indefinitely. * **Connection-scoped headers are stripped**, and the relay never routes through a forward proxy — it opens every socket itself, so the address checks above always apply. > [!WARNING] > > A call an agent makes to a service it can already reach on its own does not pass through any of > this. The Outpost governs the traffic routed through it — which is the argument for routing > everything through it or the built-in integrations rather than letting an agent reach out on a path > of its own. ## Enrolling one Creating an Outpost issues a long-lived credential, **shown once at creation**, and that credential is the Outpost's whole identity — it resolves to your account and that one Outpost. Store it before you leave the screen, then run the install command with it. Rotating the credential later issues a replacement, shown the same one time. The credential is never written to a log, a trace, or an audit record's detail. > [!NOTE] > > Rotating the credential drops the live connection. The Outpost stays down until you redeploy it with > the new value, so rotation is a planned action rather than a safe click — which is why it is a > separate permission from editing an Outpost. Labels on an Outpost scope who can see and manage it, the same way labels scope other resources. ## Next steps > **[MCP Gateway](/security-and-governance/architecture-considerations/mcp-gateway)** > > The tool surface an Outpost carries calls to. > **[Network & egress control](/security-and-governance/architecture-considerations/network-egress-control)** > > The wider network posture this fits into. > **[Deployment methods](/get-started/deployment-methods)** > > Where agents run, which decides which path applies. > **[Integration groups](/manage-your-agents/build/integration-groups)** > > Grouping the tool servers an agent may reach. # Providers Source: https://docs.komodor.com/security-and-governance/architecture-considerations/providers The models your fleet can call — through Komodor-managed access, your provider keys, or a gateway that stays in your environment. Every agent run costs a model call, and Providers is where you see which models this account can call, what each is costing, and whether it is working. The Komodor Agentic Operation Platform (KAOP) ships with managed models so agents run without any setup. You can also register your own provider account or route a self-hosted agent directly through an LLM gateway you manage. ## Three tiers, one table | Tier | What it is | What you can change | | - | - | - | | **Managed by Komodor** | Models provisioned for your account, with their budget, spend, and remaining allowance | Read-only — budget and model allowances are set by Komodor | | **Your providers** | Your own provider account, registered with your own API key | Everything: rename, update, delete, re-verify | | **Managed by me** | Your own LLM gateway, called directly by a self-hosted agent with a key that stays in your environment | The gateway, models, metadata access, and Outpost binding | The page is one flat table of **models**, not a tab per tier. Each row is a model with its provider, who manages it, its upstream model id, its spend, its usage, and its status — so the tier is just a column you can sort and filter by. A registered account with no models yet still gets a row, so it is visible rather than something you have to remember exists. Above the table sits the account's gateway budget. That strip is the only place the managed tier's account-level numbers appear. ## Bringing your own provider Supported providers are OpenAI, Anthropic, Azure, Bedrock, Vertex AI, Gemini, Mistral, Groq, DeepSeek, OpenRouter, Together AI, xAI, Cohere, Ollama, Baseten, Nebius Token Factory, and a hosted vLLM endpoint. **1. Credential** Name the account and supply its API key. The key is stored so model discovery can authenticate with it later. **2. Models** Register one or more upstream model ids against that credential. **Nothing is written until this step submits** — the credential is created, then each model against it, as one action. **3. Verify** Each new deployment is health-checked, and you can talk to it directly to confirm it answers. Verify is a step of its own rather than a result banner, because after the write *"did this work"* is a genuinely different question from *"what should I create"* — and a reader who has finished creating still has to answer it. To add models to an account you already registered, use **Add models** on that credential rather than walking the wizard again. > [!NOTE] > > Registering your own provider does not replace the managed tier — both stay callable, and an agent > picks a model from either. ## Using your own LLM gateway The **Managed by me — your own LLM gateway** tier is for self-hosted agents. The worker calls your gateway directly with your key: neither that inference key nor the inference request passes through Komodor's model gateway. The upstream provider key stays inside your gateway as well. KAOP supports LiteLLM, Hosted vLLM, Ollama, and Azure AI Foundry gateways. The first three use an OpenAI-compatible interface. Azure AI Foundry uses its own discovery route and is distinct from registering a public Azure AI Foundry account under **Add a new key**. > [!WARNING] > > Keeping Komodor out of the inference path does not turn off run reporting. Prompts, model spans, > outputs, and other evidence emitted by the worker still follow the account's normal > [data-handling rules](/security-and-governance/architecture-considerations/data-handling-redaction). ### Choose how KAOP reads gateway metadata Both modes keep inference between your worker and your gateway. The difference is whether KAOP can read metadata from the gateway. | | **Strict** | **Scoped** | | - | - | - | | What KAOP stores | The source name, endpoint, gateway kind, and model names | The same details, plus a read-scoped gateway key | | Model list | You enter the model ids | KAOP fetches the list and you choose which models to expose | | Verification and spend metadata | KAOP does not authenticate to the gateway | KAOP uses the scoped key to list models, verify reachability, and read spend | | Inference key | Stays with the worker | Stays with the worker | | Inference traffic | Worker → your gateway | Worker → your gateway | The scoped key must be read-only and non-admin. It is stored encrypted, used only for metadata operations, never delivered to a worker, and never used for inference. Do not enter your gateway's admin key or an upstream provider key. ### Register a customer-managed gateway **1. Choose Managed by me** Open **Providers**, choose **Add provider**, then select **Managed by me — your own LLM gateway**. This tier can be used only by agents you host yourself. **2. Describe the gateway** Choose its kind and strict or scoped mode, then enter a source name and gateway base URL. For a private endpoint, select the Outpost that can reach it; leave the Outpost unset for a publicly reachable gateway. **3. Supply model access** In strict mode, enter the model ids and choose **Add gateway**. In scoped mode, enter a read-scoped key and choose **Continue**, select the discovered models your agents may use, then choose **Save gateway**. An Outpost carries only metadata operations between KAOP and a private gateway: model discovery, verification, and spend reads. The self-hosted worker still reaches the gateway directly for every inference request. Make sure the gateway host is allowed by the selected Outpost before registering it. ### Use the gateway from an agent When you build or edit a self-hosted agent, select a model under **Managed by me**. Supply the **Gateway API key** in the wizard, or select **I'll provide the API key myself in the helm command** and add it when you install the agent. In either case, the key is rendered into your cluster Secret; it is not stored as a KAOP provider credential. ## Next steps > **[Credentials & secrets](/security-and-governance/identity-and-access/credentials-secrets)** > > How a provider key is stored and delivered to a run. > **[Outposts](/security-and-governance/architecture-considerations/outposts)** > > Reach a private gateway without opening an inbound path. > **[Agent spend & attribution](/manage-your-agents/optimize/agent-spend-attribution)** > > What the fleet costs, attributed per run, agent, and model. > **[Model Evaluation](/manage-your-agents/optimize/model-evaluation)** > > Decide whether a model is good enough for a given job. # Secrets & credential handling Source: https://docs.komodor.com/security-and-governance/architecture-considerations/secrets-credential-handling How a stored secret reaches the run that needs it without being exposed to a browser, a deploy, a log, or another agent. A credential is a stored secret bound to the agents allowed to use it, delivered to a run without its value being exposed. It is never shown back to you after creation, never written into a run's evidence, and never placed in the agent's prompt or model context. ## The shape of the guarantee The guarantee rests on three properties: * **A value is delivered for the run that needs it by default**, not held by the agent between runs. * **A raw value never passes through your browser or a deployment**, so it is not sitting in a console screenshot, a manifest, or a build log. * **A binding, not an account membership, grants access.** Being an agent in the account does not give you the account's secrets. ## The path a secret takes **1. You store it** A credential holds either a single secret value or a set of named keys under one name. Once stored, it is not readable back out through the console — the screen manages the credential, not its contents. **2. You bind it** A credential is bound to the agents allowed to use it. An agent with no binding to a credential cannot request it, whatever else it can do. **3. It is delivered to a run** When a bound agent starts a run that needs the credential, the value is delivered to the worker for that run. **4. The run ends and it is gone** The SDK holds a run's credentials in memory for the run only and never persists them, so a worker restart or a crash does not leave a secret on disk. > [!NOTE] > > A binding can allow **on-demand fetch**, so the agent resolves the credential outside a run — for a > value it needs at startup, such as a boot-time token. An on-demand value is held in the worker's > memory for the life of the process rather than delivered per run, and is still never written to > disk. When on-demand fetch is off, the credential is available only during a run. ## What is masked on the way out Any run-scoped secret is replaced with a redaction marker **before anything leaves the worker** — recursively, across event names, content, payloads, and metadata. A token echoed into a log line or returned inside a tool result does not reach the run record in the clear. > [!NOTE] > > This is a platform behavior, not a rule you configure. Configurable redaction — keeping a value out > of a specific tool call or model prompt — is a > [guardrail](/security-and-governance/controls/standards-guardrails), and it applies to values that > are not run-scoped secrets. ## Rotation and revocation | Action | Effect | | - | - | | **Rotate a credential's value** | Takes effect on the next run. No redeploy, no restart, no manifest change. | | **Disable or delete a credential** | Future runs that depended on it stop receiving it. | | **Rotate a worker token** | Mints a replacement and stops the old value working — update the worker's secret in the same change. | | **Revoke a worker token** | Prevents that token from authenticating, so the worker cannot start a new run. It does not terminate the worker process or stop a run already in progress. | Every one of these is recorded in the [audit log](/security-and-governance/controls/audit-log): creation, updates, value replacement, deletion, and binding changes. ## Practical guidance > [!TIP] > > Bind narrowly and rotate freely. The point of per-agent binding plus next-run rotation is that > scoping a credential tightly costs you nothing operationally — the agent that only reads gets a > read-only credential, and rotating it does not involve the agent's deployment at all. Keep a worker's token out of source control: deliver it as an environment variable or a mounted secret, and rotate it from the console rather than editing it in place. ## Next steps > **[Credentials & secrets](/security-and-governance/identity-and-access/credentials-secrets)** > > Creating and binding a credential. > **[Data handling & redaction](/security-and-governance/architecture-considerations/data-handling-redaction)** > > What else is masked, and what is not. > **[Agent identity](/security-and-governance/identity-and-access/agent-identity)** > > The worker token, and how it is managed. > **[Audit log](/security-and-governance/controls/audit-log)** > > The record of every credential change. # Approvals Source: https://docs.komodor.com/security-and-governance/controls/approvals A consequential action gets a person's decision, with the evidence needed to make it — and the decision becomes part of the run's record. An approval is how a consequential action reaches a person before it happens. Somebody approves or rejects it with the evidence in front of them, and the action goes ahead only if they allowed that exact one. ## Why a decision, not a rule A guardrail is the right control when the answer is always the same: this tool call must never happen. An approval is for the case where the answer depends on the situation — restarting this particular workload, at this hour, given what the investigation found. That distinction is why both exist. A rule that has to be right for every case gets written permissively and stops protecting anything; an approval keeps a person on the trigger for exactly the actions where judgement is the point. ## Where an approval comes from **1. A workflow step that asks a person** A workflow can contain a step whose job is to put a question to a human. The run parks there — it is not failed, and it is not quietly retried — until somebody answers, and the workflow continues from that answer. **2. A guardrail that holds a tool call** A guardrail set to **Hold for approval**, at the tool call gate, refuses the call it covers and registers an approval naming exactly that call — rather than blocking the whole class of them outright. In both cases the request carries what a decision needs: what the agent proposes to do, which agent and run it belongs to, and the record it came out of — so you are approving a specific action with its evidence, not a permission in the abstract. > [!NOTE] > > The two resume differently. A workflow step continues from the answer it was given. A held tool call > does not resume by itself: the agent was refused at the time, so the call has to be attempted again, > by that agent or by a later run. A grant is also single-use and specific to the call it names, > arguments included, so approving one attempt authorises neither a different call nor a second > attempt at the same one. ## Answering one Pending decisions collect in **Attention**, with approve and deny beside each one, and a run parked on a workflow step shows what it is waiting on — its status reads *waiting*, against the action it prepared. The platform overview carries the same queue as an **Approvals** card, counting the agent actions awaiting a human and linking straight into Attention. Approving allows exactly what was proposed and nothing wider; rejecting ends that path rather than leaving the action pending. > [!WARNING] > > Nothing pushes an approval at you today — pending decisions wait to be found rather than arriving as > a notification. If a workflow depends on a timely decision, make checking the inbox part of > somebody's routine. Approvals are answered in the console. An agent cannot approve its own held action, and there is no chat or tool path to answer one — deciding is deliberately a human-only surface. ## What the decision leaves behind Every approval and rejection becomes part of the run's durable [evidence](/manage-your-agents/run/runs-evidence), and the guardrail decision behind a held action is recorded whichever way it goes. Afterwards you can answer both questions that matter: what was proposed, and who let it happen. A run that was held for a human is also distinguishable from one that resolved untouched — so "was a person in this loop?" is a property of the run rather than something you have to reconstruct from timestamps. ## Next steps > **[Standards & Guardrails](/security-and-governance/controls/standards-guardrails)** > > The rules that decide when an action needs a person at all. > **[Runs & evidence](/manage-your-agents/run/runs-evidence)** > > Where a paused run and its decision live. > **[Audit log](/security-and-governance/controls/audit-log)** > > The record of who decided what. # Audit log Source: https://docs.komodor.com/security-and-governance/controls/audit-log An append-only record of every governed change and an explicit set of sensitive reads. The audit log answers "who changed this, and when" for your account. It records every governed change and an explicit set of sensitive reads. Find the log under **Settings → Audit log**. ## The property that matters **Entries cannot be edited or selectively deleted.** An entry leaves the log only when it ages out under retention or the account is deleted. The name on an entry is saved when it is written, not looked up at read time, so renaming a member or an agent later does not rewrite history. ## What is recorded Governed changes happen across every area of the platform. These are the common ones: | Area | Examples | | - | - | | **Members** | Invitations, suspensions, restorations | | **Service accounts** | Creation, disabling, re-enabling | | **API keys** | Creation, including which roles were granted; revocation; revealing a token | | **Authorization** | Role, policy, and grant changes; role assignments; attribute changes | | **Account** | Settings updates, trusted-domain changes | | **Agents** | Creation, enable and disable, archiving, worker token creation and revocation, label and tool-group changes, invocations, chat-session lifecycle | | **Guardrails** | Rule creation, update and deletion; approving or denying a held action | | **Skills** | Creation, update, publishing and deletion; attaching to and detaching from an agent | | **Runs** | User-initiated actions — create, revive, cancel, grade | | **Workflows** | Creation, update, deletion, and runs | | **Triggers & schedules** | Creation, update, deletion, credential rotation | | **Channels & routing** | Creation, update, deletion, pause and resume, route changes | | **Integrations** | Connecting, creating, deleting, testing a connection | | **Credentials** | Creation, update, value replacement, deletion, binding and unbinding | | **MCP servers** | Creation, update, deletion, authorization, credential binding, groups and policies | | **Knowledge bases** | Lifecycle, document ingest, reindex and delete, agent access grants | | **Quality** | Score and judge configuration changes | Each entry records the action and the area it belongs to, **who** did it — including who was acting on whose behalf, where an action was taken on someone's behalf — **when**, and whether it **succeeded or failed**. Where an update records changed fields, the entry carries the old and the new value for those fields. ## What is not recorded * **Ordinary reads.** Reading a run or listing agents is not audited, and neither is an Access explorer query — it evaluates access without changing it. Revealing an API key's token is the notable read that is recorded. * **Message content.** Chat and channel message bodies live in their own run and evidence ledgers, not here. * **Secrets, ever.** Revealing an API key records *that* the key was revealed, never the token. ## Reading the log Entries are listed newest first. Search by action, operation, or actor, filter by area and status, and sort by the When, Action, Actor, or Status column. Open an entry for its full detail — actor, operation, status, and the recorded change. Reading the log is its own capability. The built-in **Viewer**, **Owner**, and **Admin** roles can read it. That lets you give someone read-only oversight without granting permission to change anything. ## Scope The audit log covers your account and the governed actions available to your team. ## Next steps > **[Roles & permissions](/security-and-governance/identity-and-access/roles-permissions)** > > Who can read the log, and how that is granted. > **[Approvals](/security-and-governance/controls/approvals)** > > Human decisions, and where they are recorded. > **[Runs & evidence](/manage-your-agents/run/runs-evidence)** > > The other ledger — what an agent saw and did. > **[Data retention, deletion & export](/security-and-governance/architecture-considerations/data-retention-deletion-export)** > > How long-lived records differ from archive and account deletion. # Standards & Guardrails Source: https://docs.komodor.com/security-and-governance/controls/standards-guardrails The standards every new agent inherits, and the rules that block, hold, redact, or record an action at the boundary. Permissions decide what an identity may do; standards and guardrails decide how it is built and what it may do *right now*. Standards apply your organization's rules at the moment an agent is created, and guardrails inspect each action as it crosses the boundary, refusing or rewriting it before it lands. ## Standards — applied when an agent is created When an agent is created, your organization's **standards pack** applies automatically. The point is that a new agent cannot quietly ship without an owner, a budget, or grading, because those are not steps somebody has to remember. | Standard | What it settles | | - | - | | **Guardrails** | The default posture for the agent's actions — read-only work proceeding on its own, privileged diagnostics asking first, and mutations always asking. | | **Quality judges** | Which judges grade its runs, and at what sampling rate. | | **Memory & knowledge** | The history and knowledge the agent can draw on. | | **Cost budget** | A per-run and per-month ceiling. | | **Ownership** | The team that owns this agent, and where they are reachable. | | **Escalation route** | Where something goes when the agent cannot finish it. | | **Docs** | An agent card generated from the above: what it does, what it may do, and who owns it. | Each standard is listed with the value it applies, and the pack is scored out of seven — **7/7** when everything applies. The **standards scorecard** sits beside the agent summary, and that is what you confirm before creating. > [!NOTE] > > Every standard except **Docs** carries a **Waive** switch. Waiving one drops the score — 7/7 becomes > 6/7 — and the row shows **Waived** in place of its value rather than disappearing. The governance > value here is legibility, not enforcement: an agent with a waiver is fine, an agent whose waiver > nobody can see is not. ## Guardrails — enforced at the action boundary A guardrail is a rule you configure that decides whether a specific action may proceed. It is evaluated **outside the agent**: an agent cannot skip a guardrail, because the check does not run in the agent's own process. It runs at the boundary the agent's traffic has to cross. You author them under **Connect → Integrations**, in the **Govern → Guardrails** section, where **New guardrail** opens the authoring flow and the list gives each rule its gate, its matcher, and what it does **On a match**. **Policies**, beside it under Govern, is a different control and not where guardrails are written. Reading the rules and running a backtest is its own capability, held by the built-in **Viewer**, **Owner**, and **Admin** roles; creating, editing, enabling, disabling or deleting one is an administrator action, held by **Owner** and **Admin**. Without the read capability the section does not appear at all, so grant it before pointing someone at this screen. > [!WARNING] > > A guardrail governs traffic that goes through the platform's gateways. An agent reaching an external > system by some other path is outside this control — which is the strongest argument for routing > everything through the built-in integrations or the > [MCP Gateway](/security-and-governance/architecture-considerations/mcp-gateway). > > One supported topology is the exception worth naming. A self-hosted agent pointed at a colocated > [Outpost](/security-and-governance/architecture-considerations/outposts)'s local listener sends the > calls it routes locally straight into your own network, so they never reach the gateway: **Tool > call** and **Tool response** rules cannot see them, and they leave no decision record. Enabling that > listener is a deliberate deployment choice. It does not affect the other three gates — **Agent > start** is decided when the run is created, and **Model request** and **Model response** at the > model gateway. ### The five gates A guardrail stands at one gate, and the gate is fixed for the life of the rule. | Gate | What it inspects | | - | - | | **Agent start** | The payload that starts a run, as it arrives — checked before the run exists, so nothing has been stored. | | **Tool call** | A tool call on its way out, including its arguments. | | **Tool response** | What a tool returned, on its way back to the agent — the call has already run. | | **Model request** | What is being sent to a model, including the instructions the agent was given. | | **Model response** | What the model sent back — the answer has already been generated and billed. | ### Changing a rule that is already live A rule opens from its name in the list as **Edit guardrail**, and every part of it except its gate can change. The **Enabled** switch stops a rule without removing it, which is usually what you want: deleting asks first and offers to disable it instead, because a delete takes the control away with no undo. A rule declared by a [configuration cluster](/security-and-governance/identity-and-access/configuration-clusters) carries a **Cluster-managed** badge, and while that cluster owns it the switch and the delete are inactive: change the manifest instead. ### What a guardrail does on a match | Outcome | Effect | | - | - | | **Block** | The action is refused, and the agent is told which rule refused it. This is the default, because someone authoring a rule is asking for a control. At the two response gates it withholds rather than reverses: the tool has already run and its effects stand, and a model answer has already been generated and billed. What the block prevents is the agent reading it, and acting on it. | | **Hold for approval** | Available at the **Tool call** gate. The call is refused on the spot and an approval naming that exact call is registered for a person to answer; once approved, the call goes through the next time it is attempted. See [Approvals](/security-and-governance/controls/approvals) for what the decider sees. | | **Redact** | The action proceeds with every matched value replaced by `[redacted]`. A redacting rule has to be able to point at the value it removes, so at the **Tool call** gate it needs at least one argument condition: a tool pattern names an action rather than a value. If a rule matches and the value cannot be removed, the action is refused instead of going through unchanged. | | **Record only** | The action proceeds and the match is recorded — the detect-only posture, for a rule you want measured before you enforce it. | Every outcome is recorded, block and record only alike, so a rule that let an action through leaves the same trail as one that stopped it. **Recent decisions**, below the rule list, is that trail: when, which agent, the outcome, the rule that decided, and the tool it was deciding about. When more than one rule covers the same action, the most severe outcome decides: block, then hold for approval, then redact, then record only. A rule that only records can never soften one that stops the action. Redaction is the exception to a single rule deciding — every covering redaction applies, so two rules removing two different values both take effect. > [!NOTE] > > Redaction reads differently in each direction. On the way *in* — a tool result, a model answer — the > agent can see that something was removed and reason about it. On the way *out* the agent composed > the text and is not shown the rewrite, so it may go on to interpret a reply to a question it did not > actually ask. When the agent needs to act on what comes back, block is the honest choice. ### Who a rule covers Target a guardrail at explicit agents, or at a label bag the calling agent's own labels must contain. Labels are the durable option: a rule scoped to `env = production` keeps covering the right agents as the fleet changes. Set both and the rule covers an agent that matches either one. A rule with no target at all is the account-wide rule. It covers every agent in the account, including ones added later, and it covers calls that carry no agent identity — a tool call made from a person's own session, or a model request sent with the account key rather than an agent's. Naming any target excludes those, so narrowing a rule to a few agents also stops it covering them. ### Matching a call The authoring form is a picker rather than a syntax. It lists the tools your account actually has, grouped by server, so you are choosing from reality instead of guessing at tool names. A glob pattern is available as an alternative — a selection covers the tools that exist now, a pattern covers the ones that arrive later — and the form reports how many of your current tools a pattern matches. > [!WARNING] > > A pattern that matches nothing is the mistake to watch for: it produces a guardrail that looks > active and protects nothing. Check the match count the form shows you. You can also condition on a tool's **arguments**, chosen from the real argument names of the tools you picked, and on **named value classes** — a payment card number, a cloud access key — which match a kind of value rather than a string you had to write yourself. A rule can also match on judgment rather than on text. Choose **Model judgment** as the matcher, state a **Policy** in plain language — *no customer identifiers leave the account* — and pick which of your account's models is **Judged by** it. That model is shown the intercepted action and answers whether it breaks the policy. A judged policy reads the same at any of the five gates. A rule can carry both shapes at once, and they combine in opposite directions: **any** criterion matching fires the guardrail, while the conditions *within* a single criterion must all hold. An either/or is two criteria; a narrowing is two conditions on one. > [!NOTE] > > A judged policy cannot drive a redaction. It answers whether a rule was broken, not where the > offending value sits, so there is nothing to excise; the form refuses the pairing. Use a judged > policy to block, hold, or record only. > [!TIP] > > The empty state offers the three rules most accounts want first, so a first guardrail is an edit > rather than a blank page: **no destructive Kubernetes actions**, **nothing that writes to > production**, and **no payment card numbers reaching a model**. The third one is a redaction rule at > a model gate using a named value class — a useful shape to read even if you do not keep it. ### Test a rule before it is live A draft rule at the **Tool call** gate, matching on patterns, can be **backtested**: **Check the last 7 days** runs it against the tool calls your agents recorded in that window and reports how many it would have caught. That is what makes it reasonable to write a blocking rule against production: backtesting shows how the rule would have affected the recorded traffic you tested it against, rather than leaving you to discover it live. It reads what was recorded, so treat it as a sample of what happened rather than a complete account of it, and the panel says so when that history cannot be read or when it reached only the most recent calls in the window. The other gates have nothing to replay, and neither does a condition judged by a model. For those, save the rule as **Record only**, watch what it catches on live traffic, then switch it to block. ## Which one to reach for | Situation | Use | | - | - | | Every new agent must have an owner and a budget | A standard | | This class of tool call must never happen | A guardrail set to block | | A secret must not reach a third-party tool | A guardrail set to redact | | I want to measure before enforcing | A guardrail set to record only, plus a backtest for a tool-call rule | | A person should decide this case by case | [Approvals](/security-and-governance/controls/approvals) | ## Next steps > **[Approvals](/security-and-governance/controls/approvals)** > > When the right answer is a human decision rather than a rule. > **[Roles & permissions](/security-and-governance/identity-and-access/roles-permissions)** > > The permission layer a guardrail sits on top of. > **[Audit log](/security-and-governance/controls/audit-log)** > > Where guardrail decisions are recorded. > **[Data handling & redaction](/security-and-governance/architecture-considerations/data-handling-redaction)** > > What is masked without any rule at all. # How it works Source: https://docs.komodor.com/security-and-governance/how-it-works The six layers that decide what an agent may do — identity, permissions, standards, runtime guardrails, approvals, and the record of every decision. Letting an agent act on production is a governance problem before it is a technical one. The Komodor Agentic Operation Platform (KAOP) answers it in six layers: every actor has an identity, every identity holds permissions, every new agent inherits your standards, governed traffic can cross runtime gates, a consequential action can wait for a person, and governed decisions leave a record. ## The six layers > **[Identity](/security-and-governance/identity-and-access/signing-in)** > > People, service accounts, and agents are all first-class principals in an account. Nothing acts > anonymously. > **[Permissions](/security-and-governance/identity-and-access/roles-permissions)** > > Roles and grants decide which capabilities an identity holds, and on which resources. Default > deny, additive grants. > **[Standards](/security-and-governance/controls/standards-guardrails)** > > Every agent is created against your organization's standards — ownership, budget, judges, > guardrails — with waivers shown rather than hidden. > **[Runtime gates](/security-and-governance/controls/standards-guardrails)** > > Configured gateways can inspect model and tool traffic, then allow, refuse, or redact it before > it continues. > **[Approvals](/security-and-governance/controls/approvals)** > > A consequential action can wait for a person, with the evidence needed to decide. > **[The record](/security-and-governance/controls/audit-log)** > > Governed changes and selected sensitive reads are written to the audit log with their actor and > outcome. ## The principle underneath **An agent is governed the same way a person is, and by the same machinery.** An agent has its own identity, holds its own roles, and is refused by the same enforcement that refuses a human — which is what makes "what can this agent do?" a question with an answer rather than an assumption. Two properties follow from that, and they are worth stating because they are what makes the model trustworthy: * **Authorization is enforced where the request lands.** An agent cannot grant itself a capability by changing its own code. Runtime guardrails separately apply to traffic sent through configured KAOP gateways. * **The identity is the credential.** A worker's token resolves the agent and its account on its own. An agent cannot present someone else's name and be believed. ## What each layer stops | Layer | The question it answers | What it stops | | - | - | - | | Identity | Who is acting? | An action nobody is accountable for | | Permissions | Are they allowed to act on this resource? | An identity reaching beyond its job | | Standards | Was this agent built to our rules? | A new agent shipping without an owner, a budget, or grading | | Guardrails | May *this* action proceed, right now? | A permitted capability being used destructively | | Approvals | Should a person decide instead? | An irreversible action taken without a human | | Audit log | What happened, and who did it? | A change nobody can reconstruct | ## Reading order Start with [Signing in](/security-and-governance/identity-and-access/signing-in) and [Roles & permissions](/security-and-governance/identity-and-access/roles-permissions) — they establish the vocabulary every other page uses. Then read [Standards & Guardrails](/security-and-governance/controls/standards-guardrails) for the controls that act at runtime, and the Architecture considerations section for how accounts, secrets, data, and network traffic are isolated underneath. ## Next steps > **[Agent identity](/security-and-governance/identity-and-access/agent-identity)** > > How an agent authenticates and what it is permitted to do. > **[Access explorer](/security-and-governance/identity-and-access/access-explorer)** > > Check what an identity can actually do, before you change anything. > **[Agent isolation & tenancy](/security-and-governance/architecture-considerations/agent-isolation-tenancy)** > > The account boundary, and how it is kept. > **[Secrets & credential handling](/security-and-governance/architecture-considerations/secrets-credential-handling)** > > How a secret reaches a run without being exposed. # Access explorer Source: https://docs.komodor.com/security-and-governance/identity-and-access/access-explorer Ask what an identity can actually do, and test one action against one resource, before you grant or revoke anything. In the Komodor Agentic Operation Platform (KAOP), Access explorer lets you reason about access without trial and error. Its two tabs answer two different questions: **Effective access** reports everything a principal can do, with the provenance of each capability, and **Access check** dry-runs a single "can this principal do this on that?" decision using the same check the engine runs. Find it under **Settings → Access explorer**. ## Why it exists Once access is composed from roles, policies, grants, glob patterns, and label selectors, "what can this identity do?" stops being answerable by reading the configuration. A member might reach a resource through a role, through a direct grant, or through a selector that matches their own team attribute — and the union of those is the only answer that matters. That gets harder in the direction that counts most: **before** a change. Adding a policy to a role affects everyone holding it, and removing a grant may quietly break an agent that depended on it. Access explorer lets you look before you move. ## Effective access Pick a principal and you get everything it can currently do, the scope each capability applies at, and **where each one came from** — which role, policy, or direct grant produced it. Use it to: * Confirm a new member holds what you intended and nothing more. * Understand why a principal can reach a resource — the provenance tells you which grant is doing it, so you know what to change. * Check what an agent is able to do before you point it at production. ## Access check The other tab is a dry run of one decision: *can this principal perform this capability on this specific resource, right now?* It is the same check the engine runs — the same glob patterns, the same label selectors — so the answer is the answer rather than an approximation of it. > [!TIP] > > This is the fastest way to debug a label selector. A grant scoped to `team = $principal.team` fails > closed when the identity has no `team` attribute, and a check on one concrete resource shows you > that immediately. ## Who can use it Access explorer has its own permission. An administrator can grant access to the people who need to review authorization without granting permission to change roles, policies, or grants. Running an effective-access query or access check does not change the authorization configuration. ## Next steps > **[Roles & permissions](/security-and-governance/identity-and-access/roles-permissions)** > > The model this screen evaluates. > **[Service accounts](/security-and-governance/identity-and-access/service-accounts)** > > Manage the non-human identities whose access you inspect. > **[Agent identity](/security-and-governance/identity-and-access/agent-identity)** > > Checking an agent rather than a person. > **[Audit log](/security-and-governance/controls/audit-log)** > > Review changes to roles, policies, and grants. # Accounts & members Source: https://docs.komodor.com/security-and-governance/identity-and-access/accounts-members The account as an isolation boundary, belonging to several at once, and how to invite, suspend, and restore the people in one. An account is your team's space: agents, runs, integrations, credentials, keys, and members all belong to one, and data is isolated to it. You act in exactly one account at a time even when you belong to several, and nothing in one is reachable from another. ## The account boundary Everything is owned by exactly one account, and you always act in exactly one at a time. An agent, run, credential, knowledge base, or channel in one account is never visible from another — see [Agent isolation & tenancy](/security-and-governance/architecture-considerations/agent-isolation-tenancy) for how that is kept. Friendly names are scoped to the account rather than shared globally, so two teams can each have an agent called `triage` or a credential called `github` with no collision between them and no way to detect each other's. ## Belonging to more than one You can be a member of several accounts — because you work across teams, or because more than one trusts your verified email domain. When you belong to more than one, an **account switcher** lists them; picking one changes which account you are viewing and acting in, and the switch is recorded in the [audit log](/security-and-governance/controls/audit-log). With a single membership the switcher is hidden. ## Managing members **Settings → Members** lists everyone in the account. ### Inviting Invite by email. The membership is created immediately and activates on that person's first sign-in — there is no acceptance step. You can give an invitation a display name so the row is recognizable before they arrive, and assign roles as part of it. ### Suspending and restoring **1. Suspend to revoke access** Suspension takes effect immediately: live browser sessions stop working on their next request, and any API keys bound to that person stop authenticating. **2. A suspension is never undone silently** Even if the person's email domain is still trusted by the account, signing in again does not reactivate them. Only an explicit restore does. **3. History is preserved** Suspending removes access, not work: runs and other records stay attributed to them. ### Roles and attributes Each member holds one or more roles, which decide which policies — and therefore which capabilities — they receive. Roles and **attributes** are both columns on the Members screen: an attribute is a key–value pair like `team = payments`, and label-scoped grants match against it, so one grant can give everybody access to their own team's resources. Editing attributes is itself a governed capability, so it stays with administrators by default. See [Roles & permissions](/security-and-governance/identity-and-access/roles-permissions) for the full model. ## Next steps > **[Roles & permissions](/security-and-governance/identity-and-access/roles-permissions)** > > Built-in roles, custom roles, and how a grant is scoped. > **[Service accounts](/security-and-governance/identity-and-access/service-accounts)** > > The identity automation should use instead of a person's. > **[Access explorer](/security-and-governance/identity-and-access/access-explorer)** > > Check what a member can do before and after a change. > **[Audit log](/security-and-governance/controls/audit-log)** > > Invitations, suspensions, and restorations are all recorded. # Agent identity Source: https://docs.komodor.com/security-and-governance/identity-and-access/agent-identity How KAOP gives every agent its own governed identity, permissions, and revocable worker token. In the Komodor Agentic Operation Platform (KAOP), every agent is a governed principal alongside people and service accounts. It belongs to one account, acts under its own identity, and receives only the roles and credentials assigned to it. ## An agent is the actor Runs and evidence are attributed to the agent that performed the work, even when a person, schedule, or workflow initiated the run. When a person starts a run manually, the audit log also records the invocation under that person's identity. This preserves both the agent's operating history and the human action that initiated it. Agents receive a built-in **Agent** role by default. An authorized administrator can change an agent's role assignments to match its job. Use [Roles & permissions](/security-and-governance/identity-and-access/roles-permissions) to define the capabilities and resource scopes, then verify the result in [Access explorer](/security-and-governance/identity-and-access/access-explorer). ## Agent ID and display name An agent has an **Agent ID** and a **display name**. The Agent ID is a normalized slug, such as `payments-triage`, used in URLs, configuration, and by the agent itself. It is unique within the account. The display name is the more readable label shown in the console. ## How a self-deployed worker authenticates A self-deployed worker uses a **worker token** bound to one agent. KAOP resolves the agent and account from the token, then verifies that the agent identifier presented by the worker matches the token. The SDK attaches the token to registration, heartbeat, run-claim, and evidence requests. Configure the worker with environment variables or an equivalent secret-injection mechanism: | Variable | Purpose | | - | - | | `AGENTOPS_URL` | The KAOP control-plane URL. | | `AGENTOPS_WORKER_TOKEN` | The token that authenticates this agent. | | `AGENTOPS_AGENT_ID` | Optional override for the agent identifier used by a reusable worker image. It must match the token's agent. | | `AGENTOPS_WORKER_ID` | Optional identifier for one worker process or replica. | Keep the token out of source control, container images, and logs. ## Manage worker tokens Open **Settings → Agents → Worker tokens** to see each token's status, creation time, expiration, and associated agent. A token value is shown only when it is issued or rotated. **1. Issue or copy the token** Store the value in your deployment's secret manager before leaving the screen. **2. Rotate the token** KAOP issues a replacement and invalidates the previous value. Update the deployed secret before expecting the worker to authenticate with the new token. **3. Revoke the token** Prevent the token from authenticating again. Revocation does not terminate the worker process. Worker-token creation, rotation, and revocation are security-relevant changes and appear in the [audit log](/security-and-governance/controls/audit-log). ## Next steps > **[Access explorer](/security-and-governance/identity-and-access/access-explorer)** > > Inspect an agent's effective access before it runs. > **[Agent isolation & tenancy](/security-and-governance/architecture-considerations/agent-isolation-tenancy)** > > Understand account and agent boundaries. > **[Standards & Guardrails](/security-and-governance/controls/standards-guardrails)** > > Add runtime checks to governed model and tool traffic. > **[Credentials & secrets](/security-and-governance/identity-and-access/credentials-secrets)** > > Bind credentials to only the agents that need them. # API keys Source: https://docs.komodor.com/security-and-governance/identity-and-access/api-keys Create, use, and revoke KAOP bearer tokens, and choose whether each key acts as a service account or as you. An API key lets a script, CI job, or integration call the Komodor Agentic Operation Platform (KAOP) API without a browser session. Every key acts as a specific identity, so its permissions and audit attribution come from that identity. Manage keys under **Settings → API keys**. ## Create a key Choose **New API key** and configure: | Field | What it controls | | - | - | | **Name** | A recognizable description of where the key is used. | | **Acts as** | You, a new service account created for this key, or an existing service account. | | **Roles** | Required for a new per-key service account. Other choices inherit the selected identity's current roles. | | **Expires** | A preset from one hour up to 90 days, a custom date and time, or never. | KAOP displays the token at creation. It can also be revealed later from the API keys screen by you or an administrator, and reveal events are recorded in the [audit log](/security-and-governance/controls/audit-log). Store the token in a secret manager and avoid copying it into source code, build logs, or command history. ## Choose the identity | Identity | Use it when | | - | - | | **Service account** | Automation must survive team changes or needs permissions independent from a person. A new per-key service account receives the roles selected at creation; an existing one uses its current roles. | | **You** | The script is personal and should have the same access and lifecycle as your membership. You or an administrator can reveal or revoke the key. | For shared or long-running automation, prefer a service account. A personal key stops authenticating when the member behind it is suspended. ## Use a key Send the token as a bearer token on each request: ```bash theme={null} curl -H "Authorization: Bearer $KAOP_TOKEN" \ "$AGENTOPS_URL/api/v1/agents" ``` The request is authorized using the current roles of the identity behind the key. Changing an existing service account's roles changes what its keys can do without reissuing them. ## Key lifecycle * **Status** shows whether a key is active, expired, or revoked. * **Last used** helps identify keys that can be retired. * **Revoke** permanently invalidates the key. Issue and deploy a replacement first if the caller must remain available. * Revoking a key that created its own service account also disables that per-key account. Revoking a key for an existing service account or member does not disable the underlying identity. * Disabling the identity behind a key prevents that key from authenticating. Restoring a managed service account restores its still-valid keys; a revoked key stays revoked. ## Next steps > **[Service accounts](/security-and-governance/identity-and-access/service-accounts)** > > Create a durable identity for automation. > **[Roles & permissions](/security-and-governance/identity-and-access/roles-permissions)** > > Control what the identity behind a key can do. > **[APIs](/developer-tools/apis)** > > Explore the endpoints available to authenticated callers. > **[Audit log](/security-and-governance/controls/audit-log)** > > Review key creation, reveal, and revocation events. # Configuration clusters Source: https://docs.komodor.com/security-and-governance/identity-and-access/configuration-clusters Declare KAOP resources from Kubernetes manifests and understand cluster ownership, reconciliation, and edit behavior. Configuration clusters let a platform team manage supported Komodor Agentic Operation Platform (KAOP) resources from Kubernetes manifests kept in source control. An operator in your cluster reconciles those declarations through the KAOP API and records which cluster owns each resource. ## Connect a cluster Open **Settings → Configuration clusters**, choose **Add cluster**, and follow the generated instructions. **1. Save the credential** KAOP displays the cluster credential once. Store it securely before leaving the screen. **2. Create the Kubernetes Secret** Use the generated command to place the credential in a Kubernetes Secret. The credential is not embedded in the Helm command or stored as a Helm value. **3. Install the operator** Run the generated Helm command. It includes the KAOP URL, namespace, and Secret name required by the operator. **4. Confirm the first sync** Return to the configuration-clusters list and verify that the cluster reports a recent sync. The generated install commands remain available from the cluster row because they do not contain the credential value. ## Read cluster status Each cluster reports its namespace, operator version, declared-resource count, and last sync. | Status | Meaning | | - | - | | **Reconciling** | The operator is connected and applying the cluster's declarations. | | **Paused** | KAOP retains ownership information but reconciliation is paused. | | **Not connected yet** | The configuration exists, but its operator has not completed a sync. | ## Understand the Cluster-managed badge Resources declared by an operator carry a **Cluster-managed** badge, or **Cluster-managed, not enforced** when the owning cluster is paused. The effect of a console or API edit depends on the cluster's state and ownership enforcement: | State | What happens to an edit made in KAOP | | - | - | | Ownership is enforced | The edit is refused. Change the manifest instead. | | Reconciliation is active without write enforcement | The edit can succeed, but a later reconcile restores the manifest's declaration. | | Reconciliation is paused or the cluster was removed | The edit remains until reconciliation resumes, or remains unmanaged after removal. | Ownership enforcement is turned on per account, and it is off by default — reconciliation and the badge above work the same way either way, but an edit made in KAOP is only ever refused once enforcement is on for your account. Treat the manifest as the source of truth whenever a resource is cluster-managed. For the manifests themselves — the supported kinds, an annotated example, and how to read a resource's status — see [Declaring resources](/security-and-governance/identity-and-access/declaring-resources). ## Pause, resume, or delete a cluster * **Pause** stops reconciliation while preserving the cluster and ownership records. * **Resume** reconciles the manifests again and can overwrite changes made while paused. * **Delete** revokes the cluster's credential and stops managing its resources. It does not delete the declared resources from KAOP, and it cannot be undone. > [!WARNING] > > Deleting a cluster connection does not uninstall an operator that is still running in Kubernetes. > Remove or disable that deployment separately. ### Rotate the credential Rotating the credential is the only way to recover from a lost one — KAOP shows it exactly once and cannot read it back. Rotating stops the operator reconciling until it is redeployed with the new Secret, and today it is only available through the API: `POST /api/v1/iac-cluster-installs/{install_id}/credential`. ## Next steps > **[Declaring resources](/security-and-governance/identity-and-access/declaring-resources)** > > The kinds a cluster can declare, an annotated manifest, and how to read status. > **[Audit log](/security-and-governance/controls/audit-log)** > > Review cluster connection and lifecycle changes. > **[Roles & permissions](/security-and-governance/identity-and-access/roles-permissions)** > > Control who can view or manage configuration clusters. > **[Credentials & secrets](/security-and-governance/identity-and-access/credentials-secrets)** > > Manage one of the resource types a cluster can declare. > **[MCP Gateway](/security-and-governance/architecture-considerations/mcp-gateway)** > > Configure governed access to MCP servers. # Credentials & secrets Source: https://docs.komodor.com/security-and-governance/identity-and-access/credentials-secrets Store encrypted credentials, bind them to specific agents, and control how their values are delivered and replaced. The Komodor Agentic Operation Platform (KAOP) stores credentials separately from agent code and configuration. A credential is encrypted at rest, its value is write-only in management surfaces, and a binding determines which agent may receive it. Manage them from **Credentials**. ## Credential model A credential has a stable name and one of two shapes: | Shape | Use it for | | - | - | | **Single value** | One token, password, or secret string. | | **Multiple keys** | Related values that should rotate together, such as an OAuth client ID and client secret. | Credential reads return metadata such as the name, description, owner, status, labels, key names, and usage information. They do not return the stored value. Replacing the value creates the next active version while preserving the name and bindings. ## Create a credential **1. Choose New credential** Select a single value or multiple keys. For multiple keys, enter rows directly, paste JSON, or upload a `.json` or `.env` file. **2. Use a stable name** Credential names cannot be changed after creation. Choose the name your agent code and integrations should continue using through future rotations. **3. Add ownership metadata** Add a description, owner, and labels so administrators can identify the credential and apply scoped access controls. **4. Bind agents** Select the agents allowed to receive the credential, or create it unbound and grant access later. If the account already stores the same value, KAOP reuses the existing credential instead of creating another copy. The create flow identifies the credential that was reused. ## Bind only the agents that need it A credential is not delivered to an agent until a binding authorizes it. You can manage bindings from the credential's access view or from an agent's permissions and secrets view. > [!TIP] > > For a multi-key credential, restrict a binding to selected keys when the agent does not need the > whole set. A binding without key restrictions delivers every key. | Delivery mode | Behavior | | - | - | | **Run-scoped** | The credential is delivered with each run and is available while the handler executes. Use this by default. | | **On-demand** | The worker retrieves the credential outside an individual run, typically during startup. Use it only for a dependency required before a run begins. | Removing a binding prevents future authorized delivery. It does not erase a value already held by a running process, so revoke the value at its provider when exposure is possible. ## Read a run-scoped credential in Python Use the SDK accessors inside the run handler: ```python theme={null} from komodor_agentops import get_secret, get_secret_dict, get_secret_key api_key = get_secret("DATADOG_API_KEY") client_id = get_secret_key("google-workspace-oauth", "client_id") oauth = get_secret_dict("google-workspace-oauth") ``` For multi-key credentials, environment-variable names use `CREDENTIAL_NAME__KEY`. Names are normalized to uppercase with unsupported characters replaced by underscores; for example, `google-workspace-oauth` and `client_id` become `GOOGLE_WORKSPACE_OAUTH__CLIENT_ID`. Read run-scoped values inside the handler rather than at module import time. ## Masking and its limits The SDK masks exact occurrences of delivered credential values from run output, events, diagnostics, and saved state before reporting them. A value that has been encoded, split, hashed, truncated, or otherwise transformed may no longer match and may not be masked. > [!WARNING] > > Do not deliberately send credentials to models, tools, logs, knowledge bases, instructions, or > skills. Treat automatic masking as protection against accidental echoes, not as a substitute for > least privilege and safe agent code. See [Data handling & redaction](/security-and-governance/architecture-considerations/data-handling-redaction) for the boundary between credential masking, configurable guardrails, and custom network traffic. ## Replace, disable, and delete | Action | Effect | | - | - | | **Replace value** | Activates a new value without changing the credential name, labels, or agent bindings. Future runs receive the replacement. Restart a worker that loaded an on-demand value at startup. | | **Disable** | Stops future delivery while keeping the credential and its configuration. Re-enable it if the dependency is still required. | | **Delete** | Permanently removes the credential and its agent bindings. Review its usages first. | Before deletion, KAOP checks consumers such as agents, integrations, and MCP servers. Static header references must be removed before deletion. For dependent MCP servers, the deletion flow lets you choose whether to remove them or leave them configured without the credential. Platform-managed credentials may also appear in the list. They support managed KAOP services and cannot be replaced or deleted from the customer credential workflow. ## Next steps > **[Built-in integrations](/manage-your-agents/build/built-in-integrations)** > > Connect services that use stored credentials. > **[MCP Gateway](/security-and-governance/architecture-considerations/mcp-gateway)** > > Inject a credential into calls to an MCP server. > **[Secrets & credential handling](/security-and-governance/architecture-considerations/secrets-credential-handling)** > > Review the credential security architecture. # Declaring resources Source: https://docs.komodor.com/security-and-governance/identity-and-access/declaring-resources Declare KAOP resources as Kubernetes custom resources, read their status, and understand how deletion and secrets work. Declaring a resource for the Komodor Agentic Operation Platform (KAOP) means applying a Kubernetes custom resource — a `Credential`, an `McpServer`, or one of the other supported kinds — to a cluster running the operator described in [Configuration clusters](/security-and-governance/identity-and-access/configuration-clusters). The operator reconciles what you declare through the KAOP API and writes the result back onto the resource's own `status`, so the manifest is the source of truth without a separate API call of your own. ## How a declaration maps to KAOP * Every kind is in the `agentops.komodor.io/v1alpha1` API group, and every resource is namespaced. * The namespace binds the resource to one account: the cluster install running in that namespace is tied to a single KAOP account. * `spec.name` is the resource's name in KAOP and defaults to `metadata.name`. Changing `spec.name` does not rename anything — it points the declaration at a different name, so the old resource is left behind. * A field you set in `spec` is reapplied on every resync (`operator.resyncInterval`, default `5m`), so an out-of-band change to it is corrected back. A field you leave unset stays under KAOP's own control. * KAOP remains the validator. A spec the CRD's schema accepts can still be rejected — the reason appears on the resource's `Synced` condition. ## Supported kinds | Kind | Declares | Related page | | - | - | - | | `Credential` | A stored value, delivered from a referenced Secret. | [Credentials & secrets](/security-and-governance/identity-and-access/credentials-secrets) | | `EvalFlow` | An evaluation that grades a set of agents' runs against criteria, through one evaluator agent. | [Evaluations](/manage-your-agents/optimize/evaluations) | | `GoldenScenario` | A single scored test case, tagged by domain. | — | | `GoldenSuite` | A group of golden scenarios under one runner agent. | — | | `Guardrail` | A rule that blocks, holds, redacts, or records a matched action for a set of agents. | [Standards & Guardrails](/security-and-governance/controls/standards-guardrails) | | `HostedAgent` | A hosted agent's model, image, MCP servers, skills, and tools. | [Build — how it works](/manage-your-agents/build/how-it-works) | | `IncidentPipeline` | Routes an alert source to an orchestrator and specialist agents as an incident pipeline. | [Incident management & troubleshooting](/modules-and-workflows/sre/incidents) | | `IntegrationConnection` | A connection to a built-in integration provider, authenticated from a referenced Secret. | [Integrations list](/manage-your-agents/build/built-in-integrations) | | `KnowledgeBase` | A knowledge base agents retrieve from and cite. | [Knowledge base](/manage-your-agents/build/knowledge-base) | | `McpGroup` | A named bundle of MCP servers with its own allow/deny rules. | [Integration groups](/manage-your-agents/build/integration-groups) | | `McpPolicy` | A tool-call policy enforced on one or more MCP targets. | [MCP Gateway](/security-and-governance/architecture-considerations/mcp-gateway) | | `McpServer` | An upstream MCP server, its transport, and the credential it authenticates with. | [MCP Gateway](/security-and-governance/architecture-considerations/mcp-gateway) | | `Module` | Turns on a module and holds its settings. | [Modules & workflows](/modules-and-workflows/overview) | | `ModuleWorkflowTemplate` | A reusable workflow template published under a module. | [Modules & workflows](/modules-and-workflows/overview) | | `NotificationSink` | A destination — a channel or your own endpoint — that receives KAOP's events. | [Outbound webhooks](/manage-your-agents/run/outbound-webhooks) | | `ReviewWorkflow` | Binds a set of repositories and reviewer agents for automated code review. | — | | `Schedule` | Runs one agent on a recurring cron expression. | [Triggers & schedules](/manage-your-agents/build/triggers-schedules) | | `SelfHostedAgentSettings` | Settings for an agent that runs in your own infrastructure instead of KAOP's. | [Deployment methods](/get-started/deployment-methods) | | `Skill` | A versioned runbook or procedure agents can use. | [Skills](/manage-your-agents/build/skills) | | `WebhookTrigger` | An inbound endpoint that starts an agent run from an HTTP request. | [Integrations list](/manage-your-agents/build/built-in-integrations) | Every kind shares the same `spec`/`status` contract above. For the full set of fields a kind accepts, run `kubectl explain .spec` against a cluster that has the CRDs installed — each field carries its own description. ## Annotated example A `Credential` resolves its value from a Kubernetes `Secret` in its own namespace rather than holding it inline: ```yaml theme={null} apiVersion: v1 kind: Secret metadata: name: datadog-api-key namespace: agentops type: Opaque stringData: api-key: "" --- apiVersion: agentops.komodor.io/v1alpha1 kind: Credential metadata: name: datadog-api namespace: agentops spec: name: datadog-api valueSecretRef: name: datadog-api-key key: api-key ``` An `McpServer` then references that credential by name, alongside the fields that reach the upstream server itself: ```yaml theme={null} apiVersion: agentops.komodor.io/v1alpha1 kind: McpServer metadata: name: datadog-mcp namespace: agentops spec: name: datadog-mcp url: https://mcp.datadoghq.com transport: http credentialRef: datadog-api deletionPolicy: orphan ``` Apply both and read back the result. `SYNCED` is a default-visible column; `RESOURCE` needs `-o wide`: ```bash theme={null} kubectl apply -f secret.yaml -f credential.yaml -f mcpserver.yaml kubectl -n agentops get mcpservers -o wide # NAME SYNCED RESOURCE AGE # datadog-mcp True mcp_server_01h... 14s ``` ## References between resources A field ending in `Ref` — `credentialRef`, `agentRef`, `mcpGroupRef`, and the rest — resolves by the KAOP name of the resource it points at, not by the Kubernetes object name. * A reference naming something that does not exist yet reports `WaitingForDependency` on the `Synced` condition, and is retried on every resync rather than treated as a standing error. * A reference naming something that already exists but that this cluster does not declare reports `Conflict`. The existing resource is never taken over. ## Secrets Values are read from Secrets in the same namespace as the resource that references them, and the value never appears in the spec a cluster can read, nor in status. Rotation is detected through `status.secretHashes` — a per-field fingerprint of the last value pushed, so an unrotated Secret is a no-op resync rather than a repeated write. > [!WARNING] > > Applying manifests with ArgoCD, Flux, or `helm template` rather than installing with Helm against a > live cluster means the operator cannot read its own rotation-key Secret back on each upgrade. Create > that Secret yourself and name it in `rotationKey.existingSecret`, or a fresh key is generated on > every sync. ## Reading status The `Synced` condition reports whether KAOP matches the declared spec, with one of eight reasons: | Reason | Meaning | What to do | | - | - | - | | `Applied` | KAOP matches the spec. | Nothing — this is the steady state. | | `Rejected` | KAOP's own validator refused the declaration. | Read the condition's `message` for the control plane's reason, and fix the spec. | | `Conflict` | The name is already claimed by another cluster's resource, by a resource no cluster declares, or by an install whose claim this one may not take. | Resolve ownership at the resource or cluster named in the message. | | `SecretMissing` | A referenced Secret or key is not in the resource's namespace. | Create the Secret or key, or fix the reference. | | `InstallPaused` | This cluster install is paused. | Resume the install to reapply the declaration. | | `Error` | The operator could not attribute the failure to the declaration itself — an authentication failure, an exhausted retry, or a field role this build does not implement yet. | Check the operator's logs and the cluster install's credential. | | `SecretRotated` | A referenced Secret's value changed and was pushed again. | Nothing — this is a successful rotation, reported as its own event. | | `WaitingForDependency` | A `*Ref` field names a resource that does not exist yet. | Create the referenced resource; the operator retries every resync. | `status.resourceId` is the KAOP id of the managed resource. For the control plane's own wording on a failure, read the condition's message with `kubectl describe` on the resource. ## Deleting a declaration > [!WARNING] > > The default `deletionPolicy` is `delete`: deleting the Kubernetes object removes the resource from > KAOP too. Set `deletionPolicy: orphan` to leave the KAOP resource in place and release only the > operator's claim, so it can be managed elsewhere afterwards. Deletion still goes through while the > cluster install is paused — a pause stops reassertion, not an explicit delete. `kubectl delete --cascade=orphan` is about Kubernetes-owned objects and does not reach the KAOP resource either way. ## Operator scope and lifecycle * The operator watches only the namespace it is released into. Covering a second namespace means installing the chart again there with `crds.install=false`, since the CRDs are cluster-scoped and already belong to the first release. * Its permissions are namespace-scoped and narrow: no `create` or `delete` on the custom resources it reconciles, and `get` only on Secrets. * `helm uninstall` leaves the CRDs and your declared resources in place rather than deleting them. * A control plane behind an internal certificate authority needs `caBundle.existingConfigMap` pointed at a ConfigMap of PEM certificates, or the operator never becomes ready. * To check which build is running, read the image tag on the operator's pod — it is distroless, so there is no shell to exec into instead. ## Next steps > **[Configuration clusters](/security-and-governance/identity-and-access/configuration-clusters)** > > Connect a cluster and read its sync status. > **[Credentials & secrets](/security-and-governance/identity-and-access/credentials-secrets)** > > Manage credentials declared through this interface alongside the ones created in KAOP. > **[MCP Gateway](/security-and-governance/architecture-considerations/mcp-gateway)** > > Configure governed access to the MCP servers a cluster can declare. # Roles & permissions Source: https://docs.komodor.com/security-and-governance/identity-and-access/roles-permissions How KAOP combines capabilities, policies, roles, and resource scopes into default-deny access. The Komodor Agentic Operation Platform (KAOP) authorizes members, service accounts, and agents with the same default-deny model. Capabilities describe actions, policies group capability grants, roles group policies, and assignments give roles to identities. Manage the model under **Settings → Roles**, **Policies**, and **Access explorer**. ## Built-in and custom roles KAOP provides built-in roles for common responsibilities: | Role | Intended use | | - | - | | **Owner** | Full account administration. The first member receives this role, and the account must retain an owner. | | **Admin** | Delegated account administration. | | **Developer** | Build and test agents, workflows, knowledge, evaluations, and tool integrations without administering identity or secrets. | | **Operator** | Run and monitor production operations, triggers, channels, chat, and issues without administering identity or secrets. | | **Member** | Everyday use of agents, workflows, chat, knowledge, quality, and issues. | | **Viewer** | Read-only access to supported product areas; separately scoped data remains subject to its own permissions. | | **Agent** | Runtime access for agent principals without account, identity, secret, or authorization administration. | Built-in roles and their built-in policies are read-only: you can assign them, but cannot edit or delete them. Built-in role names appear with a `builtin:` prefix. Create custom policies and roles when the defaults do not match your responsibility boundaries. ## How access is composed * A **capability** is one governed action, such as invoking an agent, reading a run, or managing a trigger. * A **policy** is a named, reusable set of capability grants and their scopes. * A **role** links policies and is assigned to identities. * A **direct grant** can give a subject a role or capability over a resource scope without changing a reusable role. Access is additive. An identity receives the union of its matching grants; a request with no matching grant is denied. Adding a policy never subtracts access granted elsewhere. The console uses the same capabilities to hide or lock unavailable actions, while the API enforces authorization independently of the interface. ## Scope a grant to resources A capability says what the identity may do. Its resource scope says where it may do it. ### Account-wide scope An account-wide grant covers every resource of the applicable type in the account. Use it only when the role genuinely requires broad reach. ### Exact resource or pattern Select one resource ID, or use `*` in a pattern that must match the whole ID: | Pattern | Example match | | - | - | | `payments-*` | `payments-refunds` | | `*-staging` | `checkout-staging` | | `prod-*-worker` | `prod-us-worker` | Characters other than `*` match literally. ### Labels Label selectors scope by metadata rather than naming. For example, `team = payments` covers labeled resources for that team. A selector can also use a principal attribute: `team = $principal.team` matches resources whose `team` label equals the acting identity's `team` attribute. > [!WARNING] > > A selector that references a missing principal attribute matches nothing. Missing identity > metadata fails closed; it does not widen access. Agents, credentials, channels, knowledge bases, MCP servers, and workflows can carry labels directly. Related resources can inherit labels from their parent agent. Member and service-account attributes are managed from their identity settings. ### Expiration A grant can include an expiration time. Use expiring grants for temporary access so the permission ends without a later cleanup step. ## Verify before changing access Use [Access explorer](/security-and-governance/identity-and-access/access-explorer) to inspect an identity's effective capabilities and their provenance. Use **Access check** to test one capability against a specific resource, including its pattern and label matching, before adding or removing a grant. ## Next steps > **[Access explorer](/security-and-governance/identity-and-access/access-explorer)** > > Inspect effective access and test a single decision. > **[Agent identity](/security-and-governance/identity-and-access/agent-identity)** > > Apply roles and grants to an agent principal. > **[Standards & Guardrails](/security-and-governance/controls/standards-guardrails)** > > Add runtime controls on top of authorization. > **[Audit log](/security-and-governance/controls/audit-log)** > > Review changes to roles, policies, grants, and attributes. # Service accounts Source: https://docs.komodor.com/security-and-governance/identity-and-access/service-accounts Create durable non-human identities for automation, with independent roles, attributes, and API keys. A service account is a non-human identity in the Komodor Agentic Operation Platform (KAOP). It belongs to the account rather than to a person, so automation can keep a stable identity as team membership changes. Manage service accounts under **Settings → Service accounts**. ## Why automation should use one A service account gives a script, CI job, or integration: * Its own role assignments and resource scopes * Attribution under the automation's identity in the audit log * A lifecycle independent from the person who created it * The option to use several API keys without sharing personal credentials Prefer a service account for shared and long-running automation. Use a personal API key only when the caller should deliberately follow one member's permissions and membership lifecycle. ## Create a managed service account Choose **New service account**, enter a descriptive name, and assign at least one role. You can edit the roles and principal attributes later. Attributes can participate in label-scoped grants such as `team = $principal.team`. After creation, open [API keys](/security-and-governance/identity-and-access/api-keys) and create a key that acts as the service account. One service account can have multiple keys, which lets you rotate or revoke one integration without changing the others. ## Managed and per-key accounts The service-accounts list includes two types: | Type | Lifecycle | | - | - | | **Managed** | Created from the service-accounts screen. Its roles, attributes, and status are editable. | | **API key** | Created automatically for one API key. Its configuration is read-only and its lifecycle follows that key. | A per-key service account receives its roles when the API key is created. Revoking that key also disables its per-key account. For permissions that should evolve independently from one key, create a managed service account instead. ## Disable and restore Disabling a managed service account prevents its keys from authenticating without deleting the identity or its history. Restore it to allow still-valid, non-revoked keys to authenticate again. Past audit records and run attribution remain associated with the service account throughout. Review dependent keys before disabling an account, and use **Last used** on the API keys screen to identify active callers. ## Next steps > **[API keys](/security-and-governance/identity-and-access/api-keys)** > > Issue a bearer token for a service account. > **[Roles & permissions](/security-and-governance/identity-and-access/roles-permissions)** > > Grant only the capabilities and scopes automation needs. > **[Agent identity](/security-and-governance/identity-and-access/agent-identity)** > > Compare service accounts with runtime agent principals. > **[Audit log](/security-and-governance/controls/audit-log)** > > Review service-account and key lifecycle events. # Signing in Source: https://docs.komodor.com/security-and-governance/identity-and-access/signing-in How people authenticate through your identity provider, how a session behaves, and how someone comes to belong to an account. People sign in through your own identity provider using single sign-on, so there is no separate password for this platform to store or for you to rotate. Access is provisioned on first sign-in, so there is no separate account-creation step to complete first. ## Authentication Opening the console sends you through your organization's identity provider. On the way back, your browser holds a signed, short-lived session — nothing to configure, and no credentials held here. > [!NOTE] > > Programmatic access does not use a session. Scripts, CI jobs, and integrations authenticate with an > [API key](/security-and-governance/identity-and-access/api-keys) instead, and workers use their own > [agent credential](/security-and-governance/identity-and-access/agent-identity). ## Joining an account There is no registration step. The first time you sign in, your access is resolved from one of two paths: **1. An invitation** A teammate invites you by email from **Settings → Members**. Your membership exists from that moment and becomes active the first time you sign in with that address — there is nothing to accept. **2. A trusted email domain** An account can trust a verified email domain. If one or more accounts trust yours, signing in joins you to all of them; you land in one and reach the rest from the account switcher. If neither applies you can still sign in, but you hold no account access until you are invited. An invitation issued later takes effect on your next sign-in. The **first person in a brand-new account** becomes the account **Owner**, which holds every capability account-wide — so a fresh account can never be locked out. Everyone who joins afterwards holds whatever roles they are assigned, and an invitation can carry them. ## Sessions | Behavior | What to expect | | - | - | | **Lifetime** | A session lasts up to 24 hours. When it expires it is renewed silently against your identity provider's longer-lived session — you only see a login screen when that is gone too. | | **Signing out** | Ends both this session and your identity provider's session in that browser, so you are not signed straight back in. | | **Suspension** | If your membership is suspended, your existing session stops working on its next request rather than lasting out the day. | ## Next steps > **[Accounts & members](/security-and-governance/identity-and-access/accounts-members)** > > Managing the people in an account, and belonging to more than one. > **[Roles & permissions](/security-and-governance/identity-and-access/roles-permissions)** > > What a member can actually do once they are in. > **[API keys](/security-and-governance/identity-and-access/api-keys)** > > Access without a browser. > **[Audit log](/security-and-governance/controls/audit-log)** > > Where membership changes and account switches are recorded.