Skip to main content
The Incident management & troubleshooting module in the Komodor Agentic Operation Platform (KAOP) owns the path from an incoming alert to an understood, remediated production incident. Detection, grouping, investigation and remediation each run as steps that leave their own evidence, so the conclusion can be audited rather than trusted.

What the module takes on

Most of the work between an alert firing and a theory forming is retrieval — correlating alerts, reading metrics and logs, checking what deployed recently — and most of it repeats on the next alert. The module takes the retrieval and leaves you the decision. It groups related alerts into one incident, dispatches domain specialists to gather evidence in parallel, produces a root-cause conclusion with the evidence attached, and stages a remediation for a human to approve. Nothing reaches production without that approval.

What an incident carries

An incident carries a severity from SEV-1 to SEV-5, a status, the service, namespace, cluster and owner it concerns, when it started and how long it has been open, and — where your ticketing is connected — the ticket it is tracked under. It is the object the whole module revolves around, and it outlives the alert that created it. Its status says what is happening to it right now: An incident paused on a person does not get a status of its own. It reads Investigated and carries a Needs your answer marker, which covers a remediation choice, a guardrail hold, and a specialist waiting on a governed tool. Filter for it with the feed’s marker facet rather than by status.

Reading one incident

An incident opens into tabs. Each answers a different question, so knowing which one holds what is the difference between reading an answer and reconstructing it.
The Workflow tab’s configured-versus-actual view is the one to open when a workflow produced a disappointing result. It distinguishes a step that was misconfigured from a step that ran correctly and found nothing, which are different problems with different fixes.

The human gate

Every production mutation waits for a person. What makes that gate meaningful rather than ceremonial is what the approver is shown: the action in plain language, the literal command that will run, a risk assessment covering how reversible it is and what it touches, and the reason it was held. Approve, or reject. Afterwards a validation section reports whether the fix did what it was supposed to.

Guardrail decisions on an incident

Guardrails are configured under governance, but an incident is where you find out whether they did anything. The Guardrails tab reports how many evaluations ran in this run’s request path and how many matched, and every match is a complete record: its enforcement mode, what it did, the policy statement behind it, the standard and version it came from, the rule that matched, the principal and resource involved, how long the evaluation took, and whether it was overridable. Decisions filter by outcome — blocked, held for approval, redacted, or recorded only — which turns the tab into a direct answer to “was anything stopped during this incident, and why?”

The workflow

The module ships one workflow, Production Alert Investigation: an intake step deciding which alerts start it, a dedupe and correlate step collapsing repeats into a single incident, an investigation step where domain specialists gather evidence in parallel, and a remediation step that stages the fix for approval. A step with no agents bound to it still appears on the run and still carries its task, but produces no output — the incident reaches the next step with that stage empty, which on the investigation step means a conclusion drawn from less evidence than the workflow was designed to gather.
An account can hold more workflows than the module ships — ones you author, and ones other modules contribute. The Workflows tab lists every workflow in the account with its operating metrics and a state of active, degraded, paused or draft.

How a workflow is composed

A workflow is composed at typed stages, and a run shows the ones it used. The same component is normally used by several workflows at once.
Because remediation posture is a component, “what is the widest change this workflow could ask me to approve?” is answered by reading one slot, rather than by checking every agent bound to the workflow.

Where a change would land

Because components are shared, editing one is rarely a local decision. The Map counts the distinct components across your workflows and how many are used by two or more, and lists every shared one with the number of workflows behind it — change it once and all of them change with it. So an edit to a component used by a single workflow and an edit to one used by every workflow are visibly different decisions before you make either.

Configuring the workflow

Connecting an alert source

An alert source is an inbound endpoint: KAOP gives you a URL and a token, your monitoring tool posts to it, and the endpoint wizard lets you send or listen for a test delivery and see exactly what it would parse before you route it to a workflow. Datadog, Grafana, AlertManager, PagerDuty, OpsGenie and a generic JSON webhook are all supported. Recognized severities — P1–P5, critical/high/warning/low/info, or SEV-N — are normalized onto one SEV-1–SEV-5 scale throughout the module; anything else reads Unknown. A resolved or recovered signal from the source automatically closes the matching incident, and a later re-fire of the same alert reopens it rather than starting a duplicate (for Datadog, a new alert cycle opens a new incident, because its cycle key changes per cycle) — so the source’s own resolved/recovered events are worth wiring up, not just the firing ones. Two sources need one extra step to behave well:
  • Datadog — include $ALERT_CYCLE_KEY and $LINK in the webhook body. The cycle key keeps each alert cycle’s fires and its recovery on one incident; without it, fires are grouped by the monitor’s id or name instead. Without $LINK, the Open in Datadog link is built from the monitor id and assumes the US1 site.
  • AlertManager — point a webhook_configs receiver at the endpoint URL, pass the endpoint token as its bearer credentials, and set send_resolved: true so recovery notifications reach KAOP and close the incident automatically:
A generic webhook accepts any JSON body and reads a title, a dedup key and a severity from among a few common field names, so a sender that doesn’t match one of the named providers still works without a custom integration.

Which alerts become an incident

Not every alert should start an investigation. A workflow’s trigger filter is a list of conditions — field, operator, value — that all have to hold (match: all, the default) or that only one of has to hold (match: any), for example “SEV-2 or worse, or an alert naming one of these tier-1 services”:
A condition can read severity, title, a labels.<key>, or any path in the raw alert payload via raw.<path> — which is how a field a provider doesn’t normalize, like a service name, gets used. A condition whose field is missing from a given alert is dropped by default; turn on Route alerts missing one of these fields if a missing field should count as a match instead. Build and test the filter in the workflow wizard’s Trigger step, against a captured or edited sample payload, before it governs anything live.

Which specialists investigate

The investigation step runs whichever specialists this workflow has bound to it, set from the workflow’s own configuration — editable while it is a draft or paused, so pause an active workflow first to add or remove one. An orchestrator additionally keeps its own delegation allowlist, set on the agent itself: left empty it may call any specialist that’s online, and once you select some, anything left out is refused on every run. See Agent marketplace for that allowlist and Orchestrator for how it routes between the specialists both lists leave it.

Notifications

Sending incident events to Slack or your own systems is the same outbound webhook mechanism every module uses, scoped to this workflow’s incidents from the sink’s own Subscriptions step. The incident-lifecycle events are incident.investigation_started, incident.investigation_completed, incident.approval_needed, incident.triaged (investigation complete, on by default), incident.errored, incident.needs_attention and incident.resolved.

Asking a follow-up question

Once an incident’s investigation isn’t actively running — including while it’s parked waiting on your decision — you can ask it a question directly on the incident, and it answers from the full incident context: the alert, the findings, the root cause, and what each step did. This works on any incident a workflow run investigated. Asking a question doesn’t make a pending decision for you.

Prerequisites

Enabling it

Incident management & troubleshooting is enabled per account rather than shipped on. It appears in the AI SRE family under Modules → Overview; enabling it adds Incidents to the main menu. Enabling the module does not start it. Until an alert source is connected and an agent is bound to each step, the module is on and the incident feed stays empty.

Next steps

Alert Intelligence

Fix the noise upstream so fewer alerts become incidents.

Runs & evidence

How to read the evidence trail behind a conclusion.

APIs

List, filter and read incidents — including which are still investigating — over REST.