What the module takes on
Most of the work between an alert firing and a theory forming is retrieval — correlating alerts, reading metrics and logs, checking what deployed recently — and most of it repeats on the next alert. The module takes the retrieval and leaves you the decision. It groups related alerts into one incident, dispatches domain specialists to gather evidence in parallel, produces a root-cause conclusion with the evidence attached, and stages a remediation for a human to approve. Nothing reaches production without that approval.What an incident carries
An incident carries a severity from SEV-1 to SEV-5, a status, the service, namespace, cluster and owner it concerns, when it started and how long it has been open, and — where your ticketing is connected — the ticket it is tracked under. It is the object the whole module revolves around, and it outlives the alert that created it. Its status says what is happening to it right now:
An incident paused on a person does not get a status of its own. It reads Investigated and
carries a Needs your answer marker, which covers a remediation choice, a guardrail hold, and a
specialist waiting on a governed tool. Filter for it with the feed’s marker facet rather than by
status.
Reading one incident
An incident opens into tabs. Each answers a different question, so knowing which one holds what is the difference between reading an answer and reconstructing it.The human gate
Every production mutation waits for a person. What makes that gate meaningful rather than ceremonial is what the approver is shown: the action in plain language, the literal command that will run, a risk assessment covering how reversible it is and what it touches, and the reason it was held. Approve, or reject. Afterwards a validation section reports whether the fix did what it was supposed to.Guardrail decisions on an incident
Guardrails are configured under governance, but an incident is where you find out whether they did anything. The Guardrails tab reports how many evaluations ran in this run’s request path and how many matched, and every match is a complete record: its enforcement mode, what it did, the policy statement behind it, the standard and version it came from, the rule that matched, the principal and resource involved, how long the evaluation took, and whether it was overridable. Decisions filter by outcome — blocked, held for approval, redacted, or recorded only — which turns the tab into a direct answer to “was anything stopped during this incident, and why?”The workflow
The module ships one workflow, Production Alert Investigation: an intake step deciding which alerts start it, a dedupe and correlate step collapsing repeats into a single incident, an investigation step where domain specialists gather evidence in parallel, and a remediation step that stages the fix for approval. A step with no agents bound to it still appears on the run and still carries its task, but produces no output — the incident reaches the next step with that stage empty, which on the investigation step means a conclusion drawn from less evidence than the workflow was designed to gather.An account can hold more workflows than the module ships — ones you author, and ones other modules
contribute. The Workflows tab lists every workflow in the account with its operating metrics and a
state of active, degraded, paused or draft.
How a workflow is composed
A workflow is composed at typed stages, and a run shows the ones it used. The same component is normally used by several workflows at once.Where a change would land
Because components are shared, editing one is rarely a local decision. The Map counts the distinct components across your workflows and how many are used by two or more, and lists every shared one with the number of workflows behind it — change it once and all of them change with it. So an edit to a component used by a single workflow and an edit to one used by every workflow are visibly different decisions before you make either.Configuring the workflow
Connecting an alert source
An alert source is an inbound endpoint: KAOP gives you a URL and a token, your monitoring tool posts to it, and the endpoint wizard lets you send or listen for a test delivery and see exactly what it would parse before you route it to a workflow. Datadog, Grafana, AlertManager, PagerDuty, OpsGenie and a generic JSON webhook are all supported. Recognized severities —P1–P5, critical/high/warning/low/info, or SEV-N — are
normalized onto one SEV-1–SEV-5 scale throughout the module; anything else reads Unknown.
A resolved or recovered signal from the source automatically closes the matching incident, and a
later re-fire of the same alert reopens it rather than starting a duplicate (for Datadog, a new
alert cycle opens a new incident, because its cycle key changes per cycle) — so the source’s own
resolved/recovered events are worth wiring up, not just the firing ones. Two sources need one extra
step to behave well:
-
Datadog — include
$ALERT_CYCLE_KEYand$LINKin the webhook body. The cycle key keeps each alert cycle’s fires and its recovery on one incident; without it, fires are grouped by the monitor’s id or name instead. Without$LINK, the Open in Datadog link is built from the monitor id and assumes the US1 site. -
AlertManager — point a
webhook_configsreceiver at the endpoint URL, pass the endpoint token as its bearer credentials, and setsend_resolved: trueso recovery notifications reach KAOP and close the incident automatically:
Which alerts become an incident
Not every alert should start an investigation. A workflow’s trigger filter is a list of conditions —field, operator, value — that all have to hold (match: all, the default) or that only one
of has to hold (match: any), for example “SEV-2 or worse, or an alert naming one of these
tier-1 services”:
A condition can read
severity, title, a labels.<key>, or any path in the raw alert payload via
raw.<path> — which is how a field a provider doesn’t normalize, like a service name, gets used.
A condition whose field is missing from a given alert is dropped by default; turn on Route alerts
missing one of these fields if a missing field should count as a match instead. Build and test the
filter in the workflow wizard’s Trigger step, against a captured or edited sample payload, before it governs anything live.
Which specialists investigate
The investigation step runs whichever specialists this workflow has bound to it, set from the workflow’s own configuration — editable while it is a draft or paused, so pause an active workflow first to add or remove one. An orchestrator additionally keeps its own delegation allowlist, set on the agent itself: left empty it may call any specialist that’s online, and once you select some, anything left out is refused on every run. See Agent marketplace for that allowlist and Orchestrator for how it routes between the specialists both lists leave it.Notifications
Sending incident events to Slack or your own systems is the same outbound webhook mechanism every module uses, scoped to this workflow’s incidents from the sink’s own Subscriptions step. The incident-lifecycle events areincident.investigation_started, incident.investigation_completed,
incident.approval_needed, incident.triaged (investigation complete, on by default),
incident.errored, incident.needs_attention and incident.resolved.
Asking a follow-up question
Once an incident’s investigation isn’t actively running — including while it’s parked waiting on your decision — you can ask it a question directly on the incident, and it answers from the full incident context: the alert, the findings, the root cause, and what each step did. This works on any incident a workflow run investigated. Asking a question doesn’t make a pending decision for you.Prerequisites
Enabling it
Incident management & troubleshooting is enabled per account rather than shipped on. It appears in the AI SRE family under Modules → Overview; enabling it adds Incidents to the main menu. Enabling the module does not start it. Until an alert source is connected and an agent is bound to each step, the module is on and the incident feed stays empty.Next steps
Alert Intelligence
Fix the noise upstream so fewer alerts become incidents.
Runs & evidence
How to read the evidence trail behind a conclusion.
APIs
List, filter and read incidents — including which are still investigating — over REST.