The model
A workflow is a sequence of steps. Each step delivers a task to one or more agents. A step is the task — a named piece of work with a stated goal, a declared input shape, and a declared output shape. The agents bound to it are who does the work. Those two things are deliberately separate, and that separation is the whole point: you can read a workflow and see which agents handle which stage before anything runs. A workflow is created from a template, which defines its steps and the edges between them. The shape of the graph is fixed once the workflow exists; what you configure is which agents each step binds, and the workflow’s own lifecycle.What a step is
Every step declares a goal, an input contract, an output contract, and a wall-clock ceiling. What differs is who or what carries the task out:
A step’s ceiling is enforced twice: the agent doing the work is given the same budget, and KAOP
sweeps a step that overruns it. When an orchestrated step times out, the agent run behind it is
cancelled — so a step nothing will read stops spending tokens rather than running on.
Agent binding
Binding is the link between a step and the agents allowed to execute its task. It is a list, not a single value: one step can bind several agents, and an orchestrator then decides which of them to use for the case in front of it. Three properties follow from binding being explicit data rather than something an agent decides at run time:- It is auditable before anything runs. “What can this workflow possibly touch?” is answered by reading the bindings, not by watching a run.
- A step with nothing bound still carries its task. The step, its goal, and its contracts all still exist — there is simply no agent to execute it. Nothing is silently dropped and nothing is silently invented.
- A workflow can route around an unbound step. An edge’s condition can test whether a step has any agents bound, so a graph can skip a stage the operator deliberately left empty.
investigate goes to remediate when that step has agents bound, and straight to
postmortem when it does not — one graph, two behaviors, decided by configuration rather than by a
second template.
Binding survives an agent being re-registered: a step names the agent, not a particular deployment of
it. The flip side is that a bound agent which is not deployed is inert rather than an error — the
step’s orchestrator only reaches agents that are actually online, so that agent simply never runs.
Orchestrators and specialists
Two roles, and the distinction matters more than any other in this section. A specialist is a domain agent: a Datadog investigator, an AWS investigator, a Kubernetes investigator, a remediation agent. It knows one area deeply and is bound to the steps where that area is relevant. An orchestrator is a generalist whose job is routing. For each orchestrated step, KAOP starts one orchestrator run and hands it three things: the step’s goal, the list of agents it may use, and the output shape it owes. The orchestrator decides who to ask, in what order, and how to fold their answers into one structured result. Each specialist call is a real delegation, so each one is its own run with its own full evidence trail, linked to the orchestrator run that started it. The orchestrator’s own output additionally declares the calls it made — the agent, the tool, why it called, and what came back — so the explanation of a step’s reasoning is part of the step’s recorded answer.What an orchestrator is allowed to reach
Two independent limits compose, and neither can widen the other:
The effective scope of a step’s orchestrator run is the intersection. If the orchestrator’s
allow-list is narrowed after a workflow was configured, the step’s reach narrows with it — the
binding cannot reach past the ceiling. An orchestrator with no allow-list of its own expresses no
opinion, and the step’s binding stands alone.
Delegation-only agents
Some agents should never be startable by hand. An agent that applies changes to production is the case this exists for: it can be declared reachable only by delegation from an orchestrator or a workflow step. Every direct path — the console’s run dialog, the API, a schedule, a channel routing rule, chat — is refused for such an agent, and the picker surfaces do not offer it. The guarantee is enforced where runs are created, not by hiding the buttons. A step that binds a delegation-only agent is exempted from the orchestrator’s ceiling, so a remediation step whose orchestrator is otherwise scoped to investigators can still reach the agent its step explicitly named. The exemption only ever keeps a binding the step declared; it never adds one.How a run walks the graph
A workflow run holds a durable pointer to the step it is currently on, and that pointer is the only state that matters. Everything else follows from it:1
The run enters its current step
The step’s task is queued as a step run. An orchestrated step dispatches an orchestrator run; a
deterministic step is executed by KAOP; a decision step parks on a question.
2
The step produces an output
For an orchestrated step, that is the structured output of its orchestrator run. That output
becomes the input of whatever step comes next.
3
Edges are evaluated in priority order, and the first match wins
An edge can carry a condition over the completing step’s output, the number of attempts made at
that step, whether a sibling step has any agents bound, or whether the run has actually changed
anything yet. An edge with no condition always matches.
4
The run moves, completes, or stops for review
A matched edge moves the pointer and queues the next step. A terminal edge completes the run. If
no edge matches at all, the run stops and is flagged for attention rather than being quietly
called done.
needs_attention is a state an operator acts on, not a dead end. Resetting a run re-enters the
earliest step it never finished; steps that already completed are left exactly as they are, so the
rescue is itself auditable. A run whose every step completed has nothing to re-drive — for that, start
a fresh run with a corrected input, which gets its own id and its own history.
Retry loops
Graphs are otherwise acyclic, with one deliberate exception: an edge may point back at an earlier step when its condition bounds the attempt counter. The default incident workflow uses exactly that to re-investigate when verification fails:needs_attention instead of looping.
Crash safety
A workflow run is state in the database, not state in a process. A control-plane restart mid-workflow resumes the run at its last committed step; a completed step is never re-executed, and its side effects are never repeated. Each dispatch carries an attempt-scoped idempotency key, so a duplicated dispatch resolves to the run the first one created rather than starting a second.When a workflow needs a person
Two different kinds of waiting, kept deliberately distinct — because the operator’s question is not just “does something need me?” but “what kind of act is needed?”
Both surface in one place: Attention lists every run waiting on a human — pending approvals,
pending decisions, and runs that stopped and need review — each with the action that resolves it.
Each item carries the workflow run it belongs to, and each says how long it has been waiting.
Three guarantees about decisions are worth stating plainly:
- The options are frozen when the question is asked. What you are choosing between cannot shift under you while the run is parked, and the key you pick means exactly what you were shown.
- “None of these” is always available. A workflow can offer a standing option to apply nothing and move on, so an operator who rejects every proposal is never forced to let the question lapse.
- KAOP never chooses for you. A decision has an answer window; when it closes, the step times out and the run is flagged for attention. No option is ever picked by default.
What a workflow leaves behind
Beyond the runs of its individual steps, a workflow keeps two records of its own. Findings are a ledger, not a log. A finding carries a severity, a title, the evidence behind it, a proposed fix, and a status ofopen, fixed, or dismissed. Re-recording the same finding advances
its last_seen_at rather than creating a second row, and a finding that was marked fixed and then
seen again reopens. So “how long has this been true?” and “did it come back?” are both answerable.
Actions record what a run actually changed. Each entry is written as intended before the change
is attempted and then settled as applied or failed, with an idempotency key. That gives a
workflow two facts it could not otherwise have: whether this run changed anything at all — which a
later step can branch on — and whether one specific action has already landed, which is what makes a
retried step safe.
How a module’s workflows use this
A module is an operational outcome you own; the workflows inside it are the automation that delivers it. Everything above is the machinery those workflows run on.1
Enable the module
Modules are enabled per workspace, from Modules → Overview. Enabling one adds it to the
sidebar and makes its workflows available to operate.
2
Configure a workflow from a template
You pick the template, then make the choices that become the step bindings — which specialists
investigate, whether a remediation agent is bound at all, how long each step may take.
3
Point it at how work will arrive
A workflow names the inbound endpoint that feeds it. Endpoints are shared resources with their
own URL and token, so KAOP never mints one for you — you select an existing one, and activation
is refused for a workflow that names none.
4
Activate it
A workflow is
draft, then active, and can be paused and reactivated. Only an active
workflow starts runs, and only a draft or paused one can be deleted.investigate → remediate → verify → postmortem, with investigate and verify sharing the pool of investigators you enabled,
remediate binding only the remediation agent you picked (or nothing, for investigate-only), and
postmortem executed by KAOP itself with no agent involved. Every module’s workflows are the same
mechanism, differently configured.
Reading a workflow run
A module surfaces its own workflow runs on the record they belong to — an incident, a finding, a drift — with the step sequence, which step the run is on, and a link into each step’s own run and evidence. From there, everything in Runs & evidence applies: the orchestrator run’s transcript shows the specialists it called, and each specialist’s run is a first-class run of its own.Next steps
Modules & workflows
The module catalog, and the module-contains-workflows relationship in full.
Runs & evidence
What each step’s run records, and how to read it.
Approvals
How a held action reaches a human, and what they see.
Use specialized agents
The Komodor-built investigators and remediation agents you bind to steps.