> ## Documentation Index
> Fetch the complete documentation index at: https://docs.komodor.com/llms.txt
> Use this file to discover all available pages before exploring further.
# Orchestration
> How a workflow's steps deliver tasks to agents, how orchestrators coordinate specialists, and how a run walks the graph.
Most real operations work is too big for one agent and too structured for a free-for-all. The
Komodor Agentic Operation Platform (KAOP) resolves that with **workflows**: an ordered set of steps,
each of which delivers a task to one or more agents. This page explains steps, agent binding,
the orchestrator-and-specialist pattern, and how a module's workflows are built on all three.
## The model
**A workflow is a sequence of steps. Each step delivers a task to one or more agents.**
A step is the *task* — a named piece of work with a stated goal, a declared input shape, and a
declared output shape. The agents bound to it are *who does the work*. Those two things are
deliberately separate, and that separation is the whole point: you can read a workflow and see which
agents handle which stage before anything runs.
```mermaid theme={null}
flowchart TD
M[Module — Incident management & troubleshooting] --> W[Workflow — Production Alert Investigation]
W --> S1["1 · investigate
task: find the root cause"]
S1 --> S2["2 · remediate
task: fix it"]
S2 --> S3["3 · verify
task: confirm the fix held"]
S3 --> S4["4 · postmortem
task: write it up"]
S1 -.-> A1[Datadog Investigator]
S1 -.-> A2[Kubernetes Investigator]
S1 -.-> A3[AWS Investigator]
S2 -.-> R[Remediation agent]
```
A workflow is created from a **template**, which defines its steps and the edges between them. The
shape of the graph is fixed once the workflow exists; what you configure is which agents each step
binds, and the workflow's own lifecycle.
## What a step is
Every step declares a goal, an input contract, an output contract, and a wall-clock ceiling. What
differs is who or what carries the task out:
| Step kind | Carried out by | Typical use |
| ---------------- | ------------------------------------------------------------------- | ----------------------------------------------------------------------- |
| `orchestrated` | An orchestrator agent, working through the agents bound to the step | Investigate, remediate, verify — anything that needs judgment |
| `deterministic` | KAOP itself, with no agent and no model involved | Assembling a record, writing a summary from what earlier steps produced |
| `human_decision` | A person choosing from a set of options | "Apply remediation A, B, or none" |
A step's ceiling is enforced twice: the agent doing the work is given the same budget, and KAOP
sweeps a step that overruns it. When an orchestrated step times out, the agent run behind it is
cancelled — so a step nothing will read stops spending tokens rather than running on.
## Agent binding
Binding is the link between a step and the agents allowed to execute its task. It is a list, not a
single value: one step can bind several agents, and an orchestrator then decides which of them to
use for the case in front of it.
Three properties follow from binding being explicit data rather than something an agent decides at
run time:
* **It is auditable before anything runs.** "What can this workflow possibly touch?" is answered by
reading the bindings, not by watching a run.
* **A step with nothing bound still carries its task.** The step, its goal, and its contracts all
still exist — there is simply no agent to execute it. Nothing is silently dropped and nothing is
silently invented.
* **A workflow can route around an unbound step.** An edge's condition can test whether a step has
any agents bound, so a graph can skip a stage the operator deliberately left empty.
That last property is how "investigate only" works in practice. The default incident workflow's edge
out of `investigate` goes to `remediate` when that step has agents bound, and straight to
`postmortem` when it does not — one graph, two behaviors, decided by configuration rather than by a
second template.
Binding survives an agent being re-registered: a step names the agent, not a particular deployment of
it. The flip side is that a bound agent which is not deployed is inert rather than an error — the
step's orchestrator only reaches agents that are actually online, so that agent simply never runs.
## Orchestrators and specialists
Two roles, and the distinction matters more than any other in this section.
A **specialist** is a domain agent: a Datadog investigator, an AWS investigator, a Kubernetes
investigator, a remediation agent. It knows one area deeply and is bound to the steps where that
area is relevant.
An **orchestrator** is a generalist whose job is routing. For each orchestrated step, KAOP starts one
orchestrator run and hands it three things: the step's goal, the list of agents it may use, and the
output shape it owes. The orchestrator decides who to ask, in what order, and how to fold their
answers into one structured result.
```mermaid theme={null}
flowchart LR
Step["Step: investigate
goal + bound agents + output schema"] --> Orch["Orchestrator run"]
Orch -->|delegates| S1["Datadog Investigator run"]
Orch -->|delegates| S2["Kubernetes Investigator run"]
Orch -->|delegates| S3["AWS Investigator run"]
S1 --> Out["Structured step output
+ which specialists were called, and why"]
S2 --> Out
S3 --> Out
```
Each specialist call is a real delegation, so each one is its own run with its own full evidence
trail, linked to the orchestrator run that started it. The orchestrator's own output additionally
declares the calls it made — the agent, the tool, why it called, and what came back — so the
explanation of a step's reasoning is part of the step's recorded answer.
### What an orchestrator is allowed to reach
Two independent limits compose, and neither can widen the other:
| Layer | Means | Set where |
| --------------------------------- | ------------------------------------------------------ | --------------------------------------------------------------------------- |
| The orchestrator's own allow-list | "Which agents may this orchestrator ever delegate to?" | The orchestrator agent's own configuration, editable from its page in Fleet |
| The step's binding | "Which agents should *this step* use?" | The workflow's configuration |
The effective scope of a step's orchestrator run is the intersection. If the orchestrator's
allow-list is narrowed after a workflow was configured, the step's reach narrows with it — the
binding cannot reach past the ceiling. An orchestrator with no allow-list of its own expresses no
opinion, and the step's binding stands alone.
### Delegation-only agents
Some agents should never be startable by hand. An agent that applies changes to production is the
case this exists for: it can be declared reachable **only** by delegation from an orchestrator or a
workflow step. Every direct path — the console's run dialog, the API, a schedule, a channel routing
rule, chat — is refused for such an agent, and the picker surfaces do not offer it. The guarantee is
enforced where runs are created, not by hiding the buttons.
A step that binds a delegation-only agent is exempted from the orchestrator's ceiling, so a
remediation step whose orchestrator is otherwise scoped to investigators can still reach the agent
its step explicitly named. The exemption only ever keeps a binding the step declared; it never adds
one.
## How a run walks the graph
A workflow run holds a durable pointer to the step it is currently on, and that pointer is the only
state that matters. Everything else follows from it:
The step's task is queued as a step run. An orchestrated step dispatches an orchestrator run; a
deterministic step is executed by KAOP; a decision step parks on a question.
For an orchestrated step, that is the structured output of its orchestrator run. That output
becomes the input of whatever step comes next.
An edge can carry a condition over the completing step's output, the number of attempts made at
that step, whether a sibling step has any agents bound, or whether the run has actually changed
anything yet. An edge with no condition always matches.
A matched edge moves the pointer and queues the next step. A terminal edge completes the run. If
no edge matches at all, the run stops and is flagged for attention rather than being quietly
called done.
| Workflow run status | Meaning |
| ------------------- | ---------------------------------------------- |
| `pending` | Created, not yet entered |
| `running` | Walking the graph |
| `completed` | A terminal edge matched |
| `needs_attention` | A step failed or timed out, or no edge matched |
`needs_attention` is a state an operator acts on, not a dead end. Resetting a run re-enters the
earliest step it never finished; steps that already completed are left exactly as they are, so the
rescue is itself auditable. A run whose every step completed has nothing to re-drive — for that, start
a fresh run with a corrected input, which gets its own id and its own history.
### Retry loops
Graphs are otherwise acyclic, with one deliberate exception: an edge may point back at an earlier
step when its condition bounds the attempt counter. The default incident workflow uses exactly that
to re-investigate when verification fails:
```json theme={null}
{"op": "and", "args": [{"op": "eq", "ref": "output.validation.passed", "value": false},
{"op": "lt", "ref": "attempt", "value": 2}]}
```
Because the bound is part of the condition, the retry cap is a fact about the graph rather than an
instruction an agent has to remember. Once the bound is reached the edge stops matching, and the run
lands in `needs_attention` instead of looping.
### Crash safety
A workflow run is state in the database, not state in a process. A control-plane restart mid-workflow
resumes the run at its last committed step; a completed step is never re-executed, and its side
effects are never repeated. Each dispatch carries an attempt-scoped idempotency key, so a duplicated
dispatch resolves to the run the first one created rather than starting a second.
## When a workflow needs a person
Two different kinds of waiting, kept deliberately distinct — because the operator's question is not
just "does something need me?" but "what kind of act is needed?"
| Waiting on | What happened | What you do |
| --------------- | --------------------------------------------------------- | ----------------------------------------------- |
| An **approval** | A guardrail held an action the step's agent tried to take | Approve or deny it, with the evidence to decide |
| A **decision** | A `human_decision` step parked on a question | Pick one of the options offered |
Both surface in one place: **Attention** lists every run waiting on a human — pending approvals,
pending decisions, and runs that stopped and need review — each with the action that resolves it.
Each item carries the workflow run it belongs to, and each says how long it has been waiting.
Three guarantees about decisions are worth stating plainly:
* **The options are frozen when the question is asked.** What you are choosing between cannot shift
under you while the run is parked, and the key you pick means exactly what you were shown.
* **"None of these" is always available.** A workflow can offer a standing option to apply nothing
and move on, so an operator who rejects every proposal is never forced to let the question lapse.
* **KAOP never chooses for you.** A decision has an answer window; when it closes, the step times out
and the run is flagged for attention. No option is ever picked by default.
Answering a decision is a distinct permission from authoring a workflow, and it is audited with the
identity of whoever answered — taken from the request, never from the submitted body. A run that ever
waited on a person is permanently marked as such, so "was this automated or manual?" has an honest
answer even after the run succeeds.
## What a workflow leaves behind
Beyond the runs of its individual steps, a workflow keeps two records of its own.
**Findings** are a ledger, not a log. A finding carries a severity, a title, the evidence behind it, a
proposed fix, and a status of `open`, `fixed`, or `dismissed`. Re-recording the same finding advances
its `last_seen_at` rather than creating a second row, and a finding that was marked fixed and then
seen again reopens. So "how long has this been true?" and "did it come back?" are both answerable.
**Actions** record what a run actually changed. Each entry is written as *intended* before the change
is attempted and then settled as *applied* or *failed*, with an idempotency key. That gives a
workflow two facts it could not otherwise have: whether this run changed anything at all — which a
later step can branch on — and whether one specific action has already landed, which is what makes a
retried step safe.
## How a module's workflows use this
A **module** is an operational outcome you own; the workflows inside it are the automation that
delivers it. Everything above is the machinery those workflows run on.
Modules are enabled per workspace, from **Modules → Overview**. Enabling one adds it to the
sidebar and makes its workflows available to operate.
You pick the template, then make the choices that become the step bindings — which specialists
investigate, whether a remediation agent is bound at all, how long each step may take.
A workflow names the inbound endpoint that feeds it. Endpoints are shared resources with their
own URL and token, so KAOP never mints one for you — you select an existing one, and activation
is refused for a workflow that names none.
A workflow is `draft`, then `active`, and can be `paused` and reactivated. Only an `active`
workflow starts runs, and only a `draft` or `paused` one can be deleted.
Activating a workflow arms a graph of agents that take real actions, so it is a deliberately elevated
permission and every change to a workflow's lifecycle is audited.
The Incident management & troubleshooting module is the worked example. Its default workflow is `investigate → remediate →
verify → postmortem`, with `investigate` and `verify` sharing the pool of investigators you enabled,
`remediate` binding only the remediation agent you picked (or nothing, for investigate-only), and
`postmortem` executed by KAOP itself with no agent involved. Every module's workflows are the same
mechanism, differently configured.
## Reading a workflow run
A module surfaces its own workflow runs on the record they belong to — an incident, a finding, a
drift — with the step sequence, which step the run is on, and a link into each step's own run and
evidence. From there, everything in
[Runs & evidence](/manage-your-agents/run/runs-evidence) applies: the orchestrator run's transcript
shows the specialists it called, and each specialist's run is a first-class run of its own.
## Next steps
The module catalog, and the module-contains-workflows relationship in full.
What each step's run records, and how to read it.
How a held action reaches a human, and what they see.
The Komodor-built investigators and remediation agents you bind to steps.
## Related topics
- [Interfaces](/get-started/interfaces.md)
- [Remediation](/manage-your-agents/build/catalog/remediation.md)
- [Modules & workflows](/modules-and-workflows/overview.md)
- [Attention](/manage-your-agents/run/attention.md)
- [Azure Investigator](/manage-your-agents/build/catalog/azure-investigator.md)