Making Multi-Agent Orchestration Survive a Crash

A multi-agent task is long. An orchestrator refines a work item with the engineer, delegates pieces to specialist child agents, waits on tool calls, collects results, reviews them. That can take minutes and span dozens of steps across several services. Somewhere in the middle, a worker gets OOM-killed, a deploy restarts a pod, or the process just dies.

If the orchestration state lived in memory, everything in flight is gone. The task starts over, the engineer watches the same work happen twice, and you burn tokens redoing steps that already succeeded.

This is how an orchestrator survives that: model delegation as an explicit state machine, and persist every transition as an event. One event per transition, no polling. A crash becomes “reload the last state and continue” instead of “start over.”

To be precise about what is durable: the state machine is. The LLM call in flight at the moment of the crash is not; that step re-runs, and you pay for it again. What you never redo is any step that completed and got its event recorded, which for a task of dozens of steps is nearly all of them.

Why the naive versions fail

The first version most people build (I did too, briefly) keeps orchestration state in the orchestrator’s process. A Map of delegations, some promises awaiting child results. It works right up until the first restart, then all of it evaporates.

The common patch is polling. Persist a status column somewhere, and have the orchestrator repeatedly ask “are we done yet?” against child agents or a database. Polling has three problems:

It’s wasteful. Most polls return “nothing changed.”
It’s laggy. Progress is only noticed at the polling interval, so every transition eats up to a full interval of dead time.
It doesn’t actually fix the crash problem. The poller itself holds in-memory state about what it’s waiting for. Restart it and you’re back to reconstructing intent from scattered status rows.

The underlying mistake is treating progress as something you observe, rather than something you record.

Delegation as a persisted state machine

In this design, a delegation moves through a small, explicit set of states:

   WORK ITEM
      │  refine with engineer
      ▼
   ┌──────────────┐   delegate    ┌──────────────┐
   │ ORCHESTRATOR │ ───────────►  │ child agent  │  (specialist)
   │   agent      │               └──────┬───────┘
   └──────┬───────┘                      │ results
          │  each transition = 1 event   │
          ▼                              ▼
   ┌───────────────────────────────────────────────────┐
   │  PERSISTED STATE:  refining → delegated → running  │
   │                    → review → done                 │
   │  one event per transition (no polling)             │
   └───────────────────────────────────────────────────┘
          ▲
          │  crash + restart => reload last state => resume (not restart)

The rule is simple: a delegation cannot change state without emitting an event, and the event is the record of the transition. The event is not a side channel for observability: state and event are written in the same transaction, so the log is a faithful, ordered record of every transition, and either one can reconstruct the other. To know where a delegation stands, you read its state, or fold its events and get the same answer. To resume after a crash, you reload the current state and wait for the next event, which arrives because some other component publishes it when it finishes its part.

That last bit is what kills polling. When the child agent completes, it doesn’t sit there waiting to be asked. It publishes its completion event, and the orchestrator’s consumer picks it up and advances the machine. Progress pushes itself forward.

The substrate: outbox and idempotency

None of this works if events can be lost or duplicated in ways you can’t handle. The state machine rides on a durable message bus (Kafka, a cloud queue, or similar) plus two reliability patterns. Nothing here is bus-specific: the pattern needs only durable, at-least-once delivery, so the same design holds whatever transport you run under it.

Transactional outbox. The classic failure: you commit a state change to Postgres, then publish the event, and crash between the two. Now the delegation is running in the database but nobody was told. Or the reverse: you publish first, fail to commit, and consumers react to a transition that never happened. The outbox closes this gap. The state change and its event are written in the same database transaction, into an outbox table, and a relay ships committed outbox rows to the bus. State and event can never disagree. (Full honesty about the one poller in the building: the relay itself is a small leased worker that polls the outbox table on a sub-second interval, claims a bounded batch of committed rows, publishes them, and marks them. That is a delivery detail, and a deliberately boring one. The no-polling claim in this post is about progress: no component polls other components to reconstruct where a task stands.)

Idempotent consumers. A durable message bus gives at-least-once delivery. Redeliveries happen. So every handler is keyed on the event id: seen it before, skip; otherwise process and record it. Two details make this sound rather than decorative: the dedup record commits in the same transaction as the handler’s state change, so “processed but not recorded” cannot happen; and the dedup protects the transition itself, so any side effect the handler performs needs its own idempotency key.

   PRODUCER SERVICE                              CONSUMER SERVICE
   ┌───────────────────────────┐                ┌───────────────────────────┐
   │  BEGIN TX                 │                │  on event:                │
   │   write state change      │                │   seen this id before?    │
   │   write event -> OUTBOX   │  ── relay ──►  │     yes -> skip (no-op)    │
   │  COMMIT  (atomic)         │   MESSAGE BUS  │     no  -> process + record│
   └───────────────────────────┘   (durable)    └───────────────────────────┘

Outbox plus idempotency gives you effectively-once semantics on top of at-least-once delivery. That guarantee is what makes “one event per transition” safe to build on. A duplicate delegation.completed event is a no-op, not a double-advance of the machine.

Duplicates are the easy hazard. Two more need naming. Redelivery can arrive out of order, so the state machine rejects transitions that do not apply from the current state instead of blindly applying whatever shows up. And a handler that fails deterministically would be redelivered forever, so consumers cap redelivery and shunt the poison event to a dead-letter queue for a human to look at.

What a crash actually looks like now

Say the orchestrator service dies while three child agents are mid-task.

The children don’t notice. They finish their work and publish their completion events to the bus, which holds them durably. When the orchestrator comes back, it reloads each delegation’s persisted state (say, delegated or running), reattaches its consumers, and processes whatever accumulated while it was down. Transitions apply with a compare-and-set on the current state, so a second instance briefly alive during a deploy cannot race the machine backward. The machine advances through review to done exactly as if nothing happened, minus the downtime itself.

Nothing that completed is re-run, and recovery needs no guesswork: the persisted state says exactly which delegations are still owed an event.

The crash above is also the friendly one. The harder case is a child dying mid-task: its completion event will never arrive, and a machine that only waits will sit in running forever. Durable state does not remove the need for a liveness mechanism, a timeout or heartbeat per delegation. What it does is make that sweeper trivial to write, because “stuck” is now a queryable fact (delegations in running past their deadline) instead of a promise lost in some process’s memory.

The other thing you get for free is an audit trail. Every delegation has a complete, ordered history of what happened and when, because that history is literally the mechanism it runs on. One implementation note that matters here: the bus is a delivery mechanism, not the archive. Events also land in an append-only table, and that table, not the bus’s retention window, is what makes the history permanent.

Why not Temporal: one history instead of two

The obvious objection to all of this: durable execution runtimes already solve it. Temporal, Restate, DBOS and Inngest persist workflow state and resume after a crash, and none of them ask you to hand-roll a state machine. So why build one?

For a single service, you probably should not. Reach for the runtime.

The calculus changes once orchestration spans services. A durable execution runtime makes a workflow durable. It does not make your cross-service event publishing atomic. If several services each own their database and talk over a bus, the transactional outbox is not optional and does not go away. The runtime is therefore additive rather than a replacement: you run the outbox for domain events and the runtime for orchestration state, and the answer to “what happened to this task” now lives in two systems that can disagree.

The design in this post collapses that. One log is the state machine, the audit trail, and the integration surface at once. Other services already consume these events, because they are domain events rather than orchestration bookkeeping. There is no second history to reconcile.

The second reason is determinism. Replay-based runtimes require workflow code to be deterministic, so changing orchestration logic means versioning workflows that are still in flight. That is a manageable tax when the process is a payment or a shipment and changes a few times a year. It is a heavier one when the orchestration itself is the product and the delegation strategy changes most weeks.

Now the other side, because the trade is real. A runtime gives you several things in the box that this design leaves as homework: heartbeats and timeouts for detecting a dead child, retry policies with backoff, a console for finding stuck executions, and a test framework that can skip time. The liveness sweeper described above is one of them. Building it yourself is a cost, not a bonus.

So the rule of thumb. If you do not already run an event backbone, or orchestration durability is the only reason you would want one, use the runtime. If you already have a bus, an outbox, and services that need those events anyway, one event per transition is a small increment on infrastructure you have already paid for, and it leaves you with one history instead of two.

Takeaways

Record progress, don’t observe it. If knowing where a task stands requires asking around, a crash loses the answer. If every transition is a persisted event, the answer survives anything.
Polling is a symptom. Needing to poll usually means completion isn’t being published as an event. Fix the publishing and the poller disappears.
The reliability layer comes first. One-event-per-transition is only sound on top of outbox and idempotent consumers. Without effectively-once semantics, duplicated or lost events corrupt the machine.
Small explicit state sets beat clever implicit ones. Five named states you can fold from a log will outlive any in-memory object graph, and they make “resume” a boring operation instead of a recovery procedure.
Buy before you build. A durable execution runtime is the right default. Hand-rolling this earns its keep only when you already run an event backbone and one history is worth more than the tooling you give up.