Retries, Timeouts & Compensation
EventConductor provides built-in support for retries, timeouts, and compensation (saga pattern) — all configured declaratively in the workflow definition JSON.
Chapter 4 of EventConductor in N minutes (4:23) · English, with English and Spanish subtitles
Retries
Section titled “Retries”Set the retries field on a step to automatically re-dispatch the task when it reports an ERROR or times out.
{ "id": "charge-payment", "type": "ACTION", "name": "Charge Payment", "topic": "payment-service", "retries": 3}When a step fails:
- The orchestrator checks the retry count
- If retries remain, the step is parked in
AWAITING_RETRYfor a backoff delay, then re-dispatched with a newTaskExecutionRequestedonce the delay elapses - The attempt count is incremented
- If all retries are exhausted, the step transitions to
ERRORand the process toERROR
Backoff
Section titled “Backoff”Retries are not re-dispatched immediately — a worker that fails fast (bad config, a downstream
500) would otherwise burn the entire retry budget in milliseconds and hammer the failing
dependency in a tight loop, which defeats the purpose of retrying at all. Instead a failed step
waits out an exponentially growing, jittered delay in AWAITING_RETRY, and the timeout scheduler
re-dispatches it when the delay expires. Tune it globally:
| Property | Default | Meaning |
|---|---|---|
workflow.retry.backoff-base-ms | 1000 | Delay before the first retry |
workflow.retry.backoff-multiplier | 2.0 | Per-attempt growth (1.0 = fixed delay) |
workflow.retry.backoff-max-ms | 60000 | Cap on the delay |
workflow.retry.backoff-jitter | 0.2 | ±fraction of randomness, so a fleet that failed together does not retry in lockstep |
A manual retry (below) resets straight to CREATED with no backoff.
Timeouts
Section titled “Timeouts”Set the timeout field to limit how long a step can run. Accepts an ISO 8601 duration string or an integer in milliseconds (legacy):
| Format | Example | Meaning |
|---|---|---|
| ISO 8601 string | "PT30S" | 30 seconds |
| ISO 8601 string | "PT5M" | 5 minutes |
| ISO 8601 string | "PT1H30M" | 1 hour 30 minutes |
| Integer (ms) | 30000 | 30 seconds (legacy, still supported) |
{ "id": "external-api-call", "type": "ACTION", "name": "Call External API", "topic": "api-service", "timeout": "PT30S", "retries": 2}When a step times out:
- The step transitions to
TIMEOUT - If retries are configured, it is re-dispatched
- If no retries remain, it transitions to
ERROR
timeout: 0 (or omitting it) means no timeout.
On-timeout routing
Section titled “On-timeout routing”By default a timeout, once retries are exhausted, is a failure: the step ends TIMEOUT and the
process errors (and any already-succeeded compensable steps roll back). But a timeout is often not
an error to abort on — it is an outcome to route on. A payment-verification task nobody actions in
30 seconds should cancel the booking, not fail the process.
Set onTimeoutStepId to route forward on timeout instead of failing:
{ "id": "verify-payment", "type": "USER_TASK", "formId": "verify-payment", "timeout": "PT30S", "onTimeoutStepId": "cancel-booking", "preconditionStepId": "start"},{ "id": "cancel-booking", "type": "ACTION", "topic": "booking", "preconditions": [ { "stepId": "verify-payment", "expression": "paymentReceived == false" } ]}When verify-payment times out (after any retries), the flow moves to cancel-booking; the timed-out
step ends TIMEOUT but is not counted as a process failure — so nothing is compensated either: the
saga carries on down the on-timeout route (before 2.21.0 a routed timeout also rolled the completed
compensable steps back, and the process both compensated and reached its END). Here cancel-booking is reached two
ways — a human rejection (its precondition guard) or the timeout (onTimeoutStepId) — so both paths
converge on the same step. The pending task is cancelled when the process moves on (it disappears from
the worker/forms inbox).
On-timeout vs compensation. They are duals. Compensation runs the undo of steps that already
succeeded, in reverse, when a process fails — it rolls backward. On-timeout routes forward
to a step of your choosing, because a step that timed out never succeeded, so there is nothing of its
to compensate. Reach for onTimeoutStepId when a deadline has a defined next move (cancel, escalate,
take the default); reach for compensation when a failure must unwind committed work. A step can carry
both a timeout and a compensationStepId for the steps before it.
In the graph editor, draw an on-timeout line with shift+alt+drag from the timing-out step to its target (it renders amber with a ⏱ clock chip showing the timeout).
Compensation (Saga pattern)
Section titled “Compensation (Saga pattern)”For distributed transactions, use compensable + compensationStepId to define compensation logic:
{ "steps": [ { "id": "start", "type": "START", "name": "Start" }, { "id": "reserve-hotel", "type": "ACTION", "name": "Reserve Hotel", "topic": "hotel-service", "preconditionStepId": "start", "compensable": true, "compensationStepId": "cancel-hotel", "retries": 2 }, { "id": "reserve-flight", "type": "ACTION", "name": "Reserve Flight", "topic": "flight-service", "preconditionStepId": "reserve-hotel", "compensable": true, "compensationStepId": "cancel-flight" }, { "id": "cancel-hotel", "type": "ACTION", "name": "Cancel Hotel", "topic": "hotel-service" }, { "id": "cancel-flight", "type": "ACTION", "name": "Cancel Flight", "topic": "flight-service" } ]}A compensation step declares no preconditions. It is named by the step it undoes, and the
rollback pipeline starts it directly, so it needs no way in of its own — and a step with nothing
to wait for never starts by itself (only a START, or a WAIT_FOR_MESSAGE that begins a flow,
does). Validation knows this: a compensation step is reachable because something names it.
How it works: compensation is a whole-process saga rollback, not a per-step undo. When any step fails after exhausting its retries:
- The process enters
ERRORand the normal flow is blocked (no new steps start). - The engine collects every step that completed and is
compensablewith acompensationStepId, and runs their compensations sequentially, in reverse completion order — the latest-completed step is undone first. The step that failed is never compensated: its work did not succeed, so there is nothing of its own to undo. In the example, ifreserve-flightfails, onlycancel-hotelruns; if both reservations completed and a later step fails,cancel-flightruns first, thencancel-hotel. - Each compensation starts only once the previous one has completed, so the order is strict.
- When the whole chain completes, the process transitions to the terminal
COMPENSATEDstate (a clean rollback — distinct from a rawERROR). If a compensation step itself fails after its own retries, the chain halts and the process reaches the distinct terminalCOMPENSATION_FAILED— it is left partially rolled back, so it is not masked back into a plainERROR. This is the most dangerous saga outcome and is never silent: it increments theeventconductor.compensations.failedmetric (the one compensation metric to alert on) and logs loudly, so an operator can find the process and resolve the inconsistent state by hand.
If nothing compensable had completed when the step failed (for instance reserve-hotel itself
fails), there is nothing to roll back: no compensation runs and the process stays in plain ERROR.
Compensation on a business decision
Section titled “Compensation on a business decision”Compensation is the mechanism; who decides to trigger it is your business rule. Everything above triggers it automatically, on a failure. That is right when the steps are one technical all-or-nothing unit — charging the card and reserving the stock for one request: half of it is worth nothing, and nobody needs to be asked.
It is wrong when undoing is a decision somebody should make. A hotel that cannot book one room of
three does not silently cancel the other two: it tells the guest, proposes another room, and cancels
only if the guest says so. Write that as a flow, and let the cancel branch end with an END that
compensates:
id: booking-the-guest-decidesname: Booking where the guest decidessteps: - { id: start, type: START, name: Start } - id: rooms type: ACTION name: Book a room preconditionStepId: start compensable: true compensationStepId: cancel-room forEach: items: rooms as: room onIterationFailure: continue # keep what was booked; the step completes anyway output: roomBookings - { id: ask-guest, type: ACTION, name: Ask the guest, preconditionStepId: rooms } - { id: guest-decision, type: CHOICE, name: What did the guest say, preconditionStepId: ask-guest } - id: cancelled-by-guest type: END name: Cancelled by the guest compensate: true # undo what was booked, because the guest decided so preconditions: - { stepId: guest-decision, expression: "decision == 'cancel'" } - { id: confirm, type: ACTION, name: Confirm the booking, preconditionStepId: guest-decision } - { id: done, type: END, name: Booked, preconditionStepId: confirm } - { id: cancel-room, type: ACTION, name: Cancel the room }Reaching cancelled-by-guest cancels whatever is still in flight and runs the compensations of every
compensable step that completed, newest first — one per room that was booked, none for the room that
was not — and the process ends COMPENSATED. It is the same rollback a failure starts, with three
differences: nothing failed, so no step is ERROR and the process is not counted as failed; the
reason is recorded on the process (rollbackReason) and in the END’s log; and it is written where
the decision is, so it reads as one. In a child process it fails the parent’s PROCESS step, which
may roll the parent back in turn. See END → compensate.
Don’t get the same effect by failing a step on purpose: that is an incident in the Errors tab, a failed process in the metrics, and a flow that says the opposite of what happened.
A cancelled step whose work completes anyway
Section titled “A cancelled step whose work completes anyway”The engine cancels steps that are still running: at the end of a rollback (a sibling branch failed),
when another branch reaches END, and when an XOR join with compensateLosers goes on without
them. Cancelling races with the work. A worker that had already made the booking, or a child process
that had already completed, reports COMPLETED after the cancellation — and that report used to be
dropped as stale, leaving the booking behind with nothing to undo it.
When the step declares a compensation, that late COMPLETED is now kept as a completed run, with what
it reported (on the run, not in the process variables — the compensation sees it), and undone:
- In a failed process the rollback picks it up. If the rollback had already finished, the process
goes back to
ERROR, runs a new pass of the compensation over just that run, and endsCOMPENSATEDagain. - In a process that did not fail (
ENDreached, or anXORjoin withcompensateLosers) it is compensated as a cleanup — see below.
A late answer for a step without a compensation is still ignored, and so is one for a process an operator cancelled: cancelling a process is not a rollback. Workers should still be idempotent, and a compensation must tolerate undoing work that completed late.
A child process cut short
Section titled “A child process cut short”A PROCESS step whose child is still running when the engine cancels the step leaves that child
half done: it may have held the room and not yet confirmed it. What happens to the child depends on
why the step was cancelled.
| The parent step was cancelled because… | The child |
|---|---|
the parent is rolling back (a step failed and there is something to compensate), a forEach over PROCESS failed fast, an XOR join with compensateLosers abandoned its branch, or an END with compensateAbandoned cut its branch | is rolled back: what it has in flight is cancelled and its completed compensable steps are compensated, newest first. It ends COMPENSATED (COMPENSATION_FAILED if a compensation fails), also when it had nothing to undo. Its own children still running are rolled back the same way. |
an operator cancelled the parent, an END without compensateAbandoned cut its branch, a forEach completed early (completion: any or a condition), or the parent failed with nothing to compensate | is cancelled, as before — nothing is undone. |
A child is undone once, by one side:
- Child already
COMPLETED→ the parent’s compensation of itsPROCESSstep (compensationStepId) undoes it, as for any completed step — even when the completion is reported after the step was cancelled (above). The child itself is left alone. - Child still running → the child undoes its own steps. The parent’s
PROCESSstep staysCANCELLED, so itscompensationStepIddoes not run.
A parent that is rolling back waits for the children it rolled back: it ends COMPENSATED only once
they have, and COMPENSATION_FAILED if one of them could not finish undoing. Cancelling a process
from the console, REST or MCP is still not a rollback, for it or for its children.
Branches the flow abandons
Section titled “Branches the flow abandons”Two things end a branch without failing the process: another branch reaching END, and an XOR
join going on with another branch. What those branches completed in time stands by default —
that is how they have always behaved. Two opt-ins undo it:
| Where | Field | What it compensates |
|---|---|---|
an XOR JOIN | compensateLosers: true | The losing branches are cancelled as the join fires, and their completed compensable steps are compensated. Exactly one branch wins — the first whose link step completed (ties: the first listed in preconditionStepIds) — so a branch that completed too before the join was evaluated still loses, and is compensated. |
an END | compensateAbandoned: true | The completed compensable steps of the branches this END cut short (not on the way to it, with a step after them it cancelled). |
Work that completed late — after the cancellation — is compensated with or without them.
With compensateLosers, a losing branch’s outputs are discarded: once another branch has reached
the join, what a step of a losing branch reports is kept on its own run (its compensation sees it — the
booking id to cancel) and is not written to the process variables, so it never overwrites the
winner’s. Without compensateLosers a losing branch runs on and its outputs are written as always.
These compensations are a cleanup beside the flow, not a rollback:
- They run one at a time, newest first, each over the runs it undoes.
- They do not change the process status: an
XORcleanup runs while the process carries on, anENDcleanup runs after it isCOMPLETED. ReachingENDnever cancels a cleanup in progress. - If one fails after its retries, it is an incident — the failed compensation shows in the Errors tab for an operator — and the process neither fails nor rolls back because of it. A rollback that starts later for another reason does not halt on it either; it undoes the rest.
- A run a cleanup undid counts as undone: a later rollback does not undo it twice.
Conditional skipping
Section titled “Conditional skipping”Use preconditionExpression (a JEXL expression) to skip steps based on process variables:
{ "id": "manual-review", "type": "USER_TASK", "name": "Manual Review Required", "formId": "review-form", "preconditionStepId": "validate", "preconditionExpression": "amount > 1000 && riskScore == 'HIGH'"}If the expression evaluates to false, the step does not run and the workflow carries on without
it: nothing else can start it, and the process finishes around it. (The step execution is left
CANCELLED rather than COMPLETED — see Statuses.)
…and the other kind: a condition on one link
Section titled “…and the other kind: a condition on one link”preconditionExpression gates the step however it is reached. When what you mean is “only by
this route”, put the condition on the link instead:
{ "id": "ship", "type": "ACTION", "preconditions": [ { "stepId": "pack" }, { "stepId": "charge", "expression": "amount > 100" } ]}The two behave differently on purpose, and the difference is the whole reason both exist:
preconditionExpression (on the step) | expression (on a link) | |
|---|---|---|
| Applies to | the step, by any route | that one incoming link |
| When false | the step is skipped and the process finishes | the link is unsatisfied and the step waits |
| Re-evaluated | every tick | every tick |
A step waiting on a false link condition is released as soon as what the condition reads changes. If nothing ever changes it, the process waits indefinitely — see the caution in Step types.
Available in JEXL expressions: all process variables by name.
Running a stopped process again
Section titled “Running a stopped process again”A process that stopped — ERROR, or CANCELLED by an operator — can be put back to work in two
ways, from the process list, from the process detail, or through the MCP tools retryProcess and
restartProcess:
Retry from failure re-dispatches the steps that failed (in a cancelled process, the ones that were cancelled) and leaves the ones that already succeeded alone. The right choice when the failure was the environment.
Restart from the beginning puts every step back to the state it was created in and runs the whole process again, the successful steps included, with the variables the process was created with — re-running from the variables the failed run wrote would not be starting from the beginning. The right choice when the run itself was wrong. Both re-run the same process instance, keeping its id and business key, and both use the workflow definition as it was when that process was created.
Programmatically, publish the request rather than calling the use case, so it is carried out by the node that owns the process:
processUpstreamEventUseCase.handle(new ProcessUpstreamEventCommand( new RetryProcessRequested(processId) // or: new RestartProcessRequested(processId)));Neither is accepted for a COMPLETED or COMPENSATED process, or for one still running.
What this saves you
Section titled “What this saves you”The three mechanisms on this page — a timeout and bounded retries per step, and compensation
declared as compensable plus compensationStepId — are fields of a step here because the engine
fixes how they behave. Expressing the same behaviour in a notation that assumes nothing costs a
great deal more: the
comparison page
models this very saga in both, side by side, with both definitions downloadable.