Skip to content

Retries, Timeouts & Compensation

EventConductor provides built-in support for retries, timeouts, and compensation (saga pattern) — all configured declaratively in the workflow definition JSON.

Chapter 4 of EventConductor in N minutes (4:23) · English, with English and Spanish subtitles

Set the retries field on a step to automatically re-dispatch the task when it reports an ERROR or times out.

{
"id": "charge-payment",
"type": "ACTION",
"name": "Charge Payment",
"topic": "payment-service",
"retries": 3
}

When a step fails:

  1. The orchestrator checks the retry count
  2. If retries remain, the step is parked in AWAITING_RETRY for a backoff delay, then re-dispatched with a new TaskExecutionRequested once the delay elapses
  3. The attempt count is incremented
  4. If all retries are exhausted, the step transitions to ERROR and the process to ERROR

Retries are not re-dispatched immediately — a worker that fails fast (bad config, a downstream 500) would otherwise burn the entire retry budget in milliseconds and hammer the failing dependency in a tight loop, which defeats the purpose of retrying at all. Instead a failed step waits out an exponentially growing, jittered delay in AWAITING_RETRY, and the timeout scheduler re-dispatches it when the delay expires. Tune it globally:

PropertyDefaultMeaning
workflow.retry.backoff-base-ms1000Delay before the first retry
workflow.retry.backoff-multiplier2.0Per-attempt growth (1.0 = fixed delay)
workflow.retry.backoff-max-ms60000Cap on the delay
workflow.retry.backoff-jitter0.2±fraction of randomness, so a fleet that failed together does not retry in lockstep

A manual retry (below) resets straight to CREATED with no backoff.

Set the timeout field to limit how long a step can run. Accepts an ISO 8601 duration string or an integer in milliseconds (legacy):

FormatExampleMeaning
ISO 8601 string"PT30S"30 seconds
ISO 8601 string"PT5M"5 minutes
ISO 8601 string"PT1H30M"1 hour 30 minutes
Integer (ms)3000030 seconds (legacy, still supported)
{
"id": "external-api-call",
"type": "ACTION",
"name": "Call External API",
"topic": "api-service",
"timeout": "PT30S",
"retries": 2
}

When a step times out:

  1. The step transitions to TIMEOUT
  2. If retries are configured, it is re-dispatched
  3. If no retries remain, it transitions to ERROR

timeout: 0 (or omitting it) means no timeout.

By default a timeout, once retries are exhausted, is a failure: the step ends TIMEOUT and the process errors (and any already-succeeded compensable steps roll back). But a timeout is often not an error to abort on — it is an outcome to route on. A payment-verification task nobody actions in 30 seconds should cancel the booking, not fail the process.

Set onTimeoutStepId to route forward on timeout instead of failing:

{
"id": "verify-payment",
"type": "USER_TASK",
"formId": "verify-payment",
"timeout": "PT30S",
"onTimeoutStepId": "cancel-booking",
"preconditionStepId": "start"
},
{
"id": "cancel-booking",
"type": "ACTION",
"topic": "booking",
"preconditions": [ { "stepId": "verify-payment", "expression": "paymentReceived == false" } ]
}

When verify-payment times out (after any retries), the flow moves to cancel-booking; the timed-out step ends TIMEOUT but is not counted as a process failure — so nothing is compensated either: the saga carries on down the on-timeout route (before 2.21.0 a routed timeout also rolled the completed compensable steps back, and the process both compensated and reached its END). Here cancel-booking is reached two ways — a human rejection (its precondition guard) or the timeout (onTimeoutStepId) — so both paths converge on the same step. The pending task is cancelled when the process moves on (it disappears from the worker/forms inbox).

On-timeout vs compensation. They are duals. Compensation runs the undo of steps that already succeeded, in reverse, when a process fails — it rolls backward. On-timeout routes forward to a step of your choosing, because a step that timed out never succeeded, so there is nothing of its to compensate. Reach for onTimeoutStepId when a deadline has a defined next move (cancel, escalate, take the default); reach for compensation when a failure must unwind committed work. A step can carry both a timeout and a compensationStepId for the steps before it.

In the graph editor, draw an on-timeout line with shift+alt+drag from the timing-out step to its target (it renders amber with a ⏱ clock chip showing the timeout).

For distributed transactions, use compensable + compensationStepId to define compensation logic:

{
"steps": [
{
"id": "start",
"type": "START",
"name": "Start"
},
{
"id": "reserve-hotel",
"type": "ACTION",
"name": "Reserve Hotel",
"topic": "hotel-service",
"preconditionStepId": "start",
"compensable": true,
"compensationStepId": "cancel-hotel",
"retries": 2
},
{
"id": "reserve-flight",
"type": "ACTION",
"name": "Reserve Flight",
"topic": "flight-service",
"preconditionStepId": "reserve-hotel",
"compensable": true,
"compensationStepId": "cancel-flight"
},
{
"id": "cancel-hotel",
"type": "ACTION",
"name": "Cancel Hotel",
"topic": "hotel-service"
},
{
"id": "cancel-flight",
"type": "ACTION",
"name": "Cancel Flight",
"topic": "flight-service"
}
]
}

A compensation step declares no preconditions. It is named by the step it undoes, and the rollback pipeline starts it directly, so it needs no way in of its own — and a step with nothing to wait for never starts by itself (only a START, or a WAIT_FOR_MESSAGE that begins a flow, does). Validation knows this: a compensation step is reachable because something names it.

How it works: compensation is a whole-process saga rollback, not a per-step undo. When any step fails after exhausting its retries:

  1. The process enters ERROR and the normal flow is blocked (no new steps start).
  2. The engine collects every step that completed and is compensable with a compensationStepId, and runs their compensations sequentially, in reverse completion order — the latest-completed step is undone first. The step that failed is never compensated: its work did not succeed, so there is nothing of its own to undo. In the example, if reserve-flight fails, only cancel-hotel runs; if both reservations completed and a later step fails, cancel-flight runs first, then cancel-hotel.
  3. Each compensation starts only once the previous one has completed, so the order is strict.
  4. When the whole chain completes, the process transitions to the terminal COMPENSATED state (a clean rollback — distinct from a raw ERROR). If a compensation step itself fails after its own retries, the chain halts and the process reaches the distinct terminal COMPENSATION_FAILED — it is left partially rolled back, so it is not masked back into a plain ERROR. This is the most dangerous saga outcome and is never silent: it increments the eventconductor.compensations.failed metric (the one compensation metric to alert on) and logs loudly, so an operator can find the process and resolve the inconsistent state by hand.

If nothing compensable had completed when the step failed (for instance reserve-hotel itself fails), there is nothing to roll back: no compensation runs and the process stays in plain ERROR.

Compensation is the mechanism; who decides to trigger it is your business rule. Everything above triggers it automatically, on a failure. That is right when the steps are one technical all-or-nothing unit — charging the card and reserving the stock for one request: half of it is worth nothing, and nobody needs to be asked.

It is wrong when undoing is a decision somebody should make. A hotel that cannot book one room of three does not silently cancel the other two: it tells the guest, proposes another room, and cancels only if the guest says so. Write that as a flow, and let the cancel branch end with an END that compensates:

id: booking-the-guest-decides
name: Booking where the guest decides
steps:
- { id: start, type: START, name: Start }
- id: rooms
type: ACTION
name: Book a room
preconditionStepId: start
compensable: true
compensationStepId: cancel-room
forEach:
items: rooms
as: room
onIterationFailure: continue # keep what was booked; the step completes anyway
output: roomBookings
- { id: ask-guest, type: ACTION, name: Ask the guest, preconditionStepId: rooms }
- { id: guest-decision, type: CHOICE, name: What did the guest say, preconditionStepId: ask-guest }
- id: cancelled-by-guest
type: END
name: Cancelled by the guest
compensate: true # undo what was booked, because the guest decided so
preconditions:
- { stepId: guest-decision, expression: "decision == 'cancel'" }
- { id: confirm, type: ACTION, name: Confirm the booking, preconditionStepId: guest-decision }
- { id: done, type: END, name: Booked, preconditionStepId: confirm }
- { id: cancel-room, type: ACTION, name: Cancel the room }

Reaching cancelled-by-guest cancels whatever is still in flight and runs the compensations of every compensable step that completed, newest first — one per room that was booked, none for the room that was not — and the process ends COMPENSATED. It is the same rollback a failure starts, with three differences: nothing failed, so no step is ERROR and the process is not counted as failed; the reason is recorded on the process (rollbackReason) and in the END’s log; and it is written where the decision is, so it reads as one. In a child process it fails the parent’s PROCESS step, which may roll the parent back in turn. See END → compensate.

Don’t get the same effect by failing a step on purpose: that is an incident in the Errors tab, a failed process in the metrics, and a flow that says the opposite of what happened.

A cancelled step whose work completes anyway

Section titled “A cancelled step whose work completes anyway”

The engine cancels steps that are still running: at the end of a rollback (a sibling branch failed), when another branch reaches END, and when an XOR join with compensateLosers goes on without them. Cancelling races with the work. A worker that had already made the booking, or a child process that had already completed, reports COMPLETED after the cancellation — and that report used to be dropped as stale, leaving the booking behind with nothing to undo it.

When the step declares a compensation, that late COMPLETED is now kept as a completed run, with what it reported (on the run, not in the process variables — the compensation sees it), and undone:

  • In a failed process the rollback picks it up. If the rollback had already finished, the process goes back to ERROR, runs a new pass of the compensation over just that run, and ends COMPENSATED again.
  • In a process that did not fail (END reached, or an XOR join with compensateLosers) it is compensated as a cleanup — see below.

A late answer for a step without a compensation is still ignored, and so is one for a process an operator cancelled: cancelling a process is not a rollback. Workers should still be idempotent, and a compensation must tolerate undoing work that completed late.

A PROCESS step whose child is still running when the engine cancels the step leaves that child half done: it may have held the room and not yet confirmed it. What happens to the child depends on why the step was cancelled.

The parent step was cancelled because…The child
the parent is rolling back (a step failed and there is something to compensate), a forEach over PROCESS failed fast, an XOR join with compensateLosers abandoned its branch, or an END with compensateAbandoned cut its branchis rolled back: what it has in flight is cancelled and its completed compensable steps are compensated, newest first. It ends COMPENSATED (COMPENSATION_FAILED if a compensation fails), also when it had nothing to undo. Its own children still running are rolled back the same way.
an operator cancelled the parent, an END without compensateAbandoned cut its branch, a forEach completed early (completion: any or a condition), or the parent failed with nothing to compensateis cancelled, as before — nothing is undone.

A child is undone once, by one side:

  • Child already COMPLETED → the parent’s compensation of its PROCESS step (compensationStepId) undoes it, as for any completed step — even when the completion is reported after the step was cancelled (above). The child itself is left alone.
  • Child still running → the child undoes its own steps. The parent’s PROCESS step stays CANCELLED, so its compensationStepId does not run.

A parent that is rolling back waits for the children it rolled back: it ends COMPENSATED only once they have, and COMPENSATION_FAILED if one of them could not finish undoing. Cancelling a process from the console, REST or MCP is still not a rollback, for it or for its children.

Two things end a branch without failing the process: another branch reaching END, and an XOR join going on with another branch. What those branches completed in time stands by default — that is how they have always behaved. Two opt-ins undo it:

WhereFieldWhat it compensates
an XOR JOINcompensateLosers: trueThe losing branches are cancelled as the join fires, and their completed compensable steps are compensated. Exactly one branch wins — the first whose link step completed (ties: the first listed in preconditionStepIds) — so a branch that completed too before the join was evaluated still loses, and is compensated.
an ENDcompensateAbandoned: trueThe completed compensable steps of the branches this END cut short (not on the way to it, with a step after them it cancelled).

Work that completed late — after the cancellation — is compensated with or without them.

With compensateLosers, a losing branch’s outputs are discarded: once another branch has reached the join, what a step of a losing branch reports is kept on its own run (its compensation sees it — the booking id to cancel) and is not written to the process variables, so it never overwrites the winner’s. Without compensateLosers a losing branch runs on and its outputs are written as always.

These compensations are a cleanup beside the flow, not a rollback:

  • They run one at a time, newest first, each over the runs it undoes.
  • They do not change the process status: an XOR cleanup runs while the process carries on, an END cleanup runs after it is COMPLETED. Reaching END never cancels a cleanup in progress.
  • If one fails after its retries, it is an incident — the failed compensation shows in the Errors tab for an operator — and the process neither fails nor rolls back because of it. A rollback that starts later for another reason does not halt on it either; it undoes the rest.
  • A run a cleanup undid counts as undone: a later rollback does not undo it twice.

Use preconditionExpression (a JEXL expression) to skip steps based on process variables:

{
"id": "manual-review",
"type": "USER_TASK",
"name": "Manual Review Required",
"formId": "review-form",
"preconditionStepId": "validate",
"preconditionExpression": "amount > 1000 && riskScore == 'HIGH'"
}

If the expression evaluates to false, the step does not run and the workflow carries on without it: nothing else can start it, and the process finishes around it. (The step execution is left CANCELLED rather than COMPLETED — see Statuses.)

Section titled “…and the other kind: a condition on one link”

preconditionExpression gates the step however it is reached. When what you mean is “only by this route”, put the condition on the link instead:

{
"id": "ship",
"type": "ACTION",
"preconditions": [
{ "stepId": "pack" },
{ "stepId": "charge", "expression": "amount > 100" }
]
}

The two behave differently on purpose, and the difference is the whole reason both exist:

preconditionExpression (on the step)expression (on a link)
Applies tothe step, by any routethat one incoming link
When falsethe step is skipped and the process finishesthe link is unsatisfied and the step waits
Re-evaluatedevery tickevery tick

A step waiting on a false link condition is released as soon as what the condition reads changes. If nothing ever changes it, the process waits indefinitely — see the caution in Step types.

Available in JEXL expressions: all process variables by name.

A process that stopped — ERROR, or CANCELLED by an operator — can be put back to work in two ways, from the process list, from the process detail, or through the MCP tools retryProcess and restartProcess:

Retry from failure re-dispatches the steps that failed (in a cancelled process, the ones that were cancelled) and leaves the ones that already succeeded alone. The right choice when the failure was the environment.

Restart from the beginning puts every step back to the state it was created in and runs the whole process again, the successful steps included, with the variables the process was created with — re-running from the variables the failed run wrote would not be starting from the beginning. The right choice when the run itself was wrong. Both re-run the same process instance, keeping its id and business key, and both use the workflow definition as it was when that process was created.

Programmatically, publish the request rather than calling the use case, so it is carried out by the node that owns the process:

processUpstreamEventUseCase.handle(new ProcessUpstreamEventCommand(
new RetryProcessRequested(processId) // or: new RestartProcessRequested(processId)
));

Neither is accepted for a COMPLETED or COMPENSATED process, or for one still running.

The three mechanisms on this page — a timeout and bounded retries per step, and compensation declared as compensable plus compensationStepId — are fields of a step here because the engine fixes how they behave. Expressing the same behaviour in a notation that assumes nothing costs a great deal more: the comparison page models this very saga in both, side by side, with both definitions downloadable.