Skip to content

Reliability

An orchestrator’s only real promise is that work it accepted will finish. Everything else — the DSL, the UI, the throughput — is worth nothing if a process can quietly stop forever because a message went missing during a thirty-second broker restart.

This page is what happens when we try to break it on purpose, measured on the current version. The harness is in modules/workflow-benchmark/k8s/reliability and every number here is reproducible with two commands.

Four things must hold, and the harness computes all of them from the engine’s own tables:

Conservationprocesses in the engine = creation events the broker acknowledged
Exactly-onceno (process, step) pair has more than one execution row
Drainonce the load stops: no live process, no live step, no unsent outbox row
No poisonno outbox row parked as Error

The count of work genuinely handed over is written to the database by the load driver once a second, with acks=all and an idempotent producer. That matters: the driver is inside the blast radius, so a verdict held in its memory would be worthless the first time a scenario killed it.

Each runs while the engine is under continuous load, on a three-node Kubernetes cluster:

  • one orchestrator pod killed
  • the entire orchestrator tier killed at once
  • a rolling redeploy under load
  • the broker stopped for 90 seconds
  • PostgreSQL stopped for 90 seconds
  • the node hosting most of the engine drained
  • the workflow definition replaced while thousands of processes are mid-flight
Terminal window
export BENCH_IMAGE=<user>/eventconductor-bench:<tag>
export SOAK_RATE=4
./ec-reliability.sh deploy
./chaos.sh all
./ec-reliability.sh drain 2400

All seven scenarios recovered, and after the load stopped the engine drained completely in 144 seconds: 6,084 processes, all finished, nothing live, nothing unsent.

InvariantResult
Conservation6,086 acknowledged, 6,084 present — 2 dead-lettered (see below)
Exactly-once0 duplicate step executions
Drain0 live processes, 0 live steps, 0 unsent outbox rows
No poison0 outbox rows in Error

Recovery is defined as the finished count rising across two consecutive samples — not as pods reporting Ready, which is a weaker claim: a pod can be Ready and consuming nothing.

ScenarioProgress resumed
One pod killed54 s
Whole tier killed53 s
Rolling redeploy12 s
Broker down 90 s23 s after it returned
PostgreSQL down 90 s48 s after it returned
Node drained60 s
Definition replaced12 s

The design claims a process runs against the definition it was created with, copied into its own row, so editing a definition cannot corrupt work already in flight. Replacing a three-step definition with a four-step one while thousands of processes were running produced exactly two shapes and no third: 5,662 processes finished with the old definition and 422 with the new one. No hybrids, in either run of this test.

Four mechanisms carry the guarantees above through an outage and under contention. Each is engine default behaviour on the current version.

Workers never drop a reply. StreamBridge.send reports failure by returning false. Replies go through WorkerReply, which retries and then throws so Kafka redelivers the task rather than committing an offset for work that was never reported as done. A reply refused during a broker outage is redelivered once the broker returns. Your handlers must be idempotent. That covers the engine’s own workers too — the rule runtime answering a RULE step, and the forms engine answering a USER_TASK. The human task is the one that can least afford a lost reply: it arrives over HTTP, so no offset redelivers it, and a USER_TASK is deliberately given no fallback deadline. There the reply goes out before the task is marked complete, so a refused one leaves the task open for the person to submit again rather than stranding the process.

The outbox only marks a message Sent once the broker has it. The relay delivers before marking the row, and producer sends are synchronous by default — the framework contributes that setting itself, to every module that can publish, rather than trusting each application’s YAML — so a refused send throws and leaves the row Pending to be retried. During the verification run the outbox held 55,600 rows marked Sent against 55,601 messages on the topic: one more on the topic, which is at-least-once working correctly. Never fewer.

No live step is invisible. The deadline scan is an index range over the deadline column, so a step without a deadline would never be looked at again. The eventconductor.steps.stalled gauge counts live steps with no deadline that have waited too long, and workflow.default-step-timeout-ms gives ACTION and RULE steps a fallback deadline that hands them to the existing retry path. The fallback is off by default, and never applied to USER_TASK, PROCESS or WAIT_FOR_MESSAGE, whose waiting is unbounded on purpose.

A retryable failure is redelivered, not discarded. The engine rethrows a retryable failure — an optimistic-lock conflict, a deadlock, the database away — precisely so Kafka redelivers it, and every engine consumer now runs with an error handler that does: exponential back-off from 100 ms to 10 s, without limit, from the record that failed. Up to and including 2.32.0 the consumers ran with Spring Kafka’s default batch error handler, which retried a batch nine times with no pause and then discarded it. Two consumers contending for one process exhaust that in milliseconds, and a parallel forEach makes them contend constantly: a process’s iterations complete together, their replies arrive on upstream while the engine’s own step events for the same process arrive on outbox. The local benchmark lost 12–19% of its fan-out processes that way, stuck with iterations PENDING and nothing in the outbox. DIST-37 now runs that shape (100 processes × 50 parallel items) on every build and requires all of them to finish with no record discarded. See Consumer redelivery.

Conservation was 2 short, and both were dead-lettered rather than silently dropped — they are on the dead-letter topic with the reason attached, replayable. The reason was JpaSystemException: Unable to rollback against JDBC Connection, thrown when PostgreSQL was stopped mid-transaction: the rollback fails because the connection is gone.

Connection-level SQLExceptions are classified as retryable — matched on SQLState (class 08, class 53, and PostgreSQL’s 57P01–57P03) as well as by type. Being unable to roll back because the database has gone away is the most retryable failure this engine can have.

The section above is about an event the engine understood and could not process. There is an earlier failure: bytes that never become an event at all — JSON that does not parse, or a type this version does not know.

It reaches no handler, so none of the handling above applies to it. Spring Cloud Stream’s converter fails, and the binder’s default answer is to drop the record, commit the batch and advance the offset. Until 2.2.2 that is all that happened: no log line at any level, no dead letter, no metric, and consumer-group lag back to zero. A producer sending a thousand malformed messages saw a healthy engine that had created nothing, which reads exactly like messages that never arrived — so the search starts at the producer, then the topic, then the consumer group, and the payload is the last thing anyone looks at.

Such a record is now logged at ERROR with the reason and an excerpt of the payload, parked on the dead-letter topic as the original bytes with x-dead-letter-unreadable: true, and counted by eventconductor.events.dead.lettered like any other dead letter.

It is still skipped rather than retried, and deliberately: bytes that cannot be parsed now cannot be parsed on redelivery either, so failing the batch would stall the partition for every process behind it, for ever. What changed is that the skip says so.

Sharded message routing cannot lose a message

Section titled “Sharded message routing cannot lose a message”

Message routing sends a message to the shard that can correlate it instead of broadcasting it to all of them. The reliability question is whether narrowing the fan-out can drop a message the broadcast would have delivered. It cannot, by two properties, both of which hold with routing on:

  • The router only filters; it never decides. Every layer fails toward the broadcast that always worked: a message whose key the placement store does not own, or an expression key no shard has subscribed, is broadcast; and if a routing store is down, the lookup is caught and the message is broadcast too. A wrong or unavailable answer costs an extra query on a shard that will not match, never a lost message — the receiving shard still runs the ordinary correlation, so routing changes where a message is tried, not whether it is.
  • The per-shard filter can only ever save a query. The residual layer — a Bloom filter each shard keeps over the pairs it has waiting — is asked, of a broadcast message, “could a step here match?” A Bloom filter never answers “no” for a pair it holds, so a waiting step is never filtered away; its only possible error is the opposite one, saying “maybe” when nothing waits, which costs a query that finds nothing and drops no message. The failure that would lose a message — “did not look, and something was there” — cannot occur.
  • A routed message is as durable as any other. It rides the target shard’s upstream Kafka topic, so a shard that is down when a message is routed to it correlates the message when it returns — the same recovery the broker-outage scenario already measures.

Three chaos scenarios confirm it. Re-running the broker-down scenario (Dist06) with routing present showed recovery unchanged: the outage is ridden and progress resumes when the broker returns, exactly as with routing off. Dist21 then puts the message path itself through an outage with the filter on: 20 processes are driven to a WAIT_FOR_MESSAGE and parked, the broker is stopped, each process’s resume message is published into the dead broker, and while it is down nothing advances; when the broker returns all 20 correlate their resume and complete, the outbox drains to zero, and nothing is dead-lettered — no resume is lost across the outage, and the filter on the correlation path drops none of them.

Dist22 runs the same outage across two genuinely separate shards — each a full engine with its own database and its own upstream/outbox topics, sharing the one messages topic every shard consumes. Twelve processes are split across the shards and parked; the broker is stopped; each resume is broadcast onto the shared topic; and when the broker returns both shards receive every resume, each completes exactly its own six, both outboxes drain to zero, and nothing is dead-lettered. It is the cross-shard properties the single-node tests cannot reach, shown to survive an outage: the shared-topic fan-out, the per-shard filter ruling out the keys a shard does not hold, and recovery of the message path itself.

The subscription layer carries one semantic to know — a projection window: a step that has started waiting but whose subscription has not projected yet is not yet visible to that layer, so it is still covered by the business-key and broadcast layers, and a WAIT_FOR_MESSAGE that cannot tolerate the window can opt out of routing to force broadcast.

Restart an orchestrator — a redeploy, a crashed pod, a drained node, the whole tier at once — and every process carries on from where it was. Nothing is replayed from the beginning and nothing is lost, because the engine keeps no state that matters in memory:

  • Every transition is in the database, committed together with the events it produces (the outbox). A pod that dies between the two leaves the events Pending, and the relay sends them when a pod comes back — at-least-once, as always.
  • Waits are rows, not threads. A TIMER’s due moment, a step’s timeout or deadline, and a WAIT_FOR_MESSAGE’s subscription are stored on the step, and the scheduler finds them with an indexed query. A timer set for three days fires on time whichever pod is up then, and a message that arrives after the restart still finds the step waiting for it. (A step that started under an engine version older than those columns is armed once at startup by InFlightStepRearmRunner, which retries in the background until the database answers.)
  • Kafka remembers where each consumer was. Worker replies sent while the engine was down are consumed when it returns; with several orchestrators, the survivors take over the dead pod’s partitions at once.
  • The database may come back after the pod. With a lazy connection pool, ddl-auto: none and an explicit Hibernate dialect, an orchestrator starts with PostgreSQL unavailable and picks up its work when the database returns — the recipe is DIST-08 in TESTING.md. Without them it fails to start, and Kubernetes restarts it until the database answers.

Three things to know:

  • A task running inside the pod that died — an embedded worker, in embedded mode — is not redelivered: the step stays RUNNING until its timeout, and then retries or takes its onTimeoutStepId. Give ACTION steps a timeout, or set workflow.default-step-timeout-ms. Tasks on Kafka workers are unaffected: the task is still on its topic, or still being worked on.
  • workflow.persistence=memory keeps nothing across a restart. It is for tests and demos.
  • The database has to be durable. Everything above is only as good as the storage under PostgreSQL: on an ephemeral volume (a Kubernetes emptyDir, a container without a volume), restarting the database pod loses every process, however durable the engine is.

These are the scenarios the distributed suite breaks the engine with on every build: a pod killed mid-process (DIST-02), the broker stopped mid-process (DIST-06), a pod started without PostgreSQL (DIST-08), and a node dying inside an embedded worker (InlineCrashRecoveryE2eTest).

eventconductor.steps.stalledLive steps with no deadline that nothing will time out. Any sustained non-zero value is work that will never finish.
eventconductor.events.dead.letteredThe engine gave up on a specific event and said so. Always worth a look.
eventconductor.process.concurrent.writes.rejectedOptimistic-lock conflicts. Expected during a rebalance and under bursts of replies for one process (a parallel forEach); each one is redelivered.
eventconductor.events.redeliveredRecords the consumers’ error handler is redelivering. A trickle under contention is normal; a rate that keeps climbing on one topic — with the WARN line has failed N times and is still being retried — is a failure that is not clearing and is holding its partition.
Outbox rows not SentNormal during a broker outage and should return to zero. A sustained backlog is not normal.

synchronous_commit stays on in every measurement here. Turning it off would multiply throughput and invalidate the whole page, because the losses it permits are precisely the ones being tested for.