Skip to content

Benchmarks (local)

This page publishes one run of the local benchmark suite (scripts/benchmark.sh): six scenarios, four deployment modes, three measured repetitions each, on one developer laptop. It exists for two things:

  • comparing EventConductor with itself — one mode against another, one version against the next, one setting against another, on the same machine;
  • giving you an order of magnitude before you measure on your own hardware.

It is not a capacity claim and not a comparison with Temporal, Camunda or anyone else. Every component — engine pods, workers, load generator, PostgreSQL, Redpanda — runs on the same laptop, alongside whatever else the laptop was doing (recorded below). For what a figure from a real deployment looks like and how to read one, see Performance; for how the engines compare architecturally, Comparison.

Every worker is a no-op: it answers at once, so what is left is the engine, its database and its broker. Every ACTION step in kafka mode is a real round trip to a worker over the broker; in embedded mode it is an in-process call.

scenariodefinitionmeasures
sequentialSTART → 5 × ACTION → END (7 step executions)throughput, end-to-end latency
sagaSTART → 4 × ACTION, the 4th fails (retries: 0) → 3 compensations run → COMPENSATED (9 step executions)throughput, end-to-end latency
fan-outSTART → one ACTION with forEach over 50 items, parallel, maxConcurrency: 10 → END (53)processes/s, items/s, latency per process
timersSTART → TIMER PT1S → END, 10,000 processeshow late the timer fires (p50/p95/p99), throughput
messagesSTART → WAIT_FOR_MESSAGE (correlated by business key) → END; 10,000 processes parked, then 10,000 messagescorrelation throughput, message → process end
syncSTART → ACTION → REPLY → END, invoked through SyncInvocationService (what the REST endpoint calls, minus HTTP)request latency and requests/s at 1, 4 and 16 concurrent callers

The definitions are in modules/workflow-benchmark/src/main/resources/workflows/suite-*.json.

The four modes:

modeenginepersistencebrokerhow load enters
embedded / memory1 Spring contextin the heapnone8 caller threads call ProcessUpstreamEventUseCase
embedded / JPA1 Spring contextPostgreSQLnone8 caller threads, as above
kafka, 1 pod1 orchestrator context + 1 worker contextPostgreSQLRedpandaa Kafka producer on the upstream topic, as any external system
kafka, 2 pods2 orchestrator contexts + 1 worker contextPostgreSQLRedpandaas above
  • Warm-up, then three repetitions. Each scenario runs once at a reduced size and is discarded (JIT, connection pools, Kafka metadata), then three measured repetitions. Tables show the median of the three; throughput cells show median (min–max). Every repetition is in the JSON.
  • Each repetition starts from the same state. After a repetition the suite waits for the outbox to drain and deletes that repetition’s processes and step executions (VACUUM ANALYZE on PostgreSQL), so repetition 3 does not run against twice the rows repetition 1 did. Each mode gets a fresh database and fresh topics.
  • Throughput and latency are separate phases. Throughput: submit the whole batch at once (2,000 sequential, 1,000 saga, 100 fan-out processes) and divide by the time from the first submission to the last process’s finished timestamp. Latency: afterwards, a paced phase at half the throughput just measured (capped at 2,000/s) for 20 s. Latency measured under saturation is queue depth, not cost — Performance explains why.
  • Latency is measured from the intended send time of each process in the paced schedule to the finished timestamp the engine recorded, so a stalled driver cannot hide a stall in the engine.
  • Timers and messages are bursts by design. 10,000 processes are created at once; timer lateness is the engine’s own finishedAt − deadlineAt of each TIMER step, so it is unaffected by when the process started. For messages, 10,000 are sent at once after all 10,000 processes are confirmed waiting; “message → end” is therefore mostly time spent in the backlog, and the number to read is the correlation throughput.
  • Sync is closed-loop: 1, 4 or 16 threads each invoking, waiting for the reply and invoking again, for 15 s per level.
  • Resources are sampled every 500 ms: this JVM’s CPU (in cores) and heap, and the whole machine’s CPU (which includes the containers, the Docker VM and everything else running). Phases shorter than a second have no sample (shown as –).
  • Steps per process are counted, not assumed — from the step executions the engine wrote — and shown next to processes/s, because processes per second depends on the definition (see Performance).
MachineApple M3 Max, 14 cores (10 performance + 4 efficiency), 36 GB RAM, macOS 15.6.1
JDKAmazon Corretto 21.0.8, -Xms6g -Xmx6g -XX:+UseG1GC, one JVM for every pod, worker and the driver
DockerDocker Desktop 27.4.0, VM with 14 CPUs and 23 GB; no per-container CPU or memory limits
PostgreSQLpostgres:16-alpine (16.14), Testcontainers defaults: max_connections=100, shared_buffers=128MB, fsync=off (see caveats)
Redpandaredpandadata/redpanda:v24.1.7, Testcontainers defaults: --mode=dev-container --smp=1 --memory=1G, topics with 6 partitions
EventConductorFirst run: main at ad129191 (2.31.0) plus this page’s pull request (JSON: ad129191-dirty), for embedded/JPA and the supplementary tables. Memory-mode re-run: the same plus the in-memory fixes. Kafka re-run: 2.33.0 + the Kafka redelivery fix #565 (released in 2.33.1) + this pull request (JSON: 0bc77ca3)
Background loadload average ~7 at the start of both runs — the machine was shared with other builds, a local Kubernetes cluster and Spotlight indexing; per-repetition load average 3.5–19 in the main run and 4.9–25 in the memory-mode re-run. It is recorded per repetition in the JSON.

Engine settings are the shipped defaults except where the suite needs a value to run at all:

settingvalue
JPAddl-auto=update + the engine’s own Flyway migrations (indexes), Hibernate jdbc.batch_size=20, order_inserts/updates=true — as apps/orchestrator-standalone-app
Hikari pool10 per context
workflow.outbox-poll-interval-ms500 (default; the relay is also woken by its own pod’s writes)
workflow.outbox.batch-size / relay-concurrency100 / 0 (defaults)
workflow.timeout-scan-interval-ms10,000 (default) — decides timer lateness, see below
workflow.embedded.worker-threads0 (default: the worker runs on the calling thread)
workflow.sync.inline.enabledtrue (default)
Kafka consumersbatch-mode, concurrency 3 per binding per pod; worker concurrency 6
Kafka producersthe engine’s default synchronous sends

Medians of three repetitions; throughput as median (min–max). Latency is end to end (submission to the process’s finished timestamp) in the paced phase. “Steps/s” is processes/s × the step executions each process actually wrote. The memory-mode rows are from a re-run after the in-memory fixes described below; the kafka rows from a re-run on 2.33.0 plus the Kafka redelivery fix (#565, released in 2.33.1); the embedded/JPA rows from the first run, on 2.31.0. During the kafka re-run the machine’s load average was 5–19 except for two timer repetitions with one pod (43 and 57) and one sync repetition (28), so the kafka timer and sync rows carry more noise than the rest. Raw JSON for every run on this page: /benchmarks/2026-10-19/.

Sequential — START → 5 × ACTION → END (7 step executions)

Section titled “Sequential — START → 5 × ACTION → END (7 step executions)”
modeprocesses/ssteps/spaced at /sp50 msp95 msp99 msJVM coresmachine CPU %
embedded / memory12,514 (11,991–12,635)87,5972,0000.870.941.03––
embedded / JPA8.64 (8.26–9.91)60.54.321061421760.1842.7
kafka, 1 pod52.3 (47.8–53.4)36626.24237708510.7860.9
kafka, 2 pods65.4 (57.8–69.1)45832.72704885280.9377.3

Saga — 4 steps, the last fails, 3 compensations (9 step executions)

Section titled “Saga — 4 steps, the last fails, 3 compensations (9 step executions)”
modeprocesses/ssteps/spaced at /sp50 msp95 msp99 msJVM coresmachine CPU %
embedded / memory7,366 (7,167–7,560)66,2972,0001.001.081.17––
embedded / JPA11.3 (11.1–12.1)1025.6690.02172510.2129.2
kafka, 1 pod27.9 (24.1–31.2)25113.92554385160.5959.8
kafka, 2 pods62.3 (61.5–73.0)56131.24697528151.0376.6

Every saga process in every mode ended COMPENSATED.

Fan-out — forEach over 50 items, parallel, maxConcurrency: 10 (53 step executions)

Section titled “Fan-out — forEach over 50 items, parallel, maxConcurrency: 10 (53 step executions)”
modeprocesses/sitems/spaced at /sp50 msp95 msp99 ms
embedded / memory971 (855–975)48,543 (42,741–48,731)4858.9110.030.0
embedded / JPA1.12 (1.10–1.15)56.0 (54.9–57.5)1.00833931967
kafka, 1 pod4.04 (2.49–4.44)202 (124–222)2.021,0361,5351,986
kafka, 2 pods2.74 (2.05–2.92)137 (103–146)1.379811,4511,612

Every repetition completed 100 of 100 processes in both kafka modes. On 2.33.0 and earlier this scenario lost events in kafka mode (19 and 12 of 100 processes stuck in the first run) — see Problems the suite found; the kafka rows here are on the fix.

Timers — 10,000 processes with a TIMER PT1S, default 10 s scan interval

Section titled “Timers — 10,000 processes with a TIMER PT1S, default 10 s scan interval”
modeprocesses/slateness p50 msp95 msp99 msmax ms
embedded / memory954 (954–956)9,0759,3479,4119,470
embedded / JPA27.0 (25.9–27.4)10,60218,09619,59620,475
kafka, 1 pod125 (110–145)10,81124,39228,30429,539
kafka, 2 pods185 (173–217)9,92519,86221,84322,536

The same scenario with workflow.timeout-scan-interval-ms=1000 (-Dsuite.engine.workflow.timeout-scan-interval-ms=1000; first run, 2.31.0):

modeprocesses/slateness p50 msp95 msp99 msmax ms
embedded / memory4,134 (3,954–4,137)9971,3111,3691,416
kafka, 1 pod153 (135–161)1,25514,38117,28718,966

With a 1 s scan the median timer fires about a second late in both modes; the kafka tail is the time it takes one pod to work through 10,000 timers that all came due in the same second (~150 processes/s), not the timer itself.

Messages — 10,000 parked processes, then 10,000 messages at once

Section titled “Messages — 10,000 parked processes, then 10,000 messages at once”
modecorrelation throughput (messages/s)message → end p50 msp95 msp99 ms
embedded / memory2,163 (2,037–2,205)5.617.269.09
embedded / JPA30.9 (30.2–31.2)214,386301,953309,718
kafka, 1 pod398 (316–422)11,21522,69624,578
kafka, 2 pods638 (591–691)7,01413,86215,179

The latency columns are a 10,000-message backlog draining at the throughput on the left — in embedded/JPA, five minutes of it. Read the throughput.

Synchronous invocation — START → ACTION → REPLY → END, closed loop

Section titled “Synchronous invocation — START → ACTION → REPLY → END, closed loop”
modec=1 req/sp50 msp99 msc=4 req/sp50 msp99 msc=16 req/sp50 msp99 ms
embedded / memory3,0600.241.0410,6160.261.3816,0000.2811.1
embedded / JPA22.943.554.662.463.083.977.173.31,068
kafka, 1 pod19.448.110954.865.222293.6154499
kafka, 2 pods21.744.861.368.756.782.0104145291

No request failed or timed out in any mode. Requests always enter through the first pod; with two pods the process’s partition is owned by the other pod about half the time.

Durable commits make no measurable difference here

Section titled “Durable commits make no measurable difference here”

Sequential and sync, at a quarter of the sequential batch size, with PostgreSQL’s fsync off (the Testcontainers default used above) and on (first run, 2.31.0):

embedded / JPA processes/sp50 mskafka, 1 pod processes/sp50 mskafka sync c=1 p50 ms
fsync=off9.8512241.519442.4
fsync=on10.013039.120242.4

On this machine the cost is round trips, not disk flushes (Docker Desktop’s virtual disk is also not a production disk). On real storage with slow fsync — network block storage in particular — expect the JPA modes to be slower than this, as Performance describes.

Five consecutive batches of the sequential definition, never purged (-Dsuite.scenarios=growth):

batch 1batch 2batch 3batch 4batch 5
memory, 4,000 per batch, before the fixes1,1783891737852
memory, 20,000 per batch, after9,6489,96410,0478,92910,444
embedded / JPA, 400 per batch9.8210.210.210.49.51

(processes/s, one repetition each.)

Running the same scenarios in every mode surfaced five problems. Three were small and are fixed in the pull request that added this page; one was fixed in its own change (#565); one is a design characteristic of the mode.

Fixed — memory mode got slower with every step it had ever run. Every per-process query of the in-memory step store (findByProcessId, findPendingOrRunningByProcessId, and the due-timer and retry lookups by process) scanned the whole store, and several run on every transition — so the cost of a transition grew with the history in the heap. Five consecutive batches of 4,000 sequential processes, never purged: 1,178 → 389 → 173 → 78 → 52 processes/s before. With that fixed (an id index per process), the next scan showed up at larger sizes: findByBusinessKey, which every process creation calls, also scanned the whole store — batches of 20,000 fell from ~5,000 to ~560/s. With both indexed, five batches of 20,000 hold at ~10,000 processes/s (table above). Memory mode is what the test kit runs on, so a long test class paid for it too.

Fixed — memory-mode synchronous invocations ran one at a time. InMemoryInvocationRepository created every invocation under one monitor, and in memory mode the creation runs the whole process inline, so 1, 4 and 16 callers all got ~520 requests/s with one core busy. It now locks per idempotency key (the JPA store’s equivalent is a unique index, which never serialized different keys). The same class also scanned every retained invocation on each REPLY step, so each call got slower than the last (560 → 170 requests/s for one caller over three 45-second runs); it now indexes invocations by process. Together with the business-key index: 3,060 / 10,616 / 16,000 requests/s at 1 / 4 / 16 callers, steady across repetitions.

Fixed — the embedded/JPA outbox relay never stopped. Its thread was a while (true), so closing the engine’s Spring context left it polling a closed connection pool and logging an error every 500 ms for the life of the JVM — anything that starts more than one engine in a JVM (this suite, tests that restart an engine) inherited it. It now stops with the context.

Fixed in 2.33.1 (#565) — kafka mode dropped events under a parallel forEach. Found by this benchmark; affects 2.33.0 and earlier. In the first run, on 2.31.0, both kafka modes ended the fan-out scenario with processes stuck for good (81 of 100 completed with one pod, 88 of 100 with two), iterations left PENDING with nothing left in the outbox. The log showed the cause: with ten iterations of the same process completing at once, the pod’s consumeUpstream and consumeOutbox consumers write the same process concurrently, the optimistic lock rejects one (ConcurrentProcessAccessException, plus PostgreSQL deadlocks), the engine rethrows it as retryable so the binder will redeliver — and Spring Kafka’s default batch error handler retried a batch ten times with no back-off and then discarded it: FallbackBatchErrorHandler: Records discarded: outbox-5@31836, … — 383 records in two batches with one pod, 254 in one batch with two. #565 replaces it with an error handler that retries retryable failures with exponential back-off, never discards them, and restarts from the failed record. The kafka rows on this page are re-measured on that fix.

Historical note: on 2.33.0 and earlier, the mitigation was to give fan-out ACTION steps a timeout (or set workflow.default-step-timeout-ms) so a lost reply is retried rather than waited for forever. Upgrading removes the need.

Not fixed, by design — embedded/JPA advances everything on one thread. The embedded outbox relay delivers each message by running the engine synchronously, one message at a time, on a single thread (the class says why). Sampled during the sequential scenario, that thread spent nearly all its time waiting for PostgreSQL round trips, the JVM used 0.2 cores, and PostgreSQL counted roughly 15 commits per step execution (including the suite’s own progress polling). So embedded/JPA throughput is the inverse of the round-trip latency × statements per step — about 60–100 step executions/s here, with the CPU idle. It is the mode to pick for durability without a broker, not for throughput; making its dispatch concurrent is a separate change with its own risks.

  • Memory mode measures the engine’s own code path — no I/O at all — at 65,000–90,000 step executions/s and a millisecond per sequential process. It is a ceiling for the logic, not for any durable deployment.
  • With a database in the loop the engine is latency-bound, not CPU-bound. The JVM never used more than about one core in any JPA or kafka phase while the machine still had most of its 14 cores free. Throughput there is set by round trips (database and broker) per transition and by how many transitions are in flight — the same conclusion Performance reaches on a cluster.
  • A second pod helps throughput and costs cross-pod latency. Two pods moved sequential throughput from ~52 to ~65 processes/s and saga from ~28 to ~62, but the saga’s paced latency rose (p50 255 → 469 ms): with two pods a process’s events are handed between pods, and the relay’s wake-up signal is per pod, so the hand-off waits for the other pod’s 500 ms poll. Lowering workflow.outbox-poll-interval-ms is the documented lever. The parallel fan-out is the exception: two pods were slower (137 against 202 items/s) — ten iterations of one process completing at once is exactly the same-process write contention described below, and spreading it across two pods adds hand-offs without adding useful parallelism.
  • The redelivery fix costs paced latency where same-process writes collide. Against the first run (2.31.0, before #565), one-pod sequential throughput was similar (41 → 52 processes/s) but the paced p50 roughly doubled (204 → 423 ms) and the one-pod saga lost throughput (37 → 28). A rejected concurrent write now waits out a back-off before it is retried instead of being retried at once (and, after ten tries, dropped). Some of the difference is background load (recorded per repetition); fewer same-process write conflicts is the follow-up that would remove the rest. Both runs’ JSON is published.
  • Timer lateness is the scan interval. With the default workflow.timeout-scan-interval-ms of 10 s, a one-second timer fires 8–10 s late at the median: all 10,000 come due together and fire on the next scan. In the JPA modes the tail adds the time to work through 10,000 due timers at once. Lowering the interval is the lever — see the 1 s run below. A TIMER is accurate to its scan interval, not to the millisecond.
  • Synchronous invocation costs ~40 ms per request with a database (one worker round trip plus the reply), and scales roughly with concurrency up to the throughput the mode allows.
  • One machine. Pods, worker, driver, PostgreSQL and Redpanda share a laptop with other work. That is fine for comparing modes and builds with each other on the same machine, and it is not a scalability figure. The Performance guide covers what changes on a cluster (partitions × consumers become a first-order lever; on one machine they mostly contend).
  • fsync=off. Testcontainers starts PostgreSQL without durable commits, which flatters every JPA figure against a production database. -Dsuite.postgres.fsync=on measures with them; see the comparison below.
  • Default tuning. Every figure is at the shipped defaults (poll interval, scan interval, consumer concurrency, pool size). The existing harness in the same module shows what tuning them does.
  • Memory mode’s saturation phases last under a second, so they have no CPU samples and their throughput has the most run-to-run noise.
  • Messages and timers are bursts. Their latency columns describe a 10,000-deep backlog draining, not the cost of one correlation; read their throughput.
Terminal window
scripts/benchmark.sh # all modes and scenarios, 3 repetitions (~3 h on this machine)
scripts/benchmark.sh -Dsuite.scale=0.01 -Dsuite.reps=1 # a smoke run, a few minutes
scripts/benchmark.sh -Dsuite.modes=memory,kafka1 -Dsuite.scenarios=sequential,sync
scripts/benchmark.sh -Dsuite.engine.workflow.timeout-scan-interval-ms=1000 -Dsuite.scenarios=timers
scripts/benchmark.sh -Dsuite.postgres.fsync=on

It needs Docker and JDK 21, builds the benchmark module and what it depends on from your checkout (a reactor build — nothing is installed into ~/.m2), starts PostgreSQL and Redpanda in Testcontainers (removed when it ends), and writes benchmark.json (every repetition, every setting, the environment) and benchmark.md to modules/workflow-benchmark/target/benchmark/. Knobs: suite.modes, suite.scenarios (add growth for the history-growth check), suite.reps, suite.scale, suite.sync-seconds, suite.partitions, suite.pool-size, and suite.engine.<property> for any engine property. The script records the machine, the commit and the load average before it starts. Publish the JSON with any number you quote from it.