Architecture
EventConductor’s architecture can be summarized in one sentence: a hexagonal engine embedded in your application, whose every state transition is an immutable event written in the same ACID transaction as the state itself. Everything else — distribution, scaling, recovery — falls out of that decision.
The big picture
Section titled “The big picture”The numbered flow: ① an integration event (e.g. ProcessCreationRequested, MessageReceived, a worker report) arrives through the upstream topic or in-process; ② the engine advances its state machine and writes the new state plus the resulting domain events to the outbox table in one ACID transaction; ③ the outbox relay publishes pending events (publish-then-mark — at-least-once, never lost); ④ TaskExecutionRequested events reach workers via the downstream topic (or the step’s own topic — the built-in steps’ are served by the orchestrator itself, see Built-in workers); ⑤ workers execute your business logic and report TaskStatusChanged back through upstream — and the loop continues until an END step completes the process.
The hexagon, mapped to packages
Section titled “The hexagon, mapped to packages”The engine is strictly hexagonal — the diagram above is not marketing geometry, it is the actual package layout of workflow-engine:
| Layer | Package | Contents |
|---|---|---|
| Domain | domain/aggregates | Process, StepExecution, WorkflowDefinition — pure state machine, no framework code. Each Process carries an immutable JSON snapshot of its definition. |
| Application | application/usecases, application/services | One use case per operation (create process, update step, correlate message, retry, cancel, check timers…), plus services like ProcessAnalyticsService. Ports to the outside live in application/out. |
| Inbound adapters | infra/in | async (event consumers), rest (Git import webhook), ui (the Mateu-built management UI), mcp (the MCP server and its @Tools), scheduler (cron starts, timer & timeout scans), startup (classpath importers). |
| Outbound adapters | infra/out | persistence (JPA repositories, outbox writer, per-vendor SQL for PostgreSQL/Oracle/MariaDB/H2), async (Kafka producers / embedded relay), memory (in-memory repositories for the zero-infrastructure mode), classpath (definition loading). |
Because business logic is banned from the definition and from the engine, the domain stays small enough to reason about — and to pin down exhaustively in TESTING.md.
The life of a step
Section titled “The life of a step”sequenceDiagram
autonumber
participant U as upstream topic
participant O as Orchestrator<br/>(use cases)
participant DB as Database<br/>(state + outbox)
participant R as Outbox relay
participant D as downstream topic
participant W as Worker
U->>O: ProcessCreationRequested
O->>DB: Process + StepExecutions + events (one ACID tx)
R->>DB: poll pending outbox rows
R->>D: TaskExecutionRequested (publish, then mark Sent)
D->>W: consume task
W->>W: business logic
W-->>U: TaskStatusChanged (+ output variables)
U->>O: worker report
O->>DB: advance state machine (this pod owns the process)
Duplicates are expected and absorbed by design: a second ProcessCreationRequested with the same business key creates nothing, a duplicate dispatch for a step already past PENDING is ignored, and a late worker report on a terminal step is discarded. Idempotency lives in the engine so your workers only need to be idempotent about their own side effects.
Built-in workers: actions stay out of the core
Section titled “Built-in workers: actions stay out of the core”The orchestration core decides the flow: what runs next, guards, CHOICE/FORK/JOIN, timers, waits,
retries, compensation and incidents. It never performs a step’s action itself. A step that does
something in the outside world is a task, dispatched and answered with the same contract as an
ACTION handled by your own worker. That includes calling an API, asking a model or publishing an
event:
| Step | Task | Topic (kafka mode) | Served by |
|---|---|---|---|
HTTP_CALL | http-call@1 | http-calls | worker-http |
LLM_CALL | llm-call@1 | llm-calls | worker-llm |
PUBLISH_EVENT | publish-event@1 | event-publications | worker-events |
AGENT | run-agent@1 | agent-calls | the agent runtime (a separate deployment) |
The engine renders the request when the step starts: templates over the process’s state, and
validation such as an unknown event destination, so a bad request fails the step at once. That task
request rides the outbox with the step’s state. The I/O happens in the worker, and so do the secrets
it needs: HTTP credentials and model API keys never enter the core. Because these steps are tasks,
they get what every worker step gets, with no code of their own in the engine: timeout, retries,
incidents, cancellation, the attempt fence on late reports and per-topic rate limits. The engine
reads which step types are built-in tasks, their task ids and their default topics from one registry
(BuiltInTasks).
The point of a built-in step is that nobody writes or deploys a worker for it, so the built-in workers ship with the engine:
- Embedded mode.
worker-embeddedruns them in the application’s process:worker-eventscomes with it,worker-httpandworker-llmare added alongside, or all of them at once with theio.mateu.workflow:builtin-workersaggregator. - Kafka mode. The orchestrator image contains them. Its own pods consume
http-calls,llm-callsandevent-publications(consumer grouporchestrator-builtin-workers) and report onupstreamlike any worker. The orchestrator image is one deployable with two parts: the orchestrator, and a worker with the built-in tasks. The tasks still travel through Kafka rather than an in-process shortcut, so durability, redelivery and sharing across replicas are those of every other task. Scaling the orchestrator scales them too. - Optional standalone workers. To keep HTTP egress or model API keys out of the orchestrator’s
pods, deploy
http-worker-standalone-app/llm-worker-standalone-appand take their topics out of the orchestrator’sBUILTIN_WORKER_TOPICS. The Helm chart does this whenhttpWorker/llmWorkeris enabled. See Deployment Modes.
One event still leaves from the engine itself: the publishEvent of a user task’s escalation. It is
not a step, so it has no task to report back to, and the outbox relay sends it. It is the same
CloudEvent record the publish-event worker sends.
Distributed coordination without a coordinator
Section titled “Distributed coordination without a coordinator”There is no leader election, no consensus protocol and no engine cluster to operate.
Every event carries its process as the Kafka message key. All the events of a process therefore hash to the same partition, and a consumer group hands each partition to exactly one consumer — so exactly one pod is ever working a given process. Serialization is not something the engine arranges after the fact; it is a property of how the events are addressed. The same key gives per-process ordering, which an unkeyed topic does not provide at all.
That leaves three pieces doing the work:
- Transactional outbox — state change and event are atomic, and the relay re-delivers after any crash (at-least-once). The pod that writes a row wakes its own relay, so the poll interval is a fallback for other pods’ rows rather than latency on every step.
- Ownership by partition — described above. It is also why sharding the database scales cleanly:
shards never coordinate. This is now an opt-in, elastic capability — N shared-nothing shards, added
and removed hot, routed by config (
workflow.sharding.*), off by default. See Sharding configuration. - An optimistic version on the process and its steps, fencing the one gap ownership leaves. A
consumer group guarantees which consumer is assigned a partition, not which is still in
flight: during a rebalance the outgoing pod can be finishing a record the incoming one now owns.
A stale write is rejected instead of overwriting, and rejections are counted
(
eventconductor.process.concurrent.writes.rejected) so the assumption can be watched rather than trusted.
In embedded mode there are no partitions to own, so a row lock on the process takes their place.
The engine used to work that way everywhere, with a per-process advisory lock; ownership replaced
it in kafka mode because a lock that arranges what the log already guarantees is pure cost.
The corollary is the engine’s performance model: it orchestrates, it doesn’t execute — workers do the heavy lifting and scale horizontally on Kafka, so the only resource that grows with orchestration throughput is database write capacity. See Performance for what the engine costs per transition and how to measure it yourself.
Time, messages and rules
Section titled “Time, messages and rules”Three inbound adapters make processes react to more than worker reports:
- Schedulers —
TIMERsteps (wait for a duration or until a date in a process variable), cron-scheduled process starts (cronExpressionon the definition) and the timeout scan that drives hung tasks through retry/error/compensation. - Message correlation —
WAIT_FOR_MESSAGEsteps pause until aMessageReceivedevent with a matching correlation key (JEXLcorrelationExpression) arrives, merging its variables into the process;SEND_MESSAGEsteps emit one, so processes can signal each other without a worker in between. - Rules —
RULEsteps evaluate expression rules or decision tables from the rule catalog via the embeddablerule-runtime, locally or over REST/gRPC with a Kafka-refreshed cache.
One architecture, three sizes
Section titled “One architecture, three sizes”The same code runs in three modes selected purely by configuration — the adapters swap, the hexagon doesn’t notice:
| Mode | What changes | Infrastructure |
|---|---|---|
embedded + memory | in-memory repositories, synchronous event dispatch | none |
embedded + jpa | JPA repositories, in-process outbox relay | a database |
kafka + jpa | Kafka topics between orchestrator and workers, one owning pod per process | database + Kafka |
Details and configuration in Deployment Modes.