Skip to content

Architecture

EventConductor’s architecture can be summarized in one sentence: a hexagonal engine embedded in your application, whose every state transition is an immutable event written in the same ACID transaction as the state itself. Everything else — distribution, scaling, recovery — falls out of that decision.

Operators management UI AI agents MCP · natural language Other services integration events Your Spring Boot application orchestrator · ×n pods UI (Mateu)pages · editors RESTGit webhook (HMAC) MCP server@Tool methods Schedulerscron · timer · timeout Event consumersupstream · worker events application — use cases · services Domain Process · StepExecution WorkflowDefinition state machine only — business logic lives in workers JPA /memory+ outbox Kafkaproducerdownstream workflow-engine forms-engine rule-runtime PostgreSQL / Oracle / H2 process state + outbox table — same ACID transaction events keyed by process: one owning pod per process Kafka upstream downstream outbox events in tasks out domain events Workers ×n stateless Kafka consumers your business logic · any language scale = add instances to the group 1 2 3 4 5

The numbered flow: ① an integration event (e.g. ProcessCreationRequested, MessageReceived, a worker report) arrives through the upstream topic or in-process; ② the engine advances its state machine and writes the new state plus the resulting domain events to the outbox table in one ACID transaction; ③ the outbox relay publishes pending events (publish-then-mark — at-least-once, never lost); ④ TaskExecutionRequested events reach workers via the downstream topic (or the step’s own topic — the built-in steps’ are served by the orchestrator itself, see Built-in workers); ⑤ workers execute your business logic and report TaskStatusChanged back through upstream — and the loop continues until an END step completes the process.

The engine is strictly hexagonal — the diagram above is not marketing geometry, it is the actual package layout of workflow-engine:

LayerPackageContents
Domaindomain/aggregatesProcess, StepExecution, WorkflowDefinition — pure state machine, no framework code. Each Process carries an immutable JSON snapshot of its definition.
Applicationapplication/usecases, application/servicesOne use case per operation (create process, update step, correlate message, retry, cancel, check timers…), plus services like ProcessAnalyticsService. Ports to the outside live in application/out.
Inbound adaptersinfra/inasync (event consumers), rest (Git import webhook), ui (the Mateu-built management UI), mcp (the MCP server and its @Tools), scheduler (cron starts, timer & timeout scans), startup (classpath importers).
Outbound adaptersinfra/outpersistence (JPA repositories, outbox writer, per-vendor SQL for PostgreSQL/Oracle/MariaDB/H2), async (Kafka producers / embedded relay), memory (in-memory repositories for the zero-infrastructure mode), classpath (definition loading).

Because business logic is banned from the definition and from the engine, the domain stays small enough to reason about — and to pin down exhaustively in TESTING.md.

sequenceDiagram
    autonumber
    participant U as upstream topic
    participant O as Orchestrator<br/>(use cases)
    participant DB as Database<br/>(state + outbox)
    participant R as Outbox relay
    participant D as downstream topic
    participant W as Worker

    U->>O: ProcessCreationRequested
    O->>DB: Process + StepExecutions + events (one ACID tx)
    R->>DB: poll pending outbox rows
    R->>D: TaskExecutionRequested (publish, then mark Sent)
    D->>W: consume task
    W->>W: business logic
    W-->>U: TaskStatusChanged (+ output variables)
    U->>O: worker report
    O->>DB: advance state machine (this pod owns the process)

Duplicates are expected and absorbed by design: a second ProcessCreationRequested with the same business key creates nothing, a duplicate dispatch for a step already past PENDING is ignored, and a late worker report on a terminal step is discarded. Idempotency lives in the engine so your workers only need to be idempotent about their own side effects.

Built-in workers: actions stay out of the core

Section titled “Built-in workers: actions stay out of the core”

The orchestration core decides the flow: what runs next, guards, CHOICE/FORK/JOIN, timers, waits, retries, compensation and incidents. It never performs a step’s action itself. A step that does something in the outside world is a task, dispatched and answered with the same contract as an ACTION handled by your own worker. That includes calling an API, asking a model or publishing an event:

StepTaskTopic (kafka mode)Served by
HTTP_CALLhttp-call@1http-callsworker-http
LLM_CALLllm-call@1llm-callsworker-llm
PUBLISH_EVENTpublish-event@1event-publicationsworker-events
AGENTrun-agent@1agent-callsthe agent runtime (a separate deployment)

The engine renders the request when the step starts: templates over the process’s state, and validation such as an unknown event destination, so a bad request fails the step at once. That task request rides the outbox with the step’s state. The I/O happens in the worker, and so do the secrets it needs: HTTP credentials and model API keys never enter the core. Because these steps are tasks, they get what every worker step gets, with no code of their own in the engine: timeout, retries, incidents, cancellation, the attempt fence on late reports and per-topic rate limits. The engine reads which step types are built-in tasks, their task ids and their default topics from one registry (BuiltInTasks).

The point of a built-in step is that nobody writes or deploys a worker for it, so the built-in workers ship with the engine:

  • Embedded mode. worker-embedded runs them in the application’s process: worker-events comes with it, worker-http and worker-llm are added alongside, or all of them at once with the io.mateu.workflow:builtin-workers aggregator.
  • Kafka mode. The orchestrator image contains them. Its own pods consume http-calls, llm-calls and event-publications (consumer group orchestrator-builtin-workers) and report on upstream like any worker. The orchestrator image is one deployable with two parts: the orchestrator, and a worker with the built-in tasks. The tasks still travel through Kafka rather than an in-process shortcut, so durability, redelivery and sharing across replicas are those of every other task. Scaling the orchestrator scales them too.
  • Optional standalone workers. To keep HTTP egress or model API keys out of the orchestrator’s pods, deploy http-worker-standalone-app / llm-worker-standalone-app and take their topics out of the orchestrator’s BUILTIN_WORKER_TOPICS. The Helm chart does this when httpWorker / llmWorker is enabled. See Deployment Modes.

One event still leaves from the engine itself: the publishEvent of a user task’s escalation. It is not a step, so it has no task to report back to, and the outbox relay sends it. It is the same CloudEvent record the publish-event worker sends.

Distributed coordination without a coordinator

Section titled “Distributed coordination without a coordinator”

There is no leader election, no consensus protocol and no engine cluster to operate.

Every event carries its process as the Kafka message key. All the events of a process therefore hash to the same partition, and a consumer group hands each partition to exactly one consumer — so exactly one pod is ever working a given process. Serialization is not something the engine arranges after the fact; it is a property of how the events are addressed. The same key gives per-process ordering, which an unkeyed topic does not provide at all.

That leaves three pieces doing the work:

  • Transactional outbox — state change and event are atomic, and the relay re-delivers after any crash (at-least-once). The pod that writes a row wakes its own relay, so the poll interval is a fallback for other pods’ rows rather than latency on every step.
  • Ownership by partition — described above. It is also why sharding the database scales cleanly: shards never coordinate. This is now an opt-in, elastic capability — N shared-nothing shards, added and removed hot, routed by config (workflow.sharding.*), off by default. See Sharding configuration.
  • An optimistic version on the process and its steps, fencing the one gap ownership leaves. A consumer group guarantees which consumer is assigned a partition, not which is still in flight: during a rebalance the outgoing pod can be finishing a record the incoming one now owns. A stale write is rejected instead of overwriting, and rejections are counted (eventconductor.process.concurrent.writes.rejected) so the assumption can be watched rather than trusted.

In embedded mode there are no partitions to own, so a row lock on the process takes their place. The engine used to work that way everywhere, with a per-process advisory lock; ownership replaced it in kafka mode because a lock that arranges what the log already guarantees is pure cost.

The corollary is the engine’s performance model: it orchestrates, it doesn’t execute — workers do the heavy lifting and scale horizontally on Kafka, so the only resource that grows with orchestration throughput is database write capacity. See Performance for what the engine costs per transition and how to measure it yourself.

Three inbound adapters make processes react to more than worker reports:

  • Schedulers — TIMER steps (wait for a duration or until a date in a process variable), cron-scheduled process starts (cronExpression on the definition) and the timeout scan that drives hung tasks through retry/error/compensation.
  • Message correlation — WAIT_FOR_MESSAGE steps pause until a MessageReceived event with a matching correlation key (JEXL correlationExpression) arrives, merging its variables into the process; SEND_MESSAGE steps emit one, so processes can signal each other without a worker in between.
  • Rules — RULE steps evaluate expression rules or decision tables from the rule catalog via the embeddable rule-runtime, locally or over REST/gRPC with a Kafka-refreshed cache.

The same code runs in three modes selected purely by configuration — the adapters swap, the hexagon doesn’t notice:

ModeWhat changesInfrastructure
embedded + memoryin-memory repositories, synchronous event dispatchnone
embedded + jpaJPA repositories, in-process outbox relaya database
kafka + jpaKafka topics between orchestrator and workers, one owning pod per processdatabase + Kafka

Details and configuration in Deployment Modes.