Skip to content

Kafka Topics

Applies to workflow.mode=kafka only.

Events are serialized as JSON with a type property carrying the registered event id (kebab-case, as declared on DomainEvent) — for example "type": "process-creation-requested".

TopicDirectionDescription
outboxInternalDomain events relayed from the outbox table
upstreamInboundIntegration events from external services
downstreamOutboundTask execution requests sent to workers
http-callsOutboundHTTP_CALL requests, served by the orchestrator’s built-in workers, or optionally by http-worker-standalone-app (a step’s topic overrides)
llm-callsOutboundLLM_CALL prompts, served by the orchestrator’s built-in workers, or optionally by llm-worker-standalone-app (a step’s topic overrides)
event-publicationsOutboundPUBLISH_EVENT tasks (publish-event@1), served by the orchestrator’s built-in workers, which write the event to its destination topic (a step’s topic overrides)
agent-callsOutboundAGENT invocations (run-agent@1), served by the IA agent runtime; replies come back on upstream naming their invocationId
dead-letterOutboundEvents the engine can never process, parked unchanged for inspection and replay
process-indexInternalProcess status changes for the standalone projector (workflow.projection.mode=remote only)
human-tasksOutboundEvery change of a human task (created, assigned, claimed, released, completed, cancelled, escalated), published by forms-standalone-app (FORMS_HUMAN_TASKS_TOPIC) as a binary CloudEvent (ce_type io.eventconductor.task.<event>) for inboxes and notification centres — and for event triggers when a destination maps to it; the engine does not read it otherwise
rulesInternalRule catalog changes (rule-published, rule-deleted) published by rule-standalone-app, for rule runtimes that keep a local cache (rules.kafka-refresh=true, binding consumeRuleCatalogEvent-in-0)
destination topicsInboundDomain events (CloudEvents) on the topics of workflow.events.destinations.<name>.topic, for event triggers and onEvent waits

The engine contributes the outbox, upstream, downstream, dead-letter and process-index bindings (and the orchestrator’s two consumer groups) as lowest-precedence defaults, so an application only sets them to change them.

Purpose: Carries domain events from the outbox table to the event handlers inside the orchestrator. This is an internal topic — external services should not publish to it.

Published by: OutboxRelay (reads pending events from the DB and publishes here)

Consumed by: Orchestrator event handlers (ProcessCreatedEventHandler, StepExecutionStatusUpdatedEventHandler, etc.)

Event types (among others):

  • process-created
  • step-execution-status-changed
  • task-execution-requested
  • process-status-changed
  • message-subscription-changed

Purpose: Integration events from external services to the orchestrator.

Published by: External services, workers, and the orchestrator itself (for internal events)

Consumed by: ProcessUpstreamEventUseCase

Event types (inbound):

// Start a new process
{
"type": "process-creation-requested",
"workflowDefinitionId": "my-workflow",
"businessKey": "order-123",
"variables": [
{ "name": "orderId", "value": "123" }
]
}
// Deliver a message to waiting WAIT_FOR_MESSAGE steps
{
"type": "message-received",
"messageName": "payment-received",
"correlationKey": "order-123",
"variables": [
{ "name": "paymentId", "value": "P-9" }
]
}
// Report task completion
{
"type": "task-status-changed",
"taskExecutionId": "exec-uuid",
"status": "COMPLETED",
"variables": [
{ "name": "result", "value": "ok" }
]
}
// Append a log line to a task execution
{
"type": "task-log-emitted",
"taskExecutionId": "exec-uuid",
"messageType": "Info",
"message": "40% done"
}
// Timeout check (sent by the orchestrator's scheduler)
{
"type": "timeout-check-requested",
"processId": "process-uuid"
}
// Timer check for due TIMER steps (sent by the orchestrator's scheduler)
{
"type": "timer-check-requested",
"processId": "process-uuid"
}
// Operator commands on one process (what the UI, REST and MCP send in kafka mode)
{ "type": "process-cancellation-requested", "processId": "process-uuid" }
{ "type": "pause-process-requested", "processId": "process-uuid" }
{ "type": "resume-process-requested", "processId": "process-uuid" }
{ "type": "restart-process-requested", "processId": "process-uuid" }
{ "type": "retry-process-requested", "processId": "process-uuid" }
{ "type": "retry-step-execution-requested", "processId": "process-uuid", "stepExecutionId": "exec-uuid" }
// Manual operations (admin, audited — see Operating running processes)
{ "type": "set-process-variables-requested", "processId": "process-uuid",
"set": [{"name": "checkIn", "value": "2026-11-02"}], "remove": ["tempQuote"],
"reason": "Guest moved the stay", "requestedBy": "alice" }
{ "type": "complete-step-manually-requested", "processId": "process-uuid", "stepExecutionId": "exec-uuid",
"outputVariables": [{"name": "paymentRef", "value": "PHONE-8812"}], "reason": "Paid by phone", "requestedBy": "alice" }
{ "type": "retry-from-step-requested", "processId": "process-uuid", "stepId": "charge",
"reason": "Partner API fixed", "requestedBy": "alice" }
{ "type": "migrate-process-requested", "processId": "process-uuid", "targetVersion": 5,
"stepMapping": {"charge": "charge-card"}, "variableMapping": {"currency": "EUR"},
"reason": "v5 charges through the new gateway", "requestedBy": "alice" }
// An update of a running process (see Updating running processes) — keyed by processId; the
// updateId is its idempotency key; payload is JSON text, validated against the update's `accepts`
{ "type": "process-update-requested", "updateId": "upd-7781", "processId": "process-uuid",
"updateName": "changeAddress", "payload": "{\"address\":\"Elm Street 2\"}", "requestedBy": "billing" }
// Steps a DYNAMIC step injects into its process
{
"type": "steps-injected",
"taskExecutionId": "exec-uuid",
"processId": "process-uuid",
"stepsJson": "[ { \"id\": \"extra\", \"type\": \"ACTION\", \"topic\": \"svc\" } ]"
}

A message-received event resumes processes waiting on a WAIT_FOR_MESSAGE step with the same messageName and correlation key. Alternatives to raw Kafka delivery: POST /workflow/api/messages, the sendMessage MCP tool, a SEND_MESSAGE step in another workflow, or programmatically via ProcessUpstreamEventUseCase — see Step Types — WAIT_FOR_MESSAGE.

The operator commands are handled by CancelProcessRequestedEventHandler, PauseProcessRequestedEventHandler, ResumeProcessRequestedEventHandler, RestartProcessRequestedEventHandler, RetryProcessRequestedEventHandler, RetryStepExecutionRequestedEventHandler, SetProcessVariablesRequestedEventHandler, CompleteStepManuallyRequestedEventHandler and RetryFromStepRequestedEventHandler, which call the same use cases as the UI and MCP tools (CancelProcessUseCase, …); steps-injected is handled by StepsInjectedEventHandler. A process-update-requested is handled by ProcessUpdateRequestedEventHandler (ApplyProcessUpdateUseCase): the owner of the process applies the update under its lock and records the outcome in process_update — produced directly to the topic, with no row from a door, the update is recorded there (its payload validated first); read it at GET /workflow/api/updates/{updateId}. See Updating running processes. The commands address the process by processId; the optional businessKey of a process-cancellation-requested is not used to find it. See Starting a Process — Cancelling a process.

Purpose: Task execution requests dispatched by the orchestrator to workers.

Published by: Orchestrator (when an ACTION step is ready to execute)

Consumed by: Worker microservices

Event payload:

{
"type": "task-execution-requested",
"taskExecutionId": "exec-uuid",
"processId": "process-uuid",
"workflowDefinitionId": "my-workflow",
"stepId": "process-payment",
"taskId": "task-uuid",
"variables": [
{ "name": "orderId", "value": "123" },
{ "name": "amount", "value": "99.90" }
]
}

Workers consume this topic, perform their work, and publish a task-status-changed to the upstream topic.

A USER_TASK’s request carries its rendered assignment in the __task variable (JSON: assignee, candidateUsers, candidateGroups, priority, dueDate, followUpDate, businessKey). When one of its escalations fires, the engine sends a human-task-escalation-requested on the same topic the task went to — {taskExecutionId, processId, stepId, level, action, reassignTo, escalateTo, at} — which the forms engine applies to the task (once per level).

http-calls, llm-calls and event-publications

Section titled “http-calls, llm-calls and event-publications”

Purpose: The rendered requests of HTTP_CALL steps, the rendered prompts of LLM_CALL steps and the rendered events of PUBLISH_EVENT steps (task variable __event). They are the default topics for those steps (workflow.http.topic, workflow.llm.topic, workflow.events.worker.topic on the serving side); a step’s topic sends its requests elsewhere.

Published by: Orchestrator

Consumed by: The orchestrator itself: orchestrator-standalone-app runs the built-in workers on binding consumeWorkerEvent, group orchestrator-builtin-workers, topics BUILTIN_WORKER_TOPICS (default http-calls,llm-calls,event-publications), starting at BUILTIN_WORKERS_START_OFFSET (default earliest). Optionally, http-worker-standalone-app (http-calls, overridable with HTTP_CALLS_TOPIC) and llm-worker-standalone-app (llm-calls, overridable with LLM_CALLS_TOPIC) serve theirs instead — take the topic out of BUILTIN_WORKER_TOPICS then, since two groups on one topic run every task twice, and set latest as the start offset for the rollout that moves a topic back onto the orchestrator, or its retained tasks run again. All of them report back on upstream like any worker.

Purpose: The invocations of AGENT steps: TaskExecutionRequested with task id run-agent@1, one variable __agent (the rendered request — sealed when the process declares sensitive variables) and an invocationId. A cancellation (TaskCancellationRequested) for an agent names the same invocationId.

Published by: Orchestrator

Consumed by: The IA agent runtime, which reports back on upstream with TaskStatusChanged carrying the invocationId, a growing stateSequence and — when it reconciles a held invocation — reviewResolved: true. A reply that does not name the step’s current invocation, or repeats an older sequence, is ignored, so a redelivered reply changes nothing; a completion without exactly the step’s outputVariable as valid JSON is refused as not retryable (parked on dead-letter).

Purpose: Where an event the engine can never process is parked — one that is itself defective, or a record that could not even be read. The payload is the original event (or, for an unreadable record, its original bytes), untouched; the failure travels in headers — x-dead-letter-source (where it came from), x-dead-letter-reason, x-dead-letter-message, plus x-dead-letter-process for a readable event (which also keeps its process partition key) or x-dead-letter-unreadable: true. Replaying a dead letter is republishing it on the destination its x-dead-letter-source names.

Published by: Orchestrator

Consumed by: Nobody in EventConductor — it is for operators and tooling.

Purpose: With workflow.projection.mode=remote, the outbox relay diverts process status changes here instead of projecting them locally, so the standalone projector maintains one fleet-wide process index. Unused in the default (embedded projection) mode. See Process index.

Published by: Orchestrator (every shard)

Consumed by: projector-standalone-app (topic overridable with PROJECTION_TOPIC)

Purpose: CloudEvents from other systems that start processes (event triggers) or resume onEvent waits. The topics are not fixed: they are whatever workflow.events.destinations.<name>.topic names, for every destination some definition’s trigger or wait listens to, recomputed as definitions change.

Published by: Other systems, PUBLISH_EVENT steps, or the webhook gateway.

Consumed by: The inbound domain-event consumer, which is in the io.mateu.workflow:inbound-events-kafka module (included in orchestrator-standalone-app). See Starting on and waiting for domain events.

Every EventConductor consumer commits a record’s offset only once it has been processed, or deliberately parked on dead-letter. A retryable failure (an optimistic-lock conflict between two consumers writing one process, a deadlock, the database or broker away) is redelivered with exponential back-off and no limit; the engine’s batch consumers name the record they failed on, so the records before it are committed and the rest are redelivered from it. Nothing is discarded. Settings: Consumer redelivery.

Up to and including 2.32.0, the engine’s batch consumers ran with Spring Kafka’s default batch error handler, which retried a failed batch nine times with no pause and then discarded it (FallbackBatchErrorHandler: Records discarded: …), and record consumers (the worker-kafka task consumer) gave up after the binder’s max-attempts (3). See Reliability.

ConsumerGroupTopic
Orchestrator outbox handlerorchestrator-outboxoutbox
Orchestrator upstream handlerorchestrator-upstreamupstream
Orchestrator inbound domain events (inbound-events-kafka)eventconductor-inbound-events (workflow.events.consumer.group)the destination topics
Projector (projector-standalone-app)process-index-projector (PROJECTION_GROUP)process-index
Orchestrator built-in workers (orchestrator-standalone-app)orchestrator-builtin-workersBUILTIN_WORKER_TOPICS (default http-calls, llm-calls, event-publications)
HTTP worker (http-worker-standalone-app, optional)http-workerhttp-calls
LLM worker (llm-worker-standalone-app, optional)llm-workerllm-calls
Workers (worker-standalone-app)worker-app-groupdownstream
Worker in an application of your own (worker-kafka)${spring.application.name:eventconductor-worker}downstream
Forms engine (forms-standalone-app)forms-engine-groupdownstream
Rule runtime (rule-standalone-app)rule-engine-groupdownstream

The orchestrator’s two groups are separate on purpose: one group whose members subscribe to different topics is assigned per topic by the default range assignor, and with mixed subscriptions it leaves partitions with no consumer at all.

Multiple orchestrator instances in the same group share the load: events are keyed by process, so the consumer group gives each process’s partition to exactly one instance (a single writer per process), fenced by an optimistic-locking version on the aggregates. The singleton background jobs (timeout scan, cron) still coordinate through a short-lived database advisory lock; see the configuration reference.