Skip to content

Observability

EventConductor exposes metrics and distributed tracing through Micrometer. Both are optional and host-activated: the engine libraries carry no hard observability dependency and behave as no-ops until the host application provides the relevant infrastructure (a MeterRegistry for metrics, a tracer for tracing). The orchestrator, forms and rule standalone apps wire both.

All engines (workflow, forms, rule catalog and rule runtime) publish engine-level metrics through Micrometer. They activate automatically when a MeterRegistry bean is present in the host application — no property is needed. The easiest way to get one (plus a Prometheus scrape endpoint) is:

<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-actuator</artifactId>
</dependency>
<dependency>
<groupId>io.micrometer</groupId>
<artifactId>micrometer-registry-prometheus</artifactId>
</dependency>
management.endpoints.web.exposure.include=health,prometheus

Metrics are then available at GET /actuator/prometheus. Without a MeterRegistry each engine falls back to a no-op implementation with zero overhead — Micrometer is an optional dependency of every engine module.

Instrumentation happens at the use-case layer, so the same metrics are emitted in all deployment modes (embedded + memory, embedded + jpa, kafka + jpa).

MetricTypeTagsDescription
eventconductor.process.startedCounterworkflowDefinitionIdProcesses started
eventconductor.process.completedCounterworkflowDefinitionIdProcesses completed successfully
eventconductor.process.erroredCounterworkflowDefinitionIdProcesses finished in error (retries exhausted)
eventconductor.process.cancelledCounterworkflowDefinitionIdProcesses cancelled
eventconductor.process.durationTimerworkflowDefinitionId, outcomeProcess duration from start to final status (COMPLETED | ERROR | CANCELLED)
eventconductor.step.executionsCounterworkflowDefinitionId, outcomeStep executions finished, by outcome (COMPLETED | ERROR | TIMEOUT)
eventconductor.step.durationTimerworkflowDefinitionId, outcomeStep execution duration, from dispatch to final status
eventconductor.step.retriesCounterworkflowDefinitionId, triggerRetries performed (auto = retry policy, manual = user-initiated)
eventconductor.step.compensationsCounterworkflowDefinitionIdCompensation steps triggered after retries were exhausted
eventconductor.compensations.failedCounterworkflowDefinitionIdSaga rollbacks that could not complete — a compensation step itself failed, leaving the process partially rolled back (COMPENSATION_FAILED). Alert on any non-zero rate: it is business state left inconsistent for a human to resolve
eventconductor.process.runningGauge—Processes currently in RUNNING status
eventconductor.outbox.pendingGauge—Outbox messages waiting to be relayed (only with workflow.persistence=jpa; the in-memory mode has no outbox). Rises during a broker outage and must return to zero
eventconductor.steps.stalledGauge—ACTION and RULE steps with no deadline that have been waiting on a worker longer than workflow.stalled-step-after-ms. The one gauge to alert on — see below
eventconductor.processes.stalledGauge—RUNNING processes with no step left to run and no deadline anywhere, older than workflow.stalled-process-after-ms (scanned every workflow.stalled-process-scan-interval-ms; the ids are logged at WARN). Usually guards that leave nothing eligible after the last completed step
eventconductor.incidents.openGaugeworkflowDefinitionIdProcesses stopped on an incident right now (status ERROR), waiting for an operator — the backlog the incidents view shows. Counted every workflow.incidents.gauge-interval-ms; a definition whose incidents were all resolved reads 0 rather than disappearing. Cluster-wide: alert on max() across replicas — see below
eventconductor.process.concurrent.writes.rejectedCounter—Writes refused by optimistic locking because another writer had the process. Events are keyed by process, but a process’s step events (outbox) and its workers’ replies (upstream) arrive on different topics and so on different consumers, so a burst of replies — a parallel forEach — produces some. Each one is redelivered (see below)
eventconductor.events.redeliveredCountertopic, causeKafka records redelivered by the engine consumers’ error handler after a retryable failure (cause is the innermost exception’s simple class name). A steady trickle under contention is the design working; a rate that keeps climbing on one topic is a failure that is not clearing and is holding its partition — the WARN log line has failed N times and is still being retried names the record
eventconductor.events.dead.letteredCountersourceEvents parked on the dead-letter destination as unprocessable
eventconductor.worker.tasks.redeliveredCountertopic, causeIn a process that consumes tasks over Kafka (a worker, the rule runtime, the forms engine): tasks redelivered because their reply could not be published. Climbs during a broker outage and stops when it ends
eventconductor.worker.tasks.skippedCountertopic, causeTask records skipped because they could not be read. Should stay at zero
eventconductor.tasks.escalationsCounterworkflowDefinitionId, actionHuman-task escalations that fired (reassign | notify | escalateTo) — see Human tasks: assignment and escalation
eventconductor.processes.purgedCounterworkflowDefinitionId, statusFinished processes the retention purge deleted, with everything that belonged to them, by definition and the status they ended in. Each pass that deleted anything, or had to keep a due tree still in use, also logs one INFO line
eventconductor.processes.operationsCounteroperation, workflowDefinitionIdManual operations performed on live processes: set-variables, complete-step, retry-from-step, retry-incident (one per retried step), migrate (migrations). Each is also a line in the process log with who, what and why — see Operating running processes
eventconductor.batch.operationsCountertype, outcomeBatch operations that ended, by type (CANCEL, RETRY, RESTART, PAUSE, RESUME) and outcome: completed, failed (the batch could not go on) or cancelled
eventconductor.batch.operation.itemsCountertype, outcomeProcesses batch operations acted on, by type and outcome: succeeded, skipped (not applicable, not authorized, gone) or failed. Its rate is the runner’s throughput, bounded by workflow.batch-operations.max-per-second
eventconductor.events.triggersGaugeorigin (definition, events_file)Event triggers that can start a process, by where they are declared (sampled at most once per workflow.metrics.gauge-ttl)
eventconductor.events.receivedCounterdestination, outcome (STARTED, DELIVERED, ERROR, DUPLICATE, and with a startPolicy DEBOUNCED, THROTTLED, DROPPED, SKIPPED, REPLACED)Domain events received, counted once per receiver (trigger or wait) they reached
eventconductor.flow.startsCounterworkflowDefinitionId, scope (trigger, start), outcomeStarts that met a startPolicy: started, debounced, throttled, dropped, skipped, replaced, and released when a waiting start leaves its wait
eventconductor.dispatch.rate_limitedCountertopic, outcome (deferred, released)Tasks that waited for a permit of their topic’s rate limit, and got one
eventconductor.process.continued_as_newCounterworkflowDefinitionIdRuns a CONTINUE_AS_NEW ended, each starting the next run of its chain
eventconductor.process.business_key_reusedCounterworkflowDefinitionId, policyStarts that took a business key over from an earlier run, as the definition’s businessKeyPolicy allowed
eventconductor.outbox.pickup.latencyTimer (histogram)—jpa only: time a message waited in the outbox between commit and being claimed by a relay
eventconductor.outbox.batch.sizeSummary—Messages claimed in one relay batch
eventconductor.outbox.batch.deliverTimer (histogram)—Time spent sending one relay batch
eventconductor.outbox.relay.draining / eventconductor.outbox.relay.waitingTimer—Time per relay cycle spent draining vs waiting for work; their ratio is the relay’s duty cycle (near 1: the relay is the ceiling)
eventconductor.outbox.relay.stalledCounter—Relay passes that claimed rows and settled none — anything above zero means committed messages are not reaching the broker

Alert on eventconductor.steps.stalled. Every other metric here counts something happening; this counts work that has stopped happening where nothing in the engine will notice. A step that declares no timeout has no deadline, and the timeout scan is an index range over the deadline — so if that step’s dispatch or its worker’s reply is lost, the process stops permanently and silently. Any sustained non-zero value is work that will never finish. Give those steps a timeout, or set workflow.default-step-timeout-ms. See Reliability.

Only ACTION and RULE steps count, because only they are owed an answer by a worker. A USER_TASK waits for a person, a WAIT_FOR_MESSAGE waits for a message that may never arrive, and a PROCESS waits for its child — none of them has a deadline, and none of them is stalled. Counting those made the gauge permanently non-zero in every deployment with human tasks.

The number is cluster-wide and reported by every pod: it counts rows in a shared table rather than one pod’s share of the work. Alert on max() across replicas, not sum(), which multiplies it by the replica count. It reads zero in workflow.persistence=memory, which keeps no such table — there it means “not measured”, not “nothing stalled”.

Watch an integration with eventconductor.incidents.open. eventconductor.process.errored counts failures as they happen; this gauge says how many are still waiting for someone, so it is what a dashboard graphs and what an alert can hold on until the backlog is cleared — retried from the incidents view or with a batch operation. Like the stalled gauges it counts shared rows, so every pod reports the same figure: max() across replicas, never sum().

For a process that turns a domain event into a call to another system — where the data must not be lost — these four alerts cover the ways it goes wrong:

AlertOnMeans
Unreadable eventseventconductor.events.dead.lettered increasesAn event could not even be read and was parked on the dead-letter destination
Open incidentsmax(eventconductor.incidents.open{workflowDefinitionId="…"}) > 0Processes stopped with their retries exhausted — retry them once the cause is fixed
Degraded destinationrate of eventconductor.step.retries{trigger="auto"} well above normalThe called system is failing; incidents follow when the retries run out
Silent producerrate of eventconductor.events.received well below normal for the destinationNothing is arriving. No engine sees an event that was never published: only this alert does
MeterTypeTagsMeaning
eventconductor.sync.invocationscounterworkflowDefinitionIdInvocations that started a process (retries of an existing one are not counted)
eventconductor.sync.answerscounterworkflowDefinitionId, outcomeCallers answered, by outcome — DEADLINE means a 202
eventconductor.sync.response.latencytimer (histogram)workflowDefinitionId, outcomeRequest arriving → answer ready
eventconductor.sync.deadline.expiredcounterworkflowDefinitionIdCallers whose deadline passed before the reply
eventconductor.sync.rejectedcounterworkflowDefinitionId, reasonInvocations refused before creating anything
eventconductor.sync.waitinggauge—Callers waiting on this node
eventconductor.process.updatescounterworkflowDefinitionId, update, statusProcess updates by outcome: COMPLETED, REFUSED, NOT_ACCEPTED, FAILED (counted where the process’s owner decided)
eventconductor.sync.wakeupscounterviaHow replies reached waiting callers: registration, local, notify, poll — a rising poll share means NOTIFY is not working
eventconductor.sync.inline.stepssummary—Outbox messages handled per inline drive (the fast path)
eventconductor.sync.inline.fallbackscounterreasonInline drives that stopped early or never started: saturated, budget, claim_lost, failure
MeterTypeTagsMeaning
eventconductor.workers.http.enqueuedcountertopicTasks queued for HTTP workers
eventconductor.workers.http.activatedcountertopicTasks handed to a worker (leased)
eventconductor.workers.http.redeliveredcountertopicHand-outs after a lease expired — a worker crashed, stalled or forgot to heartbeat
eventconductor.workers.http.repliescounteroutcome (completed, failed, injected, stale, gone, cancelled)Worker replies and what became of them; stale and gone are refused late replies
eventconductor.workers.http.cancelledcounteroutcome (removed, flagged, delivered)Engine cancellations: dropped from the queue, flagged on a leased task, heard by its worker’s heartbeat
eventconductor.workers.http.rejectedcounterreason (overloaded)Activations refused with 429
eventconductor.workers.http.waitinggauge—Activations long-polling on this node
eventconductor.workers.http.queuedgaugestate (available, leased)Tasks in the queue — cluster-wide with JPA, so alert on the max across nodes, never the sum

A task waiting for an HTTP worker keeps its step PENDING, so eventconductor.steps.stalled keeps counting the ones without a timeout.

MeterTypeTagsMeaning
eventconductor.messages.routedcounterroute (placement, subscription, broadcast)Messages routed, by how the target shard was found
eventconductor.messages.placement.lookup.failedcounter—A placement lookup failed; the message fell through to the next layer
eventconductor.messages.subscription.lookup.failedcounter—A subscription lookup failed; the message fell through to broadcast
eventconductor.messages.correlation.queriescounter—Correlation queries run on this shard
eventconductor.messages.correlation.queries.skippedcounter—Broadcast messages dropped without a query because no step on this shard could be waiting for them
MeterTypeTagsMeaning
eventconductor.webhooks.receivedcounterendpoint (the configured endpoint’s name; _unknown for every URL naming no configured endpoint), outcome (published, unauthorized, rejected, unknown, error)Every webhook delivery and what became of it
MeterTypeTagsMeaning
eventconductor.http.callstimertarget (connection or host), status (code, timeout, io)Every HTTP_CALL request and how long it took
MeterTypeTagsMeaning
eventconductor.llm.call (eventconductor_llm_call_seconds_*)timerworkflow, step, connection, model, outcome (success, error, invalid_output)Every LLM_CALL model call and how long it took
eventconductor.llm.tokens (eventconductor_llm_tokens_total)counterworkflow, step, connection, model, type (input, output)Tokens used

Spring AI’s own observations are recorded besides these: gen_ai_client_operation_seconds_* and gen_ai_client_token_usage_total, by provider and model. Prompts and answers are never logged.

Tracing: the request’s span eventconductor.sync.invoke carries a link to the process’s anchored trace, and the eventconductor.sync.reply span in the process trace links back to the caller’s — so each can be reached from the other without the process becoming a child of one request.

MetricTypeTagsDescription
eventconductor.forms.task.createdCounterformIdUser tasks created
eventconductor.forms.task.completedCounterformIdUser tasks completed
eventconductor.forms.task.cancelledCounterformIdUser tasks cancelled
eventconductor.forms.task.durationTimerformId, outcomeTask duration, from its creation to its completion or cancellation (not recorded for tasks created before 2.31, which carry no creation time)
eventconductor.tasks.eventsCounterevent, formIdHuman task lifecycle events: created, assigned, claimed, released, completed, cancelled, escalated
eventconductor.forms.importedCounter—Form definitions imported (e.g. from Git)
MetricTypeTagsDescription
eventconductor.rule.catalog.savedCounterruleIdRules created/updated in the catalog
eventconductor.rule.catalog.deletedCounterruleIdRules deleted
eventconductor.rule.catalog.importedCounter—Rules imported (e.g. from Git)
eventconductor.rule.catalog.servedCounterruleId, sourceRules served to the runtime (e.g. source=grpc)
MetricTypeTagsDescription
eventconductor.rule.evaluation.countCounterruleId, ruleType, outcomeRule evaluations, by outcome (matched | nomatch | error)
eventconductor.rule.evaluation.durationTimerruleId, ruleTypeEvaluation duration
eventconductor.rule.evaluation.cacheCounterruleId, resultRule-source cache lookups (result=hit | miss)

The forms, rule-catalog and rule-runtime metrics follow the same activation rules as the workflow metrics above (a MeterRegistry bean enables them; otherwise a zero-overhead no-op is used). The rule runtime also works as a plain, non-Spring library — its metrics port then defaults to the no-op.

  • With the Prometheus registry, names are exported in Prometheus form: eventconductor.process.started becomes eventconductor_process_started_total, timers become _seconds_count / _seconds_sum / _seconds_max families.
  • Each step retry that fails again increments eventconductor.step.executions{outcome="ERROR"}, so the counter reflects attempts, not distinct steps.
  • Counters are per-node and reset on restart, as usual with Prometheus counters — use rate()/increase() over them.
  • To customize (percentile histograms, common tags, renaming), use standard Micrometer MeterFilter beans, or replace the implementation entirely by defining your own metrics bean (e.g. WorkflowMetrics).

The orchestrator, forms and rule standalone apps ship distributed tracing over OpenTelemetry (OTLP). It is disabled by default (sampling 0.0) so there is no overhead until you opt in — set the sampling probability and point it at your collector:

# fraction of traces to sample, 0.0 (off) .. 1.0 (all)
management.tracing.sampling.probability=${TRACING_SAMPLING:0.0}
# OTLP HTTP traces endpoint (OTel Collector / Tempo / Jaeger)
management.opentelemetry.tracing.export.otlp.endpoint=${OTLP_TRACING_ENDPOINT:http://localhost:4318/v1/traces}
Terminal window
export TRACING_SAMPLING=1.0
export OTLP_TRACING_ENDPOINT=http://localhost:4318/v1/traces

With tracing on, HTTP requests, Kafka (Spring Cloud Stream) publish/consume and JDBC calls are auto-instrumented. The standalone apps bundle micrometer-tracing-bridge-otel and opentelemetry-exporter-otlp; enabling tracing needs only the two properties above.

Auto-instrumentation alone does not give you one trace per process, because this engine’s asynchronous boundary is not a network call — it is a database row. A domain event is written to the outbox inside the transaction that produced it, and a relay thread publishes it some time afterwards. The instrumentation sees a write in one trace and, later, a Kafka send belonging to nothing, so the consumer on the other side starts a fresh trace: a trace per hop rather than a trace per process.

The engine therefore carries the context itself. Each outbox row stores the W3C traceparent of whatever produced the event (column trace_parent, null when nothing was being traced), and the relay publishes as a continuation of that trace. A process started by an HTTP request and finished by a worker three hops later reads as one trace.

A trace of a workflow should read the way the workflow ran: this step, then that one, then those two at the same time. That picture cannot be produced by instrumenting the code as it executes. A span is started and ended by one object on one thread, and a workflow step obliges on neither count — it starts in the transaction that dispatched it and ends wherever the worker’s reply lands, minutes or days later, across a broker and possibly a pod restart. Wrapping the running code can only ever describe the hop it is inside, which is why a trace made of dispatches and relay passes reads as a scatter of two-millisecond fragments.

So the engine does not reconstruct the process from its hops. It writes the trace out from what it durably recorded. A step execution row already carries startedAt and finishedAt; when a process reaches a terminal status the engine emits a span for the process and one for each step that ran, each covering the time that step actually took. Steps that ran one after another come out as consecutive siblings, steps that ran together as overlapping ones, and the shape falls out of the timestamps instead of having to be inferred from causality.

Order fulfilment ───────────────────────────────────────────── (the process)
├── Reserve stock ────────
├── Charge card ─────────────
├── Print label ────────── (these two ran in parallel)
└── Notify customer ──────

The process span carries eventconductor.process.id, .businessKey, .status and the workflow id and version; each step span carries eventconductor.step.id, .executionId, .status, .attempts, .type and, for an ACTION, .topic.

Which trace a process belongs to is derived from its id, not stored and not propagated — the anchor is a hash of the process id, so every pod computes the same trace id for the same process with nothing to carry and nothing to lose. That is what makes this survive the things that defeat ordinary context propagation: a rebalance, a restart, a redelivery from the dead-letter queue, a TIMER step that waits a week. A traceparent in a message header is gone the moment the message is replayed; a derived anchor was never held.

A process started from inside somebody else’s trace — a booking a CRS sent, whose trace already holds the CRS, the connector and the broker hop — can join that trace instead of starting its own, so the booking reads end to end in one waterfall: caller → engine → every worker → back.

The caller hands over its W3C context in either of two ways:

  • Kafka record headers on the ProcessCreationRequested record: traceparent, and optionally tracestate and baggage, named as the W3C specs name them, values as UTF-8 bytes.
  • The payload: ProcessCreationRequested.traceContext — {"traceparent": …, "tracestate": …, "baggage": …} — for a producer whose outbox relay cannot set headers. Optional and left off the wire when null; an engine that predates it ignores it. The payload wins when both are present.

The engine keeps the context with the process (table process_trace_context, one row per process that was given one) and uses the caller’s span as the process’s anchor in place of the derived one. Everything described below still holds — the live spans, the recorded process and step spans, one trace for the whole life of the process on every pod — it simply sits under the caller’s span, and it is stored rather than derived, so it survives restarts and redeliveries the same way. The caller’s sampled flag decides whether the process is traced.

Every record the engine publishes for the process carries the trace on: tasks (TaskExecutionRequested, straight from dispatch-step), everything relayed through the outbox, PUBLISH_EVENT events, child-process creations — each with a traceparent of the engine span that sent it, plus the caller’s tracestate and baggage passed through. Headers are written as raw bytes so any W3C propagator on the other side reads them. A worker built on worker-kafka runs the handler in a CONSUMER span eventconductor.task <stepId> under that traceparent (when it has a tracing bridge), and exposes the context to the handler as TaskContext.traceContext(). A worker with its own Spring Cloud Stream consumer joins by turning on the binder’s observation (spring.cloud.stream.kafka.binder.enable-observation=true) with a tracing bridge on the classpath. Replies need nothing: the engine continues the process’s trace from what it stored.

Only while tracing is on: with no Tracer, nothing is stored, nothing is looked up, and no header is added. workflow.tracing.join-caller-trace=false keeps every process on its derived trace even when a context arrives. Synchronous invocations keep their own design — the request and the process in separate traces, linked both ways (see Synchronous invocation).

management.tracing.sampling.probability governs process traces exactly as it governs everything else. The decision is taken from the derived trace id, by the same arithmetic the OpenTelemetry SDK’s own TraceIdRatioBased sampler uses — so the rate comes out as configured, and a process is either traced or not traced.

That last part is the point, and it matters more than the rate. The decision belongs to the process, not to the span: every pod that touches it reaches the same answer, over its whole life, across restarts and redeliveries. Sampling per span would at 10% have given you a tenth of the dispatches of a tenth of the processes — scattered fragments, and never a whole process to read. Here a traced process is traced end to end, and an untraced one costs nothing at all.

Steps that never ran — a branch abandoned when another reached END — draw nothing, so the waterfall shows what happened rather than what did not.

The live spans are still emitted, and they now join the trace of the process they are working on instead of each starting one of their own. They are what makes a process visible in the trace store while it is still running, before its own span exists:

SpanWhat it covers
eventconductor.create-processCreating a process that joined a caller’s trace
eventconductor.step-overAdvancing a process: deciding what may run now and dispatching it; tagged with the workflow id, business key, the process’s status after it and the steps it moved (eventconductor.steps.moved)
eventconductor.dispatch-stepHanding one step to a worker; tagged with the step and workflow ids
eventconductor.correlate-messageMatching an arriving message to the steps waiting for it
outbox relayPublishing one outbox row, as a continuation of the trace that produced it

correlate-message is anchored only after the match: which process an arriving message belongs to is precisely what the lookup answers, so the lookup itself runs untraced and each match is traced inside the process it turned out to belong to.

Both the propagation and the spans go through a port with a no-op default (WorkflowTracing), wired to Micrometer only when the host application brings a Tracer — the same shape as metrics. The engine libraries still run with zero observability dependencies, and nothing here can fail a workflow: a tracing error is logged at debug and the work carries on.

Metrics are exposed for Prometheus scraping by default. A deployment that already runs an OpenTelemetry collector can push them instead, over the same protocol as the traces:

management.otlp.metrics.export.enabled=${OTLP_METRICS_ENABLED:false}
management.otlp.metrics.export.url=${OTLP_METRICS_ENDPOINT:http://localhost:4318/v1/metrics}
management.otlp.metrics.export.step=${OTLP_METRICS_INTERVAL:60s}

Off by default, and independent of the Prometheus endpoint — turning it on does not turn scraping off.