Event system

Cadence uses RabbitMQ as an optional event bus for coordinating state across multiple server nodes. When RabbitMQ is available, orchestrator load/unload events, settings changes, and plugin uploads are broadcast to all connected nodes. When RabbitMQ is unavailable, the platform degrades gracefully — events are skipped and single-node operation continues.

Summary for stakeholders

• Multi-node coordination — Without a message bus, each server node would need polling or direct HTTP calls to stay in sync. RabbitMQ provides publish-subscribe coordination with zero polling.
• Graceful degradation — If RabbitMQ is unreachable at startup, the platform logs a warning and continues without event propagation. Single-node deployments work without RabbitMQ.
• Per-node queues — Each server node gets its own durable queue, ensuring every node receives every event.

Business analysis

• Settings propagation — When a sys_admin changes a global setting, all hot-tier orchestrator instances across all orgs are reloaded on every node. This ensures configuration changes take effect within seconds.
• Plugin distribution — When a plugin is uploaded, the event triggers local filesystem caching on all nodes, so the next orchestrator load finds the plugin ZIP already available.

Architecture and integration

Exchange and queue topology

flowchart LR
    P["Publisher<br/>(any node)"] --> EX["cadence.orchestrators<br/>(topic exchange, durable)"]
    EX --> Q1["cadence.orchestrators.node-1<br/>(durable queue)"]
    EX --> Q2["cadence.orchestrators.node-2<br/>(durable queue)"]
    EX --> QN["cadence.orchestrators.node-N<br/>(durable queue)"]

    Q1 --> C1["Consumer (node 1)"]
    Q2 --> C2["Consumer (node 2)"]
    QN --> CN["Consumer (node N)"]

• Exchange: cadence.orchestrators — topic exchange, durable
• Queue naming: cadence.orchestrators.{hostname} — one per node, durable
• Bindings: Each queue binds to three routing key patterns:
  • orchestrator.* — lifecycle events
  • settings.* — settings change events
  • plugin.* — plugin distribution events
• Prefetch: 10 messages per consumer

Routing keys

| Routing key | Category | Published when | | ------------------------------ | ------------- | ------------------------------------------------- | | orchestrator.load | Lifecycle | Instance loaded into pool (startup or API call) | | orchestrator.unload | Lifecycle | Instance removed from pool (deactivation or API) | | orchestrator.reload | Lifecycle | Instance config changed, pool needs refresh | | settings.global_changed | Settings | Global setting updated via admin API | | settings.org_changed | Settings | Org setting updated via org admin API | | settings.telemetry_changed | Settings | Telemetry config updated via admin API | | plugin.uploaded | Plugin | Plugin ZIP uploaded (system or org) |

Source: cadence/events/*/routing.py.

Event details

orchestrator.load

Published when an orchestrator instance is loaded into the pool. The consumer ensures plugins are local, builds resolved config, and creates or reloads the instance in the local pool.

Payload:

{
  "instance_id": "01936a8f-...",
  "org_id": "01936a90-...",
  "tier": "hot"
}

Consumer behavior (cadence/events/orchestrator/consumer.py:handle_load):

1. Fetch instance from DB
2. Ensure active plugins are available locally (ensure_plugins_local)
3. Build resolved instance config
4. If instance already in pool → reload; otherwise → create
5. Set config hash for deduplication

orchestrator.unload

Published when an instance is deactivated or explicitly unloaded.

Payload:

{
  "instance_id": "01936a8f-..."
}

Consumer behavior (cadence/events/orchestrator/consumer.py:handle_unload):

1. Check if instance is in local pool
2. If yes → remove and cleanup
3. If no → skip (already unloaded on this node)

orchestrator.reload

Published when an instance’s config changes (e.g., whoami update, plugin settings change). Includes a config_hash for deduplication.

Payload:

{
  "instance_id": "01936a8f-...",
  "org_id": "01936a90-...",
  "config_hash": "abc123..."
}

Consumer behavior (cadence/events/orchestrator/consumer.py:handle_reload):

1. Compare config_hash with local pool hash — skip if unchanged (dedup)
2. Check if instance is in local pool — skip if not loaded
3. Fetch fresh instance from DB
4. Reload instance in pool with new config
5. Update local config hash

settings.global_changed

Published when a sys_admin updates a global setting. Triggers a full hot-tier reload across all orgs.

Payload: {} (empty — the consumer re-reads from DB)

Consumer behavior (cadence/events/settings/consumer.py:handle_global_settings_changed):

1. Reload all hot-tier instances in the local pool
2. Update token TTL settings if auth-related keys changed

settings.org_changed

Published when an org admin updates an org setting. Only affects instances in that org.

Payload:

{
  "org_id": "01936a90-..."
}

Consumer behavior (cadence/events/settings/consumer.py:handle_org_settings_changed):

1. Reload hot-tier instances filtered to the affected org

settings.telemetry_changed

Published when telemetry configuration is updated. Triggers OTel provider reload without process restart.

Payload: {} (empty — the consumer re-reads from DB)

Consumer behavior: Calls reload_telemetry_callback which re-loads OTel settings from global_settings, shuts down old providers, and initializes new ones.

plugin.uploaded

Published when a plugin ZIP is uploaded (system or org). Ensures the plugin is cached locally on all nodes.

Payload:

{
  "pid": "com.example.my-plugin",
  "version": "1.2.0",
  "source": "org",
  "org_id": "01936a90-..."
}

Consumer behavior (cadence/events/plugin/consumer.py:handle_plugin_uploaded):

1. Validate source is system or org
2. Build plugin reference string
3. Call ensure_plugins_local to download/cache the plugin ZIP

Trace context propagation

Both publisher and consumer inject/extract OpenTelemetry trace context via message headers using TraceContextTextMapPropagator. This allows distributed traces to span from the API request that triggered the event through the consumer that processes it.

cadence/events/broker.py — publish

TraceContextTextMapPropagator().inject(headers)
await self._exchange.publish(Message(body=body, headers=headers), routing_key=routing_key)

cadence/events/broker.py — consume

TraceContextTextMapPropagator().extract(dict(message.headers or {}))

Demand-pool callback

When an orchestrator is loaded on-demand (cold load from DB during a chat request), the pool fires on_demand_loaded_callback. When RabbitMQ is connected, this callback publishes an orchestrator.load event so other nodes also load the instance.

Startup sequence

During the lifespan startup (cadence/core/lifespan.py):

1. RabbitMQ client connects (if CADENCE_RABBITMQ_URL is set)
2. OrchestratorEventPublisher is created and attached to app.state
3. OrchestratorEventConsumer is created with references to pool, repos, and services
4. Consumer binds to orchestrator.*, settings.*, plugin.* and starts consuming
5. Hot-tier instances are loaded (these generate orchestrator.load events to other nodes)

Graceful degradation

If RabbitMQ connection fails at startup:

cadence/core/lifespan.py

except Exception as e:
    logger.warning(f"RabbitMQ not available — events disabled: {e}")
    app.state.rabbitmq_client = None
    app.state.event_publisher = None
    app.state.event_consumer = None

The platform continues to operate as a single node. Load/unload API calls still work locally but are not broadcast to other nodes. Handlers that depend on the broker check event_publisher is not None before publishing (e.g. TelemetryService.update_config).

Acknowledgment model

The consumer uses aio-pika’s Message.process(requeue=False) context manager inside _dispatch (broker.py:207). A handler exception is caught and logged inside the dispatch body (broker.py:218-269), so the message is acked even on handler failure — failures are not requeued or retried by RabbitMQ. Recovery is expected to come from a subsequent event (e.g. a fresh orchestrator.reload after a config fix).

Troubleshooting

| Symptom | Cause | Fix | | ------------------------------------------ | ----------------------------------------------- | ------------------------------------------------------------- | | Events not propagating between nodes | RabbitMQ unreachable or wrong URL | Check CADENCE_RABBITMQ_URL; verify RabbitMQ is running | | Stale instances after settings change | Consumer not running or queue not bound | Check logs for consumer start message; verify queue bindings | | Duplicate reloads on config change | Config hash mismatch or missing hash | Verify config_hash is set on instance; check dedup logic | | Plugin not found after upload on other node | plugin.uploaded event not received | Check RabbitMQ connectivity; verify plugin routing key binding | | High memory after many reloads | Old orchestrator instances not cleaned up | Check cleanup() is called on old instances during reload |

Limitations

• No event persistence — Events are not stored in a database. If a node is down when an event is published, it misses that event. The durable queue ensures delivery to connected nodes only.
• No retry on consumer error — Failed messages are acknowledged with requeue=False. The error is logged but the event is not retried.
• Optional dependency — RabbitMQ is not required for single-node deployments. The platform works without it.

Next steps

Hot-reload and AI App pool Pool tiers, async load/unload semantics, and pool statistics.

Settings cascade Four-tier configuration resolution and settings propagation.

How the platform works Startup sequence, middleware stack, and application wiring.