Observability

Cadence exposes three observability layers: structured JSON logs (stdout), HTTP health and pool endpoints for probes, and OpenTelemetry settings adjustable via admin APIs without restarting the process.

Summary for stakeholders

• Operational spend — Logs and traces have marginal cost; aggressive tracing and high-cardinality LLM spans can raise collector and storage bills.
• Progressive rollout — stdout logs and /health first; add OTLP with sampling before full production load.

Business analysis

• SLIs — Pair log-based error rate with trace latency for chat paths when defining SLOs.
• Runbooks — Reference otel.* keys in global_settings and log pipeline routing in incident checklists.

Architecture and integration

Implementation references: cadence.domain.telemetry (otel.* keys), cadence.api.health, cadence.main logging setup.

After this section

Solution architects

• Place OTLP collectors and log pipelines in the same trust zone as the API per data classification.

Developers

• Adjust otel.* keys via admin APIs and cadence.api.telemetry metadata without guessing env-only flags for traces.

Observability layers

Layer	How to enable	What it provides
Process logs	logging.basicConfig in cadence.main (text by default)	Plain stdout logs; ship with your platform agent or sidecar
Health endpoints	Always available	Liveness (GET /health) and deeper admin health
Distributed traces	otel.enabled in global_settings (via /api/admin/telemetry or DB)	Spans across internal steps; export via OTLP when configured

Start with stdout logs + /health in staging, then enable OTLP with a low sample rate before raising to production traffic. Jumping straight to always_on tracing on busy clusters can overwhelm your collector.

Structured logging

Default API process logging is configured in cadence.main (logging.basicConfig, text format). The otel.logs_enabled setting (stored with other otel.* keys) relates to OpenTelemetry log export, not a CADENCE_LOG_FORMAT environment variable.

Stream errors on SSE chat are logged and emitted on the wire from cadence/api/chat/router.py:

cadence/api/chat/router.py

        except Exception as e:
            logger.error("Stream error: %s", e, exc_info=True)
            ...
            yield f"event: error\ndata: {json.dumps(err_data)}\n\n"

OpenTelemetry configuration

OTel settings are persisted as otel.* rows in global_settings. HTTP access: GET / PATCH /api/admin/telemetry (cadence/api/telemetry/router.py). Key metadata for validation lives in cadence/domain/telemetry/service.py:

cadence/domain/telemetry/service.py

_KEY_META: dict[str, tuple[str, str]] = {
    "otel.enabled": ("boolean", "Enable OpenTelemetry instrumentation"),
    "otel.service_name": ("string", "OTel service name reported to the collector"),
    "otel.service_version": (
        "string",
        "OTel service version reported to the collector",
    ),
    "otel.environment": ("string", "Deployment environment label"),
    "otel.exporter": (
        "string",
        "Exporter backend: console | otlp_grpc | otlp_http | none",
    ),
    # ... remaining otel.* keys — see source for the full map
}

Instrumentation flags increase cardinality

Enable otel.instrument_langchain and otel.instrument_openai_agents selectively. High-cardinality spans in production can increase collector costs significantly.

Debugging a slow chat turn

1. Note approximate time, org, orchestrator, and message id.
2. Search JSON logs for errors or stream warnings around that window.
3. If traces are enabled, open your collector and find the trace id — inspect model latency vs tool latency vs queue wait.
4. Check pool stats (GET /api/admin/pool/stats) — long queue times often indicate instances were evicted from the demand pool or not loaded.
5. If traces are missing entirely, verify otel.enabled and exporter settings via admin APIs.

Troubleshooting

Symptom	Cause	Fix
No traces	otel.enabled is false, or exporter/network misconfigured	Set otel.enabled=true; check otel.endpoint and network egress
High cardinality costs	SDK instrumentation flags enabled in production	Disable otel.instrument_langchain / otel.instrument_openai_agents
Plain text logs only	Default logging config in cadence.main	Add a log shipper or enable OTel log export via otel.logs_enabled

Next steps

Monitoring Health endpoints, pool stats, and log shipping setup.

Configuration Environment variables and infrastructure bootstrap.