Observability

Process logs, health endpoints, and OpenTelemetry configuration.

Intended audience: Stakeholders, Business analysts, Solution architects, Developers, Testers

Learning outcomes by role

Stakeholders

Explain stdout logs, health endpoints, and tracing as operational visibility investments.

Business analysts

Tie observability signals to SLIs in runbooks and incident templates.

Solution architects

Design OTLP export, sampling, and collector placement for production.

Developers

Tune process logging in `cadence.main`, OTel keys in `global_settings`, and `/api/admin/telemetry` APIs.

Testers

Validate health checks, log fields, and trace spans in staging environments.

Cadence exposes three observability layers: structured JSON logs (stdout), HTTP health and pool endpoints for probes, and OpenTelemetry settings adjustable via admin APIs without restarting the process.

Summary for stakeholders

Operational spend — Logs and traces have marginal cost; aggressive tracing and high-cardinality LLM spans can raise collector and storage bills.
Progressive rollout — stdout logs and /health first; add OTLP with sampling before full production load.

Business analysis

SLIs — Pair log-based error rate with trace latency for chat paths when defining SLOs.
Runbooks — Reference otel.* keys in global_settings and log pipeline routing in incident checklists.

Architecture and integration

Implementation references: cadence.domain.telemetry (otel.* keys), cadence.api.health, cadence.main logging setup.

Observability layers

| Layer | How to enable | What it provides | | ------------------ | ---------------------------------------------------------------------- | ------------------------------------------------------------ | | Process logs | logging.basicConfig in cadence.main (text by default) | Plain stdout logs; ship with your platform agent or sidecar | | Health endpoints | Always available | Liveness (GET /health) and deeper admin health | | Distributed traces | otel.enabled in global_settings (via /api/admin/telemetry or DB) | Spans across internal steps; export via OTLP when configured |

Start with stdout logs + /health in staging, then enable OTLP with a low sample rate before raising to production traffic. Jumping straight to always_on tracing on busy clusters can overwhelm your collector.

Structured logging

Default API process logging is configured in cadence.main (logging.basicConfig, text format). The otel.logs_enabled setting (stored with other otel.* keys) relates to OpenTelemetry log export, not a CADENCE_LOG_FORMAT environment variable.

Stream errors on SSE chat are logged and emitted on the wire from cadence/api/chat/router.py:

        except Exception as e:
            logger.error("Stream error: %s", e, exc_info=True)
            ...
            yield f"event: error\ndata: {json.dumps(err_data)}\n\n"

OpenTelemetry configuration

OTel settings are persisted as otel.* rows in global_settings. HTTP access: GET / PUT /api/admin/telemetry (cadence/api/telemetry/router.py). The update verb is PUT (full replacement of the OTel block), not PATCH. Key metadata for validation lives in cadence/domain/telemetry/service.py:

| Key | Type | Description | | -------------------------------- | ------- | ---------------------------------------------------- | | otel.enabled | boolean | Enable OpenTelemetry instrumentation | | otel.service_name | string | OTel service name reported to the collector | | otel.service_version | string | OTel service version reported to the collector | | otel.environment | string | Deployment environment label | | otel.exporter | string | Exporter backend: console | otlp_grpc | otlp_http | none | | otel.endpoint | string | OTLP collector endpoint URL | | otel.endpoint_insecure | boolean | Skip TLS verification for OTLP endpoint | | otel.headers | string | Comma-separated key=value auth headers for OTLP | | otel.traces_enabled | boolean | Enable distributed tracing signal | | otel.metrics_enabled | boolean | Enable metrics signal | | otel.logs_enabled | boolean | Enable structured logs signal | | otel.trace_sampler | string | Sampler: always_on | always_off | traceid_ratio | | otel.trace_sample_rate | float | Sampling ratio (0.0–1.0) | | otel.metrics_export_interval_ms| integer | Metrics export interval in milliseconds | | otel.instrument_langchain | boolean | Auto-instrument LangChain / LangGraph pipelines | | otel.instrument_openai_agents | boolean | Auto-instrument OpenAI Agents SDK | | otel.propagation | string | Comma-separated propagator names |

All keys are stored with overridable=false and category="telemetry" in global_settings. The Pydantic model that mirrors these keys is OtelSettings in cadence/infra/telemetry/settings.py. Field defaults from that model (the values used by the API if no global setting is present): otel_enabled=False, otel_service_name="cadence", otel_service_version="2.0.5", otel_environment="production", otel_exporter="otlp_grpc", otel_endpoint="http://localhost:4317", otel_endpoint_insecure=True, otel_headers="", otel_traces_enabled=True, otel_metrics_enabled=True, otel_logs_enabled=True, otel_trace_sampler="always_on", otel_trace_sample_rate=1.0, otel_metrics_export_interval_ms=60000, otel_instrument_langchain=True, otel_instrument_openai_agents=True, otel_propagation="tracecontext,baggage".

Debugging a slow chat turn

Note approximate time, org, orchestrator, and message id.
Search JSON logs for errors or stream warnings around that window.
If traces are enabled, open your collector and find the trace id — inspect model latency vs tool latency vs queue wait.
Check pool stats (GET /api/admin/pool/stats) — long queue times often indicate instances were evicted from the demand pool or not loaded.
If traces are missing entirely, verify otel.enabled and exporter settings via admin APIs.

Troubleshooting

| Symptom | Cause | Fix | | ---------------------- | ---------------------------------------------------------- | --------------------------------------------------------------------- | | No traces | otel.enabled is false, or exporter/network misconfigured | Set otel.enabled=true; check otel.endpoint and network egress | | High cardinality costs | SDK instrumentation flags enabled in production | Disable otel.instrument_langchain / otel.instrument_openai_agents | | Plain text logs only | Default logging config in cadence.main | Add a log shipper or enable OTel log export via otel.logs_enabled |

Next steps

Monitoring Health endpoints, pool stats, and log shipping setup.

Configuration Environment variables and infrastructure bootstrap.