How the platform works

FastAPI app factory, middleware order, lifespan startup, and API router registration.

Intended audience: Stakeholders, Business analysts, Solution architects, Developers, Testers

Redis, Postgres, and optional RabbitMQ and S3 behavior is defined at startup and in middleware; see also Configuration and Monitoring guides.

Learning outcomes by role

Stakeholders

Explain how shared infrastructure (databases, cache, optional broker) underpins uptime and cost trade-offs for a multi-tenant deployment.
Relate middleware ordering to customer-visible failures (auth, rate limits, sessions) when prioritizing incidents.

Business analysts

Document prerequisites and dependencies for operational runbooks (which subsystems must be healthy before API calls succeed).
Align acceptance language with observable HTTP outcomes (401, 403, rate limiting) tied to documented middleware behavior.

Solution architects

Map inbound middleware order, lifespan initialization, and router registration to integration boundaries and NFRs (security headers, CORS, scaling).
Justify where Redis, Postgres, RabbitMQ, and S3 sit in deployment diagrams relative to the FastAPI process.

Developers

Trace configure_* registration in cadence.main against the inbound execution order when debugging request.state and errors.
Locate register_api_routers and lifespan steps when extending startup or adding routers.

Testers

Predict failure modes from middleware order (session before rate limit, public paths, missing Redis) and design negative tests accordingly.
Correlate 401 versus 403 scenarios with authentication versus tenant and permission layers after this page.

Every request Cadence receives passes through the same fixed sequence: an outermost error handler catches anything that escapes, authentication validates the credential, the tenant layer resolves who the caller is acting for, rate limiting enforces quotas, and finally the route handler does the actual work. Understanding that sequence tells you exactly where to look when something goes wrong.

Summary for stakeholders

A single Cadence API process serves all organizations. Isolation is enforced by the middleware layers — not by separate deployments — so the health of the shared infrastructure determines what every tenant experiences.

Single deploy, many tenants — Isolation is enforced after authentication in tenant and permission layers (see Multi-tenancy).
Operational leverage — Health and behavior depend on PostgreSQL, Redis (sessions and rate limiting), and optionally RabbitMQ for orchestrator messaging and S3/MinIO for plugin storage. Gaps in those dependencies surface as structured errors or degraded features — not silent data mixing.
Risk posture — Middleware order is fixed in code. Changes to infrastructure or config alter when failures appear (for example, before or after rate limiting), which matters when triaging incidents.

Business analysis

When a request fails, the failure comes from a specific layer. A missing or invalid credential is a 401 from the authentication middleware; a valid credential without the right permission is a 403 from route-level authorization; too many requests is a rate-limit response from the Redis-backed sliding window. Knowing which layer produced which status code is what makes runbooks accurate.

Single source for “what runs first” — Product and operations can reference this page instead of ad-hoc diagrams when writing prerequisites (“Redis required for interactive JWT sessions and rate limits”).
Observable outcomes — Missing or invalid credentials (401), known user but disallowed org or role (403), and rate limiting (429) are separate acceptance themes corresponding to different middleware layers.

Architecture and integration

Request flow

flowchart LR
  Client[Client] --> MW[Middleware stack]
  MW --> R[Route handler]
  R --> MW
  MW --> Client

A request enters the outermost middleware first, passes through each layer inward to the route handler, and the response travels back out through the same layers in reverse. Each middleware can short-circuit the chain by returning a response before the request reaches deeper layers.

Inbound stack

Inbound order (top first). Registration order in cadence.main is the reverse; see Middleware chain.

Middleware chain

Starlette runs middleware in reverse registration order: the last middleware added wraps everything else and runs first on each incoming request. The application registers layers from CORS (innermost, closest to route handlers) through security headers, rate limiting, tenant context, authentication, to error handling (outermost). That means the inbound order (first to see the request) is:

| Layer | Class | Why it’s here | | -------------------- | --------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Error handling | ErrorHandlerMiddleware | Outermost so it catches exceptions from every other layer. Pure ASGI middleware (not BaseHTTPMiddleware); sets request.state.request_id (preserves incoming X-Request-ID or mints a UUID7), appends X-Request-ID to all responses, and serializes unhandled exceptions to JSON. | | Authentication | AuthenticationMiddleware | Runs before tenant resolution. Checks a public-path allowlist (PUBLIC_PATHS in cadence/core/constants/app.py plus the hard-coded /); validates Authorization: Bearer (JWT) or X-API-KEY; sets request.state.api_key_row when a key is used. Without this running first, the tenant layer would try to build a session from nothing. | | Tenant context | TenantContextMiddleware | Reads request.state.api_key_row first (API key path → synthetic ApiKeyTokenSession); otherwise decodes the JWT locally to extract jti and loads the session from Redis (session_store.get_session(jti)). The JWT secret is settings.secret_key; the third-party RSA key is not used here. | | Rate limiting | RateLimitMiddleware | Redis sliding-window (sorted set) per org/user/IP. Reads X-ORG-ID only for POST /api/chat/completion (so it can apply rate_chat_limit_rpm); for other authenticated paths it falls back to DEFAULT_RATE_LIMIT_MAX_REQUESTS (100 per 60s window). If Redis is unavailable, the middleware silently skips enforcement. | | Security headers | SecurityHeadersMiddleware | Adds X-Content-Type-Options: nosniff, X-Frame-Options: DENY, Referrer-Policy: strict-origin-when-cross-origin, Permissions-Policy: geolocation=(), microphone=(). HSTS (Strict-Transport-Security: max-age=63072000; includeSubDomains) is applied only when CADENCE_ENVIRONMENT=production. Defined inline in cadence/core/middleware_setup.py, not in the middleware/ package. | | CORS | CORSMiddleware | Innermost, closest to the route. Validates Origin against CADENCE_CORS_ORIGINS and handles preflight. Must be registered first so it becomes the innermost wrapper. |

Implementation notes

Application entry

cadence.main builds a FastAPI app, registers middleware, and mounts all routers. The lifespan=create_lifespan_handler(app_settings) argument means no request is accepted until the full startup sequence completes.

Lifespan: startup and shutdown

create_lifespan_handler in cadence.core.lifespan runs the following sequence before the server accepts any traffic:

Telemetry placeholder — A TelemetryState(enabled=False) is stashed on app.state.telemetry_state so later reloads have a slot to mutate.
Config validation — settings.validate_production_config(). In production, any insecure setting raises immediately and the process exits rather than starting with known vulnerabilities.
PostgreSQL + Redis — initialize_database_clients connects both pools and attaches clients to app.state.postgres_client / app.state.redis_client. Everything after this point can read from the database.
Repositories and services — _create_repositories builds every PostgreSQL repository, a RedisCache (namespace "cadence"), and SettingsService. It bootstraps global_settings keys that are missing (otel.exporter, access_token_ttl_seconds, refresh_token_ttl_seconds, OAuth *.enabled flags and credential keys — all written with overridable=False).
RBAC — BuiltInRBACProvider is wired with the role and user-role repositories plus the Redis client (300-second permission cache).
Telemetry bring-up — setup_telemetry and setup_log_bridge consume the DB-loaded OTel config; SQLAlchemy, Redis, HTTPX, and asyncpg instrumentors are activated.
Application services — TenantService, AuthService, OAuthService, ConversationService, PluginService, CentralPointService, ApiKeyService, TelemetryService, RbacService, and ConsentService are constructed and attached to app.state.
Orchestrator pool — OrchestratorFactory (with LLMModelFactory and plugin repo) feeds OrchestratorPool. The pool is also wired into SettingsService.
RabbitMQ (optional) — If RabbitMQClient.connect() succeeds, an OrchestratorEventPublisher and OrchestratorEventConsumer are attached and the consumer is started (binding queues to orchestrator.*, settings.*, plugin.*). On failure, all three are set to None, a warning is logged, and the platform continues without event propagation.
Plugin catalog sync — ensure_all_catalog_plugins_local pulls catalog plugin ZIPs from S3 to local cache when object storage is configured.
Hot-tier pip dependency pre-install — preinstall_hot_tier_plugin_dependencies walks the active_plugins of every hot-tier instance, reads the declared dependencies from each plugin’s metadata, and runs pip install --target so plugin imports resolve at chat time without per-request install cost.
Hot tier load — load_hot_tier_instances builds resolved configs and calls orchestrator_pool.create_instance(..., source="startup", tier="hot") for every hot/active instance.
Eviction loop — orchestrator_pool.start_eviction_loop() is called; the demand pool runs a background sweep that evicts idle instances after their TTL.
LLM instrumentors — activate_llm_instrumentors collects the set of framework_type values across hot instances and activates framework-specific LLM instrumentors.
OrchestratorService — OrchestratorService(pool, conversation_service, settings_service, stats_service, central_point_service, instance_repo) is attached to app.state.orchestrator_service.

Shutdown (after yield): event consumer stops, RabbitMQ disconnects, the eviction loop stops, the orchestrator pool runs cleanup_all(), Postgres and Redis disconnect, telemetry shuts down.

API router registration

register_api_routers in cadence.core.router mounts routers in this order:

health → oauth2 → auth → api_key → chat → orchestrator → engine → plugins → tenant → admin → telemetry → stats

Path prefixes are owned by each leaf router (/api/chat, /api/engine, /api/orgs/{org_id}/orchestrators, /api/orgs/{org_id}/plugins, /api/admin/plugins, /api/admin, /api/admin/telemetry, /api/stats, plus top-level OAuth2 routes), so registration order has no practical effect on routing. The sequence simply reflects layering by concern (infrastructure → auth → features → admin).

Verification and quality

Rate limiting skips silently without Redis. If RateLimitMiddleware cannot get a Redis client from app.state, it passes the request through without enforcing limits. Test the degraded path explicitly.
RabbitMQ failure is non-fatal. Without the broker, orchestrator event messaging is off; the API still serves all other routes.
Production startup is strict. validate_production_config will abort startup on insecure config when CADENCE_ENVIRONMENT=production. Test the startup sequence in staging before shipping config changes.

Next steps

Security and access JWT, API keys, sessions, and OAuth-related endpoints.

Multi-tenancy Organizations, X-ORG-ID, membership, and quotas.

Configuration Environment variables and production checklist.