Health & Observability

One crate, twg-observability, owns three surfaces that share an HTTP server and read from the same runtime signals: Prometheus metrics, OTEL/OTLP tracing, and health.

Health is a tree with three projections

Every stage reports into one registry — sources, decode, contract/DQ, transform DAG nodes, each sink, the recovery driver, the offset store. Each reports a state and the numbers behind it:

• ok — operating normally.
• degraded — working but impaired: a circuit breaker half-open, backpressure at Warning/Critical, elevated reject rate, rising commit latency. Visible and actionable by a human; not automatically acted upon.
• down — the stage is not functioning: source disconnected, sink circuit open, backpressure at Stuck.

Backpressure is not a separate signal. It is the connector-core four-state machine (Normal → Warning → Critical → Stuck) projected per stage, so the “growing backpressure” you want to see is the state machine’s state surfaced on each stage. Per-sink circuit-breaker state feeds sink stage health the same way.

Why three endpoints for one tree

Kubernetes probes read only the HTTP status code, never the body. The rich per-stage JSON is therefore invisible to the kubelet by design — which is what lets /health be richly honest without that honesty getting the pod killed.

curl -s /health | head -1 (or the emitted summary line) gives the overall code and a one-line story; the body underneath gives the per-stage breakdown so an operator can see immediately which stage is offline or building backpressure.

What gates what

Liveness (/live) fails only on genuinely unrecoverable in-process state. Never on a saturated sink or a rebalance — restarting does not fix a data-plane problem and wiring those in causes restart loops.

Readiness (/ready) is gated by exactly two things: source health and the primary-raw sink. If a source is down or raw cannot be written, the pod is not ready — because if raw cannot land, the source cannot safely acknowledge. Every other stage is display-only: a degraded secondary sink, a backed-up quarantine lane, a slow transform node, or lagging recovery all show in /health but never drop readiness.

Extension path, not built yet: a per-stage ignore | degrade | fail readiness policy — the monitoring twin of the sink block | isolate failure policy — for when some stage beyond source/primary-raw should gate readiness. Not needed on day one; readiness has two inputs until proven otherwise.

Metrics and tracing

Metrics export as Prometheus scrape (/metrics, disableable via config) and OTLP push (default destination Zerobus, any OTLP endpoint supported). Tracing follows the record lifecycle with context propagated through message headers, so a trace crosses the broker rather than stopping at it. Metric names and labels are defined in this crate so both transports emit identically — one dashboard works against either.

Live per-stage memory & throughput (/stats)

Beyond the continuous OTEL gauges, GET /stats serves an in-process 30-second rolling digest: per-stage throughput and average per-record / per-batch memory, as JSON. The two are not redundant — the gauges feed dashboards and alerting (Grafana computes its own windows); /stats is for the incident where you are on the pod and want it to tell you directly where its memory went in the last 30s, without depending on a metrics backend that may itself be degraded.

Per-stage memory is reported by honest attribution — queue depth × bytes, batch count × batch size, in-flight bytes — never by carving global RSS into per-stage fictions, since stages share an allocator.

The same live measurement feeds admission control: the budget uses the measured recent per-record size (p95 within the window, so a size spike does not blow the budget before a mean would catch up), with the record_size_est_bytes config as cold-start seed and floor. The static estimate self-corrects as real sizes are observed — an improvement over a fixed guess that is wrong the moment sizes drift.