Thalweg Plan-vs-Design Reconciliation

Inputs: docs/blueprints/thalweg-delivery-plan.md (v0.5, last-modified 2026-07-24) · docs/adr/README.md (ADRs 0001–0057) · drafted ADR files 0001 (Draft, v0.5) and 0003 (Accepted, v1.0) · crates/ tree (47 crates, all twg- prefixed).

Headline: The plan is one design generation behind. It was written when the whole system was framed as a single ADR (0001) covering ~18 crates; the design has since fragmented into 57 discrete decisions and 47 crates. The plan’s phase spine, layering rules, three-representation model, and fidelity approach remain correct and endorsed. What has drifted is (a) crate naming (unprefixed → twg-), (b) the crate matrix (sinks/sources/formats/contracts/DQ/observability now each have their own crate rather than being folded into a handful), and (c) three named subsystems the plan describes with stale specifics (recovery topology, metadata prefix, observability home). Roughly half the shipped crates have no phase, week budget, or exit gate in the plan at all.

Both ADR files carry an identical import note: “Treat the phase structure and crate layout as current; treat specific version pins and the single-ADR framing as pending update.” That is the reconciliation instruction in miniature — trust the plan’s shape, distrust its specifics.

A. CONFIRMED — plan decisions the ADR set still endorses unchanged

B. DEPARTED / SUPERSEDED — plan statements the design has since changed

B1. Recovery / coordination topology (the plan’s single biggest drift)

• Plan says (Phase 5, “Single-agent recovery only”): “One recovery worker needs no leases, no elections and no coordination substrate — it is the only claimant. Distributing recovery across agents is what needs coordination, and that is deferred … (ADR-0055, ADR-0056).” Recovery is framed as an internal capability of the single ingest process, unlocking the second sink.
• Current position — recovery is now a first-class deployment architecture, not an in-process worker:
  • ADR-0050: recovery is a deployment role (ingest / recovery / both); recovery windows are leased via the offset store; roles autoscale independently, to zero.
  • ADR-0057: reaffirms the plan’s gate (“recovery is a precondition for the second sink, not later hardening”) but narrows the deferral — only distributed recovery is deferred, not recovery itself.
  • ADR-0055: coordination substrate deferred, but with a chosen leading shape — an embedded replicated store, platform-independent, consensus library not written in-house.
  • ADR-0056: quorum is an additional worker role with a reduced throughput budget; modes 0 / 1 / 3+, two-node forbidden, stable assignment + graceful handoff.
  • ADR-0053: the coverage-store mechanics the plan attributes to twg-offset-store are now their own decision — position state may lag, never lead; gap-aware coverage stored with RAW, never in the sink it describes; local write-behind cache; source position only where no broker holds it.
• Impact: The plan’s ADR citations (0055/0056) are stale numbers pointing at the right idea. The rewrite must (a) split “recovery” into role-based recovery (0050, shippable in Phase 5) vs. distributed/quorum recovery (0055/0056/0057, genuinely deferred), and (b) attribute the coverage-store design to ADR-0053, not fold it into an unspecified twg-offset-store paragraph.

B2. Metadata column prefix — __kafka_ → configurable, default _twg_

• Plan says (Record-metadata section, repeatedly): reserved columns under a hard-coded __kafka_ prefix — __kafka_key, __kafka_timestamp, __kafka_offset, etc.; a payload field named __kafka_* is a hard decode error. The quality gate is named “Reserved-namespace collision” against __kafka_*.
• Current position — ADR-0051: the metadata column prefix is configurable, default _twg_; changing it on an existing target is a refused migration. The __kafka_ literal is gone (and was transport-specific, which conflicts with the transport-neutral connector goal — Pulsar metadata under a __kafka_ prefix is incoherent).
• Impact: Every __kafka_* string in the plan (metadata section, the collision gate, the examples) must become _twg_* (default) with a note that it is configurable and immutable-once-set. This is a pure find-and-replace-plus-semantics change, but it touches a named quality gate.

B3. Observability home — folded into connector-core → unified twg-observability crate

• Plan says (connector layer, Phase 5): “The instrumentation lives in connector-core, so both transports emit identical metric names.” OTel spans/metrics are a connector-core responsibility; the crate table lists connector-core deps as tower, opentelemetry, governor.
• Current position:
  • ADR-0032: a unified observability crate — metrics + OTLP + health share one surface → twg-observability.
  • ADR-0013: OTLP default, Prometheus scrape (disableable).
  • ADR-0031: health tree with three projections (/health, /ready, /live); readiness gated by source + primary-raw only.
  • ADR-0039: live per-stage memory/throughput digest — OTEL gauges + in-process /stats (30s window), honest per-stage attribution.
  • Confirmed by crate: twg-observability desc = “Prometheus metrics, OTEL/OTLP export, and the health tree with /health, /ready, /live projections.” twg-connector-core desc now reads “…flow control, observability, recovery” — it consumes observability, it no longer owns it.
• Impact: The plan must relocate OTel from a connector-core sub-bullet to a standalone crate + subsystem, and add the health tree and /stats digest, which have no presence in the plan at all (health/readiness is never mentioned). Cross-transport metric-parity gate stays, but now targets twg-observability.

B4. Sinks/sources — folded into stream-arrow → a modular source/sink crate matrix

• Plan says (crate table): stream-arrow = “BatchSink implementations, batching policy, IPC, Flight, offset correlation” — i.e. all sink concerns (including Flight and IPC) live in one crate; sources are implicit in kafka-client/pulsar-client.
• Current position — sinks and sources are each their own crate, and their existence is a decision (ADR-0012: all sources and sinks are optional and modular; replay-capable ones may hold primary-raw):
  • Sinks: twg-sink-object-store (ADR-0044), twg-sink-flight (ADR-0020), twg-sink-arrow-ipc (ADR-0026), twg-sink-postgres, twg-sink-zerobus.
  • Sources: twg-source-flight (ADR-0020), twg-source-arrow-ipc (ADR-0026), twg-source-batch (ADR-0015/0027/0033), twg-source-delta-sharing (ADR-0022).
  • twg-stream-arrow survives but is reduced to “batch sinks, IPC, Flight, offset correlation” as a shared substrate the modular sinks build on — not the home of every sink implementation.
• Impact: The plan’s Phase 5 “stream-arrow: … all four implementations (RawSink, ValueSink, ArrowSink, TeeSink)” is still valid as the core batching/correlation layer, but the plan has no concept of the 9 modular source/sink crates as separate deliverables. These need phase placement (see C and E).

B5. Single twg binary, not a library-only ecosystem

• Plan implies a library ecosystem plus a “deployable” that appears at the transform layer; there is no CLI/binary crate in the plan’s crate table, and packaging (Phase 9) produces “a binary” without naming its structure.
• Current position — ADR-0045: a single twg binary with subcommands (stream / batch / config), not separate executables. Crate: twg-cli.
• Impact: The binary is now a named crate with a defined command surface (including a whole batch mode, see C) and needs to appear in the workspace layout and packaging phase.

B6. Batch is now co-equal with streaming (not present in the plan)

• Plan says nothing — it is streaming-only end to end.
• Current position — ADR-0015: a batch CLI as a bounded source sharing the streaming pipeline; ADR-0027 Avro OCF as a batch source (shared Avro decode); ADR-0033 batch reader libraries (arrow-native + calamine + quick-xml; Polars rejected). Crate twg-source-batch reads Parquet/CSV/JSON/XML/Excel/Avro-OCF.
• Impact: This is a departure by omission — the plan asserts a streaming-only scope that the design has widened. See C for scope placement.

C. NEW / ABSENT-FROM-PLAN — subsystems with no phase, week budget, or exit criteria

Each entry: subsystem → ADR(s) → implementing crate(s) → recommended phase.

Codecs not in the plan’s Phase 7 list: twg-codec-custom (C17). Plan’s Phase 7 lists protobuf/json/xml only.

D. CRATE-LEVEL MAP — plan (~18, unprefixed) vs. actual (43, twg-)

Naming drift: The plan uses unprefixed names throughout (wire-tls, codec-core, stream-arrow). Every shipped crate is twg- prefixed. The plan already used the prefix in exactly one place — twg-offset-store in Phase 5 prose — showing the convention post-dates the plan and was applied inconsistently. The rewrite must prefix all 43.

Totals: 20 in-plan (19 renamed + 1 already-prefixed) · 26 NEW · 46 actual (of 47 workspace crates; this reconciliation table predates and omits twg-udf). The plan accounts for less than half the crate tree.

E. PER-PHASE IMPACT

Phase 0 — Foundations (2w)

• Wrong/incomplete: claims workspace-wide unsafe_code = "forbid" — contradicted by ADR-0043: twg-ffi is the sole audited unsafe exception. The rewrite must state “forbid with one crate-level allow, audited.”
• Absent scope to attach: publishing discipline (ADR-0047/0048, release-plz, dual-licence, semver checks), twg-config skeleton, twg-e2e harness scaffolding, docs-site infra (ADR-0052 — main branch protection is a gate precondition).
• Static-musl/aws-lc-rs proof stays correct.

Phase 1 — Transport primitives + value model (5w)

• Correct in spine.
• Absent scope: twg-type-map (ADR-0019) belongs here as a sans-io leaf beside the value model; twg-resource-loader (ADR-0029) foundational, pull in here or Phase 2; twg-config (ADR-0028/0030).

Phase 2 — Avro codec, all three surfaces (6w)

• Correct. Note ADR-0027 (Avro OCF) shares this decode — flag the reuse forward to batch mode.
• Absent scope: twg-contract-core/twg-contract-odcs first landing — contract-beats-inference (ADR-0035) interacts with codec schema resolution and should be designed alongside, not after.

Phase 3 — Kafka consumer, no groups (5w)

• Correct. Passthrough-ships-early claim stands.
• Absent scope: basic twg-observability metrics slice could start here (plan defers all OTel to Phase 5).

Phase 4 — Consumer groups (8w)

• Correct and unchanged. No drift. Still the hard phase per ADR-0001.

Phase 5 — Sinks, connector layer, coverage & recovery (11w) — most drift

• Wrong: “instrumentation lives in connector-core” → moved to twg-observability (B3). __kafka_ prefix → _twg_ (B2). “single-agent recovery only (ADR-0055/0056)” → role-based recovery per ADR-0050/0053/0057, distributed deferred (B1).
• Absent scope to attach: twg-observability + health tree + /stats (C7/C15); modular sink lane twg-sink-object-store/-arrow-ipc/-flight + source twg-source-flight/-arrow-ipc (C5/C6); DLQ reason-codes (C12); admission control (C14); dedup (C13); source-ack invariant hardening (ADR-0038). twg-offset-store mechanics → attribute to ADR-0053.
• Resequencing: Phase 5 is already 11 weeks and monolithic; this much added scope likely forces a split into 5a (single-sink + observability + connector-core) and 5b (coverage/recovery/multi-sink + object-store primary-raw lane).

Phase 6 — Producer, idempotence, transactions (6w)

• Correct. No drift.

Phase 7 — Ecosystem breadth (“ongoing”)

• Flagged as requested: Phase 7 is labelled “ongoing” with no exit gate — the only phase without one. This must be closed: either give it explicit exit criteria or split its contents into gated phases. It currently absorbs codecs (protobuf/json/xml), OAUTHBEARER crates, GSSAPI, and the whole AWS/Glue block, which is too much to leave ungated.
• Absent scope: twg-codec-custom (C17); the modular DB sinks twg-sink-postgres/twg-sink-zerobus (C5); the table-format block (C3) is far larger than the plan’s “Glue Data Catalog” bullet and probably deserves its own phase, not a Phase 7 sub-item.

Phase 8 — Pulsar (10w)

• Correct. Deepen with ADR-0049 (Pulsar flow control: staged subscription, topic-spread shedding, pod-seeded order, pressure-as-scale-signal) — the plan’s Pulsar gaps list predates this decision.

Phase 9 — Transform layer + packaging (12w)

• Correct in spine.
• Absent scope: twg-ffi (C8, the WASM C Data Interface boundary + ADR-0003 escape bridge); data-quality enforcement + audit (C2) run inside the DAG and have no home otherwise; twg-cli command surface + batch subcommand (B5/C4).

Cross-cutting resequencing gaps

• No headcount / parallelism assumption stated. The plan asserts “three critical paths” and “Phase 9 runs parallel to Phase 6 given separate ownership” but never states how many people/streams the timeline assumes. With 23 new crates (contracts, DQ, formats, batch, observability, 9 sink/source crates), the parallelism assumption is now load-bearing and must be made explicit — otherwise the Gantt is unfalsifiable.
• New subsystems have no Gantt bars at all: contracts, DQ, table formats, batch mode, observability/health, and the modular sink/source matrix appear nowhere in the timeline.

F. OPEN / PROVISIONAL — decisions the plan presents as settled but are genuinely open

Summary for the rewrite

1. Prefix everything twg- and expand the crate table from ~18 to 43 (§D).
2. Fix the three named departures verbatim: recovery topology (0050/0053/0055/0056/0057, not “single-agent only 0055/0056”), metadata prefix (_twg_ configurable, not __kafka_), observability home (twg-observability, not connector-core) (§B1–B3).
3. Add the absent half of the system: contracts/ODCS, data quality + audit, table formats (Iceberg-only), batch mode, the modular source/sink matrix, unified observability + health, FFI, type-map, resource-loader, config/secrets, admission control, dedup, DLQ reason-codes, releases/publishing, custom-decode (§C) — each needs a phase, budget, and exit gate.
4. Close Phase 7’s open-ended “ongoing” with an exit gate, and split the overloaded Phase 5.
5. State the parallelism/headcount assumption the timeline silently depends on.
6. Record ADR-0003 as Accepted (pin Arrow 58) and keep F1–F5 as explicitly open, not settled.
7. Reconcile the unsafe_code = "forbid" claim with the audited twg-ffi exception (ADR-0043).