Architecture Decision Records
Each ADR records one decision. Records follow the team house style and
round-trip to Confluence via mark. Status flows Draft → For Review → Accepted
(or Rejected / Superseded).
The decisions below were developed during design. Records are written per decision, not per crate — most decisions cut across several crates, and several crates carry no decision of their own.
| ADR | Decision | Status |
|---|---|---|
| 0001 | Pure-Rust, sans-io ecosystem with three co-equal representations | Planned — stub drafted |
| 0002 | Neutral value model vs. upstream value types | Planned |
| 0003 | Arrow / DataFusion version strategy | Accepted — drafted; pin to DataFusion’s Arrow major, no bridge |
| 0004 | Offset correlation and locator propagation (incl. unnest cardinality) | Planned |
| 0005 | Record metadata as reserved columns | Planned |
| 0006 | Connector flow control: state machine + admission control | Planned |
| 0007 | Transform DAG, interior sinks, and materialisation | Planned |
| 0008 | Extension model: in-process WASM vs. Flight sidecar | Planned |
| 0009 | Primary-raw role, offset colocation, and recovery topology | Planned |
| 0010 | Fresh Pulsar client vs. forking an existing crate | Planned |
| 0011 | Packaging: static musl binary primary, multi-arch OCI derived | Planned |
| 0012 | All sources and sinks are optional and modular; replay-capable ones may hold the primary-raw role | Planned |
| 0013 | Observability: OTLP default, Prometheus scrape (disableable) | Planned |
| 0014 | Data-quality enforcement: modes and split | Planned |
| 0015 | Batch CLI as a bounded source sharing the streaming pipeline | Planned |
| 0016 | Data contracts: ODCS precedence, quality-rule merge (most-restrictive wins) | Planned |
| 0017 | Unified DLQ with reason-codes (decode / contract / DQ / retry-exhausted) | Planned |
| 0018 | Contract loading: local, HTTP(S), object store (S3/ABFS) | Planned |
| 0019 | Type-mapping authority: describe-only sans-io leaf (type-map) | Planned |
| 0020 | Arrow Flight/IPC as a first-class source and sink transport | Planned |
| 0021 | Testing standard: per-crate coverage + e2e harness + encoded regressions | Planned |
| 0022 | Table-format layer: Parquet, Delta, Iceberg (read+write), Delta Sharing (read) | Planned |
| 0023 | Support one open table format — Iceberg — not two; read Delta via generated Iceberg metadata and the catalog’s Iceberg REST endpoint | Planned |
| 0024 | Catalog credential vending: UC, S3 Tables, S3/IAM, Iceberg REST | Planned |
| 0025 | Custom binary decode: batch-oriented WASM default (framing native), native compile-in as measured exception | Planned |
| 0026 | Arrow IPC (non-Flight) file + stream as first-class source/sink | Planned |
| 0027 | Avro Object Container Files as a batch source (shared Avro decode) | Planned |
| 0028 | TOML config carried verbatim by K8s (file-mount default, pointer option); no TOML↔YAML conversion | Planned |
| 0029 | Shared resource loader (local/HTTP/object-store) for config, contracts, and SQL | Planned |
| 0030 | Secret enumeration tooling (config secrets) | Planned |
| 0031 | Health tree with three projections; readiness gated by source + primary-raw only | Planned |
| 0032 | Unified observability crate (metrics + OTLP + health share one surface) | Planned |
| 0033 | Batch reader libraries: arrow-native + calamine + quick-xml; Polars rejected | Planned |
| 0034 | DQ audit trail: full-grain per-record per-rule, first-class sink, on by default | Planned |
| 0035 | Contract schema beats inference across CSV/JSON/XML batch reads | Planned |
| 0036 | Drop rules: fourth disposition, silent by default (opt-in audit), ODCS+manual merge | Planned |
| 0037 | Catalog descriptive metadata: comments/properties/tags on create+evolution, idempotent re-sync | Planned |
| 0038 | Source-ack invariant: advance offset only on durable sink ack; batch-granular default, per-record opt-in; durable mandatory for primary-raw | Planned |
| 0039 | Live per-stage memory/throughput digest: OTEL gauges + in-process /stats (30s window); honest per-stage attribution | Planned |
| 0040 | Self-correcting admission control: live measured p95 record size, static config as cold-start seed + floor | Planned |
| 0041 | Quarantine non-blocking by construction; overflow drops-with-count; recovery rehydrates (skip-DLQ opt-in) | Planned |
| 0042 | Arrow C Data Interface inbound source (trusted in-process zero-copy handoff), filling the extension matrix | Planned |
| 0043 | Single twg-ffi crate for both FFI directions; sole audited unsafe exception | Planned |
| 0044 | One sink-object-store delegating to format-parquet/delta/iceberg (format by config); plain Parquet raw lane primary-raw-eligible | Planned |
| 0045 | Single twg binary with subcommands (stream / batch / config) rather than separate executables | Planned |
| 0047 | Independent per-crate publishing under dual MIT OR Apache-2.0; own version, README, tests | Planned |
| 0054 | Content-hash dedup: opt-in, contract-gated, best-effort, exact membership, batched writes, window sized to upstream retry interval | Planned |
| 0055 | Coordination substrate deferred; embedded replicated store the leading shape, platform-independent, consensus library not written in-house | Planned |
| 0056 | Quorum as an additional worker role with reduced throughput budget; modes 0 / 1 / 3+, two-node forbidden; stable assignment and graceful handoff | Planned |
| 0057 | Recovery is a precondition for the second sink, not later hardening; only distributed recovery is deferred | Planned |
| 0053 | Position state may lag, never lead: gap-aware coverage stored with RAW (never in the sink it describes), local write-behind cache, source position only where no broker holds it | Planned |
| 0052 | Documentation site on Cloudflare Pages at docs.thalweg.dev via Git integration; apex reserved; gates protect the merge, so main must be protected | Planned |
| 0051 | Metadata column prefix configurable (default _twg_); changing it on an existing target is a refused migration | Planned |
| 0050 | Recovery as a deployment role (ingest / recovery / both); windows leased via the offset store; independent autoscaling to zero | Planned |
| 0058 | Record-metadata columns: transport-neutral _twg_source_* superset for broker-supplied fields, bare _twg_* for Thalweg-stamped values (_twg_ingest_ts / _twg_emit_ts / _twg_record_id); amends ADR-0005/0051 | Draft — drafted |
| 0059 | Public documentation rendering pipeline: self-hosted rustdoc into docs/site/api/ and mdBook into docs/site/reference/, regenerated locally and committed (Cloudflare build stays NONE); builds on ADR-0052 | Accepted — drafted |
| 0060 | Pipelined fire-and-forget sink ack: durable confirmation on a per-stream background lane, advance on the delivered offset, compile-time lane separation; refines ADR-0038’s ack invariant so it doesn’t serialise the sink | Draft — drafted |
| 0061 | Regenerate the rendered docs (rustdoc + mdBook) on main in CI after each merge and commit them back, instead of in the pre-commit hook; ends the 768-file generated-doc merge conflicts. Amends ADR-0059 | Accepted — drafted |
| 0062 | Unity Catalog as a backend module inside twg-table-catalog now, behind the TableCatalog trait; extract to a dedicated twg-catalog-unity client crate only on a stated trigger. Builds on ADR-0024/0037 | Accepted — drafted |
| 0063 | Proto-source bundling: twg-proto-bundle (+ twg proto bundle) resolves any proto source (single file / bare dir / src/main/proto repo) to one combined FileDescriptorSet .pb, single + per-package .proto, and a FQ message-type manifest; pure-Rust protox, dedup-by-import-path with identity check, round-trip-gated render. Builds on ADR-0001/0046 | Draft — drafted |
| 0064 | Cross-format schema bridge: convert schemas through a neutral hub (twg-contract-core::schema for structure, twg-type-map for leaf types); new twg-schema-bridge crate builds the protobuf → ODCS v3.1.0 spoke (twg proto bundle --emit odcs), packages preserved as ODCS multi-schema references; ODCS→proto + JSON Schema/Avro deferred. Builds on ADR-0016/0019/0063 | Draft — drafted |
| 0065 | Zerobus sink build: minimal real substrate (twg-connector-core Sink, twg-stream-arrow BatchSink, twg-offset-store gap-aware union-merge coverage); the SDK inverted behind a ZerobusTransport seam (fake-tested ADR-0060 ack core); Arrow-IPC bytes bridge the 58↔59 major gap so Arrow 59 stays confined to the SDK behind an optional databricks-sdk feature. Builds on ADR-0060/0038/0053/0062/0003 | Draft — drafted |
| 0049 | Pulsar flow control: staged subscription, topic-spread shedding, pod-seeded order, pressure-as-scale-signal | Planned |
| 0048 | Releases via release-plz: reviewed release PR, semver checks, codec version group, Conventional Commits | Planned |
| 0046 | Protobuf decode: selectively vendor a zero-copy parser (Apache-2.0), own the registry and Arrow output stage | Open — benchmark |
Genuinely open rather than merely undrafted: the protobuf decode strategy (ADR-0046 — direction chosen, licence cleared as Apache-2.0, now gated only on a benchmark against the upstream parser on our hardware and schemas).
Noted for future consideration (not yet scoped)
Captured so they are not lost; no decision or crate yet, and not committed scope:
- DuckLake as a source and sink — treated like the other database targets (goes through the sink contract / a source adapter, primary-raw eligibility assessed on its durability + replayability like any other).
- REST and WebSocket (WSS) as a source and a sink — REST polling/webhook and
WSS streaming, both directions. WSS-as-source is a streaming transport (fits
connector-core); REST-as-source is poll-or-webhook; both as sinks are push/forward targets. Locator/recovery semantics to be worked through when scoped.