ADR-0062: Unity Catalog as a table-catalog backend now; extract twg-catalog-unity on a trigger

Reference

Builds on ADR-0024 (catalog credential vending: UC, S3 Tables, S3/IAM, Iceberg REST) and ADR-0037 (catalog descriptive metadata: comments/properties/tags on create+evolution, idempotent re-sync). Interacts with ADR-0023 (single open table format — Iceberg; read Delta via generated Iceberg metadata), ADR-0019 (twg-type-map is the describe-only type authority), and ADR-0038/0057 (a sink cannot write, nor hold the primary-raw role, without a provisioned/registered table). Detailed delivery: docs/blueprints/zerobus-sink-implementation-plan.md §5.1 (the Unity Catalog / table-catalog stage).

Context

twg-table-catalog owns the transport-neutral TableCatalog trait — schema resolution, sink registration, credential vending, and descriptive metadata. Every sink depends on it: a sink only writes, so table creation, on-the-fly schema evolution, comments, and credential vending live in the catalog layer, and a sink (Zerobus in particular) is unusable without it.

Unity Catalog is the first concrete backend. Its surface is non-trivial — OAuth machine-to-machine auth with token refresh, the UC REST request/response models, credential-vending and Iceberg-REST endpoints, and retry/rate handling. That raises a crate-shape question: does UC belong inside twg-table-catalog as a backend module, or as its own client crate (twg-catalog-unity) that the catalog crate depends on — mirroring how transport clients (twg-kafka-client, twg-pulsar-client) are split out from twg-connector-core?

Deciding this prematurely cuts both ways. Extracting a crate before there is a second consumer or a large surface adds a published crate, a version cadence, and a trait/dependency boundary to maintain for no present benefit. Never extracting it risks twg-table-catalog accreting a large UC-specific client that other backends (S3 Tables, Glue, Iceberg REST) must compile past and that a would-be external consumer of “just the UC client” cannot take on its own.

Decision

Implement Unity Catalog as a backend module inside twg-table-catalog now. Extract it to a dedicated twg-catalog-unity client crate later, only when a trigger fires. The sink is written against the TableCatalog trait either way, so the extraction is a non-breaking internal refactor when it happens.

• The TableCatalog trait is the seam. Sinks (twg-sink-zerobus and others) call describe / reconcile / register_sink / vend_credentials and never name UC directly. twg-type-map remains the describe-only Arrow→dialect authority both the catalog and sinks execute against; neither hand-builds DDL.
• The UC backend delivers the ADR-0024/0037 duties driven by the incoming Arrow RecordBatch schema: create-on-absent, evolve-on-drift (additive safe / widening behind opt-in via twg-type-map::can_widen_to / else refuse loud), comments at create AND evolution (two authorities), idempotent re-sync (twg:managed/twg:comment_hash), and scoped temporary credential vending for both the write path and the Iceberg recovery read path.

Extraction trigger

Promote the UC backend module to a standalone twg-catalog-unity crate when any of these holds — record the promotion as its own follow-up when it fires:

1. A second consumer outside twg-table-catalog needs the raw UC client (e.g. a standalone provisioning CLI, or a non-sink component doing UC lookups).
2. Surface growth: the UC-specific code (auth/refresh + REST models + retry/rate handling) exceeds roughly a module’s worth and starts imposing UC-only dependencies on the catalog crate’s other backends.
3. Independent release need: UC needs to version or ship on a cadence independent of twg-table-catalog.

Until a trigger fires, one backend module is the smaller, correct commitment.

Options considered + consequences

Option 1 (chosen) — Backend module now, extract on a trigger

• Pros: smallest thing that works; no premature crate/boundary/version cost; the trait insulates every sink, so extraction is a later non-breaking refactor; matches thalweg’s deferred-work discipline (a trigger, not a guess).
• Cons: if a trigger fires soon, there is a (small, mechanical) extraction to do later.

Option 2 — Extract twg-catalog-unity up front

• Pros: clean client/backend split from day one; a would-be external consumer can take the UC client alone.
• Cons: a published crate, version cadence, and trait/dependency boundary to maintain before any second consumer or surface pressure justifies it — speculative structure. Rejected for now; revisited by the trigger.

Option 3 — UC logic in the sink

• Rejected outright. It would duplicate provisioning across every sink and couple the sink to a specific catalog, violating the trait seam and the crate-isolation standard.

Invariants pinned by this ADR

• Sinks depend on the TableCatalog trait, never on UC directly.
• twg-type-map is the only place Arrow→dialect type mapping lives; no DDL text is hand-built in the sink or the UC backend.
• Credential vending yields temporary, scoped credentials inheriting the caller’s privileges — never static secrets held by the sink.
• Extraction to twg-catalog-unity, if/when it happens, is non-breaking for sinks (same trait) and is gated on a stated trigger, not a guess.

Document version history