ADR-0064: Cross-format schema bridge — a neutral hub, with protobuf → ODCS as the first spoke

Reference

Builds on ADR-0063 (proto-source bundling to a FileDescriptorSet), ADR-0016 (ODCS precedence / quality-rule merge), and ADR-0019 (twg-type-map as the describe-only, sans-io type-mapping authority). Consumes the .pb that twg-proto-bundle produces. Touches twg-contract-core (neutral schema hub), twg-contract-odcs (ODCS emit model), and a new twg-schema-bridge crate; wired into twg proto bundle --emit odcs.

Context

The bundler (ADR-0063) turns a proto source into a FileDescriptorSet and human artefacts. The next ask is to also emit an ODCS (Open Data Contract Standard) data contract from that schema, and — later — to go the other way (ODCS → .proto

• .pb) and to import/export further formats (JSON Schema, Avro). ODCS validation has other planned uses beyond this tool.

Two things shape the design:

• A hub already exists for leaf types. twg-type-map is the sans-io type-mapping authority (ADR-0019); it already carries json_schema and avro spokes and a common-ground type. Adding format-to-format schema conversion should extend that hub-and-spoke shape, not invent a parallel one.
• Formats differ in structure, not just leaf types. Protobuf carries packages, nested messages, oneof, map, and field numbers; ODCS carries a schema[] of objects with properties[], relationships[], and logicalTypeOptions. A conversion needs a neutral structural model as well as neutral leaf types.

A pairwise (format↔format) approach costs N² adapters as formats accumulate — the exact thing a hub prevents.

Decision

Convert every schema format through a neutral hub: structure through twg-contract-core’s schema model, leaf types through twg-type-map. Each format (protobuf, ODCS, JSON Schema, Avro) is a spoke to/from that hub. Build the protobuf → ODCS spoke now in a new twg-schema-bridge crate; defer the rest.

Concrete shape:

1. Neutral structural hub — twg-contract-core::schema. A SchemaModel is a flat set of SchemaObjects (fully-qualified name, is_root, fields). A Field has a neutral LogicalType (width/signedness preserved: Int32/Int64/UInt32/ UInt64/Float/Double/Bool/String/Bytes/Timestamp/Duration/Enum/Object) and a Cardinality (Single/Optional/Repeated/Map{key}). An object-typed field is an Object{fq-name} reference — objects reference each other by name, so a multi-package/multi-message source maps onto multiple cross-referencing objects with no inlining. This is the structural half of contract-core’s eventual resolved spec; quality rules / precedence / merge stay scaffold.

2. Leaf-type hub — twg-type-map. The cross-format type vocabulary. The protobuf spoke maps proto scalars → neutral LogicalType at the proto edge (proto’s type knowledge belongs there, mirroring twg-proto-schema owning proto’s Arrow derivation, ADR-0019); the JSON Schema / Avro spokes, when built, use type-map’s existing json_schema / avro mappings. type-map is thus the leaf hub in principle, invoked by each spoke as it lands.

3. Bridge crate — twg-schema-bridge. Depends on twg-contract-core (neutral model) and twg-contract-odcs (ODCS emit model); consumes a .pb as bytes, so it does not depend on twg-proto-bundle (keeping the bundler free of the contract stack) and the bundler does not depend on it. Ships: descriptor_set_to_schema_model, schema_model_to_odcs, and the composed descriptor_set_to_odcs.

4. ODCS emit model — twg-contract-odcs::model. A serde write-model for ODCS v3.1.0 (apiVersion/kind/id/version/status, schema[], properties[] with logicalTypeOptions, relationships[]). Parsing ODCS into the resolved spec remains a separate, scaffold concern.

5. protobuf → ODCS mapping (the built spoke). Each message → one schema[] object. A message-typed field → logicalType: object plus a schema-level relationships entry (from: Object.field, to: TargetObject, type: reference) — so packages/cross-references survive as ODCS’s own multi-schema references, not lost. Width/precision is preserved in logicalTypeOptions.format (i32/i64/u32/u64/f32/f64). Google well-known wrappers unwrap to their scalar; Timestamp/Duration map across; other google.protobuf.* degrade to string.

6. CLI. twg proto bundle --emit odcs writes <name>.odcs.json. It runs at the CLI layer (bundle → bridge on the produced .pb), preserving the dependency direction in (3).

Honest losses (pinned, like ADR-0063’s “.pb is authoritative”)

protobuf → ODCS is a faithful contract projection, not a wire-faithful mirror:

• Field numbers have no ODCS equivalent — dropped. So a round-trip back to proto cannot recover the original wire format.
• oneof grouping is not expressible — members render as ordinary (optional) fields; the mutual-exclusivity is lost.
• map<K,V> has no native ODCS type — rendered as an array of {key,value} objects.
• enums have no native ODCS type — rendered as string with the allowed values preserved in the property description.

The .pb remains the authoritative, wire-faithful artefact; the ODCS is the contract/documentation view.

Options considered + consequences

Option 1 (chosen) — Neutral hub (contract-core structure + type-map leaf), bridge crate, proto→ODCS first

• Pros: N adapters not N²; consistent with the existing type-map hub; JSON Schema/Avro slot in cheaply; bundler and contract crates stay dependency-clean; the neutral schema is the foundation contract-core needs anyway.
• Cons: more upfront than a direct bridge (a neutral model to define); the neutral model must grow as exotic features (oneof, custom options) demand.

Option 2 — Direct protobuf↔ODCS bridge, no neutral hub

• Pros: least code to ship the first two flows.
• Cons: N² adapters as JSON Schema / Avro arrive; duplicates leaf-type logic that type-map already owns. Rejected — the hub is the point.

Option 3 — Put proto↔ODCS inside the existing crates (emit in proto-bundle, parse in contract-odcs)

• Pros: fewer crates.
• Cons: couples the bundler to the contract stack and contract-odcs to proto descriptor traversal; neither stays independently publishable. Rejected in favour of a dedicated bridge.

Invariants pinned by this ADR

• One hub, many spokes — every format converts through the neutral schema model; no pairwise format↔format adapters.
• Dependency direction — bridge → {contract-core, contract-odcs}; bundler depends on neither the bridge nor the contract stack; the bridge consumes a .pb as bytes.
• .pb is authoritative — ODCS is a projection; lossy directions are documented, not silent.
• Leaf-type authority is type-map — spokes map through it (proto at its own edge, per ADR-0019); no parallel type vocabulary.

Deferred work

Each entry carries a trigger; an entry without one is a wish, not deferred work.

• D1 — ODCS → proto + .pb. The reverse spoke: ODCS → neutral → generate .proto(s) → compile via compile_proto_to_bytes. Field numbers are synthesised deterministically (sequential), so output will not wire-match a pre-existing proto. Trigger: an operator needs .proto/.pb generated from an ODCS contract.
• D2 — JSON Schema and Avro spokes. Import/export through the same hub, reusing type-map’s json_schema / avro leaf mappings. Trigger: a format beyond proto and ODCS is required.
• D3 — Richer ODCS fidelity. oneof as an ODCS construct, enum allowed-values as logicalTypeOptions rather than description text, primary-key / uniqueness from contract keys, column-level (not object-level) relationships. Trigger: a consumer needs one of these expressed structurally.
• D4 — ODCS parse → resolved spec. The read side feeding validation (ADR-0016). Trigger: the contract-validation phase is scheduled.

Interaction with existing ADRs

• ADR-0063 (bundling): this ADR consumes the .pb it emits; --emit odcs extends the same twg proto bundle command.
• ADR-0019 (type-map): honoured — type-map is the leaf-type hub; the proto spoke maps at the proto edge, as proto’s Arrow derivation already does.
• ADR-0016 (ODCS precedence/merge): the ODCS emit model here is independent of the parse/merge path that ADR-0016 governs; D4 connects them.

Document version history