ADR-0063: Proto-source bundling — one FileDescriptorSet, per-package .proto, and a message-type manifest from any proto source

Reference

Builds on ADR-0001 (pure-Rust, no external toolchain on PATH) and the existing protobuf descriptor toolkit: twg-proto-schema already owns compile_proto_to_bytes (source .proto → FileDescriptorSet via protox, no protoc), build_arrow_schema (descriptor → Arrow schema), and twg-proto-flatten/twg-proto-decode consume the resulting FileDescriptorSet. Related to ADR-0046 (protobuf decode strategy).

This ADR adds the acquisition step in front of that toolkit: turning a third-party proto source tree (typically a Java/Maven/Gradle contracts repo with many src/main/proto roots) into the single FileDescriptorSet those crates already require, plus two human-facing artefacts. Implemented by a new tooling crate twg-proto-bundle and a twg proto bundle subcommand; the full build is in ../blueprints/proto-bundle-implementation-plan.md.

Context

Upstream contracts arrive as repositories of .proto files, not as compiled descriptors. The shape observed across representative sources — a small one (~9 files / 1 include root), a medium one (~130 files / 8 roots), and a large one (~530 files / 77 roots) — is a Java convention: each Maven/Gradle module carries its own .../src/main/proto include root, imports are written relative to those roots (import "example/common/id.proto"), and Google well-known types (timestamp, duration, wrappers, any) are imported. All observed files are proto3.

Three facts make naive compilation fail or mislead:

• Many include roots per repo. A single protoc/protox invocation needs the union of every src/main/proto directory on the include path, discovered, not hand-listed. 77 roots is not a config the caller should assemble by hand.
• Duplicate import paths. The same logical path resolves under multiple roots — one shared file (e.g. example/common/feed.proto) can appear under several module roots, and the large source carried 7 internal duplicates (e.g. a vendored build-staging tree alongside the canonical one). Today the colliding copies are byte-identical, but a descriptor keyed by import path cannot hold two definitions of one path, and silent “first root wins” would mask future drift into a wrong-but-compiling bundle.
• The consumer needs a type name, not just bytes. build_arrow_schema(pb, message_type, depth) and the decode path are driven by a fully-qualified message type (e.g. com.example.orders.Order). Nothing in a raw repo tells the operator which strings are valid.

Separately, “give me the merged .proto” is a common and reasonable ask, but a single compilable .proto cannot span multiple packages — the language allows one package per file, and a real source spans several (e.g. com.example.orders, example.common, example.foo). So the tool emits both: a single combined <name>.proto (compilable when the source is single-package; an explicitly-labelled combined view when multi-package) and one valid file per package. Neither replaces the .pb as the machine-consumable artefact.

Finally, the source is not always a repo. Operators also hold a single already-merged .proto, or a bare directory of .proto files with no src/main/proto convention. The tool must accept all three without a mode flag.

Decision

Add twg-proto-bundle: a pure-Rust tooling crate (and twg proto bundle subcommand) that resolves any proto source to one combined FileDescriptorSet (.pb), a single combined .proto, one proto3 file per package, and a fully-qualified message-type manifest. The .pb is authoritative and machine-consumable; the .proto renders and the manifest are derived, human-facing conveniences.

Concrete shape:

1. Source resolution — auto-detected, additive, prunable. The <SOURCE> positional resolves:

   1. a .proto file → its parent dir is the include root (only that file);
   2. a directory where --include-glob (default **/src/main/proto) matches ≥ 1 dir → repo mode, roots are the matches;
   3. otherwise → the directory itself is the lone include root (flat / bare package-tree mode).

   --include-root <DIR> (repeatable) adds extra roots on top of auto-detection — each resolved through the same glob detection — for a cross-repo dependency whose .proto live outside <SOURCE> (observed: one source imports shared types that ship in a separate repo’s contracts subtree). --exclude <name> (repeatable) prunes any subtree with that path component — needed for a vendored build-staging copy that diverges from the canonical tree (observed: a generated build directory mirroring the shared contracts).

2. Dedup by import path, identity-checked. Files are keyed by import-relative path. A path resolving under multiple roots is collapsed only if the copies are byte-identical; divergent copies are a hard error with a conflict report (path, competing roots, hashes), overridable via --on-conflict {error|first-wins}. Default error: silent drift must not corrupt the bundle.

3. Stage into one clean root, then compile. The deduped set is staged into a temp tree preserving import paths, compiled with that single dir as the sole include root via twg_proto_schema::compile_proto_to_bytes. One root removes all include-order ambiguity and makes the render deterministic. A leading UTF-8 BOM is stripped while staging — protox (unlike protoc) lexes a BOM as an invalid token, and a BOM-only difference must not read as a content conflict (observed: several source files are BOM-prefixed). protox bundles the Google well-known types and includes every reachable type (--include_imports semantics), so the .pb is self-contained.

4. .pb is the authoritative artefact. One FileDescriptorSet per invocation, holding every file in the source plus transitively-reachable WKTs. This is the exact input twg-proto-schema/twg-proto-decode already consume.

   Unresolved cross-repo dependencies are alerted, not opaque. When compilation fails because an import points at a file the source does not provide (and that is not a bundled Google WKT), the staged files’ import statements are scanned and every missing dependency is reported at once — each with the file(s) that require it — so the operator can supply them in one pass (e.g. an extra --include-root), rather than fixing them one protox-error-at-a-time. Auto-resolving those dependencies from a corpus of related repos remains deferred (D3).

5. .proto render is derived, in two forms. Both are rendered from the descriptor (maps un-synthesised from MapEntry, oneofs, proto3 optional, nested types, enums):

   • A single combined .proto — <name>.proto. When the source is single-package this is a valid, standalone-compilable file — verified end-to-end by feeding it back through the tool and diffing the recompiled type inventory (identical). When the source is multi-package, a single compilable .proto is impossible — the language allows one package per file — so the output is a combined, package-delimited view carrying a leading banner comment; it is for reading/grepping/diffing, and the .pb is the artefact to compile. The tool reports which form it produced.
   • One .proto per package — <name>/<package>.proto, importing sibling packages; Google imports centralised (a base file, import public) so no file redefines a WKT. Correctness gated by a round-trip test: recompile the render, diff the normalised type set against the source .pb.

   A per-package set only recompiles when the package dependency graph is acyclic (proto forbids circular imports; a single-package source round-trips exactly). Real multi-package sources are often cyclic (observed: two of the sources each have a mutually-recursive package pair). The bundler detects and reports the cycle; the render remains a readable view, and the .pb is authoritative regardless.

6. Message-type manifest. <name>.messages.json lists every fully-qualified message type in the set and flags root candidates (messages never used as a field type by any other message — the envelope/entry points), so the operator can read off the message_type string to hand to build_arrow_schema (e.g. com.example.orders.Order).

7. Tooling crate, not a runtime dependency. twg-proto-bundle sits above twg-proto-schema, is publish = false, and nothing on the ingest/runtime path depends on it. It is a build/operator-time artefact generator.

Options considered + consequences

Dimensions: correctness under duplicate paths, determinism, no external toolchain, operator ergonomics.

Option 1 (chosen) — Discover → dedup/identity-check → stage → compile; render per package; emit manifest

• Pros: single command for file/dir/repo sources; deterministic compile from a clean staged root; duplicate-path drift caught, not masked; pure Rust (no protoc); reuses the existing compile_proto_to_bytes; the .pb is exactly what downstream already consumes; round-trip test makes the render trustworthy.
• Cons: a stage copy per build (cheap — hundreds of small text files); a descriptor→.proto printer is real code with fidelity risk (bounded by the round-trip gate).

Option 2 — Hand the caller’s include roots straight to protox, no staging

• Pros: less code.
• Cons: duplicate import paths across roots make the result depend on include-order; a future divergence compiles silently to the wrong bytes. Loses the identity check. Rejected.

Option 3 — Shell out to protoc --include_imports --descriptor_set_out

• Pros: battle-tested compiler; free .pb.
• Cons: reintroduces an external toolchain on PATH, contra ADR-0001; still leaves discovery, dedup, per-package render, and the manifest unbuilt. Rejected.

Option 4 — Emit a single merged .proto instead of per-package

• Pros: matches the literal “one file” ask.
• Cons: a single compilable .proto is impossible for multi-package sources (one package per file); making it compile would force dropping packages and globally-unique renames across hundreds of files with colliding simple names (Metadata, Envelope, Status). Rejected as the sole form. Adopted as an additional output (<name>.proto): a genuine compilable file when the source is single-package (verified by recompiling it), and an explicitly-labelled combined view when multi-package. The per-package files and the authoritative .pb stand alongside it.

Invariants pinned by this ADR

• .pb is authoritative — the single combined .proto, the per-package .proto, and the manifest are all derived from it, never the reverse.
• Single .proto is honest about compilability — compilable only for a single-package source; a multi-package source produces a banner-labelled combined view, and the tool reports which form it emitted.
• No silent path collision — identical duplicates collapse; divergent duplicates error by default.
• Deterministic compile — from a single staged include root, not caller include order.
• No external toolchain — pure-Rust protox, consistent with ADR-0001; no protoc on PATH.
• Render is round-trip-verified for acyclic sources — an acyclic per-package render that does not recompile to the source type set is a test failure; a cyclic source is reported, and the .pb is the artefact to consume.
• Source-shape agnostic — a single .proto, a bare dir, and a src/main/proto repo all resolve through one command; cross-repo deps add roots, vendored mirrors are excluded.
• Missing dependencies are named, not opaque — an unresolved import fails with the complete list of missing files and their requirers, never a single first-encountered protox error.

Deferred work

Each entry carries a trigger; an entry without one is a wish, not deferred work.

• D1 — Custom option / extension fidelity in the render. The proto3 printer targets messages, enums, oneofs, maps, nested types, reserved, and standard field options. Trigger: a source using custom options (extensions) fails the round-trip diff.

• D2 — Compilable output for cyclic package graphs. A per-package render of a mutually-recursive package pair cannot recompile (proto forbids circular imports); today it is detected and reported, with the .pb as the artefact to consume. Trigger: an operator needs standalone-compilable .proto text for a cyclic source (would require preserving the original file boundaries, which were acyclic, rather than grouping by package).

• D3 — Auto-resolving cross-repo dependencies. Missing imports are now detected and reported (every unresolved import + its requirers), and roots/excludes are supplied explicitly. What remains deferred is resolving them automatically: given a corpus of related repos, find which one provides each missing import and add its root without the operator naming it. Trigger: a manifest/search-root of related repos exists that the tool could search transitively.

• D4 — proto2 support. All observed sources are proto3; the printer assumes it. Trigger: a proto2 source appears (adds required/groups/default-label rules).

Interaction with existing ADRs

• ADR-0001 (pure-Rust): honoured — protox, no protoc.
• ADR-0046 (protobuf decode): this ADR feeds that path; the .pb it emits and the message_type from its manifest are the inputs to build_arrow_schema / twg-proto-decode.

Document version history