Proto Bundle — Technical Implementation Plan

A crate-by-crate build plan for twg-proto-bundle and the twg proto bundle subcommand: turning any proto source (a single .proto, a bare directory of protos, or a Java/Maven/Gradle contracts repo with many src/main/proto roots) into one combined FileDescriptorSet (.pb), one proto3 file per package, and a fully-qualified message-type manifest. Grounds the decision in ADR-0063, reuses twg-proto-schema’s pure-Rust compile_proto_to_bytes (ADR-0001), and feeds the existing descriptor toolkit (build_arrow_schema, twg-proto-decode, ADR-0046).

Status. Design → build. The .pb path is the priority deliverable; the per-package .proto render and manifest are derived from it.

No sensitive information. All names in this plan are generic placeholders (com.example.*, example/common/*.proto). Worked examples describe sources only by scale — “the small single-package source”, “a large ~530-file source” — never by their real repo, package, or type names. Nothing domain-specific is embedded in the tool, its tests, or these docs.

0. What the operator runs

twg proto bundle <SOURCE> --out-dir <DIR>
     [--name <basename>]                     # default: source dir/file stem
     [--include-root <DIR>]                  # repeatable; EXTRA roots (cross-repo deps)
     [--exclude <name>]                      # repeatable; prune a subtree (e.g. a build-staging dir)
     [--include-glob '**/src/main/proto']    # repo-mode auto-detect pattern
     [--emit pb,single,proto,manifest]       # default: all four
     [--on-conflict error|first-wins]        # default: error
     [--wkt-imports central|per-file]        # default: central

Outputs in <DIR>:

Worked example — twg proto bundle <SMALL_SOURCE> --out-dir ./out --name orders (a single-package source) yields out/orders.pb, out/orders.proto (single-package ⇒ standalone-compilable), out/orders/com.example.orders.proto, and out/orders.messages.json listing com.example.orders.Order (and siblings), with the envelope/entry types flagged as roots. For a multi-package source <name>.proto is a combined view and the .pb is the artefact to compile.

1. Crate placement

New crate crates/twg-proto-bundle, publish = false (operator/build tooling, not a runtime dependency). Depends on twg-proto-schema with its compile feature. Sits above the descriptor toolkit in the layer graph; nothing depends back on it.

crates/twg-proto-bundle/
  Cargo.toml
  src/
    lib.rs        # BundleOptions, Bundle, bundle(), BundleError
    discover.rs   # source resolution -> include roots + ProtoFile{abs, import_path}
    reconcile.rs  # dedup by import path, identity check, staging tree
    compile.rs    # stage -> compile_proto_to_bytes -> FileDescriptorSet
    manifest.rs   # FileDescriptorSet -> {all, roots} FQ message types
    render.rs     # FileDescriptorSet -> one proto3 file per package
  tests/
    repos.rs      # path-gated integration over the three real repos + round-trip

Dependencies: prost / prost-types 0.14 (already the workspace proto version), walkdir, sha2 (content identity), serde + serde_json (manifest), thiserror (workspace), tempfile (staging). CLI adds clap.

2. discover — source resolution (ADR-0063 §Decision.1)

resolve_source(source, opts) -> Discovered { roots: Vec<PathBuf>, files: Vec<ProtoFile> }

• ProtoFile { abs: PathBuf, import_path: String, root: PathBuf } where import_path is abs relative to its root, forward-slashed.
• Resolution order:
  1. source is a file ⇒ root = parent (or --include-root), one ProtoFile.
  2. source is a dir + --include-root given ⇒ those roots verbatim.
  3. dir + --include-glob matches ≥ 1 dir ⇒ repo mode, roots = matches.
  4. else ⇒ source itself is the lone root.
• Walk each root with walkdir, skipping target/, build/, .git/, node_modules/; collect *.proto.
• Empty result ⇒ BundleError::NoProtoFiles.

Verified against the three repos: 1 root/9 files, 8/128, 77/534.

3. reconcile — dedup + stage (ADR-0063 §Decision.2–3)

• Group files by import_path.
• One path → many abs: read + sha256. All equal ⇒ keep first, count dropped. Divergent ⇒ BundleError::Conflict { import_path, roots, hashes } unless --on-conflict first-wins (then keep first from the first-listed root, log a warning — never silent).
• Stage: create a tempfile::TempDir; for each surviving import_path, copy its file to stage/<import_path> (creating parent dirs). Return Staged { dir: TempDir, files: Vec<String> /* import paths */ }.

Across the observed sources all collisions are byte-identical (a shared file present under two repos’ roots; ~7 internal dups in the large source) so the default error path passes; the check exists to catch future drift.

4. compile — staged tree → .pb (ADR-0063 §Decision.3–4)

• proto_paths = every staged file (absolute), include_paths = [stage.dir].
• Call twg_proto_schema::compile_proto_to_bytes(&proto_paths, &include_paths) (enable twg-proto-schema/compile).
• protox bundles the Google WKTs and includes all reachable types; the returned bytes are the FileDescriptorSet. Write to <name>.pb when pb is in --emit.
• Phase-1 assumption to confirm on first run: protox ships google/protobuf/{timestamp,duration,wrappers,any}.proto. If a compile reports a missing google/protobuf/* import, stage bundled WKT copies as a fallback (a localized change in compile.rs).

5. manifest — message-type inventory (ADR-0063 §Decision.6)

• Decode the .pb with prost_types::FileDescriptorSet.
• For each file, walk message_type recursively; FQ name = .<package>.<Msg>[.<Nested>…] (drop the leading dot for output).
• Root detection: collect every type_name referenced by any field across the set; a message whose FQ name is never referenced is a root (entry/envelope).
• Emit { "all": [...sorted...], "roots": [...sorted...] } to <name>.messages.json. Also returned in-process on the Bundle struct so tests/callers can assert on it (e.g. com.example.orders.Order present).

6. render — one .proto per package (ADR-0063 §Decision.5)

Rendered from the descriptor, not the source text, so it is canonical and package-grouped.

• Group FileDescriptorProtos by package.
• Per package emit: syntax = "proto3";, package <pkg>;, imports, then message and enum bodies.
• Imports: for each type referenced from another package, import that package’s generated file. Google WKTs: with --wkt-imports central (default), a generated base file holds the import "google/protobuf/*" lines and dependents pick them up via import public; with per-file, each file imports what it uses. No file ever redefines a WKT.
• Field rendering: scalar/message/enum types; repeated; proto3 optional (synthetic oneof unwrap); map<k,v> reconstructed from synthetic MapEntry nested messages (detect options.map_entry); oneof groups; nested messages and enums; reserved ranges/names.
• Correctness gate (test, not runtime): recompile the rendered files with protox, normalise both descriptors (sort files/fields, drop source-info), and assert equal to the source .pb. A render that does not round-trip is a test failure.

7. lib — public surface

#![allow(unused)]
fn main() {
pub struct BundleOptions {
    pub include_roots: Vec<PathBuf>,   // explicit override; empty = auto
    pub include_glob: String,          // default "**/src/main/proto"
    pub emit: EmitSet,                 // pb | proto | manifest
    pub on_conflict: OnConflict,       // Error (default) | FirstWins
    pub wkt_imports: WktImports,       // Central (default) | PerFile
    pub name: Option<String>,
}
pub struct Bundle {
    pub descriptor_set: Vec<u8>,       // the .pb bytes
    pub messages: MessageManifest,     // { all, roots }
    pub proto_files: Vec<RenderedProto>, // package -> text
    pub dropped_duplicates: usize,
}
pub fn bundle(source: &Path, out_dir: &Path, opts: &BundleOptions) -> Result<Bundle, BundleError>;
}

bundle() orchestrates discover → reconcile → compile → manifest → render, writes the --emit subset to out_dir, and returns Bundle for programmatic callers.

8. CLI wiring

twg-cli/src/main.rs is a scaffold (fn main() {}). Stand up a minimal clap command tree with proto bundle as the first real subcommand (streaming/batch stay scaffolded). main parses args into BundleOptions, calls twg_proto_bundle::bundle, prints a summary (N files, M packages, K messages, D duplicates dropped → out/…), maps BundleError to a non-zero exit.

9. Testing

• Unit: discovery per mode (file/override/repo/bare-dir); reconcile identity vs. divergence; map/oneof/optional/enum/nested rendering on small fixtures.
• Integration (tests/repos.rs, skipped unless an env var points at a source): driven by TWG_BUNDLE_TEST_SOURCE (+ optional TWG_BUNDLE_TEST_ROOTS, TWG_BUNDLE_TEST_EXCLUDE), so the suite carries no domain-specific names and runs against whatever tree the operator supplies. Assertions are structural invariants only: .pb decodes to a non-empty FileDescriptorSet; the manifest is non-empty; include-root count / dropped-duplicate count are surfaced; and a round-trip — recompile the rendered per-package set and compare the normalised type inventory to the source .pb (reported, not asserted, when the source has a package cycle).

10. Phasing

1. Phase 1 — .pb + manifest. discover → reconcile → stage → compile → manifest. Delivers the combined .pb and the message-type list (the priority). Settles the protox WKT question. Proven first on the smallest source.
2. Phase 2 — per-package render + round-trip test. The proto3 printer and its gate; run against sources of increasing size.
3. Phase 3 — CLI + docs. twg proto bundle, summary output, README/rustdoc.

11. Risks

• protox WKT bundling — Phase-1 first run confirms; cheap staged-WKT fallback.
• Render fidelity (maps, oneof, optional, custom options) — bounded by the round-trip gate; custom options deferred (ADR-0063 D1).
• Scale (~530 files / 77 roots on the largest observed source) — staging + one compile; well within protox.