Expand description
Custom binary decode. Scaffold only.
§Batch-oriented WASM is the default
Custom decoders run on the pipeline-wasm host, but the contract is batch-oriented, NOT per-record — this is what removes the decode-time latency concern. A per-record WASM call would pay the boundary crossing and marshalling once per record; at high throughput that boundary cost dominates. A transform UDF avoids this by running per-batch; custom decode does the same.
Contract shape: (&[RawFrame]) -> RecordBatch — the module receives a buffer
of N already-framed records and returns an Arrow batch, so the WASM boundary
is crossed once per batch, amortised exactly like the transform UDF. The
module iterates internally in WASM.
Framing is native, outside the sandbox: splitting the byte stream into record boundaries (length prefixes, delimiters, fixed-width) happens in native Rust on the hot path — the sandbox is never asked to do per-record stream framing. Native frames, WASM decodes the batch.
§Native compile-in is a measured exception, not a general option
For the one or two absolute-hottest formats where even amortised WASM is
measurably slower than compiled Rust AND the format is stable enough to justify
a rebuild AND you own the code, a native Decoder/ArrowDecoder impl compiled
in is permitted. This is the trusted-native corner, mirroring the C Data
Interface vs WASM split: native = trusted/compiled-in/max-throughput, WASM =
supplied-without-rebuild/untrusted.
Discipline: WASM is the default and the documented path. Native custom decode reintroduces the rebuild-to-extend friction WASM exists to avoid, so it is an explicit, justified, measured performance exception for a format you own — never the easy default. Same shape as every other named exception in the design.