Expand description
layered config load (TOML < env < CLI), K8s file-mount + pointer resolution, secret enumeration
Scaffold only — no implementation yet. See docs/blueprints for the phase this crate lands in.
§The metadata column prefix is configurable
Metadata columns default to a _twg_ prefix but the prefix is settable, since
a platform with its own column-naming conventions should not have to adopt
ours. Everything that reads or writes a metadata column resolves the prefix
from config rather than a constant — the DQ audit trail, catalog comments,
recovery queries, and the reserved-namespace collision check alike.
Validated at startup, because every failure here is cheaper before the first write than after:
- non-empty, and a legal identifier prefix in EVERY configured sink — the intersection is narrower than any one of them, so validate against all targets rather than the first;
- length-bounded, so prefix plus the longest metadata column name still fits the tightest identifier limit among the sinks;
- not a prefix of any field in the resolved contract schema, which is the collision check generalised: with a configurable prefix the reserved namespace is whatever was configured, not a hardcoded string.
Changing the prefix on an existing deployment is a breaking migration, not a setting. Previously-written tables keep the old columns; new writes create new ones; recovery reading historical raw looks for names that are no longer there. Config validation must therefore refuse a prefix that differs from the one recorded in the target’s own metadata, rather than silently writing a second, parallel set of metadata columns — which is the failure that is invisible until someone queries across the boundary.