Expand description
DQ audit-trail record shape.
The shared contract between dq-enforcer (produces verdicts) and the sinks
(persist them). Scaffold only.
An audit record is per-record, per-rule (full grain): every record’s verdict on every rule it was evaluated against. To count as an audit trail it carries:
record_id— the deterministic record id (column named under the configured metadata prefix), the join key back to rawrule_id— stable rule identifierrule_version— and the contract/rule version, since rules evolve; a verdict against an unversioned rule is worthless once the rule is redefinedrule_source— manual | odcs | merged (which authority defined it)verdict— pass | faildisposition— clean | quarantined | annotated | dropped (the outcome actually applied, not just pass/fail).droppedmeans the record matched a drop rule — a known, intended discard, distinct fromquarantined(failed a quality rule) and from a decode failure. Dropped records go nowhere: not to a clean sink, not to the DLQ (that is the point — no DLQ noise from test traffic / heartbeats).- provenance — source locator, ingest timestamp, producing pod/pipeline
The trail is append-only by construction and routed to a first-class sink
(receives = "dq_results"), so it has the SAME durability and recovery
guarantees as the data it describes — it is data, not telemetry. On by
default; disableable for non-regulated deployments.
Drop audit is OFF by default: a dropped record leaves no trace, because the
versioned drop rule is itself the audit artifact — “we discard all records
matching this versioned rule” is provable from contract/config history
without a row per record. Per-drop-rule audit = true opts a rule into
per-record drop rows where individual proof is wanted; the high-volume noise
case (heartbeats) stays silent.
Volume note: full grain is N_records × N_rules rows, typically an order of
magnitude more than data rows, so the default target is a compact columnar
sink (Delta/Iceberg/Parquet) and the record is narrow (ids + codes, no
payload).