Thalweg · API reference ← Main docs

Module databricks

Module databricks 

Source
Expand description

The real Databricks Arrow-Flight adapter (behind the databricks-sdk feature). It implements ZerobusTransport/TransportFactory over the databricks-zerobus-ingest-sdk ZerobusArrowStream.

The version bridge (ADR-0065): thalweg is on Arrow 58, the SDK on Arrow 59. Batches cross as Arrow IPC bytes via ingest_ipc_batch, so no Arrow-59 array/batch type is ever handled here — only the stream’s schema is rebuilt into the SDK’s Arrow-59 Schema at open time (to_sdk_schema).

Replay-on-failure is delegated to the SDK’s own recovery (.recovery(true) on the stream): the SDK reconnects and re-ingests unacknowledged records internally, so the sink’s pool-level rebuild only needs to reopen a fresh stream — it never re-submits batches (see pool::StreamPool::rebuild).

Structs§

DatabricksFactory
Opens Zerobus Arrow-Flight streams against a Databricks workspace.

Functions§

to_sdk_schema
Rebuild our Arrow-58 schema into the SDK’s Arrow-59 Schema. Only the column types are mapped; the array data never crosses the boundary (it goes as IPC bytes), so this covers the sink-supported scalar/temporal set.