How-to guide For operators

Connect Databricks

Point the build, the profiles and the data catalogue at a Databricks SQL warehouse, instead of the DuckDB file that ships by default.

Important

The Databricks adapter is a mock-up: written to Databricks' published API and tested as text, but never yet run against a real workspace. Treat the shape as a proposal until the Connections page shows it as live — see Pointing it at your own warehouse.

Before you start

A Databricks SQL warehouse — not a general-purpose cluster — and its HTTP path, from its Connection details tab.
A personal access token, or a service principal's. SELECT on one schema is enough to start; read-only is a supported configuration, not a degraded one.
The Administrator role, if you set this on the Connections page rather than in the environment — only an Administrator may save a connection.

Steps

Set the connection, one of two ways.

As environment variables, for a deployment:
```
export DPF_WAREHOUSE_URL=databricks://
export DPF_DATABRICKS_HOST=adb-1234567890.7.azuredatabricks.net
export DPF_DATABRICKS_HTTP_PATH=/sql/1.0/warehouses/abc123
export DPF_DATABRICKS_TOKEN=dapi…
export DPF_DATABRICKS_CATALOG=main
export DPF_DATABRICKS_SCHEMA=bronze       # optional: the session default
export DPF_DATABRICKS_READ_ONLY=1         # optional: see below
```
Or on the Connections page, to try it from the browser: click Databricks SQL, fill in the host, HTTP path, token and catalog — schema and read-only are optional — then select Save and build into it. That writes DPF_WAREHOUSE_URL for you: every later build, promotion and drift check runs against this connection, and nothing already built moves. If DPF_WAREHOUSE_URL is already set in the service's own environment, the sheet is locked; edit the environment file and restart instead.
Run .venv/bin/python -m dpf connect, or select Test on the sheet. It reads the configuration before it dials anything, so a variable you have not set is named with a sentence explaining it, rather than a connection error. A wrong hostname fails in about twenty seconds rather than the connector's ten-minute default.

Check that it worked

dpf connect prints connected to <catalog>, then the schema count, the table count, and permitted or refused — read-only for writes. Nothing it runs creates a schema, a table or a row. On the Connections page, Test shows the same answer in one line, and a connection in use carries the In use badge.

Read-only mode

DPF_DATABRICKS_READ_ONLY=1 (or read only: yes) is what a first pilot actually gets: SELECT on one schema and nothing more.

Works: dpf connect, dpf load (registers the tables already there instead of building a fixture), dpf profile, sampling and distinct-value counts, the whole source and catalogue side of the web app, authoring requirements, and drafting and accepting a data contract against real profiles.

Refused, naming the grant: build, promote and rollback. The refusal happens before anything is sent to Databricks, with a sentence such as "this Databricks connection is read-only, so it will not build dev.x. That needs CREATE TABLE and MODIFY on main" — not a driver error afterwards.

To build, the token needs USE CATALOG, USE SCHEMA, CREATE TABLE and MODIFY on the target schemas, and the dev and prod schemas have to exist already: this application will not create a schema in somebody else's catalog, and dpf connect names whichever one is missing.

Unity Catalog as the catalogue

Set DPF_CATALOG=unity to make the data catalogue read Unity Catalog through this same connection, instead of the bundled JSON export. It needs no credential of its own beyond DPF_DATABRICKS_HOST and DPF_DATABRICKS_TOKEN — one connection answers both where the data is and what is known about it. See Configuration.

If it does not work

Symptom	Fix
`dpf connect` lists variables under "not configured yet"	Set the named ones; the sentence beside each says what it expects.
It hangs, then a network error	Check the hostname for a typo — it fails in about twenty seconds rather than ten minutes, but it still has to time out first.
"this Databricks connection is read-only, so it will not build …"	Expected on a SELECT-only token. Get `USE CATALOG`, `USE SCHEMA`, `CREATE TABLE` and `MODIFY`, and unset `DPF_DATABRICKS_READ_ONLY`.
"there is no `dev` schema in `<catalog>`"	Create the schema and grant `CREATE TABLE` on it — this application will not create one in somebody else's catalog.

Pointing it at your own warehouse — what changes when the warehouse is not DuckDB.
Configuration — every variable and its default.
Command line — dpf connect's full output.
Enforce the agreement in Databricks — once connected, push the agreement's checks onto the table.