The warehouse

The warehouse is the half of the two databases that holds data rather than decisions, and it is built to stay disposable: rm -rf var/ and python -m dpf load rebuilds it byte-identical. That has to remain true, because the moment the warehouse holds something the record cannot afford to lose, it has stopped being disposable at all.

The seam

Nothing above warehouse.py imports a database driver — not pipeline.py, not the web layer, not the CLI. The file is the interface, exactly as it has always been: module-level functions, a connection as the first argument, plain Python out. The implementation lives in warehouses/, one module per engine, and warehouse.py only forwards. Deliberately not a plugin framework: there is no registry to register into and no base class with a dozen abstract methods, because two engines do not justify that machinery — the day there is a third, the one if that chooses between them grows a branch, which is a smaller change than unpicking an abstraction would be.

DPF_WAREHOUSE_URL decides which engine, and setting nothing means DuckDB at DPF_WAREHOUSE — exactly what happened before warehouses/databricks.py existed, so no configuration means no change. Connecting is read-only and deliberately cautious: it reads the configuration before dialling anything, so a variable nobody set comes back as a named variable and a sentence rather than a connection error, and it creates nothing, so it is safe to point at production — a wrong hostname fails in about twenty seconds rather than the connector's default ten-minute retry. Pointing it at a real warehouse is Connect Databricks.

Read-only mode

DPF_DATABRICKS_READ_ONLY=1 is the configuration a first pilot actually gets: a token with SELECT on one schema and nothing else. It is supported rather than tolerated.

Works. Connecting, loading (which registers the tables that are already there instead of building a fixture — see below), profiling, table sampling and distinct-value counts, the whole source and catalogue side of the web app, authoring requirements, and drafting and accepting a data contract against real profiles.

Refused, with a message that names the missing grant. Build, promote and rollback. The refusal happens in the adapter before anything is sent, so you get "this Databricks connection is read-only, so it will not build dev.x. That needs CREATE TABLE and MODIFY on main" rather than a driver error four steps later. Promotion returns it as a failed gate; the release stays approved and unpromoted.

To build, the token needs USE CATALOG, USE SCHEMA, CREATE TABLE and MODIFY on the target schemas, and the dev and prod schemas have to exist — this application will not create schemas in somebody else's catalog, and says so when they are missing.

What changes when the warehouse is not DuckDB

• Loading does not seed. The synthetic dataset in seed.py is written in DuckDB's COPY … NULLSTR dialect over a local CSV file, which a SQL warehouse cannot see. Against Databricks, Load registers the tables already in the schema and counts their rows. No source system is attached to them, deliberately: this application does not know who owns a table it found by looking, and inventing an owner would put a name on a data contract v0.1 that nobody agreed to.
• dpf demo and dpf showcase stay on DuckDB. Both build their own fixture.
• Types are translated in one place (domain/dialect.py) and the translation is lossy in ways that are written down there rather than discovered later. The headlines: VARCHAR(n) becomes an unbounded STRING and stops being enforced; DuckDB's naive TIMESTAMP becomes Databricks' zone-aware one, so the same column read through two session time zones profiles differently; TIME, UUID and JSON have no Databricks equivalent and are carried as text; unsigned integers are widened.
• Contract expressions are not translated. A mapping is authored SQL. Identifier quoting and the CAST type names are switched for the target engine; date_trunc(…) happens to work on both, and anything DuckDB-specific in a filter or a join predicate is the contract author's to fix. This is a dialect switch, not a transpiler, and calling it one would be the dishonest version.

How far it is proven

The adapter is listed on the Integrations page as a mock-up, which is that page's word for "written to the vendor's published API, never run against a tenant". The SQL it emits is tested as text; the profile body it produces is tested to be identical, key for key, to the one DuckDB produces for the same table; the type mapping is tested in both directions. None of that is the same as having authenticated against a real workspace, and the badge stays where it is until somebody has. pytest carries one integration test that skips itself unless the warehouse variables are set.

The same is true of the quality push, and more sharply: the generated DDL, DLT module and monitor definitions are asserted as text for all eight expectation kinds, and the push and the read-back are tested against a connection that records what the adapter sends — but no ALTER TABLE … ADD CONSTRAINT has ever reached a Delta table from here.