Enforce the agreement in Databricks
Compile a data contract's checks into what Databricks can actually enforce, instead of leaving them to run only here, and optionally push them onto the table.
Before you start
- A published data contract — checks compile from an accepted one; there is nothing to push before that.
- A Databricks connection, if you intend to install or check rather than only read the generated artefact — see Connect Databricks.
- To install: the connection's token needs
USE CATALOG,USE SCHEMAandMODIFY, plus ownership of the table — Unity Catalog reservesALTER TABLEfor the owner. - The
release.buildpermission.
Steps
Either from the product's Quality evidence tab, or the command line — both read the same agreement and agree on what it compiles to.
In the app: open the product, then Quality (/p/<slug>/quality). Switch between the three targets; the report and the generated artefact are pure functions of the agreement, so all three are shown whatever the warehouse is. Only Delta constraints has buttons: Install these on <table> applies the constraint set, and Check what the table enforces reads it back.
From the command line:
python -m dpf quality # constraints, with the report
python -m dpf quality --target dlt --on-error drop # a DLT module for their repo
python -m dpf quality --target monitor # custom metrics for the SDK
python -m dpf quality --push # apply the constraint set
python -m dpf quality --check # read back what is enforced
Note
Install and check run through the same Databricks connection the Connections page lists as a mock-up: tested as text, never yet run against a real workspace.
Every run prints the faithfulness report before the artefact: which checks the chosen target carries in full, which only partly, which not at all, and why. A customer who believes a check runs in their warehouse when it does not is worse off than one who knows it runs only here, so nothing is dropped quietly — a check no target can carry is named in red.
Check that it worked
Installing reports the constraint names actually applied; it is not transactional, so a set that fails halfway leaves the earlier ones installed. Checking reads the constraints back from the table's Delta properties and reports drift four ways: missing (never pushed, or dropped by hand), changed (edited in place), extra (from a superseded agreement) and unmanaged (somebody else's). Everything this application installs is prefixed dpf_.
Note
Expect the first install against a table with any history to be refused — ADD CONSTRAINT validates every row already there, and real tables usually hold one that breaks the rule.
Which target to choose
| Delta constraints | DLT expectations | Lakehouse Monitoring | |
|---|---|---|---|
| Enforced by | the engine, on every write | the DLT pipeline, and nothing else | nobody — it observes |
| On a violation | the whole write fails | fail, drop the row, or record it | a number moves |
| Carries severity | no — one, and it is fatal | yes | n/a |
| Right for | not_null, range, accepted_values |
any row rule, on DLT or Lakeflow | null_rate, row_count, trends |
What is not supported, and why
foreign_keycannot be pushed to any target. A CHECK cannot reference a second table, a DLT expectation is a predicate over one row, and a monitor aggregates over one table alone. Referential integrity stays a check this application runs.uniqueandrow_countcannot be constraints or DLT expectations — neither is visible from one row. Monitoring carriesrow_countexactly anduniqueonly approximately.null_rateis a proportion, so it belongs to monitoring and nowhere else.- A
warnis never pushed as a CHECK constraint — a CHECK has one severity, and it is fatal to the write. - A threshold above zero is never pushed as a CHECK constraint either. DLT gets it instead, as a recording
expect. expressionis emitted verbatim. It is authored SQL; Delta refuses a CHECK that aggregates, subqueries or names another table, and theALTERis where that is found out.
If it does not work
| Symptom | Fix |
|---|---|
| "this Databricks connection is read-only, so it will not …" | Expected on a SELECT-only token — see Connect Databricks. |
ADD CONSTRAINT refused on the first install |
A row already breaks the rule. Fix the data or the check, then push again — drop-first makes a second push safe. |
| "there is nothing to push for --target dlt/monitor" | Only constraints install from here; the DLT module and the monitor definition are files for the customer's own pipeline repository and SDK. |
Related
- Connect Databricks — the connection this pushes through.
- Command line — every flag.