The record

The factory keeps one record of what was asked for, agreed, built and shipped. This page is that record's model — the concepts it is made of, what is computed rather than stored, and the rules that keep it honest.

Two databases

The control plane and the warehouse are two different databases, and the split is load-bearing rather than incidental.

The control plane is a single SQLite file — var/dpf.sqlite3 by default, or wherever DPF_DB points — holding every product, requirement, contract, run, release, promotion and the audit log: the record of what was asked for, agreed, built and shipped, and who did each. The warehouse holds the data itself: DuckDB on a laptop, or a customer's Databricks SQL warehouse once one is connected. The warehouse is the seam between the two — one interface, two engines behind it — and this page does not repeat it.

Why apart: the warehouse has to stay disposable. On a laptop, deleting the DuckDB file and running python -m dpf load builds it again from nothing, and that property would not survive the audit trail moving inside it — a warehouse nobody can throw away without also losing the evidence of what shipped is not disposable, whatever the command still does. (The control plane sits in the same var/ folder by default; deleting the folder deletes the record too.) Running two gunicorn workers against one SQLite file is also why the control plane could not simply live inside the warehouse instead: SQLite's WAL mode lets a reader run while a writer holds the file, and a busy_timeout makes the loser of a write race wait rather than fail a request that happened to land mid-approval. DuckDB, deliberately, does not take writes from more than one process — a sound choice for a warehouse engine, and on its own a reason the application's writes could never share its file.

The model, drawn

Everything below lives inside that one SQLite file. The drawing is the shape; the text under it is every fact the drawing shows, stated in words.

A product is the row everything else belongs to: a slug, a name, an owner, and — because every page asks "what is live" — a cached pointer to its current release and its published contract, rather than a question answered by scanning the promotion log each time it is asked.

A product's data contract v0.1 says what it is asking each source system for, and why, in the asker's words — one section per source system, kept in the demands table, each signed by that system's owner. Asking again, or an owner answering differently the second time, writes a new version; neither ever overwrites what was there before.

A requirement belongs to a product and is a stable key, REQ-7, that owns an append-only chain of revisions. repo.save_requirement appends a revision whenever the text actually changed, and writes nothing when it has not — opening the editor and pressing save is not an event on its own. Only three states are ever typed onto a requirement by hand: proposed, accepted, retired. What is computed says what the other three are.

A proposal belongs to a product too, and is what an agent drafted — or what a person wrote into a blank one, on the steps where a model can be skipped entirely. agent_version_id pins exactly which prompt and playbook produced it, or is empty when no model was asked. Before a model answers at all, the model call behind the proposal is already on the record: llm_calls logs the request first and the response after, whichever comes back, including an error, so a crashed call still has a row. See Agents for what each one is shown and why its output is a proposal and never a commitment. A person then accepts or rejects a proposal; accepting the one kind that drafts an agreement is the only way a new contract gets made.

A contract — the data contract a use case team signs, grown from v0.1 — is two documents sharing one hash: the interface a consumer is promised, and the mapping that builds it, pinned to the requirement revisions it satisfies. A draft may be rewritten freely; accepting it assigns a semantic version and freezes the body for good — What never changes says exactly how.

A run builds and validates one contract version: the SQL that actually executed, the table it materialised, and one expectation result row per check the contract declared, each carrying the exact query that ran and how many rows failed it.

A release bundles a contract version with the run that validated it and the changelog between this version and the last, numbered once per product. Its approval is kept as a separate row — who, when, what they said — because a release can sit validated for a long time before anybody approves it, and the two are different facts about it.

A promotion is a release going live: which table it materialised into, which release it superseded, who did it. A rollback is also a promotion — of an earlier release, never a deletion of a later one — and What never changes says more.

A flag is a detected condition, raised against a product: a hash that moved, a type that no longer matches, a criterion nothing verifies. It stays open until a person resolves it, because the condition is gone, or accepts it, because it is still true and they are shipping anyway, in writing. What is computed covers detection itself.

Everything above is written by somebody, and the audit log is where that is kept: one row per transition, naming the action, the object, and the state it moved from and to. It carries two different answers to "who" — actor, the role that was acting, which a demo device may switch between four hats in one sitting, and principal, the person who actually signed in, or the name on the service token that read the API instead. Every four-eyes check in the application compares the second, never the first — see Governance for where that comparison does its work, and Identity for how a person or a token becomes either one.

What is computed rather than stored

Four facts in the record are never written by a person. They are computed every time they are read, from other facts that are.

A requirement's status. Only repo.set_lifecycle ever writes a requirement's lifecycle column, and every caller passes one of three values: proposed, accepted, retired — a person's own assessment, never a verdict about whether something downstream still holds. contracted, verified and live are computed by domain/requirement.status from what the rest of the record says: whether an accepted contract pins this revision, whether every acceptance criterion has an expectation that passed, whether that contract is the one currently live. Nothing can mark a requirement verified while its expectation is failing, because verified is not a column — it is an answer status() gives on demand.

A contract's semantic version. domain/diff.py compares the new body against its parent and classifies every change; next_version reads the number off the result. The classification is about promises, not shapes: removing a column, narrowing a type, moving the grain, or changing what an existing column computes are all major, because the interface a consumer checks can stay identical while every number under it moves — the one kind of change ordinary software versioning has no instinct for. Adding a column or an expectation, or tightening a check, is minor; rewording a description is patch. Before go-live none of that applies: there is nobody to break, so every accepted version is the next 0.x, from the 0.1 Source ended with, and the first promotion publishes the same body as 1.0.0. One change resists all three on principle: a new join is additive on paper and silently doubles every measure the moment it is not actually many-to-one, so diff.py marks it as a risk inside whichever level it lands on rather than filing it quietly under minor.

An artifact's state. clean, patched or stale, computed by repo.artifact_state from two stored texts and a hash. Regenerating a contract's SQL always refreshes generated_text; a hand edit lives in current_text and survives regeneration. The state is stale once the contract has moved past the hash the artifact was generated against, patched when the two texts disagree without that, and clean otherwise. A patch is not forbidden — it is recorded, and read as a bug report against the contract: whatever a person had to add by hand is something the contract's own vocabulary could not say.

Drift. domain/flags.py has no model anywhere in it — every condition it looks for is a comparison, which is what lets it run on a schedule rather than needing a judgement call. It names ten conditions: a source's profile hash moving since the contract was authored (source_drift), a source pinned to no profile at all (profile_unpinned), a generated artifact that predates the contract (artifact_stale) or was hand-edited (artifact_patched), a Databricks constraint the agreement calls for that the warehouse no longer enforces (quality_drift), a contract nobody has ever built (never_built), an acceptance criterion with no expectation naming it (criterion_uncovered) or with one that is failing (criterion_failing), an expectation that names no criterion at all (expectation_orphan), and whole-table conformance (conformance) — itself a family of eight checks in domain/conformance.py: a column missing or unexpected, a null inside the grain, a non-nullable column holding nulls, a declared type that no longer matches what was built, a gap in a monthly series, staleness against a freshness target, and a nullable column that happened to hold no nulls in this build. Seven of those eight can raise a flag; the last one — the nullable column with no nulls — is informational by design, and never does. What never changes covers the two ways a flag is closed.

What never changes once written

Four things, once written, are never rewritten.

A contract's canonical form, and its hash. Contract.canonical() serialises the body with sorted keys and no whitespace, excluding only version and status — the two fields accepting a draft changes without changing what the generated SQL should compute — and content_hash() is the SHA-256 of exactly those bytes. tests/test_contract.py checks both halves: the hash holds still when only version or status move, and it moves whenever meaning does, including through a re-serialisation in a different key order. Traceability lives inside this same hashed body — a column's satisfies, an expectation's verifies — so a contract is self-contained evidence rather than a row that needs a side table to mean anything. Its version changes exactly once after acceptance: the first promotion turns the 0.x it was agreed as into 1.0.0 (repo.publish_contract), the hash staying put and the 0.x kept in the log as contract.published.

A requirement's revisions. repo.save_requirement appends; there is no code path in the application that updates one. A contract accepted against revision 3 still names revision 3 after revision 4 is written, because "what did this column exist to satisfy, at the time" has to stay answerable.

The audit log. Nothing in the application issues an UPDATE or a DELETE against audit_log — not a database trigger, a test. tests/test_append_only.py greps the whole source tree for exactly those statements on every run, and separately replays the things that most look like editing history — revising a requirement, superseding a contract, rolling a release back — to check that every earlier row survives, word for word.

A promotion. Going live writes a new promotions row; rolling back writes another one, naming the release it restores, pointed at the release it supersedes — never a deletion of that release, and never an update to the row the rollback replaces. pipeline.rollback also refuses to restore a release that was never approved, or one that was never actually live: a restore that restores something that never happened is exactly the sentence an append-only record is not allowed to contain. tests/test_pipeline.py::test_rollback_restores_both_halves_exactly is the proof.

Transactions

One connection per request. create_app registers a hook that opens g.conn before any view runs; its teardown hook commits if the request ended cleanly and rolls back if it raised, then closes the connection either way — so a promotion that fails halfway through never leaves the record claiming something is live that was not. The CLI gets the same guarantee a different way: every command opens its connection through db.session, a context manager with the same commit-on-clean-exit, rollback-on-exception shape.

pipeline.promote is the clearest reason this matters. It writes to the warehouse first — a different engine, outside any SQLite transaction — and only once that succeeds does it touch the record at all: one call, repo.record_promotion, inserts the promotion row, moves the previous live release to superseded and the new one to live, repoints the product's live-release and published-contract pointers, and appends to the audit log. None of those is committed on its own; they ride on the same connection as the rest of the request and land together, or — if anything else in the request raises afterwards — not at all.

Migrations

The schema evolves by adding, never by dropping. db.init runs CREATE TABLE IF NOT EXISTS for the whole schema and then db._migrate — on every connection the application opens, which means every CLI command and every web request, not once at deploy. _migrate is necessary at all because CREATE TABLE IF NOT EXISTS does nothing to a table that already exists: a workspace running since before a column existed would otherwise just lack it. It walks a fixed table of columns added after the first release, and for each one still missing from a given table runs ALTER TABLE ... ADD COLUMN; nothing in it ever drops a column or a table. A short list of backfill statements runs alongside it, repairing rows written before a column existed from whatever they already carry — written as SQL rather than run once by hand, because a database restored from an old backup hits the same gap a live one has already closed, and migrating on every connection rather than once at startup is what lets it close there too. tests/test_db.py::test_init_is_idempotent checks the whole thing is safe to run twice: write a row, reinitialise the same file, and find the row still there.

Exports and backups

dpf export writes every one of a product's current documents — the brief, the requirements, each section of the data contract v0.1, the contract — as markdown, one file per document plus an index, to exports/<product> by default (--to names another folder). The database holds the current state; the export is what persists outside it, in a form a reviewer or a future reader does not need this application open to read.

That covers a product's documents, not the record itself. Backing up and restoring the control plane — the one file this whole page describes — is Back up and restore; the warehouse needs none of that, for the reason Two databases already gives.