Components

Where each part of the factory lives, what it is responsible for, what it must never do, and where a new change belongs — checked against the code, and held there afterwards by tests/test_architecture.py.

The four layers

Three ways in — the web application, the shell, and MCP on stdio — call pipeline.py, and nothing outside those three (plus the two scripts they each run, demo.py and showcase.py) imports it: the work that spans systems, from drafting with an agent to building in the warehouse to promoting a release, has exactly one function to call. Not every write goes through it. A read goes straight from a view or a CLI command to an adapter — repo.py for a list of products, warehouse.py for a sample of rows — and so does a decision that is one write to the record: a source owner's signature on a section of the data contract v0.1, a sign-off, a decision on a flag is a repo.py call from the view, with its permission and four-eyes checks in the view and its audit entry in the same call. The four adapter groups sit below it: the control plane, the warehouse seam, the model, and the outside connectors. domain/ sits below all of them and imports none of them — pipeline.py itself calls into domain/ directly as often as it calls an adapter, for the same reason a cashier and the till both read the same price list: wiring the two together, not choosing between them, is what the layer above does.

The ways in

dpf/web/ is the Flask application — server-rendered, no build step. create_app does, in order: construct the Flask app; resolve SECRET_KEY (refusing to start with none under DPF_ENV=production); cap request bodies at 2 MB; apply ProxyFix; wrap wsgi_app under DPF_PREFIX if the deployment is mounted on somebody else's domain; validate the identity configuration (_one_door, refusing a combination with no single right reading); install the token door (machine.install) and then exactly one people door (gate.install or entra.install); register the four blueprints; register the Jinja filters; and install the request hooks below. See Identity for what the doors (gate.py, entra.py, accounts.py, machine.py) actually decide — this page only says where they plug in.

The four blueprints:

Blueprint	Module	What it serves
views	web/views.py	Every page: thin — gather, render, redirect.
api	web/api.py	One JSON endpoint that matters: the published contract.
api_v1	web/api_v1.py	The headless read API, and MCP over HTTP at /mcp.
docs	web/docs_web.py	This documentation, at /docs — public, reads no record.

Three smaller modules round out the package: urls.py holds the one check every redirect uses to decide where it is safe to send somebody — see Security rather than a second explanation here; filters.py is formatting only, Jinja filters that decide nothing; repo_counts.py is the one query the sidebar's badges need, kept in its own module so importing the app factory never imports repo.py at module scope.

The request hooks, structurally — the request's own order is the runtime page's story, not this one's:

• before_request installs two: _same_origin refuses a cross-site POST, PUT, PATCH or DELETE; _open opens g.conn, resolves g.identity, g.principal and g.user.
• after_request (_cache_policy) sets caching headers — long for a fingerprinted static asset, none at all for anything that reads the record.
• teardown_request (_close) commits g.conn on a clean exit and rolls it back on an exception, then closes it.
• Two context processors add template globals: _who (the signed-in person, their role, can, rbac, the app version); _nav_counts (the sidebar's badge counts, via repo_counts.py).

cli.py is thin on purpose — a few lines of argument handling per command around a pipeline function, plus console.py for dependency-free terminal output. One subcommand per phase of the process (load, profile, contract, quality, …) and a handful of operational ones (connect, status, roles, reset, token, user, catalog, export, workflow, trace, flags, serve, release, docs, mcp), plus demo and showcase, the two worked examples that call pipeline.py the same way the web layer does. See the CLI reference for every flag.

mcp.py is a five-tool, read-only MCP server — search, describe, facet, gaps, products — hand-rolled JSON-RPC over stdio rather than the mcp package, because initialize, tools/list and tools/call are the whole protocol surface a read-only server needs. TOOLS describes each tool for a model choosing between them; HANDLERS maps a tool's name to the function that answers it; handle() dispatches one JSON-RPC message to one reply. The same handlers answer over HTTP, too: api_v1.mcp_http calls mcp.handle inside mcp.borrowing(g.conn), a context variable that lets a handler read the request's own connection instead of opening one — a tool added here is reachable on stdio and at /mcp without a second implementation.

The one path for work that spans systems

pipeline.py is where the control plane and domain/ meet: every operation that must leave one trail has exactly one function here, called by the web layer and the shell alike. By phase: load/profile (Source); import_requirements, pin_profiles, draft, compose, accept (Refine); build, cut_release, approve, promote, rollback (Deliver — build generates, materialises and runs every expectation; the rest cut, approve, promote or roll back the release it gates). Two more run outside that sequence, once something is live: push_quality/check_quality/ quality_target push the agreement's checks into Databricks and read them back, and detect_drift runs every drift detector. A second group answers questions rather than changing anything — coverage, criterion_checks, requirement_board, criteria, ontology — because they, too, join a stored fact to a domain/ computation and have nowhere else to live that would not duplicate the join.

What it must not do: decide anything itself, or write SQL of its own. A decision — what a semver bump should be, whether a release may promote — is answered by calling into domain/; a persistence detail is answered by calling repo.py or warehouse.py. pipeline.py wires the answers together and writes the audit trail; it does not originate either.

The adapters

The control plane

repo.py holds the queries for the product record — products, sources and profiles, demands, requirements, contracts, artefacts, runs, releases, promotions, settings, documents, descriptions, agent reference files, people. db.py owns the SQLite schema, its migrations, the session helper, and now() — a clock that a test can shift and that holds still within one audit entry. audit.py is one function, record(), called from every state transition, plus history() to read it back.

What it must not do: decide anything. accept_contract derives index tables from a contract body and requirement_evidence assembles facts for domain/requirement.py to turn into a status, and neither one is allowed to be more than that — the policy stays in domain/, where it is pure and testable without a database.

Two tables are a deliberate, narrow exception to the rule that every query lives in repo.py: agent_versions (written by roles.py) and llm_calls (written by llm.py), each a self-contained log with nothing in the rest of the schema to join against, and each committed on its own rather than on the request's transaction — so a call that never returns still leaves a record of having been asked.

The warehouse seam

warehouse.py is the interface — connect, execute, scalar, list_tables, list_schemas, table_exists, columns, row_count, profile_table, sample_rows, distinct_values, materialise, drop, dialect, seedable, describe — module-level functions, a connection first, plain Python out. warehouses/ holds the implementation, one module per engine (duckdb.py, databricks.py), each built to the same adapter shape, selected by DPF_WAREHOUSE_URL.

What it must not do: nothing above this seam may import a database driver — not pipeline.py, not the web layer, not the CLI. See The warehouse for read-only mode, what a dialect switch does and does not translate, and how far each engine is proven; this page only places it in the stack.

Talking to the model

llm.py is the one place this application calls Anthropic — a single function, run(), that logs to llm_calls before it returns, including on failure. The system prompt it sends is the pinned agent version's text, verbatim, read through roles.py; it is never assembled here. agents.py holds one function per drafting step (assess_feasibility, draft_requirements, draft_glossary, design_vault, draft_gold_model, review_sql, draft_contract, draft_demand, triage), each rendering its own grounding and calling llm.run with a schemas.py class as the expected shape — Pydantic, kept at the wire boundary, mirroring rather than replacing the frozen dataclasses in domain/. privacy.py is what every one of those grounding renderers must pass profiles and samples through first: redact_profile and redact_samples, governed by DPF_LLM_DATA.

What it must not do: commit anything. Every function in agents.py returns a proposal; a person accepts it, and only then does pipeline.py persist it — see Agents for what each step is shown and produces, and what the three privacy settings actually send.

Connectors

trackers/ turns an accepted agreement's build plan into tickets — choose()/backend() select azure_devops.py or jira.py by DPF_TRACKER, or by whichever has credentials, and push() mints a plausible local number when neither does. catalog.py defines a Catalog protocol (search, asset) and connect() picks an implementation from the CONNECTORS registry — FileCatalog, Informatica, UnityOverSQL — by DPF_CATALOG. itsm.py raises an access request in ServiceNow, raise_request(), from the same section of the data contract v0.1 rather than becoming one.

What they must not do: reach into the record or into domain/ on their own — none of the three imports repo.py, pipeline.py or anything under domain/. Each is called from pipeline.py or a view with whatever it needs already assembled, and hands back a plain result; how proven each one is (live against a real tenant, or a mock-up written to the vendor's published API) is connections.py's to report on the Connections page, not theirs to claim.

The pure core

domain/ is pure: no database, no Flask, no network. Time is passed in rather than read, with one deliberate exception — a freshness check in conformance.py asks for today's date, because "how old is the newest row" has no other honest reference point. Nothing in the package imports dpf.web, dpf.repo, dpf.db or dpf.warehouse, or anything outside itself under dpf; rows arrive as frozen dataclasses and plain dicts, and decisions leave the same way. One rule looks like an exception and is not: generate.py and expectations.py emit SQL strings and never touch a connection — the rule that every query lives in repo.py governs how this application reads and writes its own state, and the warehouse SQL these two produce is output data, the artefact the factory exists to produce, which is exactly why keeping its construction here — unit-testable, no database required — matters.

Module	What it does
conformance.py	Whether the table that got built matches the contract that described it — a table-level check beside expectations' row-level ones.
contract.py	The data contract: interface and mapping in one file, in canonical form, under one content hash.
cycle.py	How long an artefact took, and who it was waiting on.
description.py	What a column means to the business, and whether that meaning has been signed off — kept apart from the warehouse's technical facts.
design.py	The product itself: its tables, their columns, and which columns are keys.
dialect.py	What DuckDB and Databricks spell differently in SQL, written down once rather than kept as two copies.
diff.py	Two contracts in; a semver level and a changelog out.
document.py	Every artefact — brief, requirements, demand, contract — as one versioned document shape.
events.py	An audit code, rendered as the sentence a reviewer reads.
expectations.py	Every expectation kind compiled to the same one-shape failing-row-count query.
flags.py	Drift detection: arithmetic over hashes and counts, no model.
flow.py	The workflow itself — phases, steps and gates — read from dpf/workflow.yaml and validated at startup.
generate.py	Contract in, SQL out — deterministic, with no model anywhere near it.
glossary.py	The glossary: every term defined once, symmetric with the requirements that use it.
model.py	The medallion tiers — bronze, silver, gold — and the DAG between them.
ontology.py	One graph over every artefact that already exists, for search and for MCP.
profile.py	A profile's numbers, turned into observations an authoring agent or a drift check can act on.
promise.py	The design, projected into a contract — never retyped by hand.
quality.py	Acceptance criteria translated into checks Databricks itself can enforce.
rbac.py	Who may decide what — see Governance.
release.py	The release state machine and the gate that decides what may go live — see Governance.
requirement.py	Requirements, their revisions, and the status nobody types in directly.
scope.py	A draft data scope: the subjects a brief implies, before anybody has chosen tables.
status.py	One four-word vocabulary for "where is this", used on every page.
tour.py	The guided tour, as data checked against the real routes it walks.
trace.py	The traceability matrix and the coverage report.
vault.py	Data Vault as the silver layer, compiled down to ordinary model specs.

Configuration

envvars.py is the one registry of every environment variable the application reads: a frozen Var per name (its group, a sentence, its default, whether it is secret, its closed set of values if it has one, when it is required), collected in ALL. referenced() scans dpf/**/*.py and wsgi.py as text for every name actually read, so the registry cannot silently drift from the code — see the dependency rules below for the test that holds it to that. settings.py is the subset a person may type into the Connections page instead: SETTABLE names which ones, SECRET names which are write-only and never rendered back, and merged()/into_environment() enforce one precedence throughout — a deployment's own environment always wins over a value typed into a form. connections.py reports what every one of those settings connects to and how far it is proven (LIVE or MOCKUP), for the one page that answers "what does this reach out to, and which of those have you actually run?"

Agent definitions

roles.py turns a file into a versioned row: load() parses one dpf/roles/<key>.md's frontmatter (key, name, model, and optionally role, effort, output) and splits its body on a ## Playbook heading into the role's instructions and its house rules; register() gives every file a current agent_versions row, minting a new version only when content_hash() — over behaviour, not byte layout — has actually changed. Editing how an agent thinks is a diff to a markdown file in a pull request, not an UPDATE nobody can see. See Agents for what each registered role is shown and asked to produce.

The documentation engine

docs.py is this page, and every other one: pages() and nav() read docs/nav.yaml; generate() rewrites a <!-- generated: NAME --> block from docs_reference.py; fingerprint()/stamp() hold a page's covers to a hash of those files' contents, and check() runs every rule at once — front matter, drift, links and anchors, drawings, the stray heading, generated staleness — which is what python -m dpf docs check runs. docs_reference.py holds the six generators (configuration, cli, api-endpoints, events, permissions, workflow), each building its block from the code it describes rather than from a copy of it. docs_diagrams.py is the drawing language this very page uses: parse() and layout() turn a diagram fence into positioned nodes and routed links, render() into the SVG on the page, and problems() into what docs check reports — including the one check that matters most here, that a node's code: still names something that exists. version.py reads VERSION once (current(), build(), label()) so the application, /docs's own header and python -m dpf --version can never disagree. release.py is python -m dpf release: it moves CHANGELOG.md's Unreleased section into a dated one, bumps VERSION to match, and refuses to run against an empty Unreleased, because a version number claiming nothing changed is a version number the changelog cannot back.

The dependency rules, enforced

Three rules, stated here and held by tests/test_architecture.py so this drawing cannot quietly become false:

1. domain/ depends on nothing above it. Nothing under dpf/domain/ imports from elsewhere in dpf, or imports sqlite3, duckdb, databricks, anthropic, requests, urllib, httpx or flask.
2. pipeline.py has exactly the callers this page says it has. Only dpf/web/, dpf/cli.py, dpf/mcp.py, and the two scripts they run (dpf/demo.py, dpf/showcase.py), import it.
3. Each vendor SDK has exactly one home. duckdb is imported only from dpf/warehouses/duckdb.py; databricks only from dpf/warehouses/databricks.py; anthropic only from dpf/llm.py.

A pull request that adds a fourth importer of pipeline.py, has domain/ reach for repo.py, or lets a second module import duckdb fails this test before it fails a review.

Where a change goes

To add	Touch	Then
A warehouse engine	A new dpf/warehouses/<engine>.py built to the adapter shape in warehouses/__init__.py's docstring; a branch in its parse()/connect(); an entry in domain/dialect.py for what it spells differently.	Keep the vendor SDK import inside that one new file — the same rule tests/test_architecture.py holds duckdb, databricks and anthropic to. tests/test_warehouses.py exercises the adapter contract.
A tracker	A new dpf/trackers/<name>.py; a branch in trackers.choose()/backend(); its name added to DPF_TRACKER's values in envvars.py.	tests/test_trackers.py.
A catalogue connector	The Catalog protocol (search, asset) implemented somewhere; a Connector(...) entry in catalog.CONNECTORS naming what it needs and how proven it is.	tests/test_catalog.py; any new setting it reads must be in envvars.py or the next rule fires.
An agent	A new dpf/roles/<key>.md with frontmatter and a prompt; a function in agents.py calling llm.run(conn, key, …) with a schemas.py output shape; a step in dpf/workflow.yaml if it should appear in the flow.	roles.register() picks the file up with no further wiring. tests/test_roles_domain.py, and tests/test_workflow_config.py if it is also a step.
A workflow step or phase	dpf/workflow.yaml only.	domain/flow.validate() refuses a missing step, an unknown phase or a dependency out of order at startup, and tests/test_workflow_config.py exercises each way it can be wrong.
A page	A view in web/views.py calling the same pipeline function the CLI would; a template; an entry in views.EVIDENCE_TABS/EVIDENCE_ENDPOINTS if it is a step's evidence page.	tests/test_forms_render_what_they_read.py fails if the view reads a form field the template never renders; tests/test_reads_as_english.py if a raw key or path reaches a position a reader expects a name in.
A read-API endpoint	A view function, plus one Endpoint(...) row in web/api_v1.ENDPOINTS naming it.	tests/test_api_v1.py holds the route table, the OpenAPI spec and the generated reference page to that one list — an endpoint missing from it does not exist; one in it with no matching function fails at import.
An MCP tool	An entry in mcp.TOOLS (name, description, input schema); a handler function; its name added to mcp.HANDLERS.	tests/test_ontology.py asserts the tool names and the handler names are the same set. Reachable on stdio and at /mcp together, with nothing further to wire.
A setting	A Var(...) in envvars.ALL; its name added to settings.SETTABLE (and settings.SECRET if it authenticates something) if a person should be able to type it in.	tests/test_envvars.py fails the moment the code reads a name this registry does not know, or names one the code no longer reads.
An audit event	audit.record(conn, "object.verb", "object_type", …) at the point of the change; a sentence in domain/events._PHRASES if the code is shown on a phase trail.	tests/test_events.py fails for a trail-visible code with no sentence; anything else still reads, just as its own words rather than a written one.
A permission	The decision's code in domain/rbac.OWNS, naming which role(s) hold it; a sentence in rbac.MEANS.	A code left out of OWNS is refused rather than granted — rbac.can only allows what is named. docs_reference.permissions() renders whatever MEANS has, blank if it is missing, so a reviewer reading the permissions reference is what catches an empty one, not a test.