Govern APIs, Data Products and Retrieval Agents Can Trust
Turn interfaces and data into owned, versioned capabilities that applications and agents consume under identity, entitlement and evidence controls, and give agents access to meaning, limits and authority, not only bytes.
- Version
- 0.2.0
- Owner
- DEIXEI
- Last reviewed
- Reading time
- 27 min
For Cto · Ai Architect · Platform Engineer · Data Leader · Security Engineer · Risk And Assurance · Internal Audit
Sources and details
- Section
- Build the platform
- Source works
- Book 1 · Book 2 · Book 3 · Supporting Tools · Research Series · Book 09
- Source chapters
- 3, 4, 10, 12, 13, 16
- Source appendices
- 01:B
- Tool specifications
- audit_evidence_schema@0.1.0
- Refresh trigger
- source change
Applications and agents need dependable ways to use the firm’s capabilities and data. An API is a boundary of accountability: it identifies an owner, a contract, an authorisation model, a lifecycle and the evidence a consumer can rely on. A reachable database table is not a governed data product, and a retrieval index is not a substitute for source entitlements.
Agents raise the bar in a specific way. Access to data is a solved problem in most firms; access to meaning is not. What does this column represent, in what unit, is this the master or a copy, how fresh is it, what is known to be wrong with it, and may an agent use it at all? A tenured employee answers these tacitly. An agent does not ask; it proceeds (Book 1 Ch 16 §4.2). This page sets out the artefacts that make meaning, limits and authority machine-readable, and the controls that stop agents treating plausible output as record.
Standing of the content. The six-layer model, the operational antibodies, the 17-field dictionary, the Authority Process, accuracy thresholds and sizing figures are author proposals and experience-based estimates from Book 1. The minimum data estate and the reuse rules are author recommendations from Book 2. Bank A, Asset Manager X and Insurer I are composite institutions; the Book 09 material is a composite narrative. Classification, lanes and inheritance rules are on data classes and lanes; legal duties are on the regulatory map.
Why agents make the abstraction gap consequential
Every schema is a snapshot of the business at the moment it was built, and the business drifts faster than the schema is revised. Definitions change at the speed of policy, schemas at the speed of releases, master data at the speed of stewardship. The most hazardous form is a schema that looks updated while the meaning has not moved (Book 1 Ch 3 §4.2). Humans compensate silently with tacit knowledge. Agents reason over the abstraction as if it were reality, and three properties make that consequential (Ch 3 §4.3):
| Property | What happens | Control that answers it |
|---|---|---|
| Scale | A systematic misreading becomes a systematic misaction across millions of records | Record-level metadata and refusal rules; accuracy thresholds |
| Plausibility | Fluent, consistent, confident output bypasses lay review | Known-limitations register read before acting; authoritativeness field |
| Composability | One agent’s misread becomes the next agent’s premise, laundered through layers of reasoning | Lineage queryable at runtime; the Authority Process before anything becomes record |
An agent reading a stale or drifted store cannot tell from the data that it is stale; nothing in a schema announces its own obsolescence. The remedy is not a better abstraction (the next migration produces its own gap) but making the gap legible to the agent (Ch 3 §3).
Separate six data-governance layers and keep the flow one-way
Book 1 Ch 16 §4.0 identifies confusing adjacent layers (the data product with the retrieval index, agent memory with the authoritative record) as the most common cause of agents inventing facts the firm then has to defend. Give each layer one owner and one governance artefact.
| Layer | What it is | Owner | Governance artefact |
|---|---|---|---|
| Source system | System of record where data originates | System owner (business area) | Source-system change management |
| Data product | Curated, documented, versioned dataset from one or more sources | Data-product owner | Data-product SLA and the agent-readable dictionary entry |
| Semantic contract | Agreed meaning of each field, its valid values and what it does not mean | Data-product owner | The 17-field dictionary entry |
| Retrieval index | Searchable form for agents (vector, graph, structured) | Platform team | Index refresh cadence and access policy |
| Agent memory | What the agent retains within and across interactions | Agent wrangler (named owner) | The memory side of the bounded-agency envelope |
| Authoritative record | The firm’s record of decision that survives the interaction | Records owner | The Authority Process |
Flow rule. Data flows one way, from source through product and index into memory. The only return path from agent output to an authoritative record is the Authority Process. Agents may produce explanations, suggested actions, draft records, proposed classifications, recommended journal entries, drafted contracts and hypotheses; only the firm’s governed process turns them into decisions, executed actions, posted records, applied classifications, booked entries, signed contracts and confirmed facts (Ch 16 §4.4).
The Authority Process (Ch 16 §5.2) has one instance per record class, the same for agent, human, vendor or integration drafts, with named human and machine signatories at each step:
| Step | Purpose | Signatory |
|---|---|---|
| 1. Validation | Compare the draft with the master (dictionary field 8 = master) and flag mismatches | Validation tool; the producing agent’s wrangler |
| 2. Business-rule checks | Apply rules for the record class, such as finding references or four-eyes | Rules engine; rules owner; second reviewer where required |
| 3. Data-quality checks | Completeness, consistency, conformance to the schema (field 14) | DQ engine; receiving dataset’s steward |
| 4. Reconciliation | Against parents, subsidiaries, prior versions and related entities | Reconciliation tool; steward |
| 5. Human approval | A named approver with authority for the record class, rationale mandatory | Named approver; no agent substitutes here |
| 6. Audit logging | Full trail written immutably against the Audit Evidence Schema | Logging service; audit-evidence owner |
| 7. Versioning and posting | Post through the dataset’s write API, which only this process may call | Write API; data-product owner |
| 8. Separation of duties and evidence | Approver is not the wrangler; exception path with standing register, override authority and timed re-review | Control owner; internal audit |
Agents may drive steps 1–4; step 5 is human; steps 6–8 are mostly automated with human owners.
Gate agent access on a minimum data estate
Book 2 Ch 12 §7 answers what does the data estate need before agents scale? with five items and no substitutes. Use them as an admission gate per dataset:
- Contracts. Every dataset an agent touches has a versioned data contract and a named owner. No contract, no agent access.
- Semantics. Meaning is delivered machine-readably through the agent-readable dictionary into the agent’s standing context, so it does not infer meaning from column names.
- Lineage carried as evidence. Where a figure came from is referenced from the audit record’s evidence pointers, queryable rather than documented.
- Quality SLOs. Freshness, completeness and accuracy thresholds per data product, tested by the governance loop like any policy (see policy, audit and actuals). In Book 2’s worked example (Ch 12 §6, Asset Manager X, composite) a feed that misses its declared refresh window blocks the agent’s draft exactly as any other policy breach would.
- Identity-aware access. Every read resolved against the actor’s entitlement at query time, with one grounding data API in front of every grounding store so a deletion request has one place to land.
Treat APIs as products
An API record includes a named owner, purpose, schema, authentication and authorisation model, versioning and compatibility policy, rate limits, error conventions, monitoring, lifecycle state and known consumers. Use semantic versions only when the team has defined a compatible change and runs a deprecation window. A contract that many agents call cannot break casually: versioning becomes load-bearing, and a careless change is an estate-wide incident at machine speed (Book 2 Ch 12 §4). Agents do not read release notes, so a major-version migration window must be announced in the standing context agents read. Refuse to publish an API without an owner, a version and a consumer record.
Choose between sharing code and sharing a capability
| Question | Favour code reuse | Favour a running capability behind an API |
|---|---|---|
| Must the logic be identical across consumers? | A stable, low-stakes utility may tolerate independently versioned copies. | Regulated rules, prices or client-visible figures that must change consistently. |
| How often will consumers need updates? | Low change rate, few known consumers. | Frequent changes across many consumers. |
| What does a divergent copy cost? | Low impact, easy to detect. | High blast radius or hard-to-reconcile outputs. |
| Will agents consume it? | Local execution is required and the library contract is dependable. | Usually easier to discover, authorise, observe and update through a service contract. |
| Can the firm operate the runtime dependency? | Prefer local code where latency, offline use or availability rule out a call. | Only when the service team can meet the dependency’s service objectives. |
Service reuse trades copy drift for a runtime availability and ownership obligation; code reuse avoids a runtime call but makes every consumer responsible for adopting versions. Record the rationale in the architecture decision and the BOM.

The reuse decision changes the dependency graph and the way change is governed. Source: The AI Delivery Factory, Chapter 12, Figure 12.1.
Govern the fork as written policy, where agents read it. Copying is free for a coding agent, so an ungoverned estate drifts to N copies of everything. Book 2 Ch 12 §7 and Book 09 Ch 4 (composite) give the operating form:
- Rule. Anything that carries a rule or a number that must match everywhere runs once, behind a versioned contract with an owner, and its data carries an owner, a stated freshness and a stated meaning. Pure, stable, low-stakes utilities may stay as copies with a known-copies list.
- Write the decision into agent context. “There is a performance-calculation service; call it” in the agents’ standing context prevents the tenth copy.
- Count duplicates. Search the estate for an important function on a schedule; more than two copies of something that carries a rule or number marks a capability that wants to be a service. In the Book 09 composite the weekly count opens a review, not an automatic extraction, so people who will run the service decide.
- Retire copies at the speed of their own change, drifted copies first, and measure success as a falling duplicate count.
- Wrap once at the right layer. A system of record gets exactly one MCP server, owned by the team closest to it, listed with owner, version and data classification, reached only through the connector broker. A second wrapper is a contribution to the first, never a fork.
Make data contracts machine-readable
A data contract records what a data product means and what consumers may rely on: field definitions and allowed values, owner, source and lineage, quality thresholds for freshness, completeness and accuracy, sensitivity classification, retention, permitted uses, schema version, definition version and change policy. Publish it with the data product.
Keep a golden source for definitions and promote approved changes through environments. A governed source feeds analytics stores, operational stores, APIs and the agent-readable dictionary while preserving one reviewable change history. Consumer-specific schemas may differ; meaning and ownership must not silently diverge.
Fit the operational antibodies
Book 1 Ch 3 §5 names five artefacts that make the abstraction gap legible. They are principles; the dictionary below is their machine-readable form.
| Antibody | Implementation | Agent behaviour it enables |
|---|---|---|
| Metadata that travels with the data | Record-level effective-from, effective-to, definition-version, source-system, last-validated-at | Refuse to act on a record whose definition version is older than the policy under which it is asked to act |
| Queryable lineage | Lineage exposed programmatically through the catalogue (Purview Unified Catalog, AWS Glue Data Catalog with Lake Formation, Google Dataplex Universal Catalog, per Book 1 Ch 3) | Answer where did this number come from before treating it as fact |
| Named ownership | A named steward per domain, entity and authoritative source, not a role | A human escalation point when the gap is suspected; roles rotate, seats go empty |
| Known-limitations register | Per data product (below) | Read before acting; apply compensating behaviour |
| Versioned business glossary, separate from the schema | Glossary terms with their own steward and version | Translate a business question into a schema query; audit schema–meaning alignment |
Refusal rule (record-level). Before acting on a record, the agent (or the tool that serves it) compares the record’s definition-version with the definition required by the current policy. If older, the agent does not propose the low-risk outcome; it routes to the defined fallback. In Book 1’s composite KYC case, legacy records carried pre-2022 customer-type values with beneficial ownership defaulting to unknown through a catalogue rule the field did not show; the fix was a known-limitations entry, an effective-from-version on every record, and a guardrail refusing low-risk for records older than the current guideline, defaulting to enhanced due diligence (Ch 3 §6). Agent confidence fell; actual safety rose.
Known-limitations register. One short, structured register per major data product or service, in whatever catalogue the firm already runs, containing:
- what the product cannot be used for;
- what is known to be wrong;
- what is being worked on;
- what compensating controls exist;
- the named steward’s sign-off, reviewed quarterly (dictionary field 10).
The agent reads it before acting. If the catalogue cannot yet be queried by an agent, the register starts as documentation and becomes known_limitations in the dictionary entry. Book 1 Ch 3 §5 places the antibodies on the Memory side of the bounded-agency envelope: they keep the agent’s picture of the world honest about its own gaps, which no model upgrade can do.
Run the steward gap-statement exercise
For each of the three highest-stakes entities (in a large firm typically client, product or instrument, and counterparty or legal entity; in a 1,000-person specialist firm typically customer or investor, account or mandate, and transaction or policy), write a one-page, two-column gap statement: what the model says and what the business now means (Book 1 Ch 3 §7).
- The named steward attends in person, not a delegate. About ninety minutes per entity.
- The statement is descriptive, not a roadmap. If the steward cannot describe the gap, that is the finding.
- Output: the gap statement plus a first known-limitations register per entity, signed by the steward.
- Ownership: the data office with CIO sponsorship in a large firm; the CIO personally with the head of operations and one engineer in a small firm. A working group with no named owner is not acceptable.
Publish the 17-field agent-readable data dictionary
The dictionary is the semantic contract layer: one entry per dataset, machine-queryable, kept current by the owning domain, fronting both human APIs and agent MCP servers (Book 1 Ch 16 §5.1; Appendix B). Field names, types and lifecycle below are as the source gives them.
| # | Field | Type | Purpose | Lifecycle |
|---|---|---|---|---|
| 1 | dataset_id | string (catalogue ref) | Unique dataset identifier | Set at registration; immutable |
| 2 | business_name | string | Plain-language business name | Annual review |
| 3 | owning_domain | string (domain ref) | Accountable data-mesh domain | On organisation change |
| 4 | data_classification | enum (public / internal / confidential / restricted) plus sensitivity tags | Drives access and tokenisation decisions | Annual and on schema change |
| 5 | refresh_frequency | string (real-time / hourly / daily / weekly / monthly / event-driven) | How current the data is | Annual |
| 6 | lineage_upstream | array of dataset_ids | What it is derived from | On pipeline change |
| 7 | lineage_downstream | array of dataset_ids and report_ids | What consumes it | On consumer change |
| 8 | authoritativeness | enum (master / reference / authoritative-copy / derived / operational / draft / unverified) | Position in the authority hierarchy | Set at registration; changed only with governance approval |
| 9 | semantic_glossary_anchor | string (glossary term ref) | Glossary term defining business meaning | When the glossary changes |
| 10 | known_limitations | array of strings | Caveats and edge cases consumers must know | Quarterly |
| 11 | accuracy_sla | object (target %, measurement method, last-measured value, alert threshold) | The owning domain’s accuracy commitment | Measured continuously; reported monthly |
| 12 | mcp_endpoint | URL or null | Agent access path | At MCP publication |
| 13 | api_endpoint | URL or null | Human-facing API | At API publication |
| 14 | schema | string (schema ref) or inline | Technical schema | Versioned with each change |
| 15 | version | string (semver) | Contract version | Bumped on contract change |
| 16 | regulatory_class | object (DORA critical-or-important-function flag, AI Act class if applicable, GDPR sensitivity, sectoral overlays) | Regulatory frame | Annual and on regulation change |
| 17 | retention_and_agent_allowed_list | object (retention policy id, allowed-agent-uses array) | Retention and the enumerated agent uses | Annual and at incident |
Blocks: identity and ownership (1–3), sensitivity and timeliness (4–5), lineage and authority (6–9), operational contract (10–17). Map field 4 to the firm’s C1–C4 classes and carry flags as sensitivity tags, as set out on data classes and lanes.
The two load-bearing fields.
- Field 8,
authoritativeness, tells the agent this is master, do not summarise it from a knowledge-base extract; this is operational, do not treat it as master; this is unverified, do not cite it without a master cross-check. Expect resistance: domains must stop calling things master that are not. - Field 17, the agent allowed list, is enumerated and empty by default. A read tool such as
get_supplier_by_idmay be listed; a write such aswrite_supplier_recordis never granted to an agent, because only the Authority Process writes to authoritative datasets.
Enforcement points.
- The MCP server and API pull schema and tool descriptions from the entry, so the descriptions agree everywhere.
- The broker validates that the retrieved entry permits the calling envelope; the gateway validates the route; the wrangler validates the envelope mapping at attestation (Book 3 Ch 10 §7).
- Every retrieval carries the entry to the agent, and every agent action is logged with the entry’s identity, last-validated date and status, so audit can replay the meaning the agent acted on. The AES
dataEntryobject recordsdataset_id,classification,operationand optionallylineageandretention. - Require the agent to cite the dictionary entry of every dataset it reads.
- Cross-check at registration that the model usage registry’s
data_classes_touchedmatches field 4 for every dataset the agent reads.
Staleness controls (Book 3 Ch 10 §7, §9; Bank A composite): quarterly attestation by the named data owner; an 18-month maximum entry lifetime with automatic deprecation; a retrieval-blocked-stale-dictionary event when an agent reaches for an expired entry; a CDO dashboard of ageing entries. A dictionary nobody updates lies, and is worse than none because the agent acts on it confidently.
Two 17-field dictionaries in the sources. Book 1 defines the entry per dataset (fields above). Book 3 Ch 10 PR-4 defines a different 17-field entry per data class: name, description, owner, upstream lineage, sensitivity class; permitted use cases, permitted envelopes, permitted routes; retention obligation, lawful basis, data subjects, recipients, transfers, retention period (server and endpoint); last-validated date, validation cadence and status (Active / Under review / Deprecated / Withdrawn). This page treats them as two layers: the dataset entry is the semantic contract for a data product, and its field 4 points to a class entry that holds the class-wide permitted routes, envelopes and legal posture. Do not merge the field lists. Book 1 Appendix B states that a normative JSON Schema for the dataset dictionary is published at deixei.com; it is not yet among the schemas in the site’s schemas and tools reference.
Set accuracy thresholds below which agents invert value
For human consumers, quality and value move roughly together because people catch most errors. For agents the error rate is industrialised: in Book 1’s illustration, scaling a process from 1,000 to 10,000 transactions a day on 95% accurate data moves the error count from about fifty (most caught by humans) to about five hundred (none caught). Below a decision-relevant threshold, risk outruns value (Ch 16 §4.1).
| Data type | Reference threshold (author’s experience, not a standard) |
|---|---|
| Master customer data | About 99.5% record-level accuracy on the fields the agent uses |
| Reference data (currency, ISIN, LEI, country codes) | About 99.99% |
| Asset-manager composite: positions / attributions / house view | 99.95% / 99.9% / current as of this morning |
| Unstructured knowledge bases | Lower in absolute terms, higher in stakes; the agent cannot tell an obsolete chunk from an authoritative one |
Apply it: set the threshold per dataset, process and regulatory class in accuracy_sla; measure continuously; block or restrict agent uses in field 17 while measured accuracy is below threshold; improve quality before deployment, which the author treats as a precondition, not a guideline.
Pair the API and the MCP server from one dictionary entry
Data mesh, APIs and MCP are three layers of one regime per data product, not competing patterns (Ch 16 §4.3): the mesh is organisational (domains own and publish data products; the central function owns platform, dictionary and policy); the API is the human-facing technical layer; the MCP server is the agent-facing technical layer, with tool names such as get_supplier_by_id and descriptions pulled from the dictionary. One owner, one OAuth flow, one set of rate limits, one audit trail.
Procedure for the first pairing (Ch 16 §7): pick one of the three highest-stakes datasets; populate its entry; build the MCP server through the gateway and the human API through API management, both reading schema and descriptions from the entry; declare per tool whether it reads or acts so the broker can grant read tools to advisory agents and withhold acting ones (Book 2 Ch 12 §4).
Platform equivalents, as of May 2026 per Book 1 Ch 16 §4.3. [FACT-CHECK: product names, capabilities and maturity are current vendor claims.]
| Concern | Microsoft Azure | AWS | Google Cloud |
|---|---|---|---|
| Registry for data products, glossary and dictionary | Purview Unified Catalog (governance domains, data products, glossary terms) | Glue Data Catalog (technical) plus DataZone (business metadata, glossary, subscription approval) | Dataplex and Dataplex Universal Catalog (glossary, lineage, tag templates) |
| Agent-facing MCP path | API Management MCP feature | Bedrock AgentCore Gateway | Apigee MCP support with Vertex AI Agent Builder |
| AI layer | Purview AI hub; Purview SDK in Agent Framework (Ch 16 reports it released June 2026; Ch 13 reports GA announced for late June 2026) | — | — |
The author’s caution: an AI security hub is an observability and protection layer, not the agent-data primitive; the primitive is the dictionary entry per data product.
Make integration patterns explicit
Select the pattern for the interaction rather than letting each urgent feature create a private connection:
- synchronous API for timely request/response;
- events or messaging where producers and consumers should not block each other;
- managed batch or file exchange for volume and legacy interfaces;
- workflow orchestration for multi-step processes;
- data-product publication for governed datasets;
- a governed vendor integration boundary for connections outside the institution.
Give each pattern a recipe, owning team and evidence path. Agent context should point to the approved pattern and interface; a coding agent will otherwise wire a plausible connection to whatever it can see.
Onboard external feeds as data the firm depends on, not as an integration. In the Book 09 composite, a custodian feed onboarded as plumbing failed silently; the recipe was changed so that nothing is built until the sender and the firm agree a feed promise, field by field: what each record shows (for example booked versus closing balance), arrival time each business day, what a well-formed record and file look like (including a trailer record count), and what the sender does when it sends a bad one. A boundary check then refuses non-conforming files at arrival with the reason recorded. Number every file per business day so replacements apply in sequence and duplicates are refused rather than silently undoing a day’s processing. Test with deliberately malformed files before go-live.
Client-facing read-only endpoints. Where the firm already describes its reporting services machine-readably, a read-only endpoint per client, bound to that client’s mandates, resolved by the firm’s identity service and recorded per call, can reuse the reporting service behind the portal (Book 09 Ch 3, composite). In that composite the pilot ran with two institutional clients, the adoption lead compared endpoint calls with the old portal logins every week, and the endpoint was cheap because the reporting services were already described machine-readably, in contrast to an exception tool that had no such description. The chief risk officer’s team chose to govern the endpoint as a client-facing AI system, a deliberately cautious reading of the transparency duties. Whether those duties apply to a given endpoint is a per-case analysis on the regulatory map.
Put retrieval behind a governed data API
Place a grounding data API between product code and the underlying stores. It resolves the caller’s identity, selects the corpus or tenant, applies current entitlements, filters results and records what was returned. Agents, orchestrators and applications do not query vector stores directly with their own copy of the filtering logic. The filter applies whether retrieval happens before a model call or inside an agent’s loop (Paper 09 §6.1, citing Microsoft’s multitenant RAG guidance).
| Isolation choice | Appropriate starting context | Main trade-off |
|---|---|---|
| Store per tenant | Client data held as processor; information barriers; C4 corpora | More deployment and operational overhead |
| Shared multitenant store | Many internal domains at C2–C3, every query carrying an enforced tenant and entitlement filter | A filter defect exposes data across tenants; test the boundary directly |
| Shared corpus | Public or firm-wide material | Requires confidence that restricted content cannot enter |
Resolve user or delegated-user entitlements at query time. Carry source identifiers, dictionary entry and data labels through parsing, chunking, enrichment and indexing. Stamp the store’s class on every result so the gateway can apply the highest-class rule and inheritance rule 1 (Paper 09 §5.3): derived artefacts inherit the highest class of their sources. The access record captures caller, agent or product identity, corpus, permitted query context, documents returned and decision outcome without logging unnecessary sensitive content.

A single governed path centralises routing, entitlement checks and access evidence. Source: The AI Delivery Factory, Chapter 12, Figure 12.2.
Version and evaluate the retrieval pipeline
Treat an index as a build artefact. Version the configuration and source inputs for every stage, then test the index against golden questions owned with subject-matter experts.
| Phase | Versioned inputs | Example quality signal |
|---|---|---|
| Prepare | Connectors, parsers, OCR and source filters | Extraction completeness and source coverage |
| Chunk | Structure-aware or semantic strategy and parameters | Recall on representative questions |
| Enrich | Metadata and generated summaries | Filter accuracy and ranking lift |
| Embed | Model, version and dimensions | Recall against latency and cost |
| Retrieve | Search mode, result count and reranker | Precision, recall and entitlement-filter correctness |
| Prompt | Grounded-answer instructions and citation format | Groundedness and citation accuracy |
| Evaluate | Dataset version, scoring method and reviewer input | Relevance, completeness and quality by data class |
Changing a parser, chunking strategy, embedding model or index configuration creates a new retrieval release. Promote it through evaluation and approval gates and keep the previous version available for rollback where the operating model requires it.

Retrieval quality and deletion handling belong to the pipeline and the index release, not only to the prompt. Source: The AI Delivery Factory, Chapter 12, Figure 12.3.
Maintain a source-to-derived-artefact map identifying chunks, embeddings and caches derived from each document. Define how deletion, correction, changed entitlement or expired licence propagates, how query-time suppression works during the cascade, and what evidence confirms completion. Legal and privacy owners determine the retention and erasure interpretation; engineering makes the approved action executable and observable.
Roll out the dictionary in sized iterations
Book 1 Ch 16 §7 gives a sequence and author estimates (not benchmarks):
- Pick the three highest-stakes datasets: the relevant master (customer, supplier, counterparty, holdings or policy), the feed for the largest regulatory submission, and the master for the process most aggressively automated with agents.
- The owning domain populates the 17 fields; the central data function reviews; the agent platform team checks the allowed list; the CDO signs.
- Wire one MCP server and one API to one dataset from its entry.
- Run the Authority Process on one real generated artefact end to end with named signatories; time and log every step; fix gaps before scaling.
| Parameter | ~50,000-employee firm | ~1,000-employee specialist firm |
|---|---|---|
| First iteration | 8–12 weeks, team of six (data architect, two data-product engineers, agent platform engineer, governance lead, programme manager) plus domains | 4–6 weeks, team of three |
| Steady-state entries | 2,000–8,000 across 30–60 domains | 100–400 across 5–12 domains |
| MCP-published entries | 200–800 | 30–80 |
| Recurring effort | 3–5 FTE central enablement plus about 0.1 FTE per data product | 1–2 FTE |
The rest of the dictionary follows in two further iterations through the year. Compliance owns the field-to-regulation mapping and a quarterly per-domain attestation; internal audit owns a query template that reconstructs the Authority Process chain from one record reference and a half-yearly sample of posted records (Ch 16 §7).
Failure modes
| Anti-pattern | Recognise it by | Recover by |
|---|---|---|
| ETL as ground truth (Book 1 Ch 3 §8) | Asked what a value means, an engineer answers with a SQL CASE statement | Extract the definition, write it in the glossary, get it signed, version it |
| Schema as the contract | A value is renamed (retail to individual) with no schema change; integrations pass, meaning shifts | Carry definition-version beside schema version; consumers refuse definition versions they have not been validated against |
| Integration as semantic transfer | A migration ticks “no data loss” but meaning stayed with the old team | Semantic conformity test: for the highest-stakes entities, a sample of business questions must give the same answer on both sides, run by the named steward, not the migration team |
| Agent as semantic auditor | An LLM-produced schema mapping is used without validation | Treat it as generative output; it becomes authoritative only through the Authority Process |
| Agent reads the data lake without a dictionary (Ch 16 §8) | Agent reaches operational exhaust, prior agent outputs and obsolete snapshots | Revoke broad read; populate entries for needed datasets; access only through MCP; require citation of the dictionary entry |
| MCP servers but no glossary | semantic_glossary_anchor null on most entries; two domains mean different things by customer | Glossary first; anchor every entry; quarterly glossary review |
| Generative output treated as record | Drafts posted because posting is automated; confidence used as a quality gate | Reintroduce the Authority Process; reconcile the period; forbid agent writes except through it (field 17) |
| Authority Process bypassed under time pressure | Exceptions become routine | Standing exception register, named authority, timed re-review, board-visible exception counts |
| Dictionary nobody updates (Book 3 Ch 10 §9) | Last-validated dates past cadence; agent acts on a withdrawn class | Enforced validation dates, auto-deprecation, attestation in the owner’s RACI, ageing dashboard |
| Direct data access by convenience (Book 2 Ch 12 §8) | An incident finds an unknown consumer of an unexposed table | Put a data product with a contract in front; retire the direct read as debt |
| The second wrapper | Two MCP servers over one system, or broker logs showing traffic to an unlisted one | Merge into the owner’s server; broker refuses routes to unlisted servers |
| The index nobody can delete from | Erasure requires a full rebuild | Move the corpus behind the grounding API with source pointers on every fragment |
Acceptance checks
- No agent reads a dataset without a versioned contract, a named owner and a populated dictionary entry; the allowed list is enumerated and empty by default.
- Fields 8 and 11 are honestly populated for every master dataset, and measured accuracy is compared with the threshold.
- Records used by agents carry
effective-from,definition-versionandlast-validated-at, and a test proves the agent refuses a record older than the current definition. - Each high-stakes data product has a known-limitations register signed by a named steward, and agents demonstrably read it before acting.
- The three highest-stakes entities have signed gap statements.
- The glossary is versioned separately from the schema and every dictionary entry anchors to a glossary term.
- Migrations include a steward-run semantic conformity test; no unvalidated LLM mapping is in production.
- The API and MCP server for each published dataset derive schema and descriptions from one entry; tools declare read or act.
- Only the Authority Process can call write APIs on authoritative datasets; a sample of posted records shows a named step-5 approver who is not the wrangler.
- Stale entries are blocked at retrieval and raise an event.
- Known-copies lists exist for permitted code copies; the duplicate count is running and falling.
- External feeds have a signed feed promise and a boundary check proven with malformed test files.
- Retrieval uses the caller’s current entitlements, cannot bypass the grounding API, stamps store class on results, and passes cross-tenant negative tests.
- Index builds are reproducible from pinned inputs and evaluated before promotion; deletion and correction cascades cover derived data and produce evidence.
Source status. The six-layer model, antibodies, dictionary, Authority Process, thresholds and sizing are author proposals and experience-based estimates from Book 1, not independently validated; the author reports that two years of design reviews have not required adding a dictionary field or an Authority Process step. The minimum data estate, reuse rules and retrieval pipeline are author recommendations from Book 2. Bank A, Asset Manager X, Insurer I and the Book 09 firm are composites. Vendor product statements are dated and marked for verification. GDPR and retention interpretations require review by the institution’s legal and privacy owners.
Source material
- Book 1 (Building Organizations for the Age of AI), Chapter 3, Software as an Abstraction of Reality (§§3–8)
- Book 1, Chapter 16, Data as the Enterprise Multiplier (§§3–5, §7, §8)
- Book 1, Appendix B, Operating Schemas: the 17-field agent-readable data dictionary
- Book 2 (The AI Delivery Factory), Chapter 12, APIs, Data, and Integration (§4, §5, §7, §8)
- Book 3 (Deploying Claude Safely), Chapter 10, Data Classification, Retention, and DLP (§7, §9, PR-4)
- Book 09 (Everything Is Green), Chapter 3, The Quote Turned Face Down; Chapter 4, Right About the Code, Wrong About the Product; Chapters 12–13, Who Holds the Dial and The Inspection (composite narrative)
- DEIXEI Research Series, Working Paper 09, Data Classes, Processing Lanes and Controller/Processor Roles for AI (§5.3, §6.1)
- Supporting Tools, Audit Evidence Schema v0.1.0 (dataEntry)
Related research
Related guidance
-
The Anatomy of an AI ApplicationA six-tier view of what one AI product is made of, crossed by a trust and operations plane that attaches controls to every tier; what the factory already supplies per tier, how the anatomy maps to the six-layer product portfolio, and how to run an anatomy review. -
Data Classes, Processing Lanes and TokenisationClassify every model request by its data, route it only to a processing lane the class permits, keep derived artefacts inside their source class, and treat retention, roles and tokenisation as recorded, testable decisions. -
Govern MCP Connectors Through a BrokerAdmit, tier, pin and enforce every MCP server through one registry and two enforcement points, keep tokens audience-bound, govern A2A delegation through its own registry and gateway, and expose client-facing agent channels only through the same policy layer. -
Design and Operate the Model GatewayBuild the model gateway as the governed edge for every model call: per-client identity, per-request highest-class routing, tokenisation, pinned models, a conformant wire contract, a correlated evidence record, a tested concentration answer, and day-two cost, latency and reliability operations across two estates and three meters. -
Identity, Secrets and Configuration for Agent WorkloadsGive every actor one owned identity, size its authority for the worst case, separate runtime from deployment, federate instead of holding keys, refuse impersonation and silent fallback, govern configuration by class, and audit the estate until flagged identities decline. -
Build a Living Product Bill of Material and RecipeDescribe what an AI product contains and how the firm builds it in one versioned specification shared by pipelines, agents and assurance teams, including the classification line, readiness grades and reuse generations. -
Bounded-Agent PatternA class 3 agent lets a model choose the next step, but only inside a registered, validated and enforced envelope of Mandate, Means, Memory and Monitoring. -
The Audit Spine and the Evidence RecordHow to capture, join, protect, retain and prove per-action evidence: the AES v0.1.0 record contract, three capture points and one correlation ID, tiered immutable storage, retention and legal hold, replay rehearsal, the route-approval evidence gate, and the Claude-specific ways evidence goes missing. -
Model Governance: Usage Registry and Upgrade LifecycleGovern models at the grain regulators ask about: a catalogue for what exists, an 18-field usage registry for what is used, risk classified per usage, a registry the policy decision point reads by usage_id, pinned model IDs, and a four-phase upgrade lifecycle from Day −14 to Day +90 with numeric rollback thresholds. -
Evaluate, Assure and Test Agent ResilienceDefine a versioned evaluation contract, test the trajectory and the execution boundaries, run three assurance layers into one quarterly pack, map the OWASP Agentic Top 10 to controls that demonstrably fire, bring agents into threat-led penetration testing with a five-section scope, keep a continuous cadence between formal tests, land every finding as a live control-plane artefact, and engineer recovery down to a minimal AI service level on models you host.