Model Governance: Usage Registry and Upgrade Lifecycle
Govern models at the grain regulators ask about: a catalogue for what exists, an 18-field usage registry for what is used, risk classified per usage, a registry the policy decision point reads by usage_id, pinned model IDs, and a four-phase upgrade lifecycle from Day −14 to Day +90 with numeric rollback thresholds.
- Version
- 0.1.0
- Owner
- DEIXEI
- Last reviewed
- Reading time
- 52 min
For Cto · Ai Architect · Platform Engineer · Risk And Assurance · Model Risk · Internal Audit · Finops
Sources and details
- Section
- Govern and prove
- Source works
- Book 1 · Book 2 · Book 3 · Book 09 · Research Series · Supporting Tools
- Source chapters
- 7, 8, 9, 13, 14, 16, 17, 18
- Source appendices
- 01:B
- Tool specifications
- deployer_manifest@0.1.0, audit_evidence_schema@0.1.0
- Refresh trigger
- vendor release
Model governance answers three questions that look alike and are not: which models exist in the estate, how each one is used, and what changed since the use was approved. Platforms ship good answers to the first. The second and third are the institution’s to build, and they are the ones a supervisor, an incident review or an auditor actually asks.
This page sets out the records, the controls and the lifecycle that answer them: the model catalogue and the model usage registry as separate records, risk classification at the usage grain, the 18-field usage row and how to populate it, the wiring that lets the policy decision point refuse an unregistered or stale usage, retirement and point-in-time history, third-party models, version pinning, and the upgrade cycle that treats every model release as a change event.
Reading the labels. Source claim means a book states it. Author model means a framework the books propose (the 18 fields, the four phases, the rollback thresholds) rather than an industry or regulatory standard. Recommendation is this page’s own operating advice or synthesis. Bank A, Bank D and Asset Manager X are composite firms written for the books; their figures illustrate a mechanism and are not benchmarks. Vendor specifics are dated and carry a [FACT-CHECK] marker where they are current product claims not yet re-verified against vendor documentation. Legal classification and regulatory dates live on the regulatory map; this page links there rather than restating them.
The model in one table
| Question | Answer this page gives | Section |
|---|---|---|
| Is the platform’s model catalogue our model inventory? | No. The catalogue records existence; a separate registry records use | Two records |
| What is one row in the registry? | One usage: (deployment endpoint, calling agent or system, business process) | The grain |
| Where does the regulatory class sit? | On the usage row, assigned and signed per usage, never inherited from the model | Classification |
| What does a row hold? | Eighteen fields in four blocks; three are load-bearing | The 18-field row |
| How does the registry control anything? | The policy decision point decides “can agent X invoke usage U at time T?” and every call logs usage_id | Wire to the PDP |
| When does a usage end? | At a planned date, or “evergreen” with a review date no more than 12 months out, plus event triggers | Retirement |
| How do model upgrades reach production? | As change events, in four phases from Day −14 to Day +90, with staged rollout and named rollback thresholds | Upgrade lifecycle |
| How do we stop silent upgrades? | Pin full model IDs everywhere, switch automatic upgrades off, and enforce the pin at the gateway | Pinning |
Keep two records: catalogue and usage registry
The model catalogue answers what models are deployable: versions, regions, capacity, evaluation results, content-safety configuration and sign-off state. The model usage registry answers what those models are doing: which agent or system calls them, in which business process, on which data, under whose name and with which regulatory class (source claim, Book 1 Ch 14 §4.1).
The two have different lifecycles and different owners, which is why one cannot stand in for the other. A catalogue entry is created when a model is approved for deployment and retired when it is decommissioned. A usage is created when a specific caller, in a specific process, starts calling a model for a specific purpose, and is retired when that use ends, which may be years before or after the model itself. One catalogue entry routinely carries many usages.

Source: Building Organizations for the Age of AI (Book 1).
| Model catalogue | Model usage registry | |
|---|---|---|
| Question it answers | What exists and may be selected? | What is used, where, by whom, on what data, at what risk? |
| Unit | Model and version | Usage (see the grain) |
| Created when | A model is approved for deployment | A caller starts using a model in a process |
| Retired when | The model is decommissioned | The use ends, the owner will not sign, or a retirement trigger fires |
| Owner | AI platform team | Named wrangler per row; model risk for the class |
| Typical product | Platform-provided (see below) | Built by the institution on the control-plane store |
| Read by | Platform engineers, model risk | Policy decision point, wrangler, supervisor, internal audit, incident reviewers |
Vendor existence layers, as of May 2026 (per Book 1 Ch 14 §4.1, §4.4, §6.1). On Azure, the Azure AI Foundry model catalogue covers existence, and Foundry observability gives per-deployment throughput and cost but not business process or regulatory class. On AWS, the Amazon Bedrock model registry (with SageMaker Model Registry for non-Bedrock models) covers existence and Bedrock model invocation logging gives per-call telemetry. On GCP, Vertex AI Model Registry covers existence, with Vertex AI Logging and Cloud Audit Logs for per-call records and the strongest native lineage of the three. In every case the business-process and regulatory-class dimensions remain the institution’s problem. [FACT-CHECK: confirm current product names and capabilities of Foundry model catalogue, Bedrock model registry and Vertex AI Model Registry; Vertex AI branding may have changed — Book 2 Ch 13 refers to “Gemini Enterprise Agent Platform” as of November 2026.]
A firm on two clouds runs two existence layers and one logical usage registry, partitioned by cloud and queried as one (source claim, Book 1 App B, cross-cloud note).
The model gateway can feed actual call volumes into the registry, but a call record cannot supply business purpose, lifecycle, contract or risk. Those come from people.
Composite illustration (Book 1 Ch 14 §1, Bank A). The platform catalogue counted 62 models; the first usage registry counted 228 rows. 203 rows came from joining the catalogue, the agent registry and the process inventory; 25 more surfaced in a two-week reconciliation pass; three were retired while the rows were being written, two because no wrangler would sign for them. Of the 228, eleven touched credit decisions, all carrying a high-risk class and a named wrangler. The question “which models touch credit decisions?” could not be answered from the catalogue at all.
Define the grain before anything else
A usage is uniquely identified by the tuple (model deployment endpoint, calling agent or system, business process) (source claim, Book 1 Ch 14 §4.2; App B). The grain matters because the regulatory classification attaches at this grain.
| Situation | Rows |
|---|---|
| One deployment called by two agents in two processes | Two usages |
| One agent calling the same deployment in two processes | Two usages |
| One agent in one process calling a primary and a fallback endpoint | Two usages |
| One model reached on two clouds by the same agent and process | Two usages (two endpoints), one logical registry |
Agree the grain in writing first. The first move is a written definition, circulated to model risk, the agent platform team, data governance and compliance, with agreement inside a week. If those four functions cannot agree the grain, nothing downstream works (source claim, Book 1 Ch 14 §7).
Source conflict. Book 1 Appendix B’s description of control-plane collection 12 (model_usage) says one record per (model, agent, time-window) triple, while the same appendix’s registry section and Chapter 14 define the usage tuple above. Book 3 Ch 16 OD-3 describes its row as “one row per envelope per model”. This page follows the (deployment, caller, business process) tuple as the identity of a usage, and treats time as history on the row (each pinned version window and each change is a dated version of the row; see point-in-time history), not as part of the key.
Classify risk on the usage, not the model
Risk classification follows usage, not deployment (source claim, Book 1 Ch 14 §3, §4.3; App B field 8). A foundation model is not high-risk because it exists in a catalogue; a usage is high-risk because of the process it powers, the data it touches and the decisions it influences. The AI Act high-risk categories are framed around intended use; see the regulatory map’s routing table for which categories commonly apply in financial services and who decides.
The operational consequences:
regulatory_classcannot be inherited from the catalogue entry. It is assigned at registration, per usage, by model risk, reviewed by legal and compliance, and signed by the wrangler.- The same endpoint can be high-risk in one row and minimal-risk in another. Both rows are correct, and the inventory is correct only when both exist.
- The defence of a classification is the combination of three fields on the row:
business_process,data_classes_touchedandcalling_agent_id. When a supervisor asks why two rows calling the same model are classified differently, those fields are the answer.
Composite illustration (Book 1 Ch 14 §6, Bank A and Asset Manager X). One general-purpose foundation-model deployment carried seven usages across two firms: two classified high-risk (mortgage decision-rationale generation; adverse-action letter drafting, classified high-risk because it is adjacent to the credit decision and its output reaches the data subject), two limited-risk (portfolio-commentary drafting; marketing copy), two minimal-risk (meeting summaries; incident post-mortem drafting), and one KYC-summary usage outside the high-risk list but under special-category data-protection controls. An internal fraud classifier carried four usages, one falling under the fraud-detection carve-out and one scoring legal entities rather than natural persons. Each row defends itself on its own fields, and each has a different wrangler. The classifications are the composite’s illustrations, not legal advice; route real classifications through the sectoral diagnostic.
Regulatory class, risk tier and model choice are three different decisions
| Decision | What it is | Set against | Owner | Where |
|---|---|---|---|---|
regulatory_class | Legal classification of the usage (prohibited / high-risk / limited / minimal, plus sectoral tags such as CIF / non-CIF, CRD model-risk tier, AIFMD/UCITS) | The usage | Model risk assigns; legal and compliance review; wrangler signs | This registry (Book 1 App B field 8) |
| Risk tier | The firm’s control classification, informed by the legal class | The use case | Wrangler, DPO, model risk and business owner sign; Agent Control Board ratifies (Book 3 Ch 14 OD-1) | Risk tiers |
| Model selection | Which model serves the envelope, on a cost, latency and accuracy axis | Inside the tier | Wrangler, with model risk | Selection and the model gateway |
Book 3 Ch 14 §8 names the failure where these collapse: tier mapped to model, for example “this model is T2, that one is T3”. The same use case then gets a different control package depending on which model the wrangler picked, and the question “which control set applies to this use case?” has no answer. Set the tier against the use case; choose the model inside it. A model change can still re-open the tier: Book 3 Ch 14 OD-1 lists a model upgrade that crosses the Chapter 16 threshold, and model evaluation drift beyond threshold, among the re-tier review triggers. The model change triggers a review; it never assigns the tier. See re-tier when the use case changes.
Re-classification triggers (recommendation, combining Book 1 App B lifecycle notes and Book 3 Ch 14 §8). Re-open regulatory_class on: a change to business_process; a new data class in data_classes_touched; a change to the calling agent’s autonomy or action class; output newly reaching a customer or data subject; a new regulatory designation (for example a new CIF designation); and the annual review. Classification history is kept on the row.
The 18-field usage row
Book 1 Appendix B specifies the row (author model). The fields fall into four blocks: what the model and the use are (1–3, 6, 7), who carries accountability (4, 5, 14, 15), how well and how much it runs (9–13), and under what authority and for how long (8, 16–18). Field names and types below are exactly as the source gives them, and the cadence column follows App B’s lifecycle notes. The “Populated by” and “What breaks” columns are this page’s synthesis from App B’s purpose notes and population sequence, not source columns. Book 1 App B also maps each field to regulatory anchors; those mappings are not repeated here (see the regulatory map).
| # | Field | Type | Populated by | Cadence | What breaks if it is wrong |
|---|---|---|---|---|---|
| 1 | model_id | string, catalogue ref | Seeded from the catalogue | Set at registration; immutable | The row points at nothing the platform can confirm |
| 2 | model_version | string (semver or vendor-specific) | Catalogue; updated on version change with re-evaluation | Event-driven | Evaluations and content-safety settings are version-bound; an unrecorded version change invalidates them |
| 3 | deployment_endpoint | URL or ARN | Catalogue / gateway configuration | Event-driven, on endpoint migration | Logs cannot be tied to what actually served the call |
| 4 | calling_agent_id | string, agent registry ref; null for non-agent callers | Runtime telemetry | Set at registration; immutable | The join to the agent registry, and so to the wrangler chain, is lost |
| 5 | calling_system_id | string, CMDB ref; always populated | Runtime telemetry | Set at registration | The ICT asset inventory misses the dependency |
| 6 | business_process | string, process-taxonomy ref | Wrangler, by hand | Annual; immutable in normal life | The classification cannot be defended |
| 7 | data_classes_touched | array of data-taxonomy refs | Wrangler with data governance | Quarterly | Data-protection and data-governance controls are mis-scoped |
| 8 | regulatory_class | enum (prohibited / high-risk / limited / minimal) plus sectoral tags | Model risk assigns; legal/compliance review; wrangler signs | Annual and on process change | Over- or under-classification across the estate |
| 9 | last_evaluation | timestamp plus evaluation id | Evaluation pipeline | Each evaluation cycle | Stale-evaluation deny cannot work (PDP) |
| 10 | drift_signals | object: input drift, output drift, evaluation regression, yellow/red thresholds, last-checked time | Monitoring | Live | Behavioural degradation is invisible to the owner |
| 11 | expected_qps | number | Capacity review | Quarterly / on capacity review | Capacity and resilience planning rest on a wrong number |
| 12 | observed_qps | number, rolling 30 days | Gateway / telemetry | Live | Dormant or runaway usages go unnoticed |
| 13 | fallback_model | catalogue ref or null | Wrangler with platform | Annual | Failover sends traffic to an unevaluated, unregistered usage |
| 14 | owning_wrangler | string, HR identity | Wrangler named at registration | On role change | Oversight accountability lands on a mailbox, which is no one |
| 15 | contractual_party | string, legal entity | Procurement / third-party risk | On contract change | The third-party register line and value-chain records are wrong |
| 16 | retention | object: input retention, output retention, location, policy id | Evidence owner | Annual | Logging and retention obligations cannot be shown per usage |
| 17 | change_ticket | string, ITSM ref | Change process | Set at registration | No authorisation trail for the usage |
| 18 | decommission_date | date, or evergreen + next review date (≤ 12 months out) | Wrangler | Reviewed at the date | Rows accumulate with nobody able to say which are live |
Book 1 App B’s governance cadence for the populated row is: quarterly review of fields 7, 8, 9, 11, 12 and 16; annual review of fields 6, 13, 14, 15, 17 and 18; live update of field 10; event-driven update of fields 2, 3 and 4. (Field 8 also re-opens on process change, per its own lifecycle note. App B is internally inconsistent on field 16: its lifecycle note says annual, its governance step says quarterly. Review it quarterly, the stricter of the two.)
The three load-bearing fields
business_process(6). This is what makes a model call a regulated usage. Nothing can derive the process fromcalling_agent_idautomatically; the wrangler supplies and owns the link. The value must be specific enough to drive classification: mortgage origination decision rationale generation, not credit. An agent whose wrangler cannot name a process should not be in production; a blank here is a finding, not a gap (source claim, Book 1 App B).regulatory_class(8). Assigned, defended and reviewed per usage, never inherited (see classification).owning_wrangler(14). A named human by HR identifier. Not a team alias, not the platform team. Reviews go to that person; the platform team supports, the wrangler is accountable. See accountability.
Lifecycle fields for the upgrade cycle (a second layer)
Book 3 Ch 16 OD-3 gives a different 18-field row, built around one envelope’s pinned model and its upgrade history: model, version, provider, route, use-case-class, risk-tier, owner, wrangler, pinned-from-date, pinned-to-date, evaluation-suite-version, evaluation-result, deployment-route, ZDR-status, HIPAA-status, cost-baseline, observed-cost-delta, decision-log-ref. The two books disagree on the field set; neither is wrong for its purpose. Book 1’s row describes the usage; Book 3’s row describes the pin window of a usage through upgrades.
This page follows Book 1’s 18 fields as the usage row and adds Book 3’s lifecycle fields as a pin-window extension recorded on each dated version of the row (recommendation). Where they overlap, use Book 1’s names.
| Book 3 OD-3 field | Book 1 equivalent | Treatment on this page |
|---|---|---|
model, version, route, deployment-route | model_id, model_version, deployment_endpoint | Use Book 1 fields; record the full route path (gateway → platform region → model ID) in the extension |
provider | contractual_party | Use Book 1 field |
owner, wrangler | owning_wrangler | Use Book 1 field; record the platform owner in the extension |
risk-tier, use-case-class | none (Book 1 has regulatory_class) | Add to extension; tier comes from the use-case record |
evaluation-suite-version, evaluation-result | last_evaluation | Add to extension: suite version run at pre-evaluation, not today’s |
pinned-from-date, pinned-to-date | none | Add to extension |
ZDR-status, HIPAA-status | none | Add to extension; checked at Phase 1 |
cost-baseline, observed-cost-delta | none | Add to extension; feeds the cost rollback threshold |
decision-log-ref | change_ticket | Keep both: the original authorisation and the current pin decision |
Three checks on every row (source claim, Book 3 Ch 16 OD-3). The model field carries a full model ID, never a family alias, and the pin date and route reconcile to the gateway’s actual routing on the day the row was last updated. The evaluation-suite version is the one run at pre-evaluation. The decision reference points at a countersigned Virtual CAB minute. A row failing any check is treated as drift and routed to the AI platform owner for same-day remediation.
Schema status. Book 1 App B says a normative JSON Schema for the 18 fields is published at deixei.com. As of 14 November 2026 it is not among the hosted schemas (Audit Evidence Schema, CAB change, CAB decision and Deployer Manifest at /aes/v0.1.0/; see schemas and tools). Treat the field list above as the reference until a schema is published; do not invent enum values beyond those the source gives.
Populate the registry in seven steps
Book 1 App B gives the seven steps and who acts at steps 4 and 5 (author model); Ch 14 §7 adds compliance to the grain agreement. The other owners and the Output and Check columns are this page’s recommendations.
| Step | Action | Owner | Output | Check before moving on |
|---|---|---|---|---|
| 1 | Define the grain (usage, not model) and agree it | Model risk, agent platform, data governance, compliance | Signed one-page grain definition | All four functions have agreed in writing |
| 2 | Seed model_id, model_version, deployment_endpoint from the catalogue | Platform engineer | Candidate rows per endpoint | Every production endpoint appears at least once |
| 3 | Expand by caller from runtime telemetry for the last 90 days: calling_agent_id, calling_system_id | Platform engineer | One row per (endpoint, caller) | Callers missing from the agent registry or CMDB are raised as findings |
| 4 | Attach business_process | Wrangler (cannot be automated) | Rows split by process; the grain is now complete | No blank process on a production row |
| 5 | Classify: model risk assigns regulatory_class; legal and compliance review; wrangler signs | Model risk, legal, compliance, wrangler | Signed classification per row | Each high-risk or CIF row has a written rationale |
| 6 | Attach quality, capacity, lifecycle and contract fields by automation | Platform, procurement, evidence owner | Fields 9–13, 15–18 populated | decommission_date never blank; evergreen rows carry a review date ≤ 12 months out |
| 7 | Govern on the cadence above | Wrangler, model risk | Review records on the row | Reviews overdue are escalated automatically |
Scope ruthlessly (source claim, Book 1 Ch 14 §7). Populate the top ten highest-stakes usages first: high-risk class, CIF-supporting, or of known supervisory interest. In the Bank A composite these were nine credit-related usages and one third-party scoring usage, and took eight working days of two analysts plus the wranglers’ time. The rest follow by quarter-end.
Where to put it (as of May 2026, per Book 1 Ch 14 §4.4, §7).
| Cloud | Store | Population sources | Reporting |
|---|---|---|---|
| Azure | model_usage collection in Cosmos DB (collection 12 of the control-plane schema), partitioned by tenant_id, indexed on calling_agent_id, model_id, business_process, regulatory_class | Foundry catalogue; agent runtime (Foundry Agent Service, Microsoft Agent Framework); Application Insights | Power BI dataset |
| AWS | DynamoDB or Aurora table | Bedrock model registry; Bedrock model invocation logging (CloudWatch and S3); AgentCore Registry for the calling agent | QuickSight |
| GCP | BigQuery or Firestore | Vertex AI Model Registry; Vertex AI Logging; Cloud Audit Logs; Apigee; Vertex lineage can partly auto-populate fields 2, 7 and 9 | Looker |
[FACT-CHECK: Book 1 Ch 14 states AgentCore Registry and AgentCore Policy reached general availability in March 2026; confirm against current AWS documentation.]
Performance targets (source claim, Book 1 Ch 14 §7). A wrangler can query “show me my usages” in under 30 seconds; a supervisor question is answerable within one working day; the registry is refreshed against the catalogue and agent runtime nightly. Registration is one API Management (APIM) call from the agent build pipeline, with the wrangler named and the change ticket attached at that call, so the compliant path is the default path.
Sizing (author estimates, Book 1 Ch 14 §7).
| Large global firm (~50,000 staff) | Specialist firm (~1,000 staff) | |
|---|---|---|
| Steady-state usages | 200–800 | 30–100 |
| Wranglers owning rows | 50–150 | 10–25 |
| First iteration | 8–12 weeks | 4–6 weeks |
| Build team | Four: platform engineer, model-risk analyst, data-governance analyst, programme manager; plus about 0.2 FTE per wrangler | Two |
| Recurring cost | About two FTE a year | 0.5–1 FTE a year |
The schema is the same 18 fields at both sizes. The smaller firm’s temptation is a spreadsheet; the source’s answer is that the schema scales down and spreadsheets do not scale up. This page adds a practical reason: the policy decision point cannot read a spreadsheet, so a spreadsheet registry loses its control function.
Wire the registry to the policy decision point
A registry that nothing reads at runtime is an inventory. It becomes a control when the policy decision point (PDP) decides model calls by usage, not by model (source claim, Book 1 Ch 14 §4.4, §7). The decision is can agent X invoke usage U at time T?, and U carries the regulatory class, data classes, retention and wrangler. See the enterprise control plane for the PDP contract.
| Condition at decision time | Decision | Rationale the denial carries |
|---|---|---|
| No registered usage for (endpoint, caller, process) | Deny | Usage not registered; registration path |
decommission_date passed, or the usage is retired | Deny | Usage decommissioned on date; owner |
| Evaluation stale (see below) | Deny | Evaluation older than permitted; last evaluation id |
Request carries a data class outside data_classes_touched | Deny | Data class not approved for this usage |
| Request resolves to a model or version other than the row’s pin (recommendation, extending Book 3 Ch 16 §3’s refusal of unpinned aliases to the PDP) | Deny | Unpinned or drifted model |
| All conditions met | Allow, with the PDP’s normal constraints and expiry | — |
Stale evaluation. The books name the stale-evaluation deny but give no single age limit (Book 1 Ch 14 §4.4). Recommendation: treat an evaluation as stale when last_evaluation predates the current pin window’s reassessment date (Day +90 under the upgrade lifecycle), or predates the last change to model_version, or when the Day +60 attestation is overdue. Set the rule per risk tier in policy, and record the rule version on the decision.
One join key on every call. Each model invocation logs usage_id with the agent ID, timestamp, input classification, output classification and policy decision (source claim, Book 1 Ch 14 §4.4). The same log line then serves AI Act traceability and DORA incident investigation. Recommendation: carry the usage_id in the Audit Evidence Schema’s model[].registry_record field, which v0.1.0 already offers as an optional string, and make it mandatory by local profile for T2 and above. See audit evidence.
The registry is the source of truth; reconcile three places. Drift arises quietly between releases: a gateway hotfix answering a deprecation notice without an envelope amendment, or a wrangler editing a model field without going through the Virtual CAB (source claim, Book 3 Ch 16 §7). Run a reconciliation job comparing the registry pin, the gateway’s actual routing and the envelope’s Means field. The gateway reads its routes from the registry on a refresh cycle; an envelope amendment is the only path to change a Means field.
| Reconciliation finding | Action |
|---|---|
| Registry pin ≠ gateway routing ≠ envelope Means | Route to AI platform owner for same-day remediation; treat as drift |
| A family alias in any production model field | Treat as auto-upgrade exposure; remediation is a controlled change with CAB approval |
| Gateway traffic with no matching usage row | Unregistered usage; deny at PDP and open a registration finding |
Usage row with zero observed_qps over its review period | Candidate for retirement (recommendation) |
| Phase deliverable (for example Day +60 attestation) overdue | Page the wrangler; flag the envelope |
| Entitlement list in a vendor admin console differs from the registry | Reconcile line for line (Book 3 Ch 18 §5) |
Fallback routes need their own usage rows. A failover that sends traffic to an unregistered endpoint is an unapproved usage, however briefly.
Retire usages and keep point-in-time history
Retirement triggers
decommission_date forces the retirement question at registration: either a date, or evergreen with a next review date no more than 12 months out. A quarterly job escalates rows past their review date (source claim, Book 1 Ch 14 §8, anti-pattern 3; App B field 18).
| Trigger | Source | What happens |
|---|---|---|
| Review date passes without re-affirmation | Book 1 Ch 14 §8 | Escalated by the quarterly job; recommendation: PDP denies after a grace period set in policy |
| No wrangler will sign the row | Book 1 Ch 14 §1 (composite) | Retire the usage; an unowned usage cannot stay in production |
| The business process is retired | Book 1 Ch 14 §1 (composite) | Retire every usage on that process |
| The model is decommissioned | Book 1 Ch 14 §6 | All usages joined to that model retire on the same date; the registry is the only artefact that exposes this join |
| Third-party contract ends or is not renewed | Book 1 Ch 14 §6 | Align decommission_date to the contract renewal date |
| Provider deprecates or retires the pinned version | Book 2 Ch 13; Book 3 Ch 16 | Opens the upgrade cycle; see deprecation tracking |
| Pin window reaches Day +90 | Book 3 Ch 16 §4.1 | Reassessment is due, whether or not the provider has said anything |
| No traffic over the review period | Recommendation | Wrangler confirms retirement or justifies dormancy |
Re-affirmation cadence
Book 1 App B sets the field review cadences above. The Bank A composite in Book 1 Ch 14 §1 adds a re-affirmation practice: weekly wrangler attestation on the high-risk rows, monthly across the estate, and a printed diff between two dated snapshots laid side by side to show the register is right today, not on the day it was printed. App B has no “date last re-affirmed” field. Recommendation: record each re-affirmation as a dated attestation event in the row’s history rather than overwriting a single date, so that the history itself proves the cadence.
Keep point-in-time history
The question that defeats most registries is not “what is the usage?” but “what was the usage on a Tuesday three months ago?”. A firm that can produce only today’s row has no answer (source claim, Book 1 Ch 14 §7, internal-audit deliverables). Book 1 cites EBA supervisory material warning against exactly this. [FACT-CHECK: Book 1 Ch 14 cites an “EBA factsheet of 21 November 2025”; confirm the document and its wording before relying on it.]
Recommendation: make the registry append-only with a validity interval on every version of a row (valid-from, valid-to), and record who changed what, under which change ticket. Every change to a field with a review cadence, every re-classification, every pin window and every re-affirmation is a new dated version.
Internal audit’s three deliverables (source claim, Book 1 Ch 14 §7):
- An audit-evidence query template that returns, for any supervisory question, the usage tuple with wrangler, regulatory class, change ticket and data classification.
- A named-control-owner register tying each registry field to a first-line populator and a second-line reviewer, with a half-yearly reconciliation against the catalogue and the invocation logs recorded as a control test.
- A post-incident reconstruction protocol that recovers the usage record as it stood at the time of the incident and traces it to its change ticket.
Internal audit signs a half-yearly testability attestation: the registry reconciles to the agent runtime and the catalogue, and a sample of usages can be reconstructed end to end.
Third-party model usages
A model consumed from an external provider over an API may never appear in the platform catalogue. It still needs a usage row, and it is often the row that matters most (source claim, Book 1 Ch 14 §6, Group 3).
| Concern | What the row must carry | Why |
|---|---|---|
| Who provides it | contractual_party: the named third party for third-party models; the foundation-model vendor’s EU legal entity for hyperscaler-routed models; the firm itself for internal models (Book 1 App B field 15) | Drives the Register of Information line and value-chain records |
| Whose obligations | Where the firm deploys a third-party high-risk system, deployer obligations land on the firm, not the provider: use per instructions, human oversight, monitoring and logging | The obligation is invisible to the platform team without the row; see the regulatory map’s oversight and logging rows |
| Concentration | CIF flag; whether the provider is, or depends on, a designated critical ICT third-party provider (monitored by procurement) | Concentration and exit analysis per usage |
| When it ends | decommission_date aligned to contract renewal | Prevents auto-renewed contracts scored against an old evaluation (Book 1 anti-pattern 3) |
| Evidence of fitness | last_evaluation on the firm’s own scenario library, not the provider’s | The provider’s evaluation does not cover the firm’s use |
| Fallback | fallback_model, which may be a non-AI rule set | Resilience and exit |
Composite illustration (Book 1 Ch 14 §6, Bank A usage 12). An SME lending decision score from an external provider, under a third-party contract recorded in the Register of Information: high-risk class, CIF, its own scenario library (94.7% pass with one regression flagged), a rule-based legacy score as fallback, and decommission aligned to the contract renewal date. It sits in the same query and dashboard as the foundation-model rows, under the same wrangler discipline.
Supplier-hosted models reached by a floating name (Book 09 Ch 9, composite narrative). An agent reached its model through a supplier’s hosted service under a name meaning “the current model”. The firm’s rule pinned every deployment in its own subscriptions; the supplier re-pointed its name at a successor, and the agent’s judgement shifted for weeks with no release to approve. The only trace was the version string the evidence record kept on each response. The rule that followed, which this page adopts as a recommendation:
- the pinning rule reaches every model a production agent calls, wherever it is served, including inside a supplier’s SaaS;
- a supplier’s floating name is refused unless it can be fixed to a version;
- no cases, no move: no version moves under a production agent until its owner has run it against cases with known answers, and an agent with no cases does not move until it has some;
- record the model version as each response reports it in the evidence record, so a supplier-side change is at least detectable.
Treat model selection as a separate decision
Model selection happens inside a tier, never instead of one (source claim, Book 3 Ch 14 §8). A use case cannot drop a tier because a cheaper model was found. A lateral swap between model classes runs the same four phases as a version step, and is harder, because deterministic test assumptions travel less well across families than across versions (source claim, Book 3 Ch 16 §4.6).
The advisor pattern (Book 3 Ch 14 §8, Ch 16 §4.6). A mid-tier model does the work and a frontier model is attached as an advisor, invoked selectively. Book 3 reports Anthropic’s figure of a mid-tier model with a frontier advisor landing within 10% of the frontier model alone at 63% of the cost, as a vendor-internal measurement with no published methodology. The architecture transfers; the percentages do not. [FACT-CHECK: Anthropic, “Claude models explained”, 24 July 2026, figures and model names as cited in Book 3.]
Check availability and data posture at Day −14, not Day +30 (Book 3 Ch 16 §4.6, as of August 2026). Put two procurement checks in the Phase 1 ticket:
- Contractability. A model the firm cannot contract for is not a registry row, however well it evaluates. Book 3 reports that the frontier “Mythos” class was available only to selected organisations through “Project Glasswing”. [FACT-CHECK: current Anthropic model lineup and availability.]
- Zero-data-retention compatibility. Book 3 reports that the “Fable” class requires retention and is refused for zero-data-retention organisations regardless of client configuration, and that the
bestalias then resolves to Opus. For a firm that bought ZDR as a data-boundary control, the advisor pattern in its published form is unavailable. [FACT-CHECK: Fable/ZDR restriction andbestalias behaviour against current Anthropic documentation.]
Record ZDR and HIPAA eligibility for each new model ID on the registry extension at Day 0 (Book 3 Ch 16 §4.1). Routing by complexity, model entitlements and cost fallback are gateway mechanics; see the model gateway. One rule from Book 3 Ch 18 §5 belongs here: a cost-fallback substitution happens only where the mandate carries a cost-fallback-permitted flag, and the gateway logs each substitution as a policy decision. Recommendation, following Book 1’s grain: the substitute route needs its own usage row.
Run every model upgrade as a four-phase change
Why a cycle, not a project. Book 3 Ch 16 §2 and §8 record eight Claude versions in nine months and a structural change to the lineup within a year, and advises planning for a release every 8–12 weeks and 12–15 upgrade cycles over 18 months (author estimate). Two further observations from that chapter set the stakes. A version step inside one family can widen what an agent is willing to do with no change to its mandate or means: Anthropic has published a case where, after an upgrade from Opus 4 to Opus 4.5, an internal incident-response agent messaged another Claude instance to request a code fix, which was stopped only by a human pull-request review gate (Book 3 Ch 16 §2, Ch 17 §4.5). And Anthropic’s deputy CISO is cited as saying controls go obsolete roughly six months behind model evolution. [FACT-CHECK: Anthropic, “Zero risk isn’t the job”, claude.com/blog, 17 July 2026, as cited in Book 3.]
The lesson for model governance (source claim, Book 3 Ch 17 §4.5): containment that holds across a version step is structural (review gates, scoped access, agent traffic on channels humans watch), not textual. Re-test the containment posture at every new pin, not only answer quality; see evaluation and resilience. Log a near-miss of this shape as an envelope finding for the Agent Control Board.
Size the change by update scope
Not every model change needs the same treatment. Book 2 Ch 13 and Research Paper 1 §5.3 condense Microsoft’s lifecycle guidance into six update scopes (source claim; Microsoft guidance as cited by the books, accessed September 2026).
| Update scope | Example | Typically requires |
|---|---|---|
| Minor version | New dated release of the same model | Regression evaluation; prompt check |
| Variant | Faster or cheaper variant of the same family | Regression evaluation; latency and cost check |
| Major or generational version | New model generation | Full evaluation; prompt rework; possibly orchestration changes |
| Model change | Different vendor or family | Full evaluation; prompt and tool-calling rework; vendor onboarding |
| Specialised model | Domain-specific model | Re-assessed grounding needs; full evaluation |
| Hosting route change | Vendor API to private tenancy or controlled infrastructure | Infrastructure and security review; new residency assessment |
A prompt and the model version it was tested on are one change: they move together or not at all. The release unit is the pinned combination of model and version, parameters, prompt, orchestration, retrieval configuration and tool versions, recorded in a release manifest with its evaluation results (source claim, Book 2 Ch 13; Research Paper 1 §5.3). See BOM and recipes and release engineering.
The four phases
Dates are anchored to the public release (Day 0). Owners do not change between cycles (author model, Book 3 Ch 16 §4, OD-1).
| Phase | Window | Lead | Second | Sign-off | Output |
|---|---|---|---|---|---|
| 1 — Watch and pre-evaluation | Day −14 to Day 0 | AI platform owner | Head of model risk | Head of model risk | Registry update; evaluation-suite pre-run; written recommendation |
| 2 — Decision and staging | Day 0 to Day +30 | Wrangler | AI platform owner | Virtual CAB | Approved change dossier; staged rollout schedule; envelope amendment |
| 3 — Production observation | Day +30 to Day +90 | Wrangler | AI platform owner | Head of model risk (Day +60 attestation) | Day +60 suite re-run; wrangler attestation; telemetry record |
| 4 — Retention or rollback | Day +90 | Agent Control Board | Head of model risk | Agent Control Board | Logged decision (retain / roll back / carry); updated pin date; next reassessment date |
Phase 1 — watch and pre-evaluation.
- The AI platform owner watches the vendor sources weekly. Book 3 names four for a Claude estate as of August 2026: Anthropic’s release-notes overview, Microsoft Foundry’s models-from-partners page, AWS Bedrock’s model–region compatibility page, and the Claude Code settings reference that documents per-provider alias defaults. The last is the one teams forget (see alias drift). When a preview appears, open a Phase 1 ticket and notify model risk.
- On Day 0, the same working day: update the registry with the new model ID, version, provider, route, ZDR eligibility and HIPAA eligibility, leaving the pin date blank; the reconciliation job flags every envelope referencing the prior version; model risk checks out the evaluation suite at its current version and queues it against a sandbox deployment.
- Pre-evaluation runs 10–14 working days. Model risk owns the analysis; platform owns execution. The output is one recommendation: pin, pilot or migrate, on which workflows, over what cadence, with what reassessment trigger. It becomes the dossier for Phase 2.
Phase 2 — decision and staging.
- The Virtual CAB grades the change by the risk tier of the envelopes it affects. Book 3 numbers its change classes in the same “Tier” vocabulary as its risk tiers; this page keeps them apart. An upgrade affecting only T0 or T1 envelopes (internal-only, no customer impact) is a controlled change executed by the wrangler under the dossier, with the AI platform owner co-signing. An upgrade affecting any T2-or-higher envelope (customer-impacting, regulated or CIF-adjacent) takes full Virtual CAB review, second-line attestation, and a supervisor-notification assessment by the DPO and model risk. See change governance.
- Staged rollout starts the day after approval, per envelope:
| Week | Traffic on new model | Check |
|---|---|---|
| 1 | 5% | Gateway records the split; wrangler reviews the day-end evidence sample on Day +7 |
| 2 | 25% | Wrangler reviews on Day +14, with a cross-section from the prior week for comparison |
| 3 | 50% | Status to the Virtual CAB’s weekly slot; any envelope outside named thresholds pauses here, and rollback is an option, not an exception |
| 4 | 100% | Registry pin date updates to Day +28; reassessment date set to Day +90 |
- Two envelopes sharing a model never share a rollout schedule unless the Virtual CAB approves the synchronisation in writing in the dossier.
- Enforce the cadence at the gateway (source claim, Book 3 Ch 16 §7): traffic-split policies are versioned and signed, the gateway refuses to advance a split without a new policy artefact, and the AI platform owner co-signs each weekly step. A rollout collapsed to “5% on Monday, 100% on Tuesday” never runs long enough to catch tail-of-distribution regressions.
Phase 3 — production observation.
- Three telemetry streams run continuously against a 30-day baseline taken on the prior model in the fortnight before pre-evaluation: cost per task; hallucination rate against the deterministic subset, replayed weekly on fresh traffic samples; and tool-call rate and refusal rate, the behaviours most likely to drift between minor versions. Each stream’s threshold is named in the rollback plan in advance.
- On Day +60 the suite runs again: same suite version, same deterministic cases, same distributions, same scenarios. The wrangler files an attestation in the change record (re-run done, results consistent with pre-evaluation, envelope within thresholds, retention recommended through Day +90) and joins it to the registry row. If it is not filed, the reconciliation job flags the envelope and the platform owner pages the wrangler.
Phase 4 — retention or rollback. The quarterly Agent Control Board reviews the cohort and records one of three outcomes:
| Outcome | Effect |
|---|---|
| Retain | Model stays at 100% until the next release re-opens the cycle |
| Roll back | Gateway header reverts; registry pin returns to prior version; wrangler attests; evidence-portability check runs (below) |
| Carry | Retained on this envelope, with a wrangler-filed constraint: no extension to a sibling envelope until the next pre-evaluation |
Record the outcome on the registry row together with the change-record ID, the suite version used both at pre-evaluation and at Day +60, the period the telemetry was sampled over, and, for every measurement, which named threshold it was tested against. A supervisor reads this entry.
Every pin expires at Day +90 (source claim, Book 3 Ch 16 §4.1). Because provider retirement dates arrive as notices rather than a schedule, the firm sets its own reassessment date instead of inheriting the vendor’s.
Route mechanics (as of August 2026, per Book 3 Ch 16 §4.5)
The cycle is the same on every route; only the model header and the staging mechanism differ. Run one cycle per release, not one per route.
| Route | Model header | Staging | Rollback |
|---|---|---|---|
| Microsoft Foundry | Foundry deployment name | API Management policy fragment switching the upstream URL | Inverse fragment |
| Amazon Bedrock | Bedrock model identifier on the bedrock:InvokeModel call | Lambda authorizer weighting two model IDs | Invert the weight |
| Google Vertex AI | Anthropic model ID with regional endpoint per residency posture | Traffic split keyed by service account | Reverse the split (Book 3 does not state the Vertex rollback; inferred by symmetry) |
| Direct Anthropic API behind API Management | Anthropic model ID in the JSON body | API Management policy | Policy-versioned revert |
[FACT-CHECK: per-route identifier formats and staging mechanisms against current Microsoft, AWS, Google and Anthropic documentation.]
Composite illustration (Book 3 Ch 16 §1, §6, Bank A). A cross-family evaluation on three envelopes found a new tokeniser raising token counts 1.26× on the regulatory-summary corpus, compounding with list price into a 2.7× cost per task; two of 50 deterministic cases regressed (4%, below the 5% threshold but ruled material by the head of regulatory affairs, because the misread was the article number the summary existed to report); and code-review correctness improved 4.3 points on a 100-case merge-history corpus. Outcome: pin the regulatory-summary cohort on the prior model, migrate developer assistance on the four-week schedule, and hold the peer-benchmark envelope pending its own evaluation. Thresholds trigger a decision; they do not replace the business owner’s judgement on materiality.
Roll back on named thresholds and prove evidence portability
Thresholds (author model, Book 3 Ch 16 OD-2). Agree them in advance and name them in the change dossier. Rollback fires if any is exceeded during Phase 3, or if the Day +60 attestation cannot be filed.
| Threshold | Definition | Measured against |
|---|---|---|
| Cost variance | Cost per task on the new model exceeds the prior-model baseline by more than 30% on a rolling 14-day window | Gateway billing record, normalised to the envelope’s representative task |
| Evaluation regression | Deterministic-layer regression rate above 5% at the Day +60 re-run, or any single material business-outcome scenario fails | Day +60 evaluation-suite report |
| Incident-rate increase | Per-envelope incident rate up more than 100% on a rolling 30-day window against the prior model’s 30-day baseline | Incident record (incident response) |
| Attestation missing | Day +60 wrangler attestation not filed | Change record |
Rollback steps (Book 3 Ch 16 OD-2). Each step leaves a named artefact; the rollback is not complete until all six exist. OD-2 names the AI platform owner for step 1 and the wrangler for step 3, and the Virtual CAB receives step 6; the other owners marked rec. are this page’s recommendations.
| # | Step | Owner | Time limit | Artefact |
|---|---|---|---|---|
| 1 | Pause progression; revert the traffic split to 0% on the new model | AI platform owner | 30 minutes at 5% or 25%; one working hour at 50% or 100% | Gateway policy diff |
| 2 | Revert the gateway model header to the prior pinned version; log the rollback change-record ID on the policy diff | AI platform owner (rec.) | — | Rollback change record |
| 3 | Wrangler attests: threshold breached, time of breach, steps taken, registry row updated | Wrangler | — | Signed attestation |
| 4 | Evidence-portability check: replay query over the rollout cohort’s correlation IDs | Platform with evidence owner (rec.) | — | Replay output appended to the change record |
| 5 | Resume on the prior model: set the new model’s pinned-to date to the rollback date; re-open the prior version’s row | Wrangler (rec.) | After step 4 passes | Updated registry history |
| 6 | Post-rollback review at the next Virtual CAB slot: cause, evidence, control improvement filed against the suite or the staging policy | Wrangler writes it up (rec.); Virtual CAB reviews | Next standing slot | CAB minute |
The evidence-portability check is the gate (source claim, Book 3 Ch 16 §5.2, OD-2). Every audit-evidence record’s model-ID field must match the model that actually served the call according to the gateway state at that time, and the records at the rollback boundary must reference the rollback change record. A rollback that reverts the header but leaves boundary records whose model ID does not reconcile breaks the reconstruction of who did what, on whose authority. See audit evidence.
Rollback target must still be served (recommendation, from Book 2 Ch 13). You can only roll back to a version the provider still serves. Before Phase 2 approval, confirm the prior version’s retirement date falls after the end of the pin window; if it does not, the dossier must name a different fallback.
Deployment manifest rules that bind here (Supporting Tools, Deployer Manifest v0.1.0, as cited in Book 3 Ch 16 §3). required_cab_decision is always present (null only with a cited cab_exemption_rationale); verify.health_checks must be non-empty; and rollback.verification is required. Where prose and specification disagree, the specification wins. See schemas and tools.
Pin full model IDs and close alias drift
The rule. Every production model reference names a full model ID, never a family alias, in the envelope’s Means field, the release manifest, the gateway route, workstation settings and the registry row. Automatic upgrades are disabled in production, minor versions included (Book 2 Ch 13, author preference; Research Paper 1 §5.3). An alias resolving to a newer version without an envelope amendment is a policy violation: the route is configured to refuse it, the reconciliation job detects it, and the wrangler is paged (Book 3 Ch 16 §3). Remediating an aliased envelope is itself a controlled change requiring CAB approval (Book 3 Ch 16 §7).
Recognise auto-upgrade by a family alias anywhere in a production model field, by the reconciliation job finding the gateway on a different version from the registry, by an unexplained step in the token bill, or by answer quality shifting on a date matching no change in your history (Book 2 Ch 13; Book 3 Ch 16 §7).
Alias drift across routes (as of August 2026, per Book 3 Ch 16 §4.5 and Ch 18 §5)
Family aliases (opus, sonnet, haiku, fable) resolve to per-provider defaults that move on the provider’s schedule and, per Book 3, did not move together: opus resolved to Opus 5 on Bedrock and on what Book 3 calls the “Agent Platform” route, but to Opus 4.6 on Microsoft Foundry. A firm with opus in both a Foundry fleet and a Bedrock fleet is running two models under one word while its registry records one. Book 3 also reports a billing edge: an unpinned Claude Code deployment is billed at the Opus rate from version 2.1.207, because the primary default moved to an Opus-class model. [FACT-CHECK: current alias resolution per provider and the Claude Code v2.1.207 default-model change, against the Claude Code model-configuration and cloud-provider documentation.]
Claude Code settings (as of August 2026, per Book 3 Ch 8 §4.3 and Ch 16 §4.5)
| Setting | Purpose | Failure if missing |
|---|---|---|
ANTHROPIC_DEFAULT_OPUS_MODEL, ANTHROPIC_DEFAULT_SONNET_MODEL, ANTHROPIC_DEFAULT_HAIKU_MODEL | Pin full model IDs per environment, in managed settings | Aliases follow the provider’s default |
availableModels | The enterprise model allow-list | — |
enforceAvailableModels: true | Closes the picker’s “Default” option, which otherwise resolves to the account’s runtime default; Book 3 cites v2.1.175 or later | The allow-list leaks by design: users on “Default” reach a model outside the list |
/status | Shows the resolved provider, region and model | Include its output in the Phase 3 evidence sample |
A minimal managed-settings shape, with placeholders rather than real model IDs:
{
"env": {
"ANTHROPIC_DEFAULT_SONNET_MODEL": "<full-approved-model-id>",
"ANTHROPIC_DEFAULT_HAIKU_MODEL": "<full-approved-model-id>"
},
"availableModels": ["<full-approved-model-id>", "<full-approved-model-id>"],
"enforceAvailableModels": true
}
[FACT-CHECK: confirm the managed-settings key names, the env block placement, and the minimum Claude Code version for enforceAvailableModels against current Claude Code settings documentation.]
Composite illustration (Book 3 Ch 8, Bank A pilot). A workstation baseline shipped availableModels without enforceAvailableModels. For a week, four engineers whose picker sat on “Default” resolved to a model outside the approved list. The usage registry showed it before anyone reported it.
Name the enforcement boundary honestly (source claim, Book 3 Ch 8, Ch 16 §4.5). availableModels is enforced client-side, by software on an endpoint the user controls. Server-managed settings from the Anthropic admin console do not reach sessions running against Bedrock, Google’s Agent Platform or Microsoft Foundry (Book 3 Ch 8 §4.2), so those fleets are managed by MDM only, and Claude Enterprise’s server-enforced model restrictions do not apply on those routes either. Research Paper 3 (checked against vendor documentation on 10 November 2026) adds that Claude Code skips the server-managed fetch whenever a cloud-provider variable or a non-default ANTHROPIC_BASE_URL is set, so a fleet routed through the firm’s own gateway is device-policy only too (Research Paper 3). Book 3 Ch 18 §5 adds that Enterprise model entitlements can restrict models by role or organisation-wide, and the entitlement list should reconcile line for line against the registry. [FACT-CHECK: Anthropic admin post of 2 July 2026 on model defaults and entitlements, as cited in Book 3.] On every route, the gateway, not the endpoint, is the binding control: it refuses any model ID not pinned on a registered usage. Client settings reduce noise; they are not the control of record. See the model gateway and Research Paper 3.
Platform auto-upgrade settings
| Platform | Behaviour (as reported by the books) | Action |
|---|---|---|
| Azure OpenAI in Foundry, Standard deployments | Three versionUpgradeOption values: OnceNewDefaultVersionAvailable (moves within two weeks of a new default), OnceCurrentVersionExpired (moves at retirement), NoAutoUpgrade (never moves; stops working at retirement). A deployment with no value behaves as OnceCurrentVersionExpired, so an unconfigured deployment still upgrades itself at retirement | Set NoAutoUpgrade explicitly on every production deployment; track retirement so the deployment does not simply fail. Book 09 Ch 7 records a composite firm refusing seven production deployments that had no version set |
| Amazon Bedrock | Model identifiers carry the version (for example anthropic.claude-sonnet-4-5-20250929-v1:0), so calls are pinned by construction. Lifecycle states are Active, Legacy and End-of-Life; the EOL date is added to the model card when Legacy begins; after EOL requests fail and migration is not automatic | Track EOL dates as hard deadlines |
| Google Vertex AI | Each GA model ID is listed with release date, retirement date and replacement model ID; retirement dates are not moved earlier than listed | Name the exact model ID; track retirement |
Sources: Book 2 Ch 13 and Book 09 Ch 7, citing Microsoft Learn, AWS and Google Cloud documentation accessed November 2026. [FACT-CHECK: re-verify versionUpgradeOption values and default, Bedrock lifecycle states, and Vertex retirement-date policy against the vendor pages before relying on them; this page has not checked them directly.]
Track vendor deprecation and retirement
Two failures arrive from the provider side that a conventional release path never handled: the upgrade you did not choose and the retirement you did not plan for (source claim, Book 2 Ch 13). A deprecated version cannot be newly deployed; a retired one returns errors.
What to hold (recommendation). The catalogue, not the usage registry, holds deprecation facts, one record per model version per route: route, model ID, deprecation date, retirement date, replacement model ID, source page, and the date the facts were last checked. The registry then derives, for every usage, the earliest retirement date among its primary and fallback models.
What to do with it.
- The weekly Phase 1 watch updates the catalogue record. [FACT-CHECK: Book 3 Ch 16 §4.1 states that as of August 2026 Anthropic published no consolidated model deprecation schedule; check whether Anthropic’s documentation now has a deprecations page and, if so, add it to the watch list.]
- Open re-evaluation well ahead of any retirement date (Book 2 Ch 13; Research Paper 1 §5.3 factory rules). Recommendation: open Phase 1 no later than the point where a full cycle (pre-evaluation plus four weeks’ staging) can still finish before retirement.
- Because notices are not a schedule, every pin also expires at Day +90 on the firm’s own clock.
- A usage whose primary model retires before its next reassessment is escalated to the wrangler; a usage whose fallback retires is flagged too, because a fallback that no longer exists fails exactly when it is needed.
Build the evaluation suite in three layers
The suite is the operational evidence that an upgrade can be defended (author model, Book 3 Ch 16 §5). The general evaluation contract, trajectory tests and control-boundary tests are on evaluation and resilience; this section gives the model-change layer.
| Layer | Content | Detects | Owned or signed off by |
|---|---|---|---|
| Deterministic | About 50 known-good inputs with named expected outputs, drawn from the production workflow corpus; grows by one case per production incident for the next quarter | Regression: something the prior model got right that the new one gets wrong | Wrangler and business owner, per case |
| Probabilistic | Same prompts run N times, typically 20; output distribution scored for variance, refusal rate, length and confidence-marker frequency | Sensitivity: output distributions that were stable and now oscillate | Model risk (recommendation; the source names no signatory for this layer) |
| Business-outcome | 5–15 real workflow scenarios, each with explicit acceptance criteria | Fitness for the actual job; a deterministic gain with a business-outcome loss is not an improvement | Wrangler and business owner write the acceptance criteria |
Rules that make the suite defensible (source claim, Book 3 Ch 16 §5).
- Build from your own production workload. Book 3 notes Anthropic’s own guidance that public benchmarks are saturating and that customers should build custom evaluations; a leaderboard delta is not evidence for your workflow.
- Sequence. At pre-evaluation, run the layers in order, deterministic first, so a regression closes the gate before the others spend compute. At Day +60, run them in parallel with the business-outcome layer weighted.
- Ownership and versioning. The suite lives in the model-risk repository, not the platform team’s, with a version number, changelog and signed model-risk attestation per release. The version run at pre-evaluation is the version run at Day +60; changing the suite mid-cycle invalidates the comparison and forces a re-evaluation.
- Code scenarios. Where AI assistance is the default path for code change, swapping the model changes the effective authorship of everything merged afterwards, so the business-outcome layer carries a code scenario (Book 3 uses a 100-case merge-history corpus as the minimum shape).
- Containment re-test. Each new pin re-runs the control-boundary tests (Book 3 Ch 17 §4.5; see above).
- Divorce from production is the failure: a suite of synthetic or public cases that passes while production regresses. Recognise it by a case set not updated against the prior six months’ incidents and traffic.
Ownership
| Role | Owns |
|---|---|
| AI platform owner | Catalogue; vendor watch; Day 0 registry update; rollout execution and gateway policy; reconciliation job; rollback steps 1–2 |
| Head of model risk | regulatory_class assignment; evaluation suite (repository, versions, attestations); pre-evaluation analysis; Phase 1 and Day +60 sign-off |
| Wrangler (named human per usage) | business_process; signature on the classification; row reviews and re-affirmation; Phase 2–3 lead; Day +60 attestation; rollback attestation |
| Legal and compliance | Review of classifications; supervisor-notification assessment (with the DPO) for upgrades touching T2+ envelopes |
| Data governance | data_classes_touched taxonomy; agreement on the grain |
| Procurement and third-party risk | contractual_party; contract-aligned decommission dates; critical-provider monitoring |
| Virtual CAB | Grading and approval of upgrade changes; rollout synchronisation approvals; post-rollback review |
| Agent Control Board | Phase 4 decision (retain / roll back / carry) |
| Internal audit | Query template; control-owner register and half-yearly reconciliation test; reconstruction protocol; half-yearly testability attestation |
See accountability for how these seats sit in the wider ownership model.
Failure modes and recovery
| Failure | Recognise it by | Recover by | Source |
|---|---|---|---|
| Catalogue treated as model inventory | Board and supervisor reporting by model count; no answer to “which models touch decision X?” | Define the grain; build the registry; report at usage grain from the next quarterly model-risk forum | Book 1 Ch 14 §8 |
| Risk class on the model | Every credit-relevant model high-risk for every use; general-purpose models minimal by default, including their high-risk uses | Move regulatory_class to the usage row; require per-usage classification at registration | Book 1 Ch 14 §8 |
| Tier mapped to model | The same use case gets different controls depending on model choice | Set tier against use case; choose model inside it | Book 3 Ch 14 §8 |
| Registry without retirement | All rows “evergreen”; hundreds of rows, nobody sure which are live; third-party rows on old evaluations | Evergreen only with review date ≤ 12 months; quarterly escalation job | Book 1 Ch 14 §8 |
| Registry not tied to a wrangler | Owner is a team or mailbox | owning_wrangler by HR identifier; reviews go to that person | Book 1 Ch 14 §8 |
| Usage registered, tools not | Usage row with no link to the MCP tools it depends on | Cross-reference the MCP registry; one query, not two | Book 1 Ch 14 §8; see MCP and connector broker |
| No point-in-time history | Only today’s row can be produced for a past incident | Append-only, dated row versions; reconstruction protocol | Book 1 Ch 14 §7 |
| Auto-upgrade | Alias in a production field; gateway on a different version; unexplained bill step | Pin full IDs; switch auto-upgrade off; remediate as a controlled change | Book 2 Ch 13; Book 3 Ch 16 §7 |
| Supplier floating name | Version string on responses changes with no release | Extend pinning to supplier-hosted models; no cases, no move | Book 09 Ch 9 |
| Allow-list without enforcement | Users on “Default” reach unapproved models | enforceAvailableModels: true; gateway refuses unregistered IDs | Book 3 Ch 8 |
| Suite divorced from production | Suite passes, production regresses | Bind suite growth to incident review | Book 3 Ch 16 §7 |
| Staged rollout collapsed to instant | Split jumps from 5% to 100% within days | Gateway-enforced, signed traffic-split steps | Book 3 Ch 16 §7 |
| Rollback breaks evidence | Boundary records’ model ID does not reconcile with gateway state | Evidence-portability check as the rollback gate | Book 3 Ch 16 §7 |
| Registry drift | Registry, gateway and envelope disagree | Registry as source of truth; three-way reconciliation; amendment as the only path | Book 3 Ch 16 §7 |
| Behaviour widening on a version step | New actions or channels used with no envelope change | Re-test containment at every pin; log as envelope finding | Book 3 Ch 17 §4.5 |
Acceptance checks
Registry
- The grain definition is written and signed by model risk, agent platform, data governance and compliance.
- Every production endpoint and every caller seen in the last 90 days of telemetry maps to at least one usage row; unmatched callers are open findings.
- No production row has a blank
business_process,regulatory_class,owning_wranglerordecommission_date; every evergreen row has a review date no more than 12 months out. - Two rows calling the same endpoint with different classes can each be defended from
business_process,data_classes_touchedandcalling_agent_id. - Third-party model usages appear in the same registry, with
contractual_partymatching the Register of Information. - A wrangler’s “my usages” query returns in under 30 seconds; a sample supervisor question is answered within one working day.
- A usage can be reconstructed as it stood on a past date, with its change ticket.
Runtime
- A call for an unregistered, retired, stale-evaluation or out-of-data-class usage is denied with rationale (test each).
- Every model invocation record carries
usage_id, and that ID joins evidence records to the registry. - The reconciliation job runs on schedule and its last three runs show registry, gateway and envelope in agreement, or open remediation items.
- A request naming a family alias, or a model ID not pinned on the usage, is refused at the gateway.
Upgrade lifecycle
- Each open model change has a Phase 1 ticket, a pre-evaluation recommendation, and a dossier naming its rollback thresholds before staging starts.
- The gateway shows 5/25/50/100% steps with signed policy artefacts and the platform owner’s co-signature.
- The Day +60 suite run used the same suite version as pre-evaluation, and the wrangler attestation is filed.
- A rehearsed rollback produced all six artefacts, including a passing evidence-portability replay, within the stated time limits.
- Every production Azure OpenAI deployment shows
NoAutoUpgradeset explicitly; every Claude Code managed-settings profile setsenforceAvailableModels: trueand pins full model IDs. - Every pinned model and fallback has a recorded retirement date (or a recorded “none published” with the check date), and no pin is older than Day +90 without a Phase 4 decision.
Provenance and refresh
The registry, grain, classification principle, population sequence, sizing and anti-patterns come from Book 1 Chapter 14 and Appendix B. The upgrade lifecycle, evaluation suite, thresholds, rollback playbook, alias drift and Claude Code settings come from Book 3 Chapters 8, 14, 16, 17 and 18. Release pinning, update scopes and platform auto-upgrade behaviour come from Book 2 Chapter 13 and Research Paper 1 §5.3; the supplier floating-name case comes from Book 09 Chapters 7 and 9. The page restructures these for operating decisions rather than reproducing them.
Source conflicts and how this page resolves them.
- Row grain. Book 1 App B’s collection 12 description (one record per model, agent and time-window) differs from its own registry section (deployment, caller, business process). This page follows the usage tuple and keeps time as row history.
- Field set. Book 1 App B’s 18 fields and Book 3 Ch 16 OD-3’s 18 fields are different sets. This page uses Book 1’s as the usage row and Book 3’s lifecycle fields as a pin-window extension. Book 1 Ch 14 §1’s scene lists yet another set (including call counts and a re-affirmation date) and §4.2 describes blocks differently from App B; this page follows App B.
- Change-tier naming. Book 3 Ch 16 calls upgrade change classes “Tier 2” and “Tier 3”, reusing the risk-tier vocabulary. This page describes them by their effect and keeps T0–T4 for risk tiers.
Vendor claims. Model names, alias resolution, Claude Code versions and settings, Azure, AWS and Google lifecycle behaviour, and Anthropic availability and ZDR rules are dated to the book that reports them and marked [FACT-CHECK]. They change with vendor releases; this page’s refresh trigger is a vendor release. Regulatory classification and dates are on the regulatory map.
Source material
- Book 1, Chapter 14, AI Model Inventory and Foundry Governance (§3 thesis; §4.1–4.4 catalogue versus registry, grain, classification, control-plane integration; §6 composite registry; §7 Monday morning and sizing; §8 anti-patterns)
- Book 1, Appendix B, Operating Schemas: AES, Control Plane, Model Usage, Data Dictionary and Change Dossier (the 18-field model usage registry; population sequence; collection 12 model_usage)
- Book 2, Chapter 13, Release Engineering and Change Resilience (release pinning; update scopes; automatic-upgrade rule; the model that upgraded itself)
- Book 3, Chapter 8, Claude Code at the Workstation (availableModels and enforceAvailableModels; third-party-provider settings asymmetry)
- Book 3, Chapter 14, Risk Tiers and the Use-Case Taxonomy (§8 tier mapped to model; model selection orthogonal to tier)
- Book 3, Chapter 16, Model Upgrades as Change Events (four-phase lifecycle; evaluation suite; per-route mechanics; advisor pattern; OD-1 cadence table; OD-2 rollback playbook; OD-3 registry row)
- Book 3, Chapter 17, Incident Response for Agent Failures (§4.5 behaviour change after a model upgrade)
- Book 3, Chapter 18, Cost, Performance and Gateway Operations (§5 model entitlements and pinned defaults; routing by complexity)
- Book 09 (Everything Is Green), Chapter 7, The Machine That Says No (pinning production deployments)
- Book 09 (Everything Is Green), Chapter 9, One Query (supplier floating names; no cases, no move)
- Research Paper 1, A Taxonomy for a Reuse-Driven AI Factory (§5.3 model lifecycle and release pinning)
- Research Paper 3, Securing Claude Code in the Enterprise (where server-managed settings stop)
- Supporting Tools, Deployer Manifest v0.1.0 and Audit Evidence Schema v0.1.0
Related guidance
-
Design and Operate the Model GatewayBuild the model gateway as the governed edge for every model call: per-client identity, per-request highest-class routing, tokenisation, pinned models, a conformant wire contract, a correlated evidence record, a tested concentration answer, and day-two cost, latency and reliability operations across two estates and three meters. -
The Enterprise Agent Control PlaneThe firm-owned services that make every agent action identifiable, decidable, approvable and reconstructable: the ownership boundary, six planes and their contracts, the Policy Decision API with its latency budget, the persistence schema, the build sequence and the tests that show it works. -
Classify Use Cases into Risk TiersThe T0–T4 use-case taxonomy as a working decision reference: the prohibited-use gate, four intake questions, data and action boundaries per tier, the twelve-area control matrix, retention and payload rules, runtime enforcement, approval gates, regulatory routing and re-tiering. -
Route Regulatory Questions to Obligations and EvidenceA dated, article-level reference that routes AI design questions to the EU AI Act, DORA and its RTS, GDPR and sectoral rules, and turns each obligation into a control, an owner and evidence: change records under RTS 2024/1774 Art 17, the 4 h / 72 h / one-month incident clocks, the CIF hinge and its five artefacts, the three-count Register of Information reconciliation, Art 14(4) oversight controls, the Art 26(5) discharge, the FRIA gate, Art 49 and Art 50, SCHUFA-grade review and supplier controls. -
The Audit Spine and the Evidence RecordHow to capture, join, protect, retain and prove per-action evidence: the AES v0.1.0 record contract, three capture points and one correlation ID, tiered immutable storage, retention and legal hold, replay rehearsal, the route-approval evidence gate, and the Claude-specific ways evidence goes missing. -
Evaluate, Assure and Test Agent ResilienceDefine a versioned evaluation contract, test the trajectory and the execution boundaries, run three assurance layers into one quarterly pack, map the OWASP Agentic Top 10 to controls that demonstrably fire, bring agents into threat-led penetration testing with a five-section scope, keep a continuous cadence between formal tests, land every finding as a live control-plane artefact, and engineer recovery down to a minimal AI service level on models you host. -
Release Engineering for Agent-Written ChangeMove agent-speed change into production through five evidence-emitting stations: one change type per change, a reversal decided at authoring, a valid Deployer Manifest, a gateway that cannot be bypassed, hard health gates and at-most-once rollback. -
Change Governance for Agent-Enabled DeliveryRoute agent-speed change by risk without losing the approver: control values that always reach a person, a routing gate that reads the right signals, autonomy earned per change family, a Virtual CAB that prepares a complete dossier, and three bodies with defined escalation. -
Operate the Agent LifecycleInventory, register, sign, zone, review, suspend, recover and retire agents through a six-stage lifecycle with named human ownership, zone-specific authority and evidence at every transition. -
Accountability and OwnershipHold provenance and accountability apart on purpose: four named accountabilities per product, a human owner for every agent change, seven named seats, a signed Build/Run RACI and a first-line/second-line boundary that survives consolidation. -
Schemas, Conformance Packs and the ops ToolchainWhat the versioned Supporting Tools schemas hosted under /aes/v0.1.0/ require, how to validate against them, where the source specification contradicts itself, and what the planned ops CLI, Claude plugins and service conventions define. -
Respond to Agent IncidentsSort an agent failure into one of four incident shapes, suspend it in three verified layers, keep humans on every classification and disclosure decision, route one playbook to every authority that needs a cover sheet, and close each incident with a named control change.