Worked Example: An Inspection Answered From the Record
Composite supervisory walkthrough: for each question a supervisor asks about automated approval and agent actions, the evidence that answers it, the query that produces it, who runs it, and the questions the record cannot answer without a written rule.
- Version
- 0.1.0
- Owner
- DEIXEI
- Last reviewed
- Reading time
- 26 min
For Cto · Ai Architect · Platform Engineer · Risk And Assurance · Internal Audit · Compliance
Sources and details
- Section
- Worked examples
- Source works
- Book 3 · Book 09
- Source chapters
- 8, 9, 11, 13, 20
- Tool specifications
- audit_evidence_schema@0.1.0
- Refresh trigger
- source change
Case status. Everything on this page is composite. Asset Manager X (Book 09) and Insurer I (Book 3) describe no real firm, supervisor, depositary or inspection. Book 09 states plainly that nothing in its account describes how any real supervisor conducts an on-site inspection: the questions are the ones the authors would expect a well-prepared inspector to put to an instrument like this. Figures are each firm’s illustrative numbers. Legal characterisations (what is reportable, what obligation a record meets, the deadlines) are not restated here; they are on the regulatory map and the regulatory calendar.
What this page is for
The reference pages describe the record (audit evidence), the gate (change governance) and the band (control budget). This page walks the other direction: start from what a supervisor asks, and trace it to the evidence and the query that produce the answer. Use it to rehearse. For each question, you should be able to name the record, the query, the person who runs it, and the independent person who checks it, before anyone asks.
The pattern the cases share: the record answers who, what, when and on what evidence in minutes. It cannot answer on what basis unless a rule existed before the decision. Most of the hard moments in the inspection are the second kind.
The two cases in brief
| Asset Manager X (Book 09 Ch 9, 13) | Insurer I (Book 3 Ch 11) | |
|---|---|---|
| Trigger | A thematic letter on automated approval of technology change, sent to firms whose questionnaire answers said they automated approval; later an announced on-site review | A customer complaint naming an AI tool; the supervisor asks for a full reconstruction of one agent-drafted letter |
| Underlying event | An agent changed a cash-break tolerance default (a unit error made the band about a hundred times wider); a routing gate approved it alone; separately, an alerting agent’s supplier-hosted model moved version behind an unchanged name | None found; the request tested whether the firm could reconstruct |
| What was asked | Every change to reconciliation controls approved without human review since a date: approver, test evidence relied on, deployment route; whether the scope of automated approval had changed and the supervisor been told | What the customer-correspondence agent drafted and how that draft reached the moment a manager pressed send |
| Deadline / time taken | Fifteen working days allowed; answered in two days | 96 hours allowed; query in 11 seconds, packet in four hours, sent a day early |
| Evidence shape | One evidence record, one entry shape for every agent action, pipeline run, approval (human or automated), deployment and rollback, written by the act at the moment it happened, in storage that cannot be altered | Gateway, agent-end and approval-queue records joined on one correlation ID, in a SIEM hot tier and an immutable long-term store |
Book 3’s supervisor did not ask what the agent did; it asked whether the firm could tell. That is the question both cases answer.
Question, evidence, query: the master table
Each row is one supervisor question from the composites, the evidence that answers it, the query or procedure that produces it, and where the controls behind it are specified. Rows 1 to 4 come from the thematic letter and its follow-up (Book 09 Ch 9); rows 5 to 12 from the on-site review (Book 09 Ch 13); row 13 from Book 3 Ch 11; row 14 from Book 3 Ch 20.
| # | The supervisor asks | Evidence that answers | Query or procedure that produces it | Governing page |
|---|---|---|---|---|
| 1 | Describe changes to reconciliation controls approved without human review since a date: approver, test evidence relied on, deployment route | The evidence-record entry per change: initiator, executing agent and identity, envelope in force, action, data read, model and version, approver (for the gate: score, inputs, line in force), timestamps, outcome | All changes approved without a person in the window, run twice, independently, then a published funnel (see the funnel) | Audit evidence |
| 2 | Has the scope of automated approval changed since your earlier answer; when; was the supervisor told? | The record of the threshold change (who, when, under what authority, evidence attached, the objection minute one link away) and the share of change approved alone before and after | History of the gate’s configuration; share approved alone per period | Change governance |
| 3 | Who decided the threshold in force on the date of the event, and on what basis? | Who: the threshold record. Basis: only a rule written beforehand can answer this | Retrieve the threshold record; then produce the rule, if one exists | Change governance: moving the line |
| 4 | (Implicit in row 1) Are there others like it? | The full population, not only the known change | The funnel; manual reading of the residue with each record open | Audit evidence |
| 5 | Who set the gate’s line, on what basis, and from when? | Threshold record plus the shadow and live record it rested on; the rule for setting and moving the line written since | Threshold record; count of changes in the evidence window, and how many were of the kind that later failed | Change governance |
| 6 | What is your limit on the pace of change measured on, and who can move it? | The counting rule (what counts as understood) and the signed band, with who signed each | Per-domain count against the band, by the second-line analysts under access the delivery side cannot revoke | Control budget |
| 7 | Is the gate an approver independent of the people whose changes it approves, and on whose analysis? | Compliance’s acceptance note, its conditions, and any later reply or withdrawal | Compare the line the acceptance covered with the line the gate actually ran on, and the dates | Change governance: the gate as independent approver |
| 8 | What has changed since your last answers, with documents? | Change records for the gate itself: new score inputs, carve-outs, control-value list | Gate’s own change history | Change governance |
| 9 | Request-list item: management information used to monitor whether review capacity keeps pace with change | Arrival of change against rate of understanding, per domain, with the signed band shaded, twelve weeks of positions including weeks when a domain spent its budget | Weekly per-domain series from the record | Control budget, factory measurement |
| 10 | (Interview, staff alone) Walk us through a change you made: what you read, what you checked, where it shows | The approval entry with the reviewer’s note of what was checked | Retrieve the entry live, in front of the inspector | Audit evidence: twenty-change oversight sample |
| 11 | (Interview) How often do you approve a change you did not fully understand? Show us a gap | Approvals where the “what was checked” field is blank | Reviewer’s own approvals with a blank check note, last week | Audit evidence |
| 12 | Show one change the gate sent to a reviewer on test origin alone | The routing reason stored with each gate decision | Gate decisions where the only routing reason is test provenance | Change governance |
| 13 | Reconstruct what the agent drafted and walk it to the moment of sending | Gateway record, agent-end tool records, approval-queue entry with edits and justification, mail-out record | One join on the correlation ID (Book 3’s KQL below) | Audit evidence: capture points and joins |
| 14 | What is your exit strategy for the provider? | The annual exit-rehearsal close-out: use case, scope, timings, evidence-portability gaps, continuity quantification | Not a query: a rehearsal that must already have happened | Vendor concentration and exit |
The band questions the inspectors asked on day two (calibration length, conditions for moving it, equal width across domains, falling capacity under a fixed limit, who tests the counting, what the budget held back) are tabulated, with owners and what a prepared answer contains, on control budget. They are not repeated here.
Phase 1: the written question and the one query (Book 09 Ch 9)
What the record held for one change
The record’s own nine questions, applied to the four-line tolerance change, are the model for row 1. Two rows needed care when read:
| Record question | What it said | The care needed |
|---|---|---|
| Who initiated the action? | An analyst, through the standard request workflow | She was not the owner. She had asked for a feed to be onboarded; she dispatched no agent and never saw the change. The firm’s rule makes the owner whoever dispatched or merged it, and here a workflow dispatched and the gate merged. See accountability |
| Under what envelope? | May change reconciliations code and configuration; matching logic goes to a person | The envelope was silent on tolerances because a tolerance had never been called a control. The record shows the limits that were written, not the limits that were never written |
The other rows (executing agent under its own identity with a named owner, the action with its tests, the data read, the pinned model, the gate’s score and inputs and the line in force, timestamps, merged and deployed with no rollback) were complete. Book 09 notes that the gate’s designer had insisted, months earlier, that an automated approval leave the same entry a human one does; at the time it looked like tidiness. It is what made the answer possible. Record parity is specified on change governance.
Who types: the dual independent query
The person whose gate approved the change asked to run the query. The second line refused: if the answer goes to a supervisor, the owner of the control under question must not be the only one who counted. The platform lead wrote one query; the head of second-line technology risk (the CRO’s deputy) wrote her own from the question alone, without seeing his. They compared counts before anything else: both 2,870, each in under a minute.
Page recommendation: fix who types before the question arrives. Name, per control family, the second person who writes the independent query, and treat matching counts as the first acceptance check of any supervisory answer. The procedure is summarised on audit evidence.
Declared gaps
One group of entries had a blank with which model? for a fortnight: an agent had called its model through a route not yet wired to the record. Beside the blanks sat two notes from the platform lead, one declaring the gap on the day it opened and one closing it. A gap declared in advance is a chosen departure; a gap found afterwards is a finding. Book 09 treats the difference as decisive for how the answer reads. See policy, audit, actuals: declared departures.
The funnel from 2,870 to two
Read narrowly, the question had one answer. Read fully, it needed all 2,870 entries, because the only way to show there was nothing else was to look. Each step was kept so the supervisor could repeat it.
| Step | Filter | Result | How |
|---|---|---|---|
| 0 | Every change approved without a person in the window | 2,870 | Dual independent query; consistent with the gate’s expected share of roughly 1,700 changes a month |
| 1 | Who wrote the tests? | Nearly two-thirds carried no test from a person or a different agent | Join each test file to the identity on the commit that added it; the gate had never read this |
| 2 | Which changes touched a configuration file anywhere? | 63 | Filter on files changed |
| 3 | Manual reading, record open beside each change | Mostly logging levels, timeouts, retries, report column order; two declared departures with reason, reviewer and end date | Three people, one morning; minutes per change because change, tests, approver and route were already in front of them |
| 4 | Which changed a value that decides what a person sees? | 2 | Judgement during step 3 |
The second of the two was not one change but a run of weekly changes: an alerting agent re-tuned the amount below which a late-settling trade raised no alert. The value drifted a few euros either way until mid-August, then rose every week to nearly three times its July level. Asking the record with which model? showed the same model name throughout and a different version string from one Friday: the supplier hosting the agent’s model had pointed its “current model” name at a successor. The firm’s pinning rule covered deployments in its own subscriptions only. Eleven late settlements had fallen below the line; all were small and picked up next morning. No filter had been built to find this; a person reading in date order did. Model pinning and supplier-hosted names are on model governance.
Why no “did this change a control?” field. The second line asked to add a question to every entry. The platform lead refused: everything that reads the record depends on its shape never changing, and a control question answered by whoever made the change reproduces the classification error (the analyst and the supplier would both have answered no). Whether a value is a control is decided once, by the firm, in a list; the entry points at the list and the gate reads it. He won on the shape; she won on a deadline for the list. The list is specified on change governance: the control-value list.
What went in the answer, and in what order
| Decision | Who decided | Reason (source) |
|---|---|---|
| First paragraph: yes, scope changed on a date; the decision was the CIO’s; the supervisor was not told; the procedure for keeping supervisory answers current was not applied because the threshold was not treated as part of an answer; it will be from now on | CIO, in her own words | They asked whether they were told; anything else in the first paragraph would read as an excuse |
| The executive risk committee’s review of the line goes in the fourth paragraph, after the funnel | CIO, on the CRO’s suggestion to include it | True and relevant, but not mitigation for the first paragraph |
| No covering paragraph until the counting was done; no drafts circulated | CIO | Writing the reassuring sentence first means spending two days finding evidence for it |
| ”Believed to be limited” replaced | CRO | Write what we counted, not what we believe |
| Send the answer, the funnel and an offer of native records, not all 2,870 raw | Platform lead, CIO sided | Sending everything asks the supervisor to do the firm’s reading; sending the reading and the means to check it shows the firm did it |
| Incident classification recorded separately with reasons, and its outcome not used to say less | CRO | Classification is a separate decision; how it is made is on incident response |
What the record could not answer
A week later the supervisor asked who had decided the threshold in force on the day, and on what basis, and announced an on-site review. The threshold record came back in seconds and answered who: the change, the CIO’s approval under delegated authority, the re-run of the shadow and live evidence, the second-line objection one link away. It did not answer on what basis, because nothing recorded how far the line could move, on what evidence, by whose signature, or what happens if the rate it let through outran reviewers. The CIO’s reply said who, when, what she had looked at, that no rule existed, and that a proposal would go to the risk committee by year end. The firm later answered with three documents: the signed band, the procedure for setting the line within it, and the list of values the line can never decide. Keep all three versioned (change governance).
Finally, sorting the 2,870 by product showed the largest block belonged to a dashboard built fast and cleanly over the summer. Every entry proved how it had been built; nothing could say whether anyone used it. That gap is closed by usage measures, not by the evidence record (factory measurement).
Phase 2: the on-site review (Book 09 Ch 13)
Before the inspectors arrive
| Composite fact | What it tells you to prepare |
|---|---|
| Scope letter about two months ahead; sixteen written questions and a 48-item request list a week before | The written questions map to named owners; four went to the CIO |
| Answers sent on the Friday; re-read on the Monday night and nothing changed | Answers are final once counted, not polished in the last hours |
| The CEO showed the CIO a sentence she had removed from an earlier letter (“set on the basis of the evidence available at the time”) and asked her to keep it | Say what the evidence was, who declined to act on an objection and on what reading, and that you thought it enough at the time. Do not claim it was enough |
| One request-list item asked for MI on review capacity. A year earlier the only artefact was a chart a platform lead drew over a weekend and showed no one above her | The same chart, now with the signed band and twelve weeks per domain, including the bad fortnight, is the answer. Do not omit the weeks that look worst |
Day one: staff interviewed alone
Inspectors spent the first day with engineers and reviewers one at a time, with no management in the room.
- An engineer was asked to walk through a re-run she had done. She opened the evidence record and found the entry. Asked whether she had been told what to write in her approval note, she said the recipe asked what she had checked, and she answered it.
- A reviewer was asked how often she approved a change she had not fully understood. She said less often than a year before, and she could now show which ones, because the record asks what was checked and leaves a visible gap where she said nothing. Asked to show a gap, she found two of her own from the previous week.
Page recommendation: this is the strongest evidence of oversight in the case, and it cannot be coached. The control is the “what did you check” field being required by the recipe and its blanks being queryable. The monthly twenty-change sample on audit evidence is the internal rehearsal for exactly these interviews.
Day two: the threshold session
The CIO and the CRO sat with the gate’s designer, who brought the capacity chart on A3. The first three columns are Book 09’s account; the Lesson column is this page’s recommendation.
| Question | Answer given | What the inspector recorded | Lesson |
|---|---|---|---|
| Who set the line, on what? | The CIO, under delegated authority, against a written CRO objection, on four weeks of shadow and two months of live record; “there was no rule for setting it; there is one now” | The evidence window held several thousand scored changes but two of the kind that later went wrong | Count the evidence per change kind, not in total |
| What is the limit measured on; who decided what counts? | The CRO: understanding, not approvals. A human approval counts only if the record shows what was checked; a machine’s only if its tests came from someone other than the agent and it touched no control value. “I signed it; if it counts the wrong things, the finding is mine” | No comment | One named owner of the counting rule, willing to own the finding |
| Is the gate independent? | The May acceptance note, conditioned on the line compliance had been shown, and compliance’s July reply that the threshold was a gate parameter | Nobody had asked whether the accepted line was still the line the gate ran on until December: nearly five months | Encode the accepted line in the gate; a change to it triggers the key-control route |
| What does the gate read now? | What a change touches, where its tests came from, and the list of values it can never decide alone. Asked for one change routed on test origin alone, the designer found one from the previous week in under a minute | Easiest quarter of an hour of the review | A query you have run before is a query you can run in front of an inspector |
Then the band questions, which nobody had prepared answers for. The firm put them word for word at the head of the band’s year-end review terms. See control budget: the inspector’s questions as standing review terms.
During the three days the inspectors asked for three more documents, all from the evidence record; each arrived within the hour, and that was the only thing the senior inspector commented on at the close.
Act before the letter
Asked whether the firm should act on anything before the findings letter, the inspector answered: whatever you would act on if we had not come. The two owners took their halves separately.
| Owner | Action | Why |
|---|---|---|
| CRO (band) | Brought internal audit’s first review of the counting forward from summer to May, sampling the weeks nearest the edge, not the quiet ones | Answers “who tests your counting?” |
| CRO | Counted the two most different domains separately alongside the estate | Gives the year-end review two domains to compare, not one month |
| CRO | Made the held-back review a procedure step: when a domain’s budget is spent, someone outside the squad reads what queued and records whether the right things waited | Answers “what did the budget stop?” (the composite’s longest-held request was the safest one) |
| CIO (rate of understanding) | Pairing: each new reviewer in a strained domain reviews the same changes as one who can approve alone until the record shows their comments converging; two posts funded to pay for it, costed under the control budget, not hiring | Answers “when reviewers leave, who decides when the gap matters?” by raising capacity rather than asking people to read faster |
The CRO was explicit that none of this answered the inspector’s questions; it made them answerable. Report it that way. At the spring board, she reported the budget’s position herself, before the CIO, because the count was her team’s.
The board’s follow-up: show one that went right without the expert
The audit committee chair asked for a story that would have gone the same way if none of the named people had been in the building, then rejected a re-run on a quiet week as not hard enough. The answer came in April: a new custodian onboarded by an engineer working from the recipe and the record, with the experienced lead deliberately not answering questions. The evidence trail is a compact model of control-value discipline:
| Step | What the record shows |
|---|---|
| The promise first | Field by field, agreed and signed with the custodian before anything was built: which balance each record shows, arrival time each business day, what a well-formed record is, what the custodian does with a malformed one |
| Meaning boundary | ”Closing” cash balance clarified as value-dated, not booked; copying the previous custodian’s configuration would have created a daily false break on every fund in the range |
| New case without a precedent | Corrections arrive as whole replacement files. The engineer searched the previous onboarding’s record, found the lead’s reasoning on amendments, and asked the custodian to number every file per day so a replacement and a duplicate can be told apart; added a test that sends the same file twice and expects the second refused |
| Control value with unit and scope | Custodian-only tolerance override; specification named the unit and the conversion; the agent’s plan named file, scope and converted figure and stated the default would not change; the gate routed it to a reviewer because the value was on the control list; the reviewer checked the conversion against the source and the default, ran an edge-of-band test and approved with a note of what she checked; the override joined the signed control-value list that week |
| Boundary check before go-live | Four test files: two passed, two refused at the boundary (missing count trailer; disallowed date format) with the reason recorded; the custodian fixed its side before launch |
| First live month | Every break explained; replacement and duplicate handled as tested |
| Expert’s review | Read the whole record after the month closed; one question, already answered in the record |
The CIO’s note to the chair: not proof yet, but not a re-run either. The unit-and-scope rules are on change governance.
Phase 3: the single-action reconstruction (Book 3 Ch 11, Insurer I)
This is the Claude-specific version of row 13, dated as of August 2026, per Book 3 Ch 11. The stack is the composite’s Azure build; AWS and GCP equivalents are on audit evidence.
The request. Reconstruct what the customer-correspondence agent drafted in reply to one complaint, through to sending, within 96 hours.
The query (composite table names):
union AzureDiagnostics, EvidenceHub_CL, ApprovalQueue_CL
| where correlation_id_s == "inci-2026-04-04-1923-7f9d"
| order by TimeGenerated asc
Forty-seven records returned, in order, no gaps.
| Layer | Record contents in the composite | Audit questions it answers (Book 3 numbering) |
|---|---|---|
| Gateway (API Management operation log) | Workload principal; tenant; use-case class class3; route foundry-sweden-central; model claude-sonnet-4-6; tokens in and out; inbound DLP Confidential, customer-financial; DLP action tokenised with three entity replacements (name, IBAN, date of birth); policy_decision allow-with-approval; latency; retention_tier class3-6y; correlation ID | Q1, Q2, Q3, Q4, Q7, Q8 |
Agent end (SDK callbacks, corroborated by OpenTelemetry tool_decision and tool_result events) | Two registered MCP calls (case summary; policy database), allow-list pass, zero rejected calls; extended-thinking trace retained under Tier 3 policy; draft emitted | Q2, Q3, Q4, Q7 |
| Approval queue | Named manager’s UPN; queued, opened, two sentences edited (diff kept), approved; justification text | Q5, Q6 |
| Mail-out | Received and sent timestamps | Q4 |
The independent OpenTelemetry events corroborating the SDK callbacks are why the reconstruction survived a challenge on completeness. The query took eleven seconds; most of the four hours went on reading records and writing a one-page cover note in the supervisor’s language. The firm sent it a day early for margin.
What made it possible (Book 3 Ch 11): one correlation ID issued at the gateway and carried by every downstream record including the approval queue; the approval queue emitting through the same pipeline, not a separate system; tier-aware retention set at write; and a quarterly replay rehearsal against a randomly drawn ID by an engineer from outside the agent’s team. Book 3 says Asset Manager X runs that rehearsal monthly because its volumes are smaller. Without these, the honest reply is that the firm cannot produce the reconstruction in the time given. The two numberings of the nine questions (Book 3 versus the AES record) are reconciled on audit evidence.
[FACT-CHECK: the Claude model identifier, the Foundry Sweden Central route and the OpenTelemetry event names are as Book 3 Ch 11 gives them for August 2026; confirm against current Anthropic and Microsoft documentation before relying on them.]
Phase 4: what reaches the board and the supervisor alongside the record (Book 3 Ch 20)
Supervisors read the firm’s periodic reporting as well as its record. Book 3 Ch 20 adds three rules that shape what an inspector finds when they read back.
- Label every number by evidence tier. Independently attested; institutionally measured (your own systems: the only tier that should drive a decision); estimated with visible formulas (show the assumption beside the number); vendor self-reported (attributed in the sentence, never in a headline). An adjustable estimate never migrates into board reporting as measured fact. The test for any number in the pack: which system produced it, and could someone change it with a slider? See factory measurement.
- Report risk and value together. The quarterly risk report and value report are separate documents from different lines of defence, read in the same sitting. A report that shows rising adoption and value and omits incidents and near-misses invites the question it did not answer at the next dialogue.
- Record the refusals. Book 3’s Bank A said no three times in eighteen months (a shadow-AI freeze, a retired pilot, a suspended agent) and the supervisor read each as maturity. Book 09’s equivalent is the quarterly count of vetoes. A firm whose record shows no refusals protecting its stated key controls has learned something about those controls; Asset Manager X found that no refusal anywhere had protected a tolerance, alert value or limit. See policy, audit, actuals.
A supervisor who has seen the operating record mature asks what the firm is rehearsing next (Book 3’s composite answer: the exit rehearsal). An exit question answered only in the future tense is a gap (vendor concentration and exit).
A query catalogue to rehearse
Page recommendation. Prepare these as saved, versioned queries with a named runner and a named independent second writer. Field names follow the AES record where they exist; anything else is described in plain terms rather than invented.
| Query | Returns | Runner | Independent check | Rehearse |
|---|---|---|---|---|
| Machine-approved changes in a window | Every change approved without a person, with approver, tests, route | Platform | Second line writes its own from the question | Monthly drill |
| Test provenance join | Per change: did any test come from a person or a different agent? | Platform | Internal audit samples | Monthly |
| Control-value touches | Changes touching any value on the control-value list, or any configuration file | Platform | Control owner | Monthly |
| Threshold and line history | Every change to the gate’s line, with approver, evidence and objections | Gate owner | Second line | On every change |
| Model version drift | Per production agent: model name and reported version string over time | Model risk | Platform | Weekly |
| Blank oversight notes | Approvals where “what was checked” is blank, by reviewer | Second line | Internal audit | Monthly (twenty-change sample) |
| Routing reasons | Gate decisions by routing reason (score, test origin, control value, carve-out) | Gate owner | Second line | Quarterly |
| Veto count | Every refusal on the delivery path, sorted by what it protected | Second line | Internal audit | Quarterly |
| Single-action replay | All records for one correlation ID, ordered | Engineer outside the agent’s team | Assurance lead | Quarterly (randomly drawn) |
| Declared gaps | Every logged gap with open and close dates | Platform | Second line | With each answer |
Acceptance checks
- For each row of the master table, the firm can name the record, the query, the runner and the independent second writer today.
- Automated approvals leave the same entry shape as human ones, including score, inputs and the line in force.
- The record’s entry shape is fixed; “is this a control?” lives in a firm-owned list the entry points to.
- Gaps in capture are declared when they open and closed with a dated note.
- The threshold, band and never-alone list exist as signed, versioned documents, so “on what basis” has an answer.
- Any acceptance of the gate as an approver is tied to the configuration it accepted, and a change to that configuration triggers a re-review.
- The procedure for keeping answers given to the supervisor current covers gate thresholds and scope.
- A random single-action replay completes within the firm’s target, run by someone outside the agent’s team.
- Staff can retrieve their own approvals and show their own blanks without preparation.
- Every number in the board pack carries its evidence tier.
Failure modes
| Failure | Recovery |
|---|---|
| Reassuring sentence drafted first | No covering paragraph until counting is done; write what was counted |
| The control’s owner runs the only count | Dual independent query, compared before reporting |
| Answering only the known change | Start from the whole population and publish the funnel |
| Sending every raw record | Send the reading, the funnel and an offer of native records |
| Gate threshold not treated as part of the supervisory answer | Bring thresholds and scope under the procedure for keeping answers current |
| Hand-kept registers trusted over emitted records (the alerting agent’s registration still named its original model) | Reconcile hand-kept registers against the emitted record; trust the record that no one maintains |
| Screenshots and vendor dashboards offered as evidence | Per-action records in the firm’s own immutable store; label vendor figures by tier |
| Mitigation placed ahead of the admission | Admission first; context after the evidence |
Source notes
- Book 09’s record questions and Book 3’s nine questions express the same schema in different words and numbering; the crosswalk is on audit evidence.
- Composite dates in Book 09 (a threshold change in late July, a letter in late October, an on-site review in March of the following year) are narrative dates, not regulatory ones. The incident-classification footnotes in Book 09 Ch 9 carry their own legal fact-check markers; this page does not restate them.
- The master table, query catalogue and “act before the letter” owner table are this page’s synthesis; the books give the questions, answers and actions in narrative form.
Source material
- Book 09 (Everything Is Green), Chapter 9, One Query (the supervisor's letter, the dual query, the funnel from 2,870 to two, declared gaps, the follow-up on the threshold)
- Book 09 (Everything Is Green), Chapter 13, The Inspection (sixteen written questions, the request list, the on-site interviews, the band questions, acting before the letter, the custodian onboarding)
- Book 09 (Everything Is Green), Chapter 8, The Number That Was Too Good (the unit and scope error behind the tolerance change)
- Book 3 (Deploying Claude Safely), Chapter 11, Evidence Capture and the Audit Schema (§1 and §8 Insurer I reconstruction; §5 capture points; PR-1, PR-4)
- Book 3 (Deploying Claude Safely), Chapter 20, The 18-Month Roadmap and Reporting to the Board (§5 and §5.1 what the board sees and the evidence hierarchy; OD-2)
Related guidance
-
The Audit Spine and the Evidence RecordHow to capture, join, protect, retain and prove per-action evidence: the AES v0.1.0 record contract, three capture points and one correlation ID, tiered immutable storage, retention and legal hold, replay rehearsal, the route-approval evidence gate, and the Claude-specific ways evidence goes missing. -
Change Governance for Agent-Enabled DeliveryRoute agent-speed change by risk without losing the approver: control values that always reach a person, a routing gate that reads the right signals, autonomy earned per change family, a Virtual CAB that prepares a complete dossier, and three bodies with defined escalation. -
Control BudgetAn author proposal for a CRO-signed limit on how fast change may arrive relative to the firm's capacity to understand it: the CIO runs the line, the CRO signs the band, only changes with a full record count, two weeks outside the band queues intake automatically, quarter-end widening is signed in advance, and someone outside the squad reviews what the budget held back. -
Close the Policy, Audit and Actuals LoopRun governance as a standing comparison of declared policy against emitted records, route only the exceptions to the second line, and govern the policy packs that drive the comparison. -
Accountability and OwnershipHold provenance and accountability apart on purpose: four named accountabilities per product, a human owner for every agent change, seven named seats, a signed Build/Run RACI and a first-line/second-line boundary that survives consolidation. -
Measure the FactoryRead usage first, then the seven-dimension scorecard; emit a served-request event that separates human from agent callers; label every reported number by evidence tier; watch the two leading numbers of a control gap; treat an unexplained sharp improvement as an alarm; and never turn a participation metric into a target. -
Route Regulatory Questions to Obligations and EvidenceA dated, article-level reference that routes AI design questions to the EU AI Act, DORA and its RTS, GDPR and sectoral rules, and turns each obligation into a control, an owner and evidence: change records under RTS 2024/1774 Art 17, the 4 h / 72 h / one-month incident clocks, the CIF hinge and its five artefacts, the three-count Register of Information reconciliation, Art 14(4) oversight controls, the Art 26(5) discharge, the FRIA gate, Art 49 and Art 50, SCHUFA-grade review and supplier controls. -
Model Governance: Usage Registry and Upgrade LifecycleGovern models at the grain regulators ask about: a catalogue for what exists, an 18-field usage registry for what is used, risk classified per usage, a registry the policy decision point reads by usage_id, pinned model IDs, and a four-phase upgrade lifecycle from Day −14 to Day +90 with numeric rollback thresholds. -
Respond to Agent IncidentsSort an agent failure into one of four incident shapes, suspend it in three verified layers, keep humans on every classification and disclosure decision, route one playbook to every authority that needs a cover sheet, and close each incident with a named control change. -
Vendor Concentration and ExitMeasure concentration on three axes against 70% and 85% discussion triggers, keep a tested successor for every material model usage, record eight contract-cleavage points per route, prove evidence portability with a 15-item checklist, and rehearse exit on one material use case every year with a filed close-out report. -
Worked Example: A Migration Through the FactoryA composite data-centre exit walked as a reusable decision trail: purpose sheet, recipe boundary, inventory as a discovery control, the control loop, transform review with adversarial cases and business reconciliation checks, rehearsals run by role, and the effort test that tells a factory from a pile of tools.