Run the AI Delivery Factory
Operate demand, capacity, the two worker classes, exceptions and evidence as one service: read two leading numbers, keep a handover backlog, gate construction on three conditions, run queues and alerts at agent volume, and count unused products as cost.
- Version
- 0.2.0
- Owner
- DEIXEI
- Last reviewed
- Reading time
- 49 min
For Cto · Cio · Ai Architect · Platform Engineer · Product Leader · Risk And Assurance · Internal Audit
Sources and details
- Section
- Operate and assure
- Source works
- Book 1 · Book 2 · Book 3 · Supporting Tools · Research Series · Book 09
- Source chapters
- 1, 2, 3, 6, 7, 8, 10, 12, 13, 16, 21, 24
- Tool specifications
- audit_evidence_schema@0.1.0, cab_decision_record@0.1.0
- Refresh trigger
- source change
The factory is a service with a finite human review capacity, shared platform responsibilities and a growing stream of machine-prepared work. Running it means choosing what enters, limiting how much is active, routing work to the right worker class and risk lane, watching the numbers that move first, and learning from outcomes. The objective is useful, supportable change with accountable decisions and evidence, not maximum ticket or deployment volume.
This page is the operating manual. The architecture of the factory is on factory model; the full measurement scorecard is on factory measurement; the signed control-budget instrument is on control budget. Regulatory obligations and their dates are on the regulatory map and are not restated here.
Decisions this page supports
| Decision | Owner (recommended) | Section |
|---|---|---|
| Is the control gap open in my estate? | Head of the factory, with the platform lead | Two leading numbers |
| What is our first stated limit on the rate of change? | CRO signs; CIO runs | First proxy |
| Should this be built at all? | Business owner, accepted at intake | Intake |
| Which work moves from people to agents next? | Senior owner of the handover backlog | Handover backlog |
| What does the operating roadmap carry this quarter? | Head of the factory | Operating roadmap |
| Which queues may run without a human reading them? | Operational owner, per queue | Operations at agent volume |
| Which anomalies page whom, at what severity? | Group Risk owns the severity policy | Exception engine |
| Keep, fix or retire a product nobody uses? | Product lead, with the adoption reader | Unused products |
Read two leading numbers first
Throughput metrics (deployment frequency, test pass rate, change-failure rate) can all be green while the review queue grows unseen. Book 09 (Introduction, Ch 1) names the reason and the two numbers that show it first.
Why the standard dashboard misses it
- Merge-anchored lead time hides the queue. If lead time is measured from merge to production, the days a change waits for a reviewer before merge are not in it. Lead time is also computed only over changes that shipped, so work still waiting is absent from the figure.
- Change-failure rate counts only failures someone noticed. A wrong change that nobody detects (for example, a widened tolerance that silently clears real breaks) never enters it.
- The arrival line is measured everywhere, the capacity line almost nowhere (Book 2 Ch 1 §5). Before AI assistance the two ran together; afterwards arrival bends upward while review capacity stays nearly flat. The widening area between them is the control gap. Adding reviewers or approval steps raises the flat line only slightly; the gap closes from the arrival side, by routing low-risk, well-evidenced change to automated approval that still emits a full evidence record (change governance).
Recommendation (this page): anchor lead time at pull-request open, not at merge, and report the age of open pull requests every morning alongside it.
The two numbers
| Number | Definition | Source |
|---|---|---|
| Review wait | Median age of open pull requests waiting for review, read daily and reported weekly | Repository platform: review requests and timestamps |
| Uncommented large approvals | Share of approvals on large changes that carry no review comment. Book 09’s closing workbook defines “large” as a change over twenty files | Repository platform: approvals and comments |
Rules for reading them (Book 09 Introduction and Closing):
- Split every series between the teams that use coding assistants most and those that use them least. A firm-wide average hides the few domains that matter.
- Ask for the weekly line since before the assistants arrived, not last month’s figure. A single month tells you where you are; the line tells you the direction.
- Do not average across teams and do not send the numbers upward as a single score.
Interpretation matrix
The first three readings are Book 09’s (Introduction, Closing); the fourth row and the actions column are this page’s synthesis.
| Review wait | Uncommented large approvals | Reading | Action |
|---|---|---|---|
| Rising | Flat | Capacity problem | Add or train reviewers (slow); reduce intake; route more low-risk change to the automated lane |
| Rising | Rising | Reviewers are passing work they have not read; the records will not say so | Treat as a control finding: draw the twenty-change sample now; throttle intake (first proxy) |
| Flat in heavy-adoption teams | Flat | Something is working | Find out why and copy that team’s practice |
| Falling sharply, unexplained | Any | Possible over-routing or a broken signal | Treat as an alarm, not a result (sharp improvement) |
Neither number is a verdict. Teams differ in their work, and an uncommented approval of a large mechanical change (a rename across many files, a dependency update) can be entirely sound. A gap between heavy and light teams is a reason to read a sample of the changes behind it.
Three diagnostic questions, including who produces change
Run these once to establish the picture, then keep the answers current (Book 09 Ch 1; Book 2 Ch 1 §5, §7):
- How fast is change arriving, and from where? Pull ninety days of pull-request volume per team. Then build an inventory of change producers: everything that now generates change (code, tests, infrastructure templates, runbooks, migration scripts, data mappings, API definitions, change requests) and every agent that commits, with the named person who answers for each. A blank owner is a finding.
- How fast can the firm understand change, and is that keeping pace? Review latency and reviewer load, heavy versus light teams on one chart. Add approval exceptions (reviewers assigned outside the owning team) and security waivers: they show where review is being worked around. Look at how long the fastest approvals on large changes took. A lengthening queue and a minutes-long approval of a large change are the same finding seen from two ends.
- Is the rate of understanding measured at all? Ask internal audit what comes back when it samples for evidence of oversight. If the answer is screenshots, understanding is being taken on trust. If audit has not asked, draw the sample yourself.
The twenty-change sample (Book 09 Closing). Someone outside the teams draws twenty changes at random from last month, including some a machine approved. For each, fill five columns from the record, not memory: who approved; what they checked (a comment, an answered checklist, a test they added; “reviewed it” does not count); who made the tests (a person, a different agent, or the same agent that made the change); minutes from open to approval; and whether it altered a limit, threshold or tolerance that decides what a client or investor sees. Count the blanks in what they checked and the rows where tests came from the same source as the change. Expect about two hours.
The 90-day baseline split by adoption
Book 2 (Ch 1 §7) sets the first measurement before any policy work:
| Pull for the last 90 days | Split |
|---|---|
| Pull-request volume, review latency, reviewer load, failed checks, approval exceptions, deployment frequency, change-failure rate, security waivers | High-adoption teams versus low-adoption teams |
Put the two review-latency curves on one chart. If the high-adoption curve bends upward while the low-adoption curve stays flat, that is the control gap and the evidence to fund closing it. The same baseline feeds the control-budget calibration on control budget and the scorecard on factory measurement.
Count the time it takes to say yes
Refusals are visible; waiting is not. A request queued between two teams is not a refusal, so most estates keep no count of it (Book 09 Ch 7). Book 09 Ch 12 (composite, Asset Manager X) prices putting one agent into production across five engineering teams under three leaders (cloud, identity, network, security engineering, AI): each request took three to five working days to reach the front of a queue in which it was nobody’s whole job, so one agent waited fifteen to twenty-five working days. A separate composite agent, registered with an owner and limits within an hour, waited over three weeks for a network path and a gateway rule owned by two different teams, each waiting for the other.
Recommendation (this page): add a time-to-yes measure: for every enablement request that crosses a team boundary (network path, gateway rule, identity grant, broker rule), record request time, each hand-off and the permission time. Report the oldest open requests by name beside the refusal counts, and treat any request open for more than a locally set number of working days as an escalation. Book 09’s composite answer was to hold rules as code in a repository the delivery department deploys from, with the owning team’s approval as the only gate and a stated turnaround; the ownership of the rule does not move.
State a first proxy for the control budget
The full instrument is a CRO-signed ratio of change arrival to evidence-quality-weighted control capacity, with automatic consequences (Book 2 Ch 24 §4.9). Its counting rule, band setting, calibration and reporting route are on control budget. Running the factory needs two things from it immediately.
Start with a first proxy (Book 2 Ch 1 §7). You do not need a precise number to begin. The firm must accept that a control-capacity limit exists and name a first proxy: a review latency per domain past which that domain is deemed to be operating beyond safe capacity. Put it in front of the same committee that reads the throughput numbers, on the same cadence, so the two are read together. Write a structural counter-incentive into the programme’s funding: the committee that funds the AI programme on throughput also reviews the limit and can throttle the programme.
Reconcile the latency caveat. Review latency alone can be held down by approving without reading, which is exactly the pathology the second leading number exposes. The sources layer the measure as it matures; this page follows the layering rather than picking one:
| Stage | Capacity measure | Source |
|---|---|---|
| First proxy | Review latency threshold per domain | Book 2 Ch 1 §7 |
| Interim guard | Proxy read together with the uncommented-approval share | Book 09 Introduction (this page’s pairing) |
| Full instrument | Arrival against capacity counted only from reviews that left a full evidence record, evaluations that ran and samples actually read; never raw approvals per hour | Book 2 Ch 24 §4.9; Book 09 Ch 12 (share of arriving changes with a full record) |
Automatic consequences once the budget is spent. Intake throttles (new initiatives queue rather than enter) and the autonomy ladder freezes: no agent gains autonomy until the budget recovers, while demotions remain available. The consequences take effect without a meeting; the roadmap review then argues about funding more control capacity or accepting less arrival, not about whether to slow down. This is an author operating model, not a regulatory formula. A Book 09 composite set its first band at 75 (in any week at least three in four changes arriving in a domain leave a full record), one point below the worst week of the best month in the firm’s own record; that number is illustrative, not a benchmark.
Scale agents only as fast as controls (Book 1 Ch 1 §7.3, §8.1). Recognise the failure when agent count rises faster than wrangler count, or the change board still lists agents as “applications”. Recovery: a hard pause on the backlog of new agents (not on running agents), install inventory and wranglers, and resume only when controls are at least one quarter ahead of the deployment plan. Report agent count and named-wrangler count as a pair; see agent lifecycle for the inventory.
Intake: decide whether to build
Ask the business owner to bring the observed problem and its baseline, not only a proposed tool (Book 2 Ch 24 §4.3). Requests arrive solution-shaped (“we need a dashboard”) or as symptoms (“the close is late again”); a factory with abundant construction will otherwise industrialise the symptom.
- Problem pass first. Start from the business’s own defect data: breaks, exception queues, manual touches, rework, late reports, complaints. Chart it; a Pareto chart usually shows one bar carrying most of the pain.
- Spread causes across six categories a business person can argue with: people, process, policy, data, system, measurement. Run five whys down the branch that carries the volume.
- Every “because” carries an evidence pointer (the break report, exception log, feed timestamps, count of manual touches) and a named human accepts the finished chain in writing before it routes. An agent-drafted chain with no pointers is plausible fluency, not analysis.
- Route before you build: process cause → business process owner; data cause → data owner; policy ambiguity → policy owner; reusable capability gap → platform or foundation backlog (this page’s addition); only the remainder → product discovery and delivery intake.
- Split the labour: agents prepare evidence (pull queues, count touches, chart breaks, assemble candidate chains with pointers); humans decide which bar matters, which cause is real, and whether the answer is a process change or a product.
Intake is a series of increments (business analysis, technical design, architecture, solution shaping, UI), never a single pass; a complete specification produced in one sitting is a confident description of the wrong thing. For an AI product the solution-shaping pass starts from the nearest recipe family at the lowest agency class that meets the need, with the classification line written into the BOM (choose the AI product shape, BOM and recipes). Intake consumes scarce people (business owner, data owner, integration owner), so it sits on the operating roadmap and is planned against capacity.
At intake capture the affected users and service, expected value and measurement window, data and dependency classes, candidate recipe or existing capability, testability (which test techniques the chosen stack permits; decide there will be tests and of what kind before choosing a technology that resists them), required owners, the human duties the recipe needs (supervisor, golden-record annotator, feedback provider, with hours against each), operational support and the lowest suitable agency class. An initiative without a named business owner, a measurable problem or a viable control route waits or is declined with the reason recorded. The output is a committed intent.md accepted by the product owner (Book 2 Ch 24 §4.3, Ch 15).
One baseline, read three times
The bar charted in the problem pass is the expected value (Book 2 Ch 24 §4.3; Ch 21 §4.4). No new instrument is needed:
| Reading | When | Question |
|---|---|---|
| 1. Expected value | At intake | How much will this bar shrink, by when? |
| 2. Value from usage | From the first week in production | Is the product used, and is the bar moving? (factory measurement) |
| 3. Kill clause | On the stated date | Has the bar shrunk? If not, the bet failed, however well built or warmly received |
Three conditions before construction starts
Book 09 (Ch 10) turns a product that shipped flawlessly against a problem that had already disappeared into three pre-build conditions. Book 2 (Ch 21 §4.4) supplies the mechanism. Record all three on the product’s page before the first line is built:
| # | Condition | Evidence that satisfies it | Failure it prevents |
|---|---|---|---|
| 1 | The number to move, stated and measured now | The baseline from the problem pass, measured at the start of construction, not at the request date | Building against an expired problem (a request can be overtaken by another change, including an agent the firm already deployed) |
| 2 | A dated stop condition, signed by the person who asked for it | A date and a condition, both specific, set before construction; the condition is the baseline bar re-read | A kill clause negotiated at the point of disappointment rather than set at the point of enthusiasm |
| 3 | Proven switch-off | A retirement recipe run once against the prototype, before anything crosses into production, timed | A “bet” that cannot be stopped, which is an irreversible commitment with optimistic paperwork |
Retirement in a regulated firm means data to dispose of, integrations to unwind, entitlements to revoke, agent identities to retire, evidence to close and retain, a register entry to amend, and possibly a supervisory conversation if the product was ever assessed as supporting a critical or important function. You can retire cheaply only what the BOM completely enumerates (BOM and recipes). If the retirement recipe does not exist, price the bet as irreversible.
Triage and route
Triage is a human decision supported by machine-prepared evidence. A practical routing record contains:
| Decision | What the triage owner records |
|---|---|
| Problem and value | Baseline, affected service, owner, expected outcome, stop date and condition |
| Risk and authority | Data and service impact, agency class, identity and approval needs |
| Route | Existing capability, process/data/policy remediation, product lane or platform lane |
| Readiness | Dependencies, test strategy, acceptance evidence, retirement recipe and named operators |
| Capacity | Estimated reviewer, specialist and platform effort; active work displaced or deferred |
| Decision | Accept, return for evidence, redirect, defer or reject; decision owner and rationale |
Use risk and reversibility to choose the lane. A low-impact, bounded, repeatable change may move through automated checks with sampled human review. A change affecting customer outcomes, regulated records, material access, production data or a critical service requires the relevant accountable reviewers and explicit release authority; ship any carve-out that keeps critical-function changes away from automated approval before raising the routing line (change governance). The lane is set by the operation’s impact and authority, not by whether an agent wrote the code.
Plan on one capacity-bounded operating roadmap
The operating roadmap is the standing instrument a running department uses to decide and re-decide what to build. It is distinct from the one-time transformation roadmap (implementation roadmap), and it never completes (Book 2 Ch 24 §4.2). Three properties:
- Hierarchical: business outcomes at the top, initiatives under each outcome, concrete work under those; it reads across every backlog, pod and product.
- Capacity-bounded: the floor has a finite number of production lines and pods; what it can carry in a quarter is a hard number. A roadmap that ignores capacity is a wish-list.
- Permanently under discussion: the content organises the argument; humans decide. A review that produced no change should be interrogated, not celebrated.
Investment profiles. Each initiative carries a profile as well as a priority. A front-loaded initiative takes a full pod now because its value evaporates if it slips; a slow build takes, for example, half a pod over six months because there is no cliff. Similar total capacity, opposite shape; show both against the same finite floor and choose. Book 2’s composite quarter front-loaded one change, set another to a slow build, and pulled two initiatives whose products telemetry showed were no longer used, with a written record of what was deferred and why.
Three altitudes. The same roadmap is argued at three levels, which must reconcile:
| Altitude | Who argues | About what |
|---|---|---|
| Portfolio | Executive committee | Which outcomes get capacity at all, in the language of value and risk appetite |
| Outcome | Product leads | Sequence of initiatives within an outcome |
| Floor | Pods | Concrete work and its dependencies |
Keep five shared planning surfaces distinct (Book 2 Ch 24 §5): intake for incoming demand and analysis; the operating roadmap; a product backlog per product for delivery and support; the platform backlog for reusable components and shared controls; and the machinery backlog for the factory’s own tools and workflows. The roadmap reads across the backlogs; it does not replace them. Name an owner for each, keep them on surfaces agents can read and write (if agents cannot post state to the product backlog, fix that first), and route a verified shared cause to the platform or machinery owner rather than duplicating fixes. Size: at a few thousand staff the roadmap review is a standing fortnightly meeting with named backlog owners; at a small firm the five collapse onto two or three surfaces and the CTO chairs a weekly conversation. The standing cadences and initiative register are on implementation roadmap.
Keep a handover backlog
The handover curve (Book 2 Ch 3 §4.3, §5) plots the maturity of the structure given to agents (schemas, standing context, encoded skills) against the share of delivery work agents execute. Work moves over in a predictable order: repetitive, well-bounded operations first; judgement-laden work last or never. The curve is a control metric, not a technology metric, and it is many curves, one per station of the estate (agent readiness).

Source: The AI Delivery Factory, Chapter 3.
| Zone | What the agent does | Boundary that must be built |
|---|---|---|
| 1 Assisted | Accelerates a human who stays in the loop for every step | A human reading the diff |
| 2 Delegated | Performs a whole operation end to end; the human reviews the outcome | All three: described by a schema the agent can read; runs inside a bounded-agency envelope; emits evidence to the Audit Evidence Schema |
| 3 Autonomous within guardrails | Initiates and completes recurring operations within machine-enforced, continuously tested limits | Most of the factory’s apparatus; earned rung by rung on the autonomy ladder (agent lifecycle) |
An activity in Zone 2 whose evidence you cannot produce is not in Zone 2; it is Zone 1 with the safety off.
The sheet
Start with the teams the organisation considers exemplary and refuse “good engineers” as the explanation; their templates, review habits, runbooks and deployment patterns are the candidate recipe. List a week of one team’s work and mark each activity repetitive or judgement-laden. The repetitive column is the handover backlog. Keep it as one sheet the team can see:
| Column | Content |
|---|---|
| Activity | One repetitive operation (e.g. nightly break triage, routine schema migration) |
| Current zone | 1, 2 or 3 |
| Missing input | Schema, context or skill: the three-input diagnostic for a stuck activity |
| Named owner | The person (wrangler) who will hold it once it moves |
| Gate status | Whether the Zone 2 conditions (schema, envelope, evidence) hold |
| Last moved / pulled back | Date and reason |
Recommendation (this page): add the last two columns so the monthly review reads from the sheet rather than from memory. The first four columns are Book 2’s.
Monthly review
Once a month someone senior enough to say no reviews what has moved zone, what is queued to move, and what has been pulled back, and treats a pulled-back activity as information, not failure. Run the backlog like a risk register: named owners, a review rhythm, and a bias toward moving one activity properly over moving ten hopefully. The rule at every size: move operations, never judgement, and never move an operation to Zone 2 until the three conditions hold. Write the line into the recipe where a task that looks mechanical is a judgement (Book 2’s composite moved matching-logic changes back to Zone 1 after a green-tested agent redesign silently changed a tolerance).
Sizing (Book 2 Ch 3 §7): at large scale, run it as a portfolio exercise across three exemplary teams in different domains with named wranglers for the first Zone 2 handovers and no Zone 3 anywhere yet; at a couple of thousand staff, move one repetitive operation to Zone 2 properly and let it become the reference.
Run the automation factory with two worker classes
The automation factory provisions the governed starting point for every product. Its seven jobs and three architectural claims are on factory model; the execution architecture (source control as the source of truth, isolated runtimes, substitutable model, locked toolchains, controls that execute at four points) is on enterprise control plane. This section is how to operate the floor.
Two classes of worker, one provisioning path
| Worker class | Does | Toolchain (Book 2 reference model) | Output |
|---|---|---|---|
| Pipeline workers (deterministic scripts: Terraform, Bicep, Ansible, CLI) | Scaffold structure and provision controls exactly as the recipe says | Fixed by the recipe | Repository, environments, identity, pipeline, evidence package |
| Solution-development agents | Write features, tests and refactors inside the provisioned workspace | Locked service stack, e.g. Python and Rust for logic, a fixed component library for UI, Terraform for infrastructure | A pull request |
| Service-management agents | Watch telemetry, correlate events, analyse incidents, recommend and (at a granted rung) remediate | Locked operations stack, e.g. Bash, PowerShell, Python, cloud CLI, Terraform | An assessment, recommendation or bounded remediation |
Book 2 (Ch 16 §4.8) calls the two agent fleets “swarms”: one builds the product, the other keeps it running. They share the floor, the provisioning path, the evidence road and the wrangler discipline, so there is one governance regime, not two. An agent needs what a new human developer gets on day one, provisioned identically: a repository with the same branch policies, a locked toolchain pinned to approved versions, standing context files (CLAUDE.md or AGENTS.md) committed to the repository, policy checks on its output, and its own claimable identity so the record shows which agent, on whose behalf, under which wrangler (Book 2 Ch 16 §4.3). The workstation baseline is on coding-agent workstation.
The safe operating loop
Every unit of agent work runs the same loop. Each step names its evidence.
- Provision before generation. The workspace, its controls and its evidence bindings exist before either fleet produces an artefact. Do not dispatch the first agent until evidence emission is wired (Book 2 Ch 16 §8: “pave first, then dispatch”).
- Define. The wrangler writes the task and constraints (a spec under the recipe; for service management, the known runbook for the service class).
- Dispatch. The agent runs in its isolated runtime with only the egress, credentials and tools the recipe allows, never on a laptop with a developer’s credentials.
- Emit at the seams. The pull request carries the agent’s identity claim (
executing_agent.agent_id_claim); each pipeline step emits a signed action record; each model call records model, prompt reference and outcome. Evidence is pre-attributed, not reconstructed (audit evidence). - Test against the estate’s history. Both fleets run the shared evaluation harness before merge alongside the product’s own spec-generated tests (evaluation and resilience).
- Review and merge. The human gate is on output, not effort: the wrangler judges the change, not an archaeology of what the agent touched. Service-management agents recommend unless the service’s rung allows bounded remediation.
- Feed failure back. No incident review closes until its failing case is reduced to an evaluation case and contributed to the harness; contributions to shared assets go as pull requests to the asset’s trusted committer.
- Upgrade as a factory job. When a recipe changes, the factory dispatches a solution-development agent into each product created from it; each opens a pull request under the product team’s wrangler, and honest failures become harness cases.
Book 2’s composite (Asset Manager X, illustrative) contrasts a ticket-queue path across five teams with a median of just over five weeks to a runnable environment and no structured evidence, against a provisioned workspace and first governed feature in under a day; and an estate-wide recipe upgrade across 41 products in a week (34 merged cleanly, 5 needed local rework, 2 failed honestly and were contributed as evaluation cases). Treat these as illustrations of shape, not benchmarks.
Provider notes (dated)
As of 2026, per Book 2 Ch 16 §4.9–§4.10. Book 2’s reference model uses Claude as the substitutable intelligence layer and Azure-hosted Ubuntu runtimes (Container Apps or dedicated VMs) with Entra Agent ID for claimable agent identity; on AWS, ECS or Fargate in an egress-controlled VPC with IAM task roles or IAM Roles Anywhere, and Bedrock session-tagged identifiers as the identity-claim source; on GCP, Cloud Run or GKE in a dedicated project with Workload Identity Federation, and the Gemini Enterprise Agent Platform’s agent identity. [FACT-CHECK: Entra Agent ID availability and scope; Bedrock session-tag mapping to agent identity; the statement that the Gemini Enterprise Agent Platform absorbed the former Vertex AI agent tooling as of April 2026, against Microsoft, AWS and Google documentation.] Book 2’s practical advice holds regardless of vendor: the isolation primitive varies most between providers and must be read at the provider level; prove the factory on one cloud, then port the modules.
Start-up steps and ownership
- Pick one common product type and map every manual setup step (tickets, hand-offs, waits) with elapsed time, rework and control risk against each.
- Automate the step where friction and control risk are both high (repository with branch policies, pipeline, secret store, logging, evidence publishing); do not skip evidence publishing.
- Provision agent workspaces as tenants of the same path; move any laptop-based agents into purpose-built runtimes before scaling.
- Seed the shared harness from the last three production incidents with a software cause, and ask one business expert for twenty verified golden records.
- Close any path by which an unregistered model, agent or tool can obtain a credential or route.
- Name the automation factory as a product: an accountable owner, a backlog (the machinery backlog), a support model, a versioning approach and an adoption metric. The catalogue-to-context pipeline and the harness are factory products under the same ownership.
- Wrap existing pipelines, deployers and ITSM at the seams rather than replacing them.
Failure modes to watch: scaffolding without lifecycle management (products drift from the recipe with no upgrade path); automation without ownership (scripts forked because nobody maintains a canonical version); generating faster than the platform can govern (evidence assembled after the fact, agent pull requests with no agent identity); unconstrained agent workspaces (every service structured differently); runtime coupled to corporate endpoints.
Control flow across lanes
Keep the delivery path visible as stations rather than one undifferentiated queue:
- Shape: validate problem, owner, baseline, stop condition, product boundary and acceptance conditions.
- Prepare: select a supported capability or recipe; confirm identity, data, network and environment prerequisites.
- Build and test: execute work in an isolated workspace; retain source changes, test results, evaluation results and reviewer decisions.
- Review and release: apply policy and risk gates at the point of change; record any required human approval and the exact artefact and target.
- Operate: watch service and business outcomes, incidents, cost and use; route issues to the product, platform or control owner.
- Learn: convert verified recurring causes into a tested shared capability, recipe, policy or practice. Recurrence opens a judgement by the foundation’s owners (factory architect, platform lead), who may amend, add a control or decline with the reason written down; do not promote automatically.
Work can proceed at machine speed between human decision points, but decision ownership remains explicit. Keep request, implementation, independent control approval and production operation appropriately separated for the change’s risk; the exact rules come from institutional policy.
Work in progress and service commitments
Set work-in-progress limits by the scarce constraint: reviewers, domain experts, integration owners, platform engineers or release windows. Starting more work does not increase throughput when the next station is full. Show queues and blocked work by lane and cause; when a limit is reached, finish, remove or deliberately reprioritise before admitting more.
Define service levels as local commitments, not generic industry numbers. For each lane publish: an acknowledgement target and the clock’s start/stop rules; target time to triage and the evidence needed to begin it; expected review and release windows, with dependencies excluded transparently; severity-based incident response and escalation coverage; and ownership when a request is blocked, returned or outside the factory’s remit. Report elapsed time and waiting time per station; do not improve a chart by excluding difficult work without disclosing the rule.
Reviewer load and attrition
A routing gate works by sending the hard changes to people. That concentrates difficulty: a senior reviewer whose day was a mix of easy and difficult changes now sees only difficult ones, each taking longer (Book 09 Ch 6). A queue count measures how many changes wait, not what each costs the reviewer who reads it. In Book 09’s composite, two of the twelve senior reviewers carrying the strained domains resigned within a fortnight citing load, and replacing them was slow because a new reviewer took months to be trusted to approve alone. The queue climbed again.
Measure, per strained domain (the measures are this page’s recommendation; the failure is Book 09’s):
| Measure | Why |
|---|---|
| Review effort per change (time from first open to decision, by risk score band) | Shows rising cost per review that a queue count hides |
| Reviews per reviewer, and the share carried by the top names | Shows load sitting on a dozen people |
| Time for a new reviewer to approve alone in the domain | The replenishment rate of control capacity; it moves without anyone deciding |
| Reviewer departures and absences in strained domains | Leading indicator that capacity will fall |
Recovery (Book 09 Ch 13): pair every new reviewer in a strained domain with one who can approve alone, on the same changes, until the record shows their comments converging. Pairing costs experienced reviewers’ time, so fund it explicitly (the composite funded two extra posts) and report the cost under the control-budget line, not under hiring, because it buys control capacity.
Operations at agent volume
Incident management, change management and application support all remain; what changes is volume and precision (Book 2 Ch 24 §4.6).
Set the human/API dial per queue
Triage can run as an agent on a short cadence (Book 2 cites a ten-minute cadence), and the party on the other side of a ticket is increasingly another agent. At the limit, a queue with no human in the loop becomes a plain software queue behind an API. As of November 2026, Book 2’s stated position is to keep tickets monitored by experts who read the agents’ exchange so escalation is fast. Treat the choice as a dial, decided per queue and recorded:
| Queue characteristic | Dial position | Example |
|---|---|---|
| Low stakes, high volume, well understood, reversible | Collapse to API early | Routine dependency or retry requests between internal services |
| Moderate stakes or novel | Agents work it; an expert reads the conversation and can intervene | Application-support tickets |
| Touches client money, can move data across a boundary, or a supervisor would want to see it | Human reads the conversation, possibly permanently | Payment, client-data or cross-entity requests |
Record for each queue: dial position, rationale, the operational owner, review date, and the signal that would move it. The mistake is to collapse everything because the API exists.
Every alert carries an action
Business alerts matter as much as IT alerts. An alert that a data feed has not arrived by its expected time is not an observation; it must name an action: call this named person, trigger a retry, escalate to the counterparty for a cross-company integration. Acceptance rule (from Book 2’s worked example): the alert template will not save without an action and a named owner.
Playbooks run at a rung
Agents can now execute playbooks, not just read them. That is a capability, not a licence: playbook execution is bound to the rung of the autonomy ladder the department has granted, per runbook and per domain, never autonomous by default (agent lifecycle). The warning sign is a non-trivial remediation nobody was watching that happened to be fine.
Telemetry loops and problem management
Agents can analyse telemetry continuously and file bugs: product defects to the product backlog, shared-component problems to the platform backlog. They can also read exception queues and break reports and surface problem candidates into intake; a named human originator still owns the intent. Both are generating loops and need loop guards.
Agents are good at hypothesis testing but can spend many tokens disproving causes before finding the quick fix. Collapse the hypothesis space with record-anchored five whys: after an incident the evidence is the audit spine and telemetry (trace, call stack, change record, deployment timestamps); each “because” names its record, a hypothesis with no record is dropped, and a human accepts the chain before anything is fixed or filed. Route a cause confined to one product to that product’s backlog; route a shared cause (the same missing validation in a second and third product) to the platform or machinery backlog as a change to a recipe, template, policy pack or global internal practice.
Loop guards
Any agent or workflow that can generate follow-on work needs a bounded loop (Book 2 Ch 24 §4.7). The canonical failure: low-quality data produces errors no code fix can resolve; telemetry files bugs, agents attempt fixes that cannot hold, errors recur, and the queue fills with the same tickets opened and closed repeatedly.
| Guard | Setting |
|---|---|
| Counter | Stop after N iterations |
| Time limit | Stop after N minutes |
| Operation limit | Stop after N tokens or N actions |
| Recognition rule | This error class is not code-fixable: route to a human, do not retry |
| Token gauge alarm | Alert when tokens spent climb without a matching curve of work resolved |
Stop the loop before retrying, preserve inputs and outputs, and require a person to decide whether to resume after a material exception. A queue full of repeated, unresolved tickets is an incident, not throughput. In Book 2’s composite the token-gauge alert caught a custodian-feed data fault after two days; the pod tagged the error class not-code-fixable, stopped retries, collapsed the tickets into one escalation with an action, and then routed the class-level fix (schema validation at ingestion, a data contract before onboarding, alert template requiring action and owner) to the ingestion recipe on the platform backlog.
Run the exception engine and alert hub
Supporting Tools specifies two services that make anomalies in the evidence stream visible and routed. Both API specs are Draft 1 (2026-05-16), and the decision behind the alert hub (D29) has status Proposed; field names below are from those drafts and may change. [FACT-CHECK: confirm the field names and SLO targets against the current published Supporting Tools release before treating them as normative.]
Exception engine: anomaly catalogue (v0.1)
One watcher per anomaly kind reads the append-only evidence store (the spec’s “lake”) and the SIEM feed (Microsoft Sentinel in the spec); a triage engine sets severity, deduplicates related anomalies, attaches the runbook reference and emits the alert through the hub.
anomaly_kind | Default severity | Default owner | Detects |
|---|---|---|---|
heartbeat.missed | P2 | platform-soc | Forwarder silent beyond threshold (default 300 s from a registered leaf) |
signature.invalid | P1 | sec-soc | An event with an invalid signature; every one is an anomaly |
pack.stale | P3 | platform | A leaf using a policy pack past expiry |
pack.recalled.in_use | P1 | risk + sec-soc | A leaf using a recalled pack |
deploy.without_cab | P1 | platform-soc + risk | A deployment start with no backing change-decision chain |
cab.attestation.overdue | P2 | acb-chair | Emergency-change attestation past the 24-hour window |
key.rotation.overlap_exceeded | P1 | sec-soc | Old signing key used past the overlap window |
replication.lag.breach | P2 | platform | Replication lag over threshold |
clock.skew.anomaly | P2 | platform-soc | Leaf-to-edge clock skew over threshold |
session.incomplete | P3 | wrangler + platform | A session opened and never closed |
query.cost.exceeded | P3 | platform-finops | A query over the cost ceiling |
verify.dlq.spike | P1 | sec-soc | Verification dead-letter-queue spike |
m1.specialist.timeout.rate | P2 | ai-platform | Virtual CAB specialist timing out above threshold |
m4.rollback.false_positive.rate | P2 | ai-platform | Auto-rollback false positives above threshold |
Exception record fields: exception_id, anomaly_kind, severity, first_seen_at, last_seen_at, occurrence_count, scope (for example agent and entity), runbook_ref, named_owner, alert_routed_record_id. Lifecycle: acknowledge (acknowledger and note), then resolve with resolution_class of fixed, false_positive, superseded or duplicate and a rationale. Roles separate watchers, SOC readers, audit readers (who also see resolved exceptions), acknowledgers and the runbook maintainer (Group Risk with IT Standards). Each anomaly kind needs a runbook; the spec marks the runbooks as still owed.
| Engine SLO | Target | Window |
|---|---|---|
| Detection latency (event arrival to exception) | p99 ≤ 60 s | 7 days |
| Alert routing latency | p99 ≤ 60 s | 7 days |
| False-positive rate per kind | ≤ 5% | 30 days |
| Coverage of named kinds | 100% | Continuous |
| Runbook completeness | 100% | Continuous |
The engine watches itself: a watcher that stops emitting is an anomaly, because silence means the detector died. Failure handling: evidence store unreachable → skip cycle, emit exception.watcher.degraded, back off; watcher crash → restart from last checkpointed window; runbook link broken → exception.runbook.stale; flood → suppress duplicates in window and emit one summary per minute. Business-level exceptions (policy versus actuals) are a separate queue on policy, audit and actuals.
Alert hub and severity policy (D29)
A single hub routes alerts from every centre service by severity and audience: paging (SOC), ITSM tickets, chat cards (for example a change-board chair hand-off) and email (advisory). The draft names PagerDuty, ServiceNow, Microsoft Teams and email as the destinations, configured per tenant.
- Group Risk owns the severity policy (D29.1): a versioned, signed mapping from source and alert type to severity and destinations. Rules evaluate top to bottom, first match wins; a catch-all routes anything unmatched to a P3 digest. A producer’s
claimed_severityis a suggestion the policy may override; producers err high when unsure. (v1 ships with inline configuration; the signed pack follows.) - Every routed alert is evidence:
alert.routedis emitted for each inbound alert, carrying the policy version. - Receipt is required (D29.2): paging acknowledgement within a 30-second SLO; ITSM state polled and back-referenced; chat cards acknowledged by button; email counts as delivered, never as acknowledged. No receipt within the SLA emits
alert.no_receiptand escalates severity by one tier.
| Hub SLO | Target |
|---|---|
| Inbound routing latency | p99 ≤ 5 s (7 days) |
| Delivery latency to destination | p95 ≤ 10 s (7 days) |
| Receipt confirmed within 30 s | ≥ 99% (7 days) |
alert.routed per inbound alert | 100% |
| Severity-policy version drift, source versus hub | 0 minor versions, checked daily |
Failure handling: destination unreachable → retry with back-off, then alert.delivery_failed and fan out to the next destination; policy pack expired → alert.policy_stale, route on last-known policy; flood from one source → suppress with alert.suppressed and one flood_summary per minute; duplicate idempotency key → return the existing alert. Open in the draft: routing around unavailable on-call staff, correlation of related alerts, per-business-unit tenancy and test alerts that do not pollute the trail.
Exceptions, incidents and escalation
An exception is a visible state with an owner and next action, not a bypass. Stop the affected lane when an identity is unknown, evidence is missing, a policy check is unavailable, a test or evaluation result breaches its gate, requested authority exceeds the approved scope, or the rollback path is unproven. Preserve the inputs and event trail; notify the named service owner and the relevant security, risk or business contact according to the severity policy above.
For a production incident, first contain the affected identity, route or release; preserve evidence; assess user and service impact; then restore using the pre-agreed procedure (incident response). Do not let an agent’s diagnostic loop make an unbounded series of hypotheses or changes. After recovery, route a product defect to the product backlog and a shared control or platform defect to that capability’s owner.
Escalate when a decision exceeds delegated authority, a service objective is at risk, a control has failed or been bypassed, impact is uncertain, a third party is implicated, or the work affects a critical service or customer outcome. The escalation record names the decision-maker, current containment, evidence available, next update time and any work stopped. A line manager or change board cannot substitute for the accountable business or control owner.
Holds create side doors. Book 09’s composite (Ch 8) put a hold on agent-written changes in two domains; an engineer then had an agent produce urgent fixes in a personal workspace outside the hold and shipped them under her own name. When you impose a hold, publish the front door for urgent changes (who can approve, with their name beside the engineer’s) at the same time, and watch for agent sessions producing code where no pipeline runs.
Treat an unexplained sharp improvement as an alarm
A falling error or break count is usually read as success. When a routing gate or an agent is letting the wrong things through, the leading sign can be a smooth improvement with no step, no outage and no incident (Book 09 Ch 8: a tolerance widened about a hundredfold by an unconverted unit cleared real breaks for weeks while the “breaks reaching the team” line fell and the “auto-matched” line rose by the same amount).
- Name the alarm number per domain: the measure that would move if the gate were passing the wrong things (breaks reaching people, exceptions raised, manual touches, alerts per feed).
- Alarm on sharp, unexplained improvement, not only on deterioration. Check comfortable explanations (volume change, quieter period, an agent improving) against dates; if none changed in the week the line moved, investigate.
- Read what was cleared automatically. Book 09’s composite recovery: the list of items cleared automatically is read on the first day of every close, sorted by amount, by two people, and signed. Name the practice, not the person who found the problem.
- List the control values (tolerances, thresholds, limits, materiality levels, alerting rules) where the gate reads them, and send every change to one to a reviewer however small the diff (change governance).
- Test falling error rates against independent evidence before treating them as proof that controls work.
Carry unused products as cost
Usage is the gauge that moves first: a defect rate takes a release, a business outcome a quarter or two, usage the first week (Book 09 Ch 10). Every product should emit one usage event each time it does its job, defined in the foundation so it is the default (application foundation); the event contract and value-from-usage gauge are on factory measurement.
Operating rules:
- Read usage first at the quarterly product review, before delivery numbers, for every product.
- Count system callers as well as people. A product whose sign-ins fall while calls from other services and agents rise is growing, not declining; a people-only gauge under-reads the products other systems depend on.
- Read usage for products, never for people. No measure of usage or participation (active users, suggestion-acceptance rate, agents’ share of commits) becomes a target in anyone’s objectives. Participation is the easiest thing for people or agents to manufacture, and a targeted gauge measures obedience.
Product cost register. Carry every product that shipped and is not used at its running cost, with a decision against it:
| Field | Content |
|---|---|
| Product | Name and register ID |
| Users / calling systems last month | From the product’s own usage event where it exists; from sign-ins otherwise (say which) |
| Running cost | Support rota, change slot, register entry, reviewer attention per change, model and hosting cost |
| Decision | Keep, fix or retire |
| Owner | Named person |
| Date | When the decision will be executed or revisited |
The model bill is not the cost: a product with a trivial token bill can still consume a reviewer for every change made to it. Retire by recipe, timed, from the product’s own record of what it contains; the time taken is the true price of every other product on the list. Report the register to the board beside what was shipped (“I’m showing them I count it”).
Promotion must be cheaper than evasion. An unregistered tool with eight daily users is the strongest evidence of need available; the answer is a sandboxed, time-limited prototype tier the platform can see and a crossing into production measured in days because the foundation supplies identity, access, telemetry, deployment and evidence by inheritance (application foundation). Measure paved and shadow time-to-result side by side; a factory slower than the shadow road has conceded the one axis shadow AI wins on (Book 2 Ch 24 §8).
Measure portfolio reuse
Research Paper 1 proposes six portfolio metrics for a reuse-driven factory (§6). They are the author’s definitions and have not been benchmarked; the 90-day window is a starting value to tune.
| Metric | Definition | Direction |
|---|---|---|
| Time from idea to governed AI in production | Median days from intake to production approval | Down |
| Reuse ratio | Share of components in new applications that come from the catalogue | Up |
| Blueprint coverage | Share of new applications instantiated from a G4 blueprint | Up |
| Control inheritance rate | Share of required controls satisfied by inherited platform services | Up |
| Contribution rate | Assets promoted from delivery teams to the catalogue per quarter | Up |
| Orphan rate | Catalogue assets with no active owner or no evaluation run in 90 days | Down |
Recommendation (this page): review them monthly at the factory operations meeting, owned by the platform lead (Research Paper 1’s adoption roadmap tracks reuse ratio and control inheritance monthly). A second consumer or third recurrence is a signal that opens a promotion decision by the people who will own the operating cost; it is not an automatic promotion rule. Reuse generations G0–G4 are defined on BOM and recipes.
Inventory shadow AI tools
The agent inventory is on agent lifecycle. Before it, find the tools: capability already running that never passed through a decision. Tools arrive three ways (Book 3 Ch 2): procurement (slow, visible), pilots that quietly graduate (fast, partly visible) and individual initiative (invisible until something breaks). The third is where shadow agents live.
Discovery joins (Book 3 Ch 2, composite method). Cross-check, for the last calendar year:
| Source | What it reveals |
|---|---|
| SaaS procurement records from finance | Corporately and departmentally procured tools |
| CASB / browser telemetry | Domains in use, including tools with no procurement record |
| Identity-provider application registry, plus legacy SSO logs pre-dating any migration | Tools wired to corporate identity |
| File-sharing inventory | Documents holding prompts, outputs or shared workbooks |
| Expenses coded as software or information services above a threshold (the composite used €10,000 per cost centre) | Tools bought on personal or departmental cards |
| Direct ask to every desk, twice (email and in person) | What telemetry misses |
| Embedded agentic features in already-procured SaaS (Book 1 Ch 1 §8.2) | Agents that arrived as product features |
Record per tool: name and vendor; desks and named users; procurement route (corporate, departmental, pilot exception, free tier, personal expense, none); data that passes through it and whether any sub-processor is unregistered with data protection; whether accounts are personal or corporate; who signed procurement; who carries the risk. Book 3’s composite took three people two weeks and found fourteen tools across twenty-two desks through six procurement routes, including unregistered sub-processors handling client identifiers. Then decide per tool: consolidate onto the default path, keep as a permitted specialised exception, or retire. Treat the result as an authority problem, not a tools problem: the inventory exists to establish who decided.
Prove the evidence record with a 90-day pilot
Before scaling agents, prove that the evidence record works in production. Supporting Tools’ pilot plan (D16, published 2026-05-14; no pilot unit had been identified at publication, so there are no results to cite) is a usable template. Adapt the rows; record any change to the structure as a decision.
Scope and funding. One business unit, 5–10 production agents across several teams. Fund from the group IT or cloud centre-of-excellence budget with compliance co-signing (framed as platform investment with regulatory benefit).
| Success measure | Target | Notes |
|---|---|---|
| Time-to-evidence for a regulatory query | ≤ 5 minutes | Baseline is hours to days of manual reconstruction |
| Completeness (share of agent actions producing a valid evidence record that reaches the store) | ≥ 99.5% | Below 99% means evidential gaps |
| Cost per agent per month (compute, storage, ops time) | No target; measured | Input to the scale-out case |
Excluded on purpose: developer satisfaction, lines of code, agent count.
| Phase | Days | Gate |
|---|---|---|
| 1 Scope and install | 0–15 | One record from one agent reaches the immutable store, is counter-signed and retrieved by the second-line reviewer (pass/fail) |
| 2 Instrument the cohort | 15–45 | Completeness ≥ 95%; first exception detected and triaged; otherwise descope or extend |
| 3 Integrate downstream | 45–75 | SIEM feed live; time-to-evidence ≤ 15 minutes on a synthetic query; completeness ≥ 99%; first independent audit report |
| 4 Review and decide | 75–90 | Written go/no-go from the sponsor with second-line and internal-audit sign-off: scale out, pause and remediate, or stop. Verbal sign-offs do not count |
Pre-declared exit triggers: cost overrun above 50% by day 45 → pause for replan; completeness below 90% by day 30 → escalate, stop if unresolved by day 60; performance impact above 5% on instrumented systems → escalate to the vendor; adoption refusal by two or more agent teams → root-cause, descope if unresolved within 14 days. Hitting a trigger is the pilot working.
Roles: sponsor (2 h/week), technical pilot lead (full time), 1–2 platform engineers (full time), a wrangler liaison per agent team (4 h/week per agent), second-line reviewer who designs the time-to-evidence tests (4 h/week), and an internal-audit observer from day one (2 h/week; non-negotiable). Cadence: a weekly 30-minute review (completeness, exception triage, blockers, gate readiness) and a 60-minute review at each phase boundary.
Keep a minimal AI service level
In a disaster-recovery scenario or acute incident there may be no agents available. If recovery depends on agents and their external models are unreachable, model unreachability becomes part of the disruption (Book 2 Ch 24 §4.8). State, alongside ordinary service levels, which agent-supported functions survive with no connectivity to any external model provider and at what reduced capability. Keep at least one internal model runnable in your own infrastructure for the recovery path. As of November 2026 Book 2 names Anthropic’s models as its reference third-party models for normal operation.
Size the internal floor modestly: insurance for the sensitive slice of work and the recovery scenario, not a cheaper way to run the factory. Change sourcing decisions driven by control (secrecy, residency, resilience) on events, not on price. Exercise the fallback and the manual route. Where to bring this inside the firm’s continuity obligations is on the regulatory map; the local-model design is on evaluation and resilience; exit planning is on vendor concentration and exit.
Operating cadence and ownership
| Cadence | Who | Reads | Decides |
|---|---|---|---|
| Daily / continuous | Service owners, operational owner, agent-fleet operator | Alerts and receipts, failed jobs, blocked work, token gauge, open-PR age | Contain and resolve service impact; stop loops |
| Weekly | Factory operations | Two leading numbers (split), intake, WIP and ageing, exceptions by age, incidents, time-to-yes, capacity | What to pause, which queue dial to move, what to route to a shared backlog |
| Monthly | Senior owner of the handover backlog; platform lead | Handover sheet; portfolio reuse metrics; twenty-change sample (drawn by someone outside the teams) | What moves zone, what is pulled back; promotion decisions |
| Quarterly | Product review (usage first), operating roadmap review | Usage per product; cost register; investment profiles | Keep / fix / retire; capacity per outcome |
| Governance cadence | Product, platform and control owners; CRO for the budget | Control-budget position, policy changes, material risk exceptions | Band and policy changes; funding of control capacity |
Calibrate frequencies to service criticality and institutional policy. The product team owns product outcomes and its production service. Factory operations owns intake, flow visibility, capacity and routing. Platform owners maintain shared paved roads and repair common causes. Security, risk, privacy and other control owners set or interpret controls within their mandates. Named agent wranglers remain accountable for agent behaviour within their delegated scope. Book 2 (Ch 24 §7) and Book 09 (Closing) require seven named owners before the factory counts as more than a pilot: each agent’s wrangler, the head of the factory, the factory architect, the product lead, the business analyst, the agent-fleet operator and the operational owner; one person may hold several seats if it is declared (accountability). No team can outsource its decision responsibility to a dashboard, agent or workflow tool.
Failure modes
| Failure | How you recognise it | Recovery |
|---|---|---|
| Green dashboard, growing queue | Lead time anchored at merge; open-PR age on no screen | Anchor at PR open; publish the two leading numbers, split |
| Approvals outrunning understanding | Review wait and uncommented large approvals rising together | Twenty-change sample; throttle intake; reviewer pairing |
| Scaling pilots without scaling controls | Agents outnumber wranglers; agents listed as applications | Pause the backlog, not running agents; resume one quarter ahead |
| Ungoverned steepness | Cannot say in one query what an agent did last Tuesday | Stop advancing the handover curve until every handed-over activity has an envelope and evidence |
| Moving judgement, not operations | Agent output merged for questions of taste and trade-off | Write the line into the recipe; move the activity back to Zone 1 |
| Industrialised symptom | Intake problem statement is the solution with “we need” in front; no evidence pointers | Re-run the problem pass; route causes out; fire the kill clause |
| Roadmap frozen into a plan | Nobody pushes back in the review | Make the re-decided trade-offs the deliverable |
| Gate concentrates hard reviews | Senior reviewer departures; time-to-approve-alone in months | Measure effort and load; fund pairing under the control budget |
| Runaway loop | Token spend climbing, resolutions flat | Loop guards; recognition rule; route to a human |
| Playbooks off the leash | Unwatched non-trivial remediation that happened to be fine | Bind execution to the granted rung |
| Too-good number | Smooth unexplained improvement | Alarm number per domain; two-person signed reading |
| Shipped, not used | Delivery green, usage absent | Usage first; cost register; retire by recipe |
| Usage as a target | Usage in personal objectives | Remove it; read for products only |
| Waiting nobody counts | Requests open between teams for weeks | Time-to-yes measure; rules as code with owner approval |
| Silent detector | Watcher stops emitting | Engine treats watcher silence as an anomaly |
Acceptance checks
- Median open-PR review wait and the uncommented-approval share on changes over twenty files are reported weekly, split between heavy and light assistant users, with the series back to before the rollout.
- A twenty-change sample is drawn monthly by someone outside the teams, with the five columns filled from records.
- An inventory of change producers exists, and every committing agent has a named person beside it.
- A first control-capacity proxy per domain is stated and sits in the same committee pack as throughput; agent and wrangler counts are reported as a pair.
- Every product in construction has a measured baseline, a dated stop condition signed by the requester, and a retirement recipe that has been run once.
- The handover backlog sheet exists for at least one exemplary team and is reviewed monthly; pulled-back activities are recorded.
- No agent runs outside a factory-provisioned workspace; agent pull requests carry an agent identity claim; the automation factory has a named owner, backlog and version.
- Every queue has a recorded dial position and operational owner; no alert template saves without an action and owner; playbook execution is bound to a granted rung.
- Every agent loop has at least one tested stop condition, and the token gauge alerts on divergence from resolved work.
- Every anomaly kind has a severity, owner and runbook reference; the severity policy is versioned and owned by Group Risk; alert receipt is confirmed or escalated.
- Each domain has a named alarm number for unexplained improvement; automatically cleared items are read and signed by two people.
- Usage is read first for every product, for system callers as well as people, and never appears as a personal target; unused products carry a keep/fix/retire decision, owner and date.
- Time-to-yes for cross-team enablement requests is recorded and the oldest open requests are named weekly.
- A minimal AI service level is written down and its local-model fallback has been exercised.
- The seven named owners are written down in one place, with multiple seats declared.
Source notes and differences
- Composites. Asset Manager X, its people and all its numbers (Book 2, Book 09, Book 3) and “Bank A” in the Supporting Tools examples are composites describing no real firm. Figures quoted here illustrate shape and are not benchmarks.
- Control-capacity measure. Book 2 Ch 1 proposes review latency as a first proxy; Book 2 Ch 24 requires an evidence-quality-weighted ratio and rejects raw approval counts; Book 09 Ch 12 counts the share of arriving changes that left a full record. This page presents them as maturity layers; the full instrument is on control budget.
- Terminology. Book 2’s “autonomy ladder rungs” and Book 09’s “advise / act with approval / act alone” describe the same ladder; Book 2’s “seven names” and Book 09’s “seven owners” are the same seats. Book 2 uses “Virtual CAB” and “audit spine”; Book 09 uses plainer terms (“the evidence record”).
- Author proposals. The control budget, the handover curve and zones, the two leading numbers and their matrix, the station scoring, the portfolio metrics and the pilot plan are author operating models, not regulatory requirements or industry standards. Measures marked “this page’s recommendation” are syntheses added here.
- Supporting Tools status. The exception engine and alert hub specs are Draft 1 and D29 is Proposed; the pilot plan’s runbook links and public endpoints are not verified. Schema IDs in the frontmatter are source identifiers, not claims of a published endpoint.
- Not covered here. The seven-dimension scorecard, event contract, token economics and shipped-but-unused valuation are on factory measurement; gateway dashboards on model gateway; the gate carve-out on change governance.
This page remains a draft.
Source material
- Book 2, Chapter 1, The Day the Review Queue Broke (§5 control gap and three diagnostic questions; §7 90-day baseline split by adoption, first proxy for a control-capacity limit; §8)
- Book 2, Chapter 3, Why Craft Alone No Longer Scales (§4.3 handover curve; §5 three zones and Zone 2 conditions; §7 handover backlog sheet and monthly review; §8)
- Book 2, Chapter 16, The Automation Factory (§4.2–§4.4 second worker class, provisioning, pre-attributed evidence; §4.6 shared evaluation harness; §4.8 two swarms; §4.9–§4.11 execution architecture, provider view, wrapping; §5–§8)
- Book 2, Chapter 21, Product Development at Agent Speed (§4.4 kill clause and retirement recipe)
- Book 2, Chapter 24, Running the Factory (§4.2 operating roadmap; §4.3 intake; §4.6 operations at agent volume; §4.7 loop guards; §4.8 minimal AI service level; §4.9 control budget; §5 five backlogs; §6 worked example; §7–§8)
- Book 1, Chapter 1, The Adoption Gap (§7.3 control plane before scale; §8.1 scaling pilots without scaling controls; §8.2 embedded agents)
- Book 3, Chapter 2, Why Claude Becomes the Default (The Inventory; Why This Chapter Matters: three arrival routes)
- Book 09, Introduction, The Tuesday Everything Was Green (two leading numbers and their reading)
- Book 09, Chapter 1, The Day the Review Queue Broke (merge-anchored lead time; three questions for this week)
- Book 09, Chapter 6, Two Speeds (gate concentrates hard reviews; reviewer resignations; time to approve alone)
- Book 09, Chapter 7, The Machine That Says No (waiting between queues is not counted)
- Book 09, Chapter 8, The Number That Was Too Good (sharp improvement as an alarm; two-person cleared-break reading)
- Book 09, Chapter 10, Is Anyone Using It? (usage first, never a target; unused products carried as cost; three pre-build conditions)
- Book 09, Chapter 12, Who Holds the Dial (cost of crossing team boundaries; first control-budget band, composite)
- Book 09, Chapter 13, The Inspection (reviewer pairing funded under the control-budget line)
- Book 09, Closing, An Afternoon With a Pencil (twenty-change sample; large change over twenty files; seven owners)
- Research Paper 1, A Taxonomy for a Reuse-Driven AI Factory (§6 portfolio metrics)
- Supporting Tools, docs/api/engine_exception.md, Exception engine API (Draft 1, 2026-05-16; §3–§8)
- Supporting Tools, docs/api/m9_notification_hub.md, M9 Notification + Alert Hub API (Draft 1, 2026-05-16) and planning/decisions_log.md D29
- Supporting Tools, docs/pilot_path/pilot_path.md, Pilot Path — 90-Day Plan for the Audit Spine (D16)
Related research
Related guidance
-
The AI Delivery Factory: Operating ModelThe factory's six connected parts, the control gap it closes, the handover curve that governs what agents may take over, and how to start one without rebuilding your tooling. -
The Enterprise Agent Control PlaneThe firm-owned services that make every agent action identifiable, decidable, approvable and reconstructable: the ownership boundary, six planes and their contracts, the Policy Decision API with its latency budget, the persistence schema, the build sequence and the tests that show it works. -
Control BudgetAn author proposal for a CRO-signed limit on how fast change may arrive relative to the firm's capacity to understand it: the CIO runs the line, the CRO signs the band, only changes with a full record count, two weeks outside the band queues intake automatically, quarter-end widening is signed in advance, and someone outside the squad reviews what the budget held back. -
Measure the FactoryRead usage first, then the seven-dimension scorecard; emit a served-request event that separates human from agent callers; label every reported number by evidence tier; watch the two leading numbers of a control gap; treat an unexplained sharp improvement as an alarm; and never turn a participation metric into a target. -
Operate the Agent LifecycleInventory, register, sign, zone, review, suspend, recover and retire agents through a six-stage lifecycle with named human ownership, zone-specific authority and evidence at every transition. -
Respond to Agent IncidentsSort an agent failure into one of four incident shapes, suspend it in three verified layers, keep humans on every classification and disclosure decision, route one playbook to every authority that needs a cover sheet, and close each incident with a named control change. -
Evaluate, Assure and Test Agent ResilienceDefine a versioned evaluation contract, test the trajectory and the execution boundaries, run three assurance layers into one quarterly pack, map the OWASP Agentic Top 10 to controls that demonstrably fire, bring agents into threat-led penetration testing with a five-section scope, keep a continuous cadence between formal tests, land every finding as a live control-plane artefact, and engineer recovery down to a minimal AI service level on models you host. -
Build the Standard Application FoundationRun the shared identity, telemetry, deployment, data and AI capabilities as a funded product that humans and agents both start from, with a contract, evidence-computed trust, a cheap promotion boundary and reuse measured by consumption. -
Coding-Agent Workstation Baseline (Claude Code)Put Claude Code on developer workstations and build agents as privileged automation: one device-delivered managed policy per surface, lock flags that stop rules merging upward, command hooks for enforcement and HTTP hooks only for evidence, the OS sandbox where it runs, telemetry by policy, and the workstation recorded as a processing location. Dated to November 2026. -
Prepare a Governed Landing Zone for AI DeliveryDesign the management-group hierarchy, enforce guardrails in deny mode, give humans and agents the same paved workspace, integrate the eight agent-platform components, close the perimeter, and prove the zone is done with seven observable exit criteria and a second-line sign-off. -
Build a Living Product Bill of Material and RecipeDescribe what an AI product contains and how the firm builds it in one versioned specification shared by pipelines, agents and assurance teams, including the classification line, readiness grades and reuse generations. -
The Agent-Speed SDLCA committed artefact chain, governed context, trusted tests and eight SDLC controls that let coding agents work quickly while intent, review, identity and evidence stay connected. -
Change Governance for Agent-Enabled DeliveryRoute agent-speed change by risk without losing the approver: control values that always reach a person, a routing gate that reads the right signals, autonomy earned per change family, a Virtual CAB that prepares a complete dossier, and three bodies with defined escalation. -
Accountability and OwnershipHold provenance and accountability apart on purpose: four named accountabilities per product, a human owner for every agent change, seven named seats, a signed Build/Run RACI and a first-line/second-line boundary that survives consolidation. -
Close the Policy, Audit and Actuals LoopRun governance as a standing comparison of declared policy against emitted records, route only the exceptions to the second line, and govern the policy packs that drive the comparison. -
The Audit Spine and the Evidence RecordHow to capture, join, protect, retain and prove per-action evidence: the AES v0.1.0 record contract, three capture points and one correlation ID, tiered immutable storage, retention and legal hold, replay rehearsal, the route-approval evidence gate, and the Claude-specific ways evidence goes missing. -
Route Regulatory Questions to Obligations and EvidenceA dated, article-level reference that routes AI design questions to the EU AI Act, DORA and its RTS, GDPR and sectoral rules, and turns each obligation into a control, an owner and evidence: change records under RTS 2024/1774 Art 17, the 4 h / 72 h / one-month incident clocks, the CIF hinge and its five artefacts, the three-count Register of Information reconciliation, Art 14(4) oversight controls, the Art 26(5) discharge, the FRIA gate, Art 49 and Art 50, SCHUFA-grade review and supplier controls. -
The Implementation RoadmapSequence the factory as a dependency graph, not a calendar: the closure order for the governance gap, a written portfolio threshold, six overlapping phases over eighteen months with entry conditions, exit evidence, 'not yet' lists and pass/soft-fail/hard-fail gate questions, tranche funding released on exit evidence, a programme RACI, size, maturity and sector calibration with four signed trade-offs, and the first week, thirty days and ninety days in dependency order.