Measure the Factory
Read usage first, then the seven-dimension scorecard; emit a served-request event that separates human from agent callers; label every reported number by evidence tier; watch the two leading numbers of a control gap; treat an unexplained sharp improvement as an alarm; and never turn a participation metric into a target.
- Version
- 0.1.0
- Owner
- DEIXEI
- Last reviewed
- Reading time
- 36 min
For Cto · Cio · Ai Architect · Platform Engineer · Product Leader · Risk And Assurance · Internal Audit · Finance
Sources and details
- Section
- Operate and assure
- Source works
- Book 2 · Book 3 · Book 09 · Book 1
- Source chapters
- 1, 8, 10, 13, 17, 20, 21, 24
- Tool specifications
- audit_evidence_schema@0.1.0
- Refresh trigger
- quarterly
A delivery scorecard can be entirely green while the factory loses money. Book 2 opens its measurement chapter with exactly that case: a product with lead time down, daily deployments, flat change failure and 96% evidence completeness, which nine of 41 provisioned users had opened in the last thirty days (Book 2 Ch 17 §1, composite). Every dimension the firm measured said it had done well. None said whether anyone used what it shipped.
This page sets out how to measure an AI delivery factory so that the review catches that case in the first weeks, and catches its control-side twin: change arriving faster than reviewers can stand behind it while the throughput chart improves.
Reading the labels. Source claim means a book states it. Author model means a framework the books propose, such as the seven dimensions, the five agent-era measures, the four evidence tiers or the intent coverage bands. None of these is an industry standard or a supervisory expectation. Recommendation is this page’s own synthesis. Asset Manager X and Bank D are composite firms written for the books; their figures illustrate a method and are not benchmarks. Vendor product facts are dated and carry [FACT-CHECK] where they have not been re-verified against vendor documentation. Legal duties, such as data protection and operational-resilience rules, are stated only on the regulatory map; this page links there.
Decisions this page supports
| Decision | Who decides | Section |
|---|---|---|
| What does the factory review read first? | CIO, chairing the factory review | Read the review in this order |
| What does every shipped product emit so that usage can be measured? | Platform owner of the application foundation | The served-request event |
| Which numbers go on the scorecard, and which are targets? | Head of the factory, with the CRO for the control lines | The seven-dimension scorecard |
| Is a product paying back, or should it be held, fixed or written down? | Product owner, with finance | Cash controls and the shipped-unused balance |
| Is discovery keeping up with the agent lane? | Head of product or discovery | Validated-intent lead time |
| Is the control gap open? | CIO and CRO together | Two leading numbers |
| Can this number go in the board pack, and with what label? | Author of the pack; CRO countersigns | The evidence tiers |
| May we switch on per-user contribution metrics? | DPO, with HR and employee representatives | Per-user metrics |
Read the review in this order
The ordering is the control. Book 2 and Book 09 both end their usage chapters by changing the quarterly review so that usage is read first, for every product (Book 2 Ch 17 §6; Book 09 Ch 10). Book 2 adds that delivery is read second and that no product may close the review green on delivery alone.
Recommendation. The five-part agenda below is this page’s arrangement of the sources; only “usage first, delivery second” is a source rule. It places the control position ahead of delivery diagnostics so that the control gap is not read as an afterthought.
- Value and use. For each product, users or calling systems last month, read from the product’s own event, against the baseline set at intake. See intake on Running the Factory.
- Inheritance and consumption. What the product inherited from shared assets, and who consumes the shared shelf.
- Control position. The two leading numbers, the control budget’s position by domain, evidence completeness. See Control Budget.
- Delivery diagnostics. Flow, quality, resilience, security, developer experience.
- Cash and inventory. Token cost per served request, the shipped-unused balance and the patience runway.
Three rules apply to every line (author model, Book 2 Ch 17 §4 and §8; Book 09 Ch 10):
- Read the pattern, not a single number. Green flow with flat value means the factory is efficiently producing waste. Rising value with falling evidence means the firm is winning the market and losing the supervisor.
- No composite index. Do not roll the dimensions, or per-station readiness outcomes, into one factory number. Averaging hides the variance that sets throughput: one unready shared station queues every change that touches it. If the review discusses why the index moved rather than what the stations did, the index has taken over. Per-station readiness is set out on Agent Readiness.
- No participation metric is ever a target. Suggestion acceptance rate, agents’ share of commits and the AI-assisted-change ratio are diagnostics of the delivery system. Once one appears in an objectives letter or on a board KPI slide, people manage the number and it stops measuring adoption. Book 2 cites the classic software factories, which wrote reuse targets and component-registration quotas into engineers’ evaluations (Book 2 Ch 17 §4). Book 09 replays the argument when the CIO proposes active-user targets for product leads and loses it (Book 09 Ch 10, composite).
The seven-dimension scorecard
Book 2 reads the factory across seven dimensions. Six measure how well the factory runs. The seventh, value, measures whether what it ships is worth having, and the other six exist to explain it (Book 2 Ch 17 §4, Fig 17.3).
| Dimension | Core measures | Measure the agent era adds or changes | How to read it |
|---|---|---|---|
| Flow | Lead time from intent to production; deployment frequency; batch size; queue depth | Review latency by team | Authoring is no longer the constraint; review is. Read per team, never as a firm-wide average. |
| Quality | Defect escape rate; test effectiveness; architecture conformance; dependency health | AI-assisted-change ratio; architecture-drift findings linked to generated artefacts | The ratio shows how far the handover to agents has actually gone, as opposed to what licence counts imply. Drift shows whether agents follow your patterns. |
| Resilience | Change-failure rate; time to restore; rollback success; incident recurrence | Rework-and-restart rate | A healthy restart rate is the factory cutting losses early. Zero is a warning, not a triumph. |
| Security | Vulnerability age; policy violations; secrets findings; access exceptions; supply-chain risk | Security findings in generated code and configuration, tracked separately from human-authored change | Shows whether assistants introduce a class of defect human reviewers used to catch by habit. |
| Evidence | Share of deployments with a complete evidence record; audit response time; expired-exception count | Evidence completeness for AI-assisted change | The roll-up of whether Audit Evidence Schema records are produced at all. |
| Developer experience | Setup lead time; paved-road adoption; cognitive load; time on undifferentiated work | Treat as market share against the shadow road | If the governed path is slower than the ungoverned tool, people leave it regardless of its evidence. Also leads quality and flow by about two quarters. |
| Value | Usage by humans and by other agents and services; the outcome usage produces | Agent callers counted alongside human sessions | A value line that counts only human sessions under-reads the capabilities other agents depend on. |
Name the four keys correctly
The four delivery-performance measures from the DevOps Research and Assessment programme (deployment frequency, lead time for changes, change-failure rate, time to restore) sit in the flow and resilience rows. They are research-derived measures; nobody supervises them. The EU Digital Operational Resilience Act is a binding regulation that says nothing about deployment frequency. On any slide a risk officer will read, label the measures the four keys or delivery-performance metrics and keep the bare word “DORA” for the regulation (Book 2 Ch 17 §4; Book 09 uses “the DevOps research programme” for the same reason). A CRO who reads “DORA change-failure rate” will reasonably ask which article it maps to, and the answer is none. For the regulation itself, see the regulatory map.
The five agent-era measures and their traps
Each measure has a way of reading it that inverts its meaning. Write the trap beside the measure on the scorecard (author model, Book 2 Ch 17 §4).
| Measure | Definition | Trap | Honest reading |
|---|---|---|---|
| Review latency by team | Time a change waits for and spends in review, per team | Celebrating a falling firm-wide lead time while one team’s queue grows a week deep | Per team only; the drowning squad hides inside the average |
| AI-assisted-change ratio | Share of merged change that carried a generated artefact | Making it a target, so engineers accept suggestions to hit the number | Diagnostic only; it stops being true the day it enters an objectives letter |
| Rework-and-restart rate | How often a thread of work is deleted and restarted from an adjusted spec | Driving it to zero and rewarding squads that never restart | Pair with guidelines harvested per restart: did each restart teach the factory something it wrote down? See Agent-Speed SDLC |
| Override-and-rejection rate | How often a human rejects or materially edits an agent’s suggestion | Reading near-zero as excellence when it is often a review function that has stopped reviewing | Read beside defect escape: falling overrides with rising escaped defects means review has gone to sleep |
| Evidence completeness for AI-assisted change | Share of small, fast agent changes carrying their full record | A healthy aggregate while the newest, fastest squad drops records | Read per squad, weighted towards the highest-frequency squads |
Book 2 adds a practitioner caution, explicitly not a measured claim: a large share of reported AI quality problems turn out to be evaluation problems, such as the wrong benchmark, an unrepresentative test set, or golden records nobody re-scored. Check the instrument before blaming the model. See Evaluation and Resilience.
Goodhart at machine speed
Agents make proxy-satisfying output close to free: coverage percentages, documentation counts and readiness checklists can all rise while the estate degrades. The defence is to measure outcomes, not artefacts, per station. For agent readiness, the ground truth is each station’s outcome record, meaning task success rate, intervention rate, restart rate and review effort per change, queried from the evidence record the firm already keeps (Book 2 Ch 17 §4). Readiness routes the lane only. The risk tier authorises autonomy.
The served-request event
Source claim (Book 2 Ch 17 §4 and §7; Book 09 Ch 10). Usage is the gauge that arrives first. Defect rates take a release to show and business outcomes take a quarter, but usage moves in the first week. The instrument is one event, emitted every time a shipped capability does its job, built into the common starting point every application inherits. Telemetry bolted on later measures what is easy rather than what matters.
What the event must carry
The sources fix the content of the event, not its field names: which capability, which caller, a person or another agent or system, and at what model cost (Book 2 Ch 17 §4; Book 09 Ch 10). The last two rows of the table below (time, and an outcome proxy) are this page’s recommendation, drawn from Book 2’s instruction to weight served requests by a value proxy the business accepts (Ch 17 §5). Name the fields in your foundation’s telemetry contract and version it with the foundation. Do not add it to the Audit Evidence Schema: the served-request event is high-volume usage telemetry, while the evidence record is the durable action record (see Running the Factory on keeping the two apart).
| Must carry | Why |
|---|---|
| Which capability or product served the request | Roll-up is per capability, not per host or per cluster |
| Which caller | Needed to count active consumers and to attribute agent-to-agent value |
| Caller type: a person, or another agent or system | Without it, a capability consumed as an API by other agents is invisible. This is the field most often missed. |
| Model cost of serving the request (tokens, or cost) | Feeds token cost per served request and payback |
| Time | Trend and grace-period calculations |
| Outcome, where the business has agreed a value proxy (for example a task completed, report generated or transaction processed straight through) | Turns a request count into a value-weighted count |
Recommendation. Make the event a property of the application foundation, so a product on the paved road emits it from day one without anyone asking. A product that reaches production without it fails the foundation’s acceptance check.
Failure modes the sources record
- Sessions, not callers. In the composite, the client-reporting product’s event was written before agents were consumers. Two downstream onboarding agents calling its endpoint appeared in request logs but not in the value roll-up, so the fastest-growing part of its usage was invisible (Book 2 Ch 17 §6). Book 09 tells the same story: sign-ins to the reporting service were falling while calls to it, from clients’ assistants and two onboarding agents, rose steadily (Book 09 Ch 10, composite).
- No event at all. The product recorded nothing about its own use, so a usage count took most of a week and had to be rebuilt from sign-ins on the identity service (Book 09 Ch 10, composite). Use sign-ins as a stopgap only, and say so in the pack.
- No baseline. Usage was low, but nobody had ever stated how many users were expected or what number the product was meant to move. Read usage against the baseline captured at intake, not against hope (Book 2 Ch 17 §6; Book 09 Ch 10).
Retrofitting caller type product by product is expensive. Add it at the foundation layer before the agent-to-agent channel grows (Book 2 Ch 17 §7).
Where the signal comes from, by cloud
As of August 2026, per Book 2 Ch 17 §4. None of the three gives you the value dimension out of the box; each gives you the raw signal. [FACT-CHECK: current Azure Monitor/Application Insights, Amazon CloudWatch (RUM, X-Ray, embedded metric format) and Google Cloud Monitoring/Logging/Trace capabilities and names against vendor documentation]
| Cloud | Primitives the source names | Where the served-request event lands |
|---|---|---|
| Azure (reference architecture) | Azure Monitor with Application Insights: request counts, per-endpoint volumes, user sessions, custom events | Custom event emitted by the application |
| AWS | CloudWatch metrics and logs; CloudWatch RUM for real-user monitoring; X-Ray for tracing | Custom metric, or an embedded-metric-format log line |
| Google Cloud | Cloud Monitoring and Cloud Logging, with Cloud Trace | Custom metric or structured log |
The failure mode is the same on all three: plenty of infrastructure telemetry, no usage telemetry, and an empty value line because nobody made emitting the event the default.
Consumption, not construction
Shared-services programmes failed because they counted components published. Count consumption instead. These metrics come off the bill of materials, the catalogue and the evidence record, so they need no new instrumentation (author model, Book 2 Ch 17 §5; BOM mechanics on BOM and Recipes).
| Metric | Definition | Reading |
|---|---|---|
| Inherited share | Fraction of a shipped product inherited by reference (foundation nodes, recipes, policy packs, evaluation cases, consumed capabilities) rather than written, measured off the BOM, per production line | Candidate headline number for the factory. No published benchmark exists, so the trend is the instrument. |
| Active consumers per catalogue asset | Production consumers, not registrations | Zero production consumers two quarters after publication puts the asset up for review |
| Time-to-first-consumption | Publication to first governed production consumer | The catalogue’s own lead time |
| Clone rate (counter-metric) | Duplication telemetry read against consumption | Inherited share and clone rate rising together means compounding and sprawl at once |
| Data-side twins | Time from data contract published to first governed consumer; consumers per data product | An unconsumed data contract after two quarters goes to the same review |
| Control inheritance rate | Share of a product’s required controls (from its risk tier and policy pack) whose evidence comes from an inherited asset | Should trend up. A perfect reading usually means the required-control list was trimmed. |
| Orphan rate | Share of catalogue assets with no active owner, or no evaluation run in 90 days | Should trend down. Work the list from the most-consumed orphan down: reassign, retire or quarantine, each with a date. |
The composite shows why consumption cannot replace usage. The unused reconciliations product had the highest inherited share on the floor, which is why it shipped so fast. Compounding made it cheap to build. It could not make it worth building (Book 2 Ch 17 §6).
Cash controls: token economics
Frame (author model, Book 2 Ch 17 §5). Token spend is the factory’s cash-flow statement; value from usage is its P&L. Cash is a fact; value rests partly on proxies the business agreed to accept. Run the token gauge and the loop-escape conditions as treasury limits with automatic consequences, not as performance metrics. That makes the CFO the gauge’s owner rather than its auditor. Spend caps and refusal at the limit live in the model gateway. Loop stop conditions are on Running the Factory.
Where tokens are a material delivery cost, run token economics as four moves:
- Attribute. Both build tokens (agent-assisted delivery) and run tokens (inference per served request) belong to the capability, not to a shared cloud bill. Without attribution there is an invoice, not an economics.
- Measure payback. Served requests over time, weighted by the agreed value proxy, against attributed tokens. The key ratio is token cost per served request. Falling cost with rising usage means the capability is paying back. Fixed cost with flat usage is a cost trap. A capability with almost no served requests has an undefined ratio, and you cannot compute payback on something nobody uses.
- Correlate with outcome. Where usage sits close to revenue, line the delivery timeline up against the outcome the capability was built to move (fee income, retention, straight-through volume).
- Respect the limits. Present correlation as correlation and name the confounders. The instrument allocates the next tranche of investment; it does not manufacture attribution.
A directional split of token spend across the ten most important products, refined over two quarters, is more useful than a precise figure that arrives too late (Book 2 Ch 17 §7). Per-task model choice, the four sourcing gates and the self-hosting breakeven test belong to the sourcing decision; see Build, Configure or Consume.
Book 09 adds a reason to put model cost on the usage event: the price of a given level of model capability is falling fast, so building gets cheaper every quarter while the cost of owning what was built (support rota, change slot, register entry, reviewer attention) does not fall. Usage is the instrument that separates the two (Book 09 Ch 10).
The shipped-unused balance and the patience runway
Shipped-unused balance (author model, Book 2 Ch 17 §5). A standing scorecard line, run as an impairment review rather than a wall of shame.
| Element | Rule |
|---|---|
| Entry condition | Capability past its grace period with served requests below threshold. Set both the grace period and the threshold locally; the source fixes neither. |
| Valuation | Run cost: licence, tokens, support, and its place in the change calendar (and, per Book 09 Ch 10, its register entry and the reviewer attention each change to it consumes) |
| Decision | Hold, fix or write down, with an owner and a date on every entry |
| Cadence | Every factory review, as a standing line |
| Write-down | Reclaims run cost and review attention. It is not an admission of failure. |
Composite. The reconciliations product went onto the balance with one question attached (does the process it served still exist?), a bias towards write-down, and a four-week deadline on the answer (Book 2 Ch 17 §6). In Book 09’s December board deck, the closing slide lists every product shipped that year with its users or calling systems for the month. Unused products are carried at running cost, each with decision, owner and date, and the reconciliations screen is marked retire (Book 09 Ch 10, composite).
Patience runway (author model, Book 2 Ch 17 §5). The quarters of board patience remaining at current burn and currently demonstrated value, updated at every review. Money out comes before money back by two to three quarters, and evidence is the interim dividend that keeps the funding rolling. A phase that produces no exit evidence visibly shortens the runway before the board says so. It is the first number read in the annual continue/shrink/expand decision on the factory; see Implementation Roadmap.
Validated-intent lead time and the intent coverage ratio
The delivery scorecard says nothing about whether the intent the factory delivers was worth validating. Book 2 Ch 17 reserves a line for it and Ch 21 supplies two numbers, read together (author model, Book 2 Ch 21 §5):
- Validated-intent lead time. Median elapsed time from a problem entering discovery to a bet card with fields 1 to 5 complete (problem, who it serves, validation evidence with a dated baseline, moat rank, learning horizon) and admitted to the agent lane. This is the discovery-side lead time. It ends where the flow dimension’s intent-to-production lead time begins; do not merge them.
- Intent coverage ratio. Validated intent admitted per period divided by agent-lane capacity consumed per period, both in the same unit (bet cards or specs, consistently).
| Ratio | Meaning | Action |
|---|---|---|
| Comfortably above 1 | Discovery is ahead; the lane is the constraint (the pre-agent world) | Invest in lane capacity |
| At or below 1 | Discovery is the constraint, which is where a working factory ends up | Normal; protect discovery capacity |
| Far below 1 with the lane running flat out | The lane is being filled with unvalidated intent | Confirm in the field-3 (validation evidence) column of the bet cards; stop admitting cards with field 3 blank |
Do not set an agent-lane utilisation target. The “empty-lane panic” anti-pattern is a CIO treating a dip in lane utilisation as waste. An idle lane costs tokens you do not spend. A full lane running on unvalidated intent costs review attention, run cost and a permanent operational tail. Show the coverage ratio to the same audience that sees utilisation (Book 2 Ch 21 §8). The opposite failure is discovery theatre: validated-intent lead time lengthens without admitted intent getting any better, and discovery artefacts hold sentiment but no observation. Fix it by giving field 3 a shape: what was observed, how many users, over what period, and what would have falsified the hypothesis.
Composite reading. At Asset Manager X’s first portfolio review, validated-intent lead time was 31 days and the coverage ratio 0.6, meaning the factory was still building faster than the firm could decide what was worth building (Book 2 Ch 21 §6). If the line cannot be instrumented yet, reserve it on the scorecard marked “not yet instrumented”. An empty line states what the firm intends to know (Book 2 Ch 17 §5). The bet card itself belongs to product discovery practice and is out of scope here.
Two leading numbers for the control gap
The delivery scorecard reads throughput. The control gap opens on the other side, when change arrives faster than the people who put their names to it can understand it. Book 09 makes two numbers the first thing to pull, and makes reporting them weekly the first of its ninety-day lines (Book 09 Introduction; Closing).
| Number | Definition | Source records |
|---|---|---|
| Median open-PR wait | Median age of open pull requests waiting for review, taken each morning | Repository platform: review requests, comments, approvals |
| Uncommented approvals on large changes | Share of approvals on changes over twenty files that carry no review comment | Same |
How to pull them (source rules, Book 09 Introduction and Closing):
- Split every number between the teams that use the coding assistants most and those that use them least.
- Ask for the weekly series since before the assistants arrived, not last month’s figure. One month shows where you are; the line shows the direction.
- Use ninety-day windows for comparisons. A month is noise; a year hides the bend.
- Report both every week to the risk committee. The CRO sees them before the dashboard does.
- Put reviews per reviewer beside them, to see whether the load sits on a dozen names.
Composite signature. Median open-PR age held at about a day and a half for two years, then rose to more than three days over six weeks with no step in it. On changes over twenty files in the heaviest-adopting domains, uncommented approvals went from one in twenty to one in six. One team of eight went from about fifteen pull requests a week to thirty-three with the same reviewers, while the firm-wide rise of about two-fifths looked like the business case arriving early (Book 09 Ch 1).
The twenty-change sample
The two numbers show strain; the sample shows whether oversight happened. Someone outside the teams draws twenty changes at random from last month, including some a machine approved if any machine does. Fill five columns from the record only (Book 09 Closing):
| Column | Fill with |
|---|---|
| Who approved it | Person or mechanism |
| What they checked | A review comment, a checklist answered, or a test they added. “They reviewed it” is not an answer. |
| Who made its tests | A person, a different agent, or the same agent that made the change |
| Minutes from opening to approval | From the repository |
| Did it alter a control value? | A limit, threshold, tolerance, or anything else that decides what a client or investor sees |
Two counts come out: rows where what they checked is blank (oversight that is a name on a button) and rows where tests came from the same source as the change (testing that is a change agreeing with itself). Repeat the sample monthly. The sample is about two hours’ work.
Interpretation
| Reading | Meaning | Basis |
|---|---|---|
| Wait rising, uncommented large approvals flat | A capacity problem. You can hire into it, slowly. | Source claim (Book 09 Introduction) |
| Wait and uncommented large approvals rising together | People are passing what they have not read, and the records will not show it | Source claim (Book 09 Introduction) |
| Wait rising in heavy teams, uncommented large approvals climbing, and the sample’s “what they checked” column mostly empty | The control gap is open. Move to a first proxy and the control budget. | Source claim (Book 09 Closing) |
| Heavy teams’ wait flat | Find out why and copy that team. In the composite it was the team with the longest review comments and the most uniform repositories. | Source claim (Book 09 Closing; composite, Ch 1) |
| Wait falling while uncommented approvals rise | Latency improved because review got thinner. Do not read this as capacity gained. | Recommendation |
| Uncommented approvals high on large mechanical changes (renames, dependency updates) | Can be sound. Read a sample before concluding. | Source caveat (Book 09 Introduction) |
No single number is a verdict; the combination is the finding, and a gap is a reason to read a sample of the changes behind it (Book 09 Introduction; Closing). The composite shows why: a reviewer approved a thirty-one-file rename in about four minutes, without comment. Re-reviewed properly, it took fifty-two minutes and turned up a nightly export in another team’s repository that still read the setting by its old name. The change turned out to be correct, because the agent had added an alias, but neither the reviewer nor the record could have said so at the moment of approval (Book 09 Ch 1).
Treat an unexplained sharp improvement as an alarm
Source claim (Book 09 Ch 8). For each domain, name the number that would move if the routing gate were letting the wrong things through, and treat a sharp, unexplained improvement in it as an alarm rather than a result. Composite. At a quarter-end close, the cash breaks reaching a human fell from sixty-odd to three. The falling break count on the wall had been applauded. The cause was a four-line tolerance change, approved automatically and tested only by tests generated from the same mistaken specification, which widened the default tolerance about a hundredfold for most of the firm’s cash accounts.
Recommendation. The steps below operationalise the Book 09 rule. Step 4 is the composite’s own checklist change (Book 09 Ch 8). The expected range in step 2 and the owners are this page’s additions.
| Step | What to do | Owner |
|---|---|---|
| 1 | For each domain, write down its alarm number: the count that falls when the control is silently failing (for example auto-cleared exceptions, breaks escalated to people, alerts raised, items rejected) | Domain product owner, with the second line |
| 2 | Set an expected range from the domain’s own record and state the conditions that legitimately move it (for example quarter-end volume). Quarter-end should produce more to disagree about, not less. | Domain product owner |
| 3 | When the number improves sharply without a recorded cause, open an investigation, not a celebration | Factory operations |
| 4 | Read the list the automation cleared: on the first day of every close, sort by amount; two people read it and sign. Composite name: the cleared-break reading. | Squad lead |
| 5 | Trace the cause back to the change record: owner field, test origin, and whether the change touched a control value | Platform lead |
Book 09 Ch 8 lists four further checks for any firm running a routing gate. Pull ten changes approved without a person and read the owner field; a service account, mechanism or mailbox where a person should be is a finding. For the same ten, ask who wrote the tests. List the values that decide what people see and check how change policy classes them; route every change to them to a reviewer. Find who last moved the gate’s threshold, on what evidence, and where that is recorded. Find every automation that answers an outside party on the firm’s behalf and check that it is on the register. Gate design is on Change Governance; this page only adds the alarm number to the scorecard. Falling error rates should always be tested against independent evidence (see Running the Factory).
Agent count against wrangler count
Source claim (Book 1 Ch 1 §7.3, §8.1). Scale agents only as fast as controls scale. The warning sign is agent count rising faster than wrangler count, often with the change board still listing agents as “applications”. Book 1’s sizing for a 50,000-person firm is one named individual wrangler per agent, with no wrangler holding more than five to seven agents. Recovery: put a hard pause on the agent backlog (not on agents already running), install the inventory and wranglers, and resume only when controls are at least one quarter ahead of the deployment plan.
Recommendation. Put three numbers on the control line of the scorecard each month: agents in production (from platform logs, not the register), certified wranglers holding at least one agent, and the maximum agents per wrangler. Book 09’s equivalent acceptance test is that the register and the logs give the same agent count and every entry names a person, not a team (Book 09 Closing, statement 1). Wrangler duties are on Accountability and Agent Lifecycle.
The board-reporting catalogue: a second layer
Book 3 organises the metrics a Claude programme reports to its board into eight categories, each with a named owner and cadence (Book 3 Ch 20 §5.1, OD-3). It answers a different question from Book 2’s scorecard (“is the AI programme controlled and worth it?” rather than “is the factory compounding?”), so read the two as layers: the catalogue feeds the board reports, and the scorecard runs the factory review. Where they overlap (cost, evidence, incidents), use one definition and one source.
| Category | Example metrics | Owner | Cadence | Nearest scorecard dimension |
|---|---|---|---|---|
| Adoption | Active users; active agents; business units onboarded; approved prompts used; share of AI-assisted work inside approved channels with usable evidence | AI Platform Product Owner | Monthly | Value; developer experience |
| Value | Hours saved; cycle-time reduction; quality improvement; cost avoidance; revenue enablement | Business owner per use case | Quarterly | Value |
| Control | Registered-agent coverage; evidence completeness; unresolved exceptions; review timeliness; attestation completion | AI Platform Owner | Monthly | Evidence |
| Risk | Incidents; near-misses; policy violations; prompt-injection attempts; data-loss events | CRO via CISO and Model Risk | Quarterly | Security; resilience |
| Model performance | Evaluation pass rates; hallucination rate; source-grounding quality; deterministic-test regression count | Head of Model Risk | Quarterly | Quality |
| Cost | Token spend; cost per use case; cost per successful task; budget exceptions | AI Platform Owner | Monthly | Cash controls |
| Training | Completion rate by role; failed knowledge checks; refresher compliance | Head of HR / Learning, with the AI Platform Owner | Quarterly | (none; add it) |
| Shadow AI | Discovered tools; remediated tools; blocked domains; migration to approved path | CISO with the AI Platform Owner | Quarterly | Developer experience (the shadow road) |
The leading indicator (source claim). The share of AI-assisted work happening inside approved channels with usable evidence is read first. If it is rising, the rest of the catalogue can be trusted; if it is falling, the rest is decorative. “Number of prompts” is not a useful early metric.
Two reporting rules from Book 3 Ch 20 apply to both layers. The quarterly risk report (owned by the CRO) and value report (owned by business owners, with the AI Platform Owner co-signing) are separate documents, and the value report is read alongside the risk report (Book 3 Ch 20 OD-2), so a use case with strong value and a recurring near-miss stays visible instead of being averaged into one colour. And adoption, value, risk and control are read together or not at all: a pack that reports rising adoption and value but omits incidents and near-misses is the “board reporting that hides risk” anti-pattern. The full reporting grid is on Implementation Roadmap.
Label every reported number by evidence tier
Author model (Book 3 Ch 20 §5.1). Board packs fail when a vendor console screenshot reaches the management body with the authority of a measurement it never had. Every number in a pack states its tier:
| Tier | What belongs here | May drive a decision? | Where it may appear |
|---|---|---|---|
| 1. Independently attested | Third-party benchmarks, externally verified certifications and assurance reports (Book 3 gives a SOC 2 Type II report and a proctored credential as examples) | Citable as evidence; the source reserves decision-driving for tier 2 | Anywhere, cited |
| 2. Institutionally measured | Numbers your own systems produced: gateway token spend, registry coverage, evidence completeness, incident counts, evaluation pass rates on your own suites | Yes. The only tier that should drive a decision. | Anywhere. The metrics catalogue belongs here. |
| 3. Estimated with visible formulas | Anything built on an adjustable assumption, including vendor value calculators | No. Never carries a budget decision. | Context in the value report, with the assumption in the same cell. Never in the strategic-posture section of the annual board paper. |
| 4. Vendor self-reported | Vendor claims about their own product’s effect | No | Attributed in the sentence, never in a headline |
The template rule. An adjustable estimate never enters board reporting as measured fact. Add a source column to the local metrics catalogue and fill it in. Any row whose source turns out to be a vendor console estimate has been mis-filed: it moves to tier 3 and carries its assumptions into the pack. The test for any number in a deck is: which system produced this, and could someone change it with a slider? Recovery for a pack that failed the test is to apply the tier labels retroactively to the last four quarters (Book 3 Ch 20 §7).
Claude-specific (as of August 2026, per Book 3 Ch 13 §5.1 and Ch 20 §5.1).
- The Claude admin console’s Value tab (Enterprise) estimates productivity lift, cost per commit and annual value. Anthropic states that the formulas are visible and the inputs adjustable, which makes these figures tier 3.
[FACT-CHECK: Value tab availability, plan scope and what it computes, against current Anthropic admin documentation] - Contribution metrics (beta on Team and Enterprise since 29 January 2026) match Claude Code sessions to GitHub commits and pull requests, count only code with high confidence of Claude Code involvement, and require the Claude GitHub App and GitHub Analytics. The method is stronger, but it is still the vendor’s attribution of its own product. Treat it as tier 3 at best until your own records corroborate it, and as tier 2 only for figures you reproduce from your own repository and gateway data.
[FACT-CHECK: whether contribution metrics has left beta, its plan availability, and its export/API and retention behaviour] - Anthropic’s headline figures (67% more pull requests merged per engineer per day; 70–90% of code written with assistance) are Anthropic’s internal results: tier 4. Cite them as vendor testimony with the label, or not at all.
Per-user contribution metrics
Source claim (Book 3 Ch 13 §5.1; Ch 20 §5.1). Attributing AI-assisted work to individual employees is employee-monitoring data before it is productivity data. Book 3 also notes that the vendor documentation says nothing about export, API access or retention of this per-user telemetry (as of August 2026). Rules 2 and 3 are Book 3’s; rules 1, 4 and 5 follow the composite Asset Manager X configuration and Book 3’s DPO review cadence, adopted here as recommendations:
- Default to team-aggregate granularity, with per-user attribution switched off (the composite’s setting).
- Run the data-protection impact assessment before enabling per-user attribution, not after the first question about how a number in the deck was produced.
- Consult the works council where one exists. Book 3 states that in Germany, Austria and the Netherlands this consultation is a condition of the feature going live.
[FACT-CHECK: legal review of works-council consultation as a go-live condition in DE, AT and NL]Confirm the position for your jurisdictions with counsel; the legal basis is on the regulatory map. - Store the DPIA reference against the tenant configuration record, as the composite does.
- Re-review the configuration annually. The DPO re-checks contribution-metrics and usage-telemetry settings against the DPIA and the works-council agreement, and confirms per-user attribution is still off unless separately agreed (Book 3 Ch 13 OD table).
[FACT-CHECK: whether per-user attribution in Claude contribution metrics can be disabled tenant-wide by policy rather than by convention]
This aligns with the Goodhart rule above. Participation metrics are diagnostics of the delivery system, never a way to rank people (Book 2 Ch 17 §8). Using them on individuals turns measurement into surveillance and destroys the trust the metrics depend on.
The control budget’s meter
The scorecard carries the control budget’s position as a line. The budget itself (who signs it, how it is counted, what happens when it is spent) is on Control Budget. One rule belongs here because it is a measurement rule (source claim, Book 2 Ch 17 §5 and Ch 24 §4.9): control capacity is weighted by evidence quality, never counted as raw approvals per hour. A reviewer approving forty changes an hour is not forty times the capacity of one who approves carefully. Count only reviews that left their full record, evaluations that actually ran and samples that were actually read. A budget with a flattering meter is worse than none, because it authorises throughput the controls cannot cover.
Ownership and cadence
Recommendation. These tables assign the sources’ measures to roles and rhythms. Source-fixed cadences are the weekly two numbers and monthly sample (Book 09 Closing), the quarterly review (Book 2 Ch 17) and the catalogue cadences (Book 3 Ch 20 OD-3); the rest is this page’s allocation.
| Role | Owns |
|---|---|
| CIO / head of the factory | The factory review and its order; the scorecard; the decision that no product closes green on delivery alone |
| Platform owner (application foundation) | The served-request event contract and its default emission; usage roll-up per capability |
| Adoption lead / usage analyst | Reading usage per product each week; the shipped-unused balance entries |
| Product owner | Baseline at intake; hold/fix/write-down decisions; the domain’s alarm number |
| Finance (CFO’s team) | Token attribution and the cash-control view; the patience runway with the head of the factory |
| Head of product / discovery | Validated-intent lead time and the intent coverage ratio |
| CRO and second line | Seeing the two leading numbers weekly; the control budget meter; countersigning tier labels in board packs |
| Internal audit | The twenty-change sample’s independence; testing the counting behind control lines |
| DPO | DPIA before per-user metrics; granularity setting |
| Cadence | What is read |
|---|---|
| Each morning | Median open-PR wait (automated) |
| Weekly | Two leading numbers by heavy/light split; control budget position per domain; usage per product |
| Monthly | Twenty-change sample; agent and wrangler counts; Adoption, Control and Cost catalogue categories |
| Quarterly | Full scorecard review, usage first; shipped-unused balance; patience runway; separate risk and value reports; Value, Risk, Model performance, Training and Shadow AI categories |
| Annually | Factory continue/shrink/expand decision, reading the runway first |
At your size (source claim, Book 2 Ch 17 §7). In a large bank, the value dimension is a platform product: a served-request standard in the foundation, a central usage roll-up, and an adoption lead per product family. In a firm of a few thousand (composite Bank D), it is one dashboard the head of technology reads himself: ten products, one usage curve each, token cost per served request beside it, and one inherited-share number for the year. Only the number of readers changes.
Failure modes and recovery
| Failure | How you recognise it | Recovery |
|---|---|---|
| Productivity-only measurement | Every slide is throughput; the review can end without anyone asking about use | Usage becomes the first column and the first question |
| Counting construction | Usage measured in provisioned seats; catalogue measured in entries published | Value-weighted served requests; consumption metrics; review any asset with no production consumer after two quarters |
| Composite index | The review discusses why the index moved | Refuse the aggregation; one headline per dimension, per-station outcomes beneath |
| Participation metric as a target | An assistant-usage number appears in objectives or individuals defend it | Withdraw it as a target; keep it as a diagnostic |
| Restart rate driven to zero | Squads nurse bad threads; defect escape rises a release later | Reward the early restart; read guidelines harvested per restart |
| Event counts sessions, not callers | Request logs and the value roll-up disagree | Add caller type at the foundation; re-instrument the affected products |
| Green number nobody questions | A sharp fall in exceptions with no recorded cause | Alarm number per domain; two-person reading of what automation cleared |
| Vendor estimate in the board pack | Nobody can say which system produced a number, or it has a slider | Tier labels and a source column; relabel the last four quarters |
| Per-user metrics switched on by default | No DPIA reference on the tenant record | Revert to team-aggregate; run the DPIA; consult employee representatives |
| Utilisation target on the agent lane | Field 3 blank on admitted bet cards | Show the coverage ratio beside utilisation; remove the target |
Acceptance checks
A practitioner can test the measurement system against these:
- The quarterly review agenda puts usage before delivery, and the minutes show no product closed green on delivery alone.
- Every product on the foundation emits a served-request event carrying capability, caller, caller type (person or agent/system) and model cost. A product launched last quarter shows agent callers in its roll-up if it has any.
- Each of the ten most important products has seven scorecard columns filled from records, with a value and an evidence column, and each measure has an owner, definition, source, cadence and action threshold.
- No participation metric appears in any objectives letter or board KPI slide as a target.
- Every entry on the shipped-unused balance has a hold/fix/write-down decision, an owner and a date.
- Token cost per served request is computed for every product where tokens are a material cost.
- The scorecard carries a validated-intent lead time and intent coverage ratio line, even if it reads “not yet instrumented”, and there is no agent-lane utilisation target.
- Median open-PR wait and uncommented approvals on changes over twenty files are reported weekly, split heavy/light, to the risk committee; the twenty-change sample is drawn monthly by someone outside the teams.
- Each domain has a named alarm number with an expected range, and the cleared list is read and signed by two people at each close.
- Agent count from platform logs equals the register count; no wrangler holds more agents than the firm’s stated maximum.
- Every number in the last board pack carries a tier label; no tier-3 figure sits in the strategic-posture section or carries a budget decision.
- Per-user contribution metrics are off, or a DPIA reference is stored on the tenant configuration record and employee representatives have been consulted where required.
Provenance and refresh
The seven dimensions, agent-era measures, Goodhart rule, served-request event, consumption metrics, cash controls, shipped-unused balance and patience runway are synthesised from Book 2 Ch 17. The intent metrics come from Book 2 Ch 21. The evidence tiers, eight-category catalogue and per-user caution come from Book 3 Ch 13 and Ch 20. The two leading numbers, the sample, the alarm rule and the usage-first ordering come from Book 09 (Introduction, Ch 1, Ch 8, Ch 10 and Closing). The agent/wrangler signal comes from Book 1 Ch 1.
Source conflict noted. Book 3 Ch 13 §5.1 attributes the 67% and 70–90% headline figures to Anthropic’s 2 July 2026 admin-visibility announcement, while Book 3 Ch 20 §5.1 places them alongside the 29 January 2026 contribution-metrics announcement. The tier-4 label applies either way; confirm the originating announcement before citing either figure.
Refresh quarterly, and on any change to the Claude admin console’s reporting features or to the books’ measurement chapters. This page remains a draft.
Source material
- Book 2 (The AI Delivery Factory), Chapter 17, Measuring the Factory (§4 three loops, value from usage, seven dimensions, AI-specific measures and their traps, Goodhart rule, DORA naming, usage signal on Azure/AWS/GCP; §5 compounding scorecard, consumption metrics, cash controls, token economics, shipped-unused balance, patience runway, metered lines; §6 Asset Manager X review; §7 Monday actions; §8 anti-patterns)
- Book 2 (The AI Delivery Factory), Chapter 21, Product Development at Agent Speed (§5 validated-intent lead time and intent coverage ratio; §6 worked reading; §8 empty-lane panic and discovery theatre)
- Book 2 (The AI Delivery Factory), Chapter 24, Running the Factory (§4.9 control budget and its measurement)
- Book 3 (Deploying Claude Safely), Chapter 13, Wranglers, Product Teams and RACI (§5.1 contribution metrics, DPIA and works council, the Value tab)
- Book 3 (Deploying Claude Safely), Chapter 20, The 18-Month Roadmap and Reporting to the Board (§5.1 evidence hierarchy; §7 board reporting that hides risk; OD-2 reporting grid; OD-3 metrics catalogue)
- Book 09 (Everything Is Green), Introduction (the two numbers) and Chapter 1, The Day the Review Queue Broke
- Book 09 (Everything Is Green), Chapter 8, The Number That Was Too Good (checks before it happens to you)
- Book 09 (Everything Is Green), Chapter 10, Is Anyone Using It? (usage first; the usage event; participation is never a target)
- Book 09 (Everything Is Green), Closing, An Afternoon With a Pencil (records table, twenty-change sample, ninety-day line 1)
- Book 1 (Building Organizations for the Age of AI), Chapter 1, The Adoption Gap (§7.3 wranglers within one quarter; §8.1 agent count against wrangler count)
Related guidance
-
Run the AI Delivery FactoryOperate demand, capacity, the two worker classes, exceptions and evidence as one service: read two leading numbers, keep a handover backlog, gate construction on three conditions, run queues and alerts at agent volume, and count unused products as cost. -
Control BudgetAn author proposal for a CRO-signed limit on how fast change may arrive relative to the firm's capacity to understand it: the CIO runs the line, the CRO signs the band, only changes with a full record count, two weeks outside the band queues intake automatically, quarter-end widening is signed in advance, and someone outside the squad reviews what the budget held back. -
Build the Standard Application FoundationRun the shared identity, telemetry, deployment, data and AI capabilities as a funded product that humans and agents both start from, with a contract, evidence-computed trust, a cheap promotion boundary and reuse measured by consumption. -
Design and Operate the Model GatewayBuild the model gateway as the governed edge for every model call: per-client identity, per-request highest-class routing, tokenisation, pinned models, a conformant wire contract, a correlated evidence record, a tested concentration answer, and day-two cost, latency and reliability operations across two estates and three meters. -
The Agent-Speed SDLCA committed artefact chain, governed context, trusted tests and eight SDLC controls that let coding agents work quickly while intent, review, identity and evidence stay connected. -
Change Governance for Agent-Enabled DeliveryRoute agent-speed change by risk without losing the approver: control values that always reach a person, a routing gate that reads the right signals, autonomy earned per change family, a Virtual CAB that prepares a complete dossier, and three bodies with defined escalation. -
The Audit Spine and the Evidence RecordHow to capture, join, protect, retain and prove per-action evidence: the AES v0.1.0 record contract, three capture points and one correlation ID, tiered immutable storage, retention and legal hold, replay rehearsal, the route-approval evidence gate, and the Claude-specific ways evidence goes missing. -
Accountability and OwnershipHold provenance and accountability apart on purpose: four named accountabilities per product, a human owner for every agent change, seven named seats, a signed Build/Run RACI and a first-line/second-line boundary that survives consolidation. -
Build a Living Product Bill of Material and RecipeDescribe what an AI product contains and how the firm builds it in one versioned specification shared by pipelines, agents and assurance teams, including the classification line, readiness grades and reuse generations. -
Evaluate, Assure and Test Agent ResilienceDefine a versioned evaluation contract, test the trajectory and the execution boundaries, run three assurance layers into one quarterly pack, map the OWASP Agentic Top 10 to controls that demonstrably fire, bring agents into threat-led penetration testing with a five-section scope, keep a continuous cadence between formal tests, land every finding as a live control-plane artefact, and engineer recovery down to a minimal AI service level on models you host. -
The Implementation RoadmapSequence the factory as a dependency graph, not a calendar: the closure order for the governance gap, a written portfolio threshold, six overlapping phases over eighteen months with entry conditions, exit evidence, 'not yet' lists and pass/soft-fail/hard-fail gate questions, tranche funding released on exit evidence, a programme RACI, size, maturity and sector calibration with four signed trade-offs, and the first week, thirty days and ninety days in dependency order. -
Build, Configure or ConsumeThe layered sourcing decision: a signed portfolio threshold (8 products, 24 months, 3 shared patterns including shared data) decides whether building is available at all; Research Paper 7's four gates decide each capability; Book 2's five weighted dimensions score the contested cases. Covers the annual estate sort, the audit-log export clause for rented workflows, the model-sourcing breakeven test and register, and who owns what is built. -
Route Regulatory Questions to Obligations and EvidenceA dated, article-level reference that routes AI design questions to the EU AI Act, DORA and its RTS, GDPR and sectoral rules, and turns each obligation into a control, an owner and evidence: change records under RTS 2024/1774 Art 17, the 4 h / 72 h / one-month incident clocks, the CIF hinge and its five artefacts, the three-count Register of Information reconciliation, Art 14(4) oversight controls, the Art 26(5) discharge, the FRIA gate, Art 49 and Art 50, SCHUFA-grade review and supplier controls. -
Bounded-Agent PatternA class 3 agent lets a model choose the next step, but only inside a registered, validated and enforced envelope of Mandate, Means, Memory and Monitoring.