Browse the reference ⌄
Implementation Guide published

Release Engineering for Agent-Written Change

Move agent-speed change into production through five evidence-emitting stations: one change type per change, a reversal decided at authoring, a valid Deployer Manifest, a gateway that cannot be bypassed, hard health gates and at-most-once rollback.

Version
0.2.0
Owner
DEIXEI
Last reviewed
Reading time
34 min

For Cto · Ai Architect · Platform Engineer · Product Leader · Risk And Assurance · Change Manager

Sources and details
Source works
Book 1 · Book 2 · Book 3 · Supporting Tools · Research Series · Book 09
Source chapters
7, 8, 9, 11, 13, 15, 16, 17
Tool specifications
deployer_manifest@0.1.0
Refresh trigger
source change

Full source list ↓ Source index →

AI-assisted delivery raises the number of plausible changes arriving at production. It does not raise the number of people who can review them. Release engineering is the practice that keeps each change controlled, observed and reversible at that volume. It is the technical work of versioning, storing, promoting, deploying, verifying and reversing an artefact. It is not release management (scheduling, stakeholder sign-off, communication). When firms try to solve an engineering problem with a meeting, the meeting becomes the bottleneck that agent-speed delivery then overruns (Book 2 Ch 13 §4).

The page is organised around the decisions a platform team, a product team’s wrangler and the second line each have to make before a change reaches traffic. Where the books and the Supporting Tools specification disagree, the difference is stated and the page says which version it follows.

A six-stage delivery loop separates human decisions from agent-executed work and commits an artefact at each transition.

The committed artefact chain: every transition is recorded, while the required human decisions remain explicit. Source: The AI Delivery Factory, Chapter 15, Figure 15.1.

The three answers every change carries

Before a change enters the path it must answer three questions in machine-readable form (Book 2 Ch 13 §3):

  1. What type of change is this? Exactly one type (see one change type per change).
  2. How does it reverse? A named mechanism suitable for that type, decided at authoring and tested (see reversal plan by change type).
  3. What does healthy look like while it rolls out? A change-specific metric enforced as a hard gate at the first rollout stage.

A change that cannot answer all three does not enter the path. The answers live in the Deployer Manifest, not in someone’s memory.

The release path: five stations

Every change, human- or agent-authored, travels the same stations. No change skips one, and each one emits evidence as it runs (Book 2 Ch 13 §5).

The stations and their evidence are from the source; the owner column is this page’s suggested allocation, drawn from the Book 2 worked example (wrangler defines healthy, platform engineering enforces it).

#StationWhat happensEvidence emittedOwner (suggested)
1Thread and branchThe unit of work is isolated: one agent thread, one branch, one workspace. A drifting thread is deleted and re-dispatched from an adjusted spec rather than steered back (thread restartability, agent-speed SDLC).Branch, spec reference, provenance (which artefacts an agent produced, against which spec)Dispatching wrangler
2Merge gate (route by risk)The change is scored and routed, not read in full by a human: low risk to automated approval, higher risk to human review or the Virtual CAB. Provenance is a routing input: an agent-drafted schema migration earns extra test data and a second reviewer; an agent-drafted copy change does not.CAB Decision Record with reasoning, for every route including automated onesChange Risk Index owner; approvers per change governance
3Progressive rolloutExposure widens in stages: canary or ring, then cohorts, then full. Never all of production at once.One record per stagePlatform engineering
4ObserveHealth checks defined before deployment run as hard gates at each stage. Healthy advances; unhealthy stops and reverses.Health-check results per stageWrangler (definition), platform (enforcement)
5Reversal decisionThe reversal recorded in the manifest is executed by the path without a human inside the window, and the reversal itself is verified.Rollback start, verification per check, rollback endPlatform; wrangler attests

Two properties turn the diagram into a control. First, the audit trail is a by-product of the stations running; nothing is assembled after the event. Supervisors increasingly expect evidence produced as the path ran, not a ticket written afterwards (Book 2 Ch 13 §4, stated as the author’s supervisory experience, not a quoted rule). Second, the merge gate is where scarce review meets tripled change volume. A gate that tries to inspect everything becomes a rubber stamp. The scoring mechanism belongs to the Change Risk Index and change governance. The regulatory basis for controlled, independently approved, reversible ICT change is on the regulatory map.

The readiness card

Print it. A change that fails any line does not enter the path (Book 2 Ch 13 §7):

  • One type. The change is a single change type. Logic plus migration plus configuration is three changes and travels as three.
  • A reversal, named and tested. Rollback, rollforward, flag-off, or a named compensation with an owner, and the reversal has been run, not just written.
  • A definition of healthy. A named metric that shows the change is hurting, enforced as a hard gate at the first stage.
  • A bounded first exposure. Canary or ring, never the whole estate.
  • Evidence by default. Routing decision, rollout stages, health results and any reversal are emitted by the path.

For an AI product, replace and add lines:

  • The pinned combination is the one type. The release manifest names every pinned element and automatic model upgrades are off (see model releases).
  • The failure-mode register was re-scored against this manifest and the detection column still holds (see failure-mode register).

First diagnostic (recommendation, Book 2 Ch 13 §7). Take your ten most recent change-caused incidents and ask of each: what change type was it; was it bundled with another type; was the reversal decided before deploy; did the rollout limit blast radius; did the path emit evidence as it ran; did approval land on the real risk? Bundling is the usual finding. The highest-leverage first fix is one recipe change: make change-type classification and a matching reversal plan a deployment precondition, and make the first-stage health check an automated gate.

One change type per change

Large bundles combine things that should be reasoned about separately. They raise review burden, blur what the tests proved and make rollback ambiguous. Cheap generation makes this worse, because an agent produces the application change, the migration and the configuration change in one thread and it feels efficient to ship them together. Each part looks plausible to a reviewer; together they may not be reversible (Book 2 Ch 13 §4, §8).

Rule. One change, one type, one reversal plan. A thread that produced several types is split before the merge gate into separate changes on separate paths, joined by a common change_id. This matches the Deployer Manifest’s own design decision D19.1: one artefact per manifest, with multi-artefact orchestration handled above the schema at the CI/CD layer, where each step has its own manifest, apply and verify (UC-deploy-7).

Exception: the AI release unit. A prompt and the model version it was evaluated against are one change of one type, not a bundle. They move together or not at all (Book 2 Ch 13 §4).

Metric. Batch size is the first number to watch when agents arrive. Track it with deployment frequency, change-failure rate and time to recover. Healthy agent adoption shows smaller changes shipped more often with a falling failure rate. Larger changes shipped faster with a rising failure rate means the path has inherited the cost of cheap generation (Book 2 Ch 13 §7).

Acceptance check (this page’s recommendation). The merge gate refuses a change whose diff touches more than one change type unless it is a declared AI release unit.

Decide the reversal at authoring time

Reversibility cannot be designed during an incident. It is a property chosen before deploy and recorded in the change (Book 2 Ch 13 §4). The two families are rollback (return to a previous known-good version) and rollforward (advance to a fixed version). At high cadence, rollforward becomes the normal answer for stateless services, but not for everything.

Author’s rule (Book 2 Ch 13 §4, marked as preference). The recipe classifies the change type and requires the reversal mechanism that type supports. An irreversible change cannot deploy without an explicit, named compensation and a named owner for it.

Change typeReversal mechanismWhat must exist before deploy
Stateless application versionRollback to prior artefact, or rollforwardPrior artefact still deployable; digest recorded
Application logic behind a feature flagFlag-off, no redeployFlag provisioned; prior code path intact; flag change itself governed
Database schema changeCompensating (“down”) migration, not a revertThe down migration written the same day as the up, and tested
Data backfillIdempotent re-run, or quarantine behind a flag until provenProof of idempotency; quarantine mechanism
Traffic routing changeRoute awayPrior route still healthy and addressable
Secret or certificate rotationRotate back to the prior credentialPrior credential deliberately kept valid for the window
External side effect (message to counterparty, file to custodian, payment)Usually none; compensation onlyNamed compensation (correcting message, manual intervention, call), named owner, acceptable recovery time
Model / prompt / retrieval combinationRevert to prior pinned combination at the gatewayPrior model version still served by the provider; evidence portability check (see below)

The failure that hurts most is the change that assumed a rollback it did not have: the schema migration that “would just be reverted” and could not be. Reversal is rehearsed on a schedule, not first attempted on incident day (Book 2 Ch 13 §8).

Feature flags as reversal and as change. Flags decouple deploy from release and give the finest-grained reversal; without them the path can only reverse a whole deployment, which is too coarse at agent volume (Book 2 Ch 13 §4). But a flag that changes security posture, data access or regulatory behaviour is a governed change, not a preference: versioned, environment-scoped, reviewed, and not silently flippable by an agent. An agent flipping such a flag is a control-plane action (Book 2 Ch 11). Flags are provisioned by the application foundation; see application foundation and identity and secrets for the configuration controls, including detecting drift between declared and running configuration.

The Deployer Manifest: what makes a deployment valid

The Deployer Manifest is the structured record that drives a deployment through plan → preflight → apply → verify → rollback. Schema: /aes/v0.1.0/deployer_manifest.json (JSON Schema draft 2020-12, status “agreed and published” per D19). When ops deploy apply runs, the manifest becomes the action.parameters of an Audit Evidence Schema record with kind=deployer.apply; see audit evidence for the envelope and schemas and tools for the wider tool set. Where prose and spec disagree, the spec wins (Book 3 Ch 16 §3).

Validity rules

A manifest is valid for deployment only if all of these hold. The first four are the invariants the deployment gateway enforces (D23).

#RuleFieldSourceRefusal code (M3 draft)
1A CAB decision is always named, or a cited exemption is given. The key is always present; it may be null only if cab_exemption_rationale is non-empty and cites the policy that allows the exemption. There is no “no CAB decision”, only a decision or a documented exemption.preflight.required_cab_decision, preflight.cab_exemption_rationaleD19.3, schemam3.required_cab_decision_missing
2Health checks are non-empty. A deployment with no verification is forbidden.verify.health_checks (minItems: 1)D19, schemam3.health_checks_empty
3Rollback is verified. Every rollback runs its own checks.rollback.verificationD19, D23, UC-deploy-5m3.rollback_verification_empty
4A named accountable human. A deployment without one cannot run.metadata.agent_wrangler_upnD19m3.wrangler_upn_missing
5One artefact per manifest.artefact (single object)D19.1—
6The referenced CAB decision resolves, is not no-go, has all go-with-conditions conditions fulfilled=true, and is not stale.resolved from the evidence lake at preflightM3 §3.2, UC-deploy-2m3.cab_decision_missing, m3.cab_decision_no_go, m3.cab_condition_unfulfilled, m3.cab_decision_stale
7backend.type is in the enum and backend.config validates against config_schema_ref if given.backendD19.2, D23.3m3.backend_config_invalid

Source difference: rule 3. The published schema v0.1.0 requires rollback.verification to be present but sets no minItems, so an empty array validates. D23, the M3 conformance vectors and UC-deploy-5 all require it to be non-empty. This page follows the gateway rule: treat an empty rollback.verification as invalid, and enforce it at the gateway (or with an additional lint) until the schema adds minItems.

Source difference: staleness. M3 and UC-deploy-2 refer to a cab_decision_max_age “on the manifest”, but schema v0.1.0 has no such field and forbids additional properties. Configure the staleness limit in the gateway policy, not in the manifest, until a schema version adds it. M3 lists backend-specific staleness as an open question.

Validity is necessary, not sufficient. A non-empty verify.health_checks can still be trivial: a single http probe on a load balancer returning 200 passes while the application is broken. Review health checks at the Virtual CAB the way model risk reviews prompts; the check is the control (Book 2 Ch 9).

Field reference

Required top-level keys: schema_version, manifest_id, artefact, target, backend, preflight, apply, verify, rollback, metadata. Optional: promotion. additionalProperties is false throughout.

SectionFields (required in bold)Notes
artefactname, version, kind, source, digest.sha256 (64 hex) / digest.sha512 (128 hex)kind ∈ container_image, helm_chart, terraform_module, bicep_module, function_app, static_files, custom. Recommendation: always record a digest so the deployed object is provably the tested object.
targetenvironment, systems (≥1), regions
backendtype, config, config_schema_ref (URI)type ∈ helm, kubernetes, terraform, opentofu, bicep, azure_cli, aws_cli, gcp_cli, docker, ssh, custom. config is untyped at schema level.
preflightchecks, required_chk_guidance (array of guidance ids), required_cab_decision (string or null), cab_exemption_rationaleConditional: null decision ⇒ rationale required, minLength: 1.
applyintent, program, args, timeout (ISO 8601 duration), env
verifywait_window (ISO 8601), health_checks (≥1)
rollbackintent, program, args, timeout, verification, automatic_triggersTriggers: condition_kind is metric only in v0.1.
promotionfrom_environment, to_environment, approval_required (boolean)
metadataowner_team, agent_wrangler_upn, tags

The shared check type

One check definition is used in three places: preflight.checks, verify.health_checks and rollback.verification. It is a discriminated union on kind (D19.4):

kindRequired fieldsOptional
httpid, description, kind, url (URI), expected_status (100–599), timeoutexpected_body_fragment, required (default true)
commandid, description, kind, program, expected_exit_code, timeoutargs, required
metricid, description, kind, query_kind (kql, promql, other), query, expected_condition, evaluation_windowrequired

An automatic rollback trigger (automaticTriggerMetric) needs id, description, condition_kind: "metric", query_kind, query and evaluation_window. It has no threshold field: the breach condition must be expressed in the query itself (the published example filters DriftPct > 5). The M4 orchestrator’s registration payload, by contrast, carries an explicit threshold (op, value) plus evaluate_window_seconds and evaluate_after_apply_seconds. Whoever registers the trigger with M4 has to translate between the two; record how in the platform’s runbook.

A minimal valid skeleton

Keys are from schema v0.1.0; values are illustrative, modelled on the published cs_v2.4.0 example (a composite bank).

{
  "schema_version": "0.1.0",
  "manifest_id": "depl-<date>-<product>-<version>",
  "artefact": { "name": "<product>", "version": "2.4.0", "kind": "container_image",
                "digest": { "sha256": "<64 hex>" } },
  "target":   { "environment": "production-eu-west", "systems": ["<agent or service id>"] },
  "backend":  { "type": "helm", "config": { "release": "<name>", "namespace": "<ns>" } },
  "preflight": {
    "checks": [ { "id": "rollback-loaded", "description": "Prior release present and ready",
                  "kind": "command", "program": "helm", "args": ["history", "<name>"],
                  "expected_exit_code": 0, "timeout": "PT2M" } ],
    "required_chk_guidance": ["<guidance id>"],
    "required_cab_decision": "<change_id>"
  },
  "apply":  { "intent": "deploy <product> 2.4.0", "program": "helm", "args": ["upgrade", "--install", "..."], "timeout": "PT30M" },
  "verify": { "wait_window": "PT10M", "health_checks": [
    { "id": "business-metric", "description": "Change-specific health signal within baseline band",
      "kind": "metric", "query_kind": "kql", "query": "<query>", "expected_condition": "<condition>",
      "evaluation_window": "PT1H" } ] },
  "rollback": { "intent": "roll back to 2.3.1", "program": "helm", "args": ["rollback", "<name>"], "timeout": "PT15M",
    "verification": [ { "id": "post-rollback", "description": "Prior behaviour restored",
      "kind": "http", "url": "https://<host>/healthz", "expected_status": 200, "timeout": "PT2M" } ],
    "automatic_triggers": [ { "id": "metric-breach", "description": "Breach of agreed threshold",
      "condition_kind": "metric", "query_kind": "kql", "query": "<query that returns rows only on breach>",
      "evaluation_window": "PT1H" } ] },
  "metadata": { "owner_team": "<team>", "agent_wrangler_upn": "<named human UPN>" }
}

Change-specific health. Choose the health metric for the change, not a generic error rate. In Book 2’s composite (Asset Manager X, reconciliations squad), a matching-rule change was gated on the fund-accounting match rate staying within two points of its 90-day baseline through a 1% canary; the rollback verification required confirmation that flag-off restored the prior match rate. The canary breached, the path drained the 1% and turned the flag off, and the record was attached to the change for the wrangler to decide on fix-forward the next morning (Book 2 Ch 13 §1, §6; composite example).

Deploy lifecycle: refusals and failure outcomes

Every phase emits an AES record, including no-op phases (UC-deploy conventions). The phases, records and exit codes for ops deploy (UC-deploy-1 to 5):

PhaseRecordsRefuse or fail whenWhat happens next
Plandeploy.planManifest invalid (exit 3); backend unsupported (exit 2); any invariant violated (refusal)Nothing applies
Preflightdeploy.preflight; deploy.preflight.exempt warning when a null decision uses an exemptionCAB decision missing, no-go, unfulfilled condition, stale; any preflight check fails (exit 4)Nothing applies
Applydeploy.apply.start, deploy.apply.step (if backend reports), deploy.apply.endOutcome partialAutomatic rollback per manifest
Outcome unknown (e.g. network failure mid-apply)No automatic retry. P1 alert; a human decides
Verifydeploy.verify per checkAny check fails; a metric check whose query is unreachable is unknown and treated as fail (fail-closed)Automatic rollback
Rollbackdeploy.rollback.start, deploy.rollback.verify per check, deploy.rollback.end (succeeded / partial / failed)Rollback verification failsOutcome rolled-back-with-issues; P1 alert; no automatic retry; manual review

Why no retry on an unknown outcome: the system does not know whether the change landed. Retrying an apply of unknown state, or re-running a rollback that may have half-completed, can move the system further from any known-good state. A human with the deploy chain (GET /v0/deploys/{deploy_id} or by change_id) decides.

Acceptance checks for the platform team.

  • A manifest lacking any of the four invariants is refused at plan, with the specific refusal code recorded as m3.refusal.
  • An unknown apply raises a P1 and never retries.
  • Every deployment can be shown end-to-end by change_id (UC-deploy-6), and the read is itself audited (deploy.show).

A deployment gateway that cannot be bypassed

Client-side tooling enforces nothing on its own: a team can recompile or rename the binary. The Supporting Tools design therefore places a Deployer Backend Gateway (M3) between every client and every backend (D23, status Proposed; M3 API Draft 1, May 2026; a design, not a released service). The client is convenience; the gateway is the authoritative enforcement point. Any team writing its own client still has to call the gateway to deploy, and the API management layer in front refuses unsigned client requests.

Design properties to require of whatever you build or buy:

PropertySpecification (M3 / D23)
PlacementEdge tier, per entity and per jurisdiction (D23.1)
DefaultRefusal for missing fields. No permissive mode (D23.2)
Correctness target100% refusal correctness, continuous: every invalid manifest refused, no valid one refused. Verified by the conformance harness on every release
Conformance vectors (minimum)Empty verify.health_checks; empty rollback.verification; null CAB decision without rationale; missing agent_wrangler_upn; CAB decision not in the lake; http check missing url; backend.type outside enum
Dependency failureBackend unreachable → refuse with m3.backend_unavailable, no silent fallback. Evidence lake unreachable at preflight → refuse with m3.lake_unreachable; no apply without a confirmed CAB decision
Rolesdeploy.invoker.team, deploy.invoker.agent, deploy.observer, and deploy.emergency (time-bounded, four-eyes: two authenticated principals)
Service levelsManifest validation p99 ≤ 250 ms; apply availability ≥ 99.9% (30 days); verify-fail to rollback-start p95 ≤ 30 s

Open question in the source. M3 asks whether deploy.emergency should allow an emergency deploy with a null CAB decision and an exemption rationale filed within 24 hours. The draft recommends yes, using the CAB template-rule mechanism. Until your governance ratifies that, an emergency deploy still needs a cited exemption at deploy time. See change governance for the emergency-change path.

Run the gateway’s conformance suite with the conformance harness and keep the badge current.

Automatic rollback, at most once

Health gates only protect production if a breach triggers reversal without a person watching a graph. The Supporting Tools design gives this to an Auto-Rollback Orchestrator (M4; D24, status Proposed; API Draft 1).

Registration. At apply time the gateway registers the manifest’s rollback.automatic_triggers with M4, scoped to that deployment_id (default watch lifetime 24 h, lifetime_seconds: 86400), recorded as rollback.signal.registered. No triggers, no registration: the manifest still applies, but rollback is manual only (D24.1). Triggers are unregistered once verify passes.

Evaluation. A worker evaluates each active trigger every 60 s against the evidence lake (preferred) or, optionally, a Microsoft Sentinel feed for telemetry that is not in AES records; the fallback is configured per trigger (D24.3, M4 §4).

At-most-once. A double rollback is worse than a missed one: reverting past the target state can leave a system unrecoverable. The rules (M4 §6):

  1. M4 generates a rollback_id (UUIDv7) when it decides to fire.
  2. It persists (rollback_id, deployment_id, signalled_at) before sending.
  3. Failed delivery is retried with the same rollback_id until acknowledged or the retry budget runs out; then a P1 alert (rollback.signal.unacknowledged) and watching stops.
  4. The receiving gateway also deduplicates by rollback_id within 24 h; a replay returns the first response and emits m3.rollback.signal.duplicate.
  5. The leaf emits its own deploy.rollback.* records; the orchestrator does not emit on its behalf (D24.2).

Never fire without a measurement. If the lake is unreachable, skip the cycle and emit rollback.eval.degraded. On a time-skew anomaly in the measurement, refuse to fire (rollback.eval.skew_anomaly). If the Sentinel feed is down, fall back to lake-only and mark the trigger degraded.

Manual emergency signal. A human can force a signal through the emergency role with four-eyes approval, recorded as rollback.signal.manual.

Calibration and exit criterion. Targets: end-to-end rollback (signal to deploy.rollback.end) p99 ≤ 4 min; at-most-once correctness 100%; false-positive rate ≤ 1 per 1,000 deployments over 60 days. More than one unwanted rollback per 1,000 deployments by day 60 is a pilot exit signal. Threshold calibration is a monthly responsibility of the Agent Control Board (D24.4). M4 recommends against threshold autotuning in v0.1 so thresholds stay explicit and auditable, and recommends per-region evaluation so one bad region does not roll back all regions (open questions).

Progressive delivery and staged rollout

Never expose all of production at once. The health check at each stage is a hard automated gate. A “green” pipeline proves the change is well-formed, not that it is safe on real traffic, so no green build may bypass the first rollout stage (Book 2 Ch 13 §4, §8).

PatternUse whenReversal
Blue-greenYou can run two production environmentsShift traffic back to the warm old environment
CanaryReal traffic can be sliced; widen only while health holdsDrain the slice
RollingInstances replace incrementallyStop and replace back
RingsInner ring of internal users, then widening production cohorts, each a gateHalt at the ring
Feature flagDeploy dark, release to cohort/account/nobodyFlag-off

Managed equivalents (as of November 2026, per Book 2 Ch 13 §4). AWS CodeDeploy supports blue-green and canary for ECS and Lambda with rollback wired to CloudWatch alarms, and AWS AppConfig carries flags with deployment strategies and a rollback trigger. GCP Cloud Deploy provides delivery pipelines, canary strategies and promote/rollback across targets. On Azure, Book 2 uses Azure DevOps or GitHub Actions, deployment rings, Azure Deployment Environments and Azure App Configuration for flags. Book 2’s judgement: CodeDeploy is the most mature at alarm-driven rollback, Cloud Deploy has the cleanest promotion model, and Azure integrates most tightly with the identity and evidence planes. Choose for where your evidence must land. [FACT-CHECK: current CodeDeploy, AppConfig, Cloud Deploy and Azure progressive-delivery capabilities against vendor documentation.] Smaller firms should use these managed features and wire automatic rollback to an alarm on the one metric that shows harm, rather than building the path from first principles. The discipline is the same as a large firm’s, with fewer stages and fewer critical functions in scope.

Staged rollout per envelope for model changes

For a model upgrade, Book 3 sets a slower, per-envelope schedule enforced at the model gateway (Book 3 Ch 16 §4.2, §7, OD-1; Bank A is a composite):

WeekTraffic on new model, per affected envelopeCheck
15%Wrangler reviews day-end evidence sample on Day +7
225%Wrangler review on Day +14, compared with a cross-section of the prior week
350%Status to Virtual CAB at the weekly slot; any envelope outside thresholds paused here
4100%Registry pin date updated; reassessment set to Day +90

Enforcement rules: the rollout is per envelope, and two envelopes sharing a model never share a schedule unless the Virtual CAB approves the synchronisation in writing. The gateway’s traffic-split policies are versioned and signed. The gateway refuses to advance the split without a new policy artefact, and the AI Platform Owner co-signs each weekly step. That closes the “5% Monday, 100% Tuesday” anti-pattern, where staging becomes theatre and a tail-of-distribution regression never has time to show.

Layering with Book 2. Book 2 Ch 13’s canary (e.g. 1% of batch traffic, gated within minutes) is the general mechanism for code, configuration and the pinned AI release unit. Book 3’s four-week 5/25/50/100% schedule is the policy for a provider model upgrade in a governed envelope, where outputs are probabilistic and regressions surface slowly. Use both: the fast canary catches gross breakage; the staged schedule catches drift. The full Day −14 to Day +90 upgrade lifecycle, owners and attestations are on model governance.

When the change is a model

For an AI product the unit of release is a pinned combination: model and version, parameters, prompt, orchestration logic, retrieval configuration and index, and tool versions. Changing any one is a release, recorded in the release manifest (an extension of the BOM; see BOM and recipes) with the evaluation results the combination earned (Book 2 Ch 13 §4).

Release rules (Book 2 Ch 13 §4; recommendations unless marked).

  • Prompts and model configuration live in an external, versioned store, never in application code, and always deploy with the model version they were evaluated on.
  • A router splits traffic between two complete combinations; the model gateway honours each pin so no user meets an untested pairing.
  • For a model release, health is groundedness against golden records, refusal rate, latency and cost per call. A model regression rarely shows as errors; it shows as fluent, worse answers.
  • Author’s absolute rule: automatic model upgrades are disabled in production, minor versions included. Book 3 Ch 16 §3 goes further: auto-upgrade (a family alias resolving to the latest version without an envelope amendment) is a policy violation; the route is configured to refuse it, reconciliation detects it and the wrangler is paged.
  • You can only roll back to a version the provider still serves. Track deprecation and retirement dates and open re-evaluation ahead of them.

Model update scopes and the evaluation each needs

Book 2 condenses Microsoft’s eight life-cycle update scopes into six (Book 2 Ch 13 §4). Classify every model change as one of these, the same way the recipe classifies any change type:

ScopeExampleMinimum evaluation before release
Minor versionNew dated release of the same modelRegression evaluation; prompt check
VariantFaster or cheaper variant in the familyRegression evaluation; latency and cost check
Major / generationalNew model generationFull evaluation; prompt rework; possibly orchestration change
Model changeDifferent vendor or familyFull evaluation; prompt and tool-calling rework; vendor onboarding
Specialised modelDomain-specific modelRe-assess grounding needs; full evaluation
Hosting route changeVendor API to private tenancy or controlled infrastructureInfrastructure and security review; new residency assessment

Book 3 Ch 16 §4.6 adds that a lateral switch between model classes runs the same lifecycle and is harder, because deterministic test cases transfer less well across families than across versions.

Pinning on each route (vendor-specific, dated)

  • Azure OpenAI / Foundry (as of November 2026, per Book 2 Ch 13 fn). Standard deployments carry versionUpgradeOption: OnceNewDefaultVersionAvailable, OnceCurrentVersionExpired or NoAutoUpgrade. An unset value behaves as OnceCurrentVersionExpired, so an unconfigured deployment still upgrades itself at retirement. Set NoAutoUpgrade explicitly on every production deployment and check it in preflight. [FACT-CHECK: versionUpgradeOption values and default behaviour against current Microsoft Learn.]
  • Amazon Bedrock (as of November 2026, per Book 2 Ch 13 fn). Model identifiers carry the version, so calls are pinned by construction; end-of-life is a hard date you must track, after which requests fail and migration is not automatic. [FACT-CHECK: Bedrock model lifecycle states and EOL behaviour.]
  • Google Vertex AI / Gemini Enterprise Agent Platform (as of November 2026, per Book 2 Ch 13 fn). Each model ID is published with release and retirement dates; name the exact ID in production. [FACT-CHECK: current Vertex model-versions page.]
  • Claude across routes (as of August 2026, per Book 3 Ch 16 §4.5). Family aliases (opus, sonnet, haiku, fable) resolve to per-provider defaults that do not move together, so one alias can mean two models in two regions. Pin full model IDs (e.g. claude-sonnet-4-6) in every envelope’s Means field. For Claude Code fleets, pin per environment with ANTHROPIC_DEFAULT_OPUS_MODEL, ANTHROPIC_DEFAULT_SONNET_MODEL and ANTHROPIC_DEFAULT_HAIKU_MODEL in managed settings, constrain with availableModels, and set enforceAvailableModels: true. availableModels is enforced client-side and third-party-provider fleets do not receive server-managed settings, so on those routes the gateway is the control that holds. Book 3 also notes that Anthropic published no consolidated deprecation schedule as of August 2026; Bank A (composite) treats every pin as expiring at Day +90 regardless. [FACT-CHECK: Claude Code model-configuration settings, alias defaults per provider, and Anthropic deprecation publication against docs.claude.com.]

Rollback triggers and evidence portability for model releases

Book 3 names three rollback thresholds, agreed in the change dossier before rollout (Book 3 Ch 16 OD-2): cost per task more than 30% above the prior-model baseline over a rolling 14 days; deterministic regression above 5% at the Day +60 re-run, or one material business-outcome scenario failing; per-envelope incident rate up more than 100% over a rolling 30 days. Reverting the split takes effect within 30 minutes at 5% or 25%, and within one working hour at 50% or 100%.

A model rollback is complete only when the evidence-portability check passes: a replay over the cohort’s correlation IDs shows every audit record’s model-ID field matches the model that served the call, and the boundary records reference the rollback change record. Without it, the “who did what on whose authority” reconstruction breaks at exactly the moment a supervisor will look (Book 3 Ch 16 §5.2). The full six-step playbook and its artefacts are on model governance; evaluation-suite design is on evaluation and resilience.

The failure-mode register per AI product

The release manifest says what was pinned and what it scored. It does not say how the product fails. The failure-mode register does: a version-controlled file in the product repository beside intent.md and spec.md, listing failure modes and scoring each for severity, occurrence and detection (FMEA). It is a file in Git; no vendor tool is needed (Book 2 Ch 13 §4).

Starter failure modes for an AI product (source list): fabricated answer presented as fact; prompt injection via retrieved content or tool output; data crossing a classification boundary into a context that should not hold it; drift after a model or index change; upstream model or tool unavailable; agent acting outside its mandate.

Suggested row layout (this page’s proposal; the source names the scores and owner but not column keys):

ColumnContent
modeFailure mode in one line
ownerNamed person; a line with no nameable owner is deleted
severityOn the scale agreed once with the second line, so 9 means the same in every product
occurrenceEstimated frequency
detectionLikelihood the factory catches it before a client does
action_priorityPriority from an action-priority table, not a multiplied RPN
controlsEvaluation gate, output filter, drift monitor, human checkpoint that move the detection score
manifest_refRelease manifest version this row was last scored against

Rules (Book 2 Ch 13 §4, §8).

  • Re-score whenever the release manifest changes. A new model, prompt, index or tool version changes both what can fail and what would catch it. Make the re-scored register a required, dated artefact of any release that changes a pinned element.
  • Rank by action priority, never by multiplying the three numbers. Multiplication lets a rare, hard-to-detect severity-10 mode rank below a weekly nuisance. The AIAG & VDA FMEA Handbook (2019) replaced RPN with action priority. In AI products the severity-10 lines tend to be the quiet ones: a wrong number that leaves the building, a classified record in a shared index.
  • Use detection as the argument for a control. A mode the factory cannot detect scores badly however rare it seems; that number is the case for adding an evaluation gate or a checkpoint. It gives the product team and the second line one shared currency.
  • Keep it light. The change dossier references the register by evidence pointer; no new dossier field and no change to the evidence envelope. Scoring facilitation sits inside existing roles; no new FMEA coordinator.

Stale register test. The register’s last commit predates the last three release-manifest entries, or every detection score is still the workshop number. A stale register is worse than none because the second line reads it as a live control. Recovery: tie the re-score to the manifest and delete ownerless lines.

What it does not do. The register is an input to the ICT risk-management framework, to a data-protection impact assessment where one is required, and to the AI Act risk-management system for high-risk systems. It satisfies none of them on its own (Book 2 Ch 13 §4, flagged by the author for legal review). Articles and application dates are on the regulatory map.

Shared capabilities: a concentration you chose

A shared capability consumed by many products, including the model gateway itself, trades many small uncorrelated failures for one large correlated one. That trade is usually right but must be paid for (Book 2 Ch 13 §4):

  • Hyrum’s Law. With enough consumers, every observable behaviour becomes a dependency, and agents create consumers faster than people did. Contract tests become the only version boundary that holds.
  • Release treatment. Every change to a shared capability routes as high blast radius at the merge gate by construction; the Change Risk Index weights consumer count the way it weights a data migration. Roll out against the health of consumers, not only the service’s own health. In Book 09 Ch 7’s composite, any service with more than a handful of callers rolled out against its callers’ health.
  • Reversal tested against the consumer estate. A rollback that is clean for the service and breaks forty callers is not clean.
  • Concentration entry (author’s practice, marked as supervisory expectation, not obligation). For shared capabilities supporting a critical or important function, hold an internal concentration analysis: what depends on it, blast radius, resilience, how to operate through its failure. This is an internal analogue, not the DORA third-party register; keep that distinction intact (see regulatory map).

Split agentic and privileged CI pipelines

When a coding agent runs non-interactively on a build agent, it becomes an unattended actor with pipeline privileges. The governing threat is a pull request whose content steers an agent that holds a deployment credential (Research Paper 3 §8).

Pipeline rules (Research Paper 3 §8.2, as of November 2026):

  1. One job, one disposable instance. No long-lived build agents running the coding agent. Prefer self-hosted scale-set agents that reimage after each job, with container jobs for the strongest isolation; restrict egress on the subnet, because network restriction is not a property of the container job.
  2. Policy in the image, not the repository. The CI profile is a managed-settings.json in the image’s system directory, setting allowManagedPermissionRulesOnly (pipeline --allowedTools and repository rules are ignored), allowManagedHooksOnly (repository hooks do not run), an empty managed-mcp.json (repository MCP servers do not load), and allowedProviders restricted to the pipeline’s provider. Changes to pipeline definitions and .claude/ need reviewer approval before any build runs them.
  3. Split the pipeline. No deployment-capable credential in any job that runs an agentic step on untrusted input. The unprivileged agentic stage (review, triage, proposed change) produces artefacts; a privileged stage with no agent in it consumes them behind a human or deterministic gate. Treat the artefact as attacker-influenced text.
  4. Model credentials from the secret store, preferring workload identity over API keys where the route supports it.
  5. Human review for effects. Code changes arrive as a pull request, enforced by branch policy.
  6. Telemetry is mandatory, with pipeline and project attributes.
  7. No bypass mode needed. dontAsk with managed allow rules gives a fixed envelope.

Why policy cannot live in the repository (as of November 2026, per Research Paper 3 §8). A claude -p session runs the hooks in a project’s .claude/settings.json and connects servers in .mcp.json without a trust dialog, so a pull request that adds a hook asks the pipeline to run a command. --allowedTools only pre-approves tools; restricting the set requires --tools, which does not cover MCP tools (the paper’s example adds --disallowedTools "mcp__*"). From version 2.1.302 a run with no mode set starts in Manual mode, so a call that needs approval is denied; dontAsk makes that denial explicit. [FACT-CHECK: claude -p hook and MCP behaviour, --tools / --allowedTools semantics, default permission mode from version 2.1.302, and the managed-settings keys above, against current Claude Code documentation.] The deployment-credential boundary for build steps is also on landing zone.

Public case: the Amazon Q supply-chain incident (July 2025)

Book 1 Ch 8 §6 reads the public Amazon Q prompt-poisoning incident (OWASP Agentic Top 10 Appendix D) as a release-path failure. A merged pull request inserted a destructive instruction into the assistant’s prompt-construction code, and the release was published to the marketplace. AWS stated the payload did not execute as intended, but the merge and the release had both happened. Book 1 identifies four independent gates that would each have stopped it. Read them as release-engineering requirements:

GateRelease-path requirement (Book 1)Acceptance check (this page’s suggestion)
Release-branch mergeSeparate, elevated deployment identity for release merges, a named approver, a change reference, and a content check that runs on the merge bundle, not on the original PRGate log shows the check ran against the merged tree hash
PublishA separate publish identity with its own approval and a source-to-binary equivalence check between merged source and published artefactArtefact digest in the manifest matches a reproducible build of the tagged source
Developer runtime profileThe agent’s local profile holds no production-touching role by default; production access only through just-in-time elevation the agent cannot triggerAccess review of developer profiles
Workstation egressDeny-by-default egress with allow-listed endpointsEgress policy evidence

The first two belong on this page. The last two belong to identity and secrets and the landing zone, and to managed settings for coding agents on agent-speed SDLC.

Branches, tags and thread hygiene

Book 2 carries forward the author’s 2024 “Release Flow” model (Book 2 Ch 13 §4): an unstable trunk kept deployable but not sacred; short-lived feature branches; long-lived release branches named by major and minor, with patch left open; fix branches cut from the release branch; tags marking what shipped. Use semantic versioning so the version string signals blast radius (MAJOR breaking, MINOR compatible feature, PATCH fix).

  • “Main equals production” is a fallacy. A branch is a place to work, not a record of what shipped. Find what is running from tags and the deployed manifest’s artefact digest, never from a branch head.
  • Thread hygiene. An agent thread may live two hours and produce a branch, migration and tests. Isolate every thread on its own branch and workspace. When steering a drifted thread costs more than restarting, delete it and re-dispatch from an adjusted spec.
  • The branching model is agent input. Write it into the standing context agents read before working, or the agent will invent its own conventions and the release path will inherit them.

Release identity and pinning

This section is this page’s synthesis of the pinning and provenance rules above (Book 2 Ch 13 §4), stated as recommendations. Before promotion, record source revision, dependency and model/tool versions, build environment, test and evaluation results, approvals and the Deployer Manifest. The deployed object must be the object that passed the checks; rebuilding or changing configuration after approval breaks that link. Pin everything that can change runtime behaviour: application artefacts, infrastructure definitions, prompt and skill bundles, agent settings, model routes, tool definitions and policy packs (pack freshness rules are on policy, audit and actuals). Identify which values may change at run time and capture those changes as evidence. Detecting drift between declared and running configuration is on identity and secrets.

Keep one process for all change: human- and agent-authored change travel the same path (Book 2 Ch 13 §5). Do not create a separate, permanent “AI code” process. Record how the work was produced, then apply controls to the artefact and its risk. Recommendation: every agent-generated change also has a committed plan, a review record, and checks the agent cannot alter to make its own result pass. One named human is accountable per merge, and that name is in the merge record whatever agents reviewed first; agentic code review supports the reviewer but does not discharge four-eyes (Book 3 Ch 13 §4.3). The review lanes are on agent-speed SDLC; see also change governance and accountability.

Anti-patterns and recovery

Anti-patternRecognise itRecover
Green-build complacency”The pipeline is green” is used to skip the canaryMandatory first-stage gate no green build can bypass
Bundling under cheap generationOne change touches more than one change typeClassify type in the recipe; split onto separate paths
Untested rollbackManifest rollback.verification empty or waivedGateway refuses empty verification; rehearse reversal on a schedule
Trivial health checkOnly an http 200 on a load balancerReview checks at the Virtual CAB; require a change-specific metric
Manual release heroicsThe runbook is a person’s nameMove coordination into automated gates, rollback and evidence
Gate that inspects instead of routesReview latency rising, approvals getting shorterRoute on the Change Risk Index; let bounded agency carry low-risk change
Model that upgraded itselfQuality shifts on a date with no release; nobody knows which version served a tracePin the version; disable auto-upgrade; track retirement
Staged rollout collapsed to instant5% one day, 100% the nextGateway refuses to advance without a signed policy artefact
Stale failure-mode registerLast commit predates the last three manifest entriesTie re-score to manifest changes; delete ownerless lines
Retry on unknown outcomeAutomation re-applies after a mid-apply network failureunknown raises a P1 and stops; a human decides

Close the loop

Connect release approval and deployment records to the change identifier and the evidence the system generated. A successful pipeline proves the configured checks ran; it does not prove the design was right or the outcome safe. Feed monitoring and incident records back into evaluation cases, failure-mode scores and, where needed, new intent and controls. See evaluation and resilience.

Source status. This page synthesises the cited books and the Supporting Tools specification. The Deployer Manifest schema v0.1.0 is published; the M3 gateway and M4 orchestrator are proposed designs (Draft 1, May 2026) with open questions, not running services. Thresholds named here (5/25/50/100%, 30% cost, 5% regression, 100% incident rate, ≤1/1,000 false positives) are the sources’ proposals and need workload-specific calibration. Vendor product claims carry [FACT-CHECK] markers and dates.

Source material

  • Book 1, Chapter 8, Identity and Network: The First Two Control Planes (§6, Amazon Q worked example)
  • Book 1, Chapter 17, Change Control in the Age of AI
  • Book 2, Chapter 9, The Audit Spine (§4.3, the Deployer Manifest)
  • Book 2, Chapter 11, Identity, Secrets and Configuration (configuration and feature flags as change)
  • Book 2, Chapter 13, Release Engineering and Change Resilience
  • Book 2, Chapter 15, The SDLC at Agent Speed
  • Book 3, Chapter 13, Wranglers, Product Teams and RACI (§4.3, one accountable human per merge)
  • Book 3, Chapter 16, Model Upgrades As Change Events (incl. Operating Discipline OD-1, OD-2)
  • Book 09, Chapter 7, The Machine That Says No (shared services rolled out against callers' health)
  • Supporting Tools, Deployer Manifest schema v0.1.0 and README; decisions log D19, D23, D24
  • Supporting Tools, API drafts M3 Deployer Backend Gateway and M4 Auto-Rollback Orchestrator (Draft 1, May 2026)
  • Supporting Tools, use cases UC-deploy-1 to UC-deploy-7 (ops deploy)
  • Research Paper 3, Securing Claude Code in the Enterprise, §8 CI/CD

← All AI Delivery Factory guidance

On this page

  1. The three answers every change carries
  2. The release path: five stations
    1. The readiness card
  3. One change type per change
  4. Decide the reversal at authoring time
  5. The Deployer Manifest: what makes a deployment valid
    1. Validity rules
    2. Field reference
    3. The shared check type
    4. A minimal valid skeleton
  6. Deploy lifecycle: refusals and failure outcomes
  7. A deployment gateway that cannot be bypassed
  8. Automatic rollback, at most once
  9. Progressive delivery and staged rollout
    1. Staged rollout per envelope for model changes
  10. When the change is a model
    1. Model update scopes and the evaluation each needs
    2. Pinning on each route (vendor-specific, dated)
    3. Rollback triggers and evidence portability for model releases
  11. The failure-mode register per AI product
  12. Shared capabilities: a concentration you chose
  13. Split agentic and privileged CI pipelines
  14. Public case: the Amazon Q supply-chain incident (July 2025)
  15. Branches, tags and thread hygiene
  16. Release identity and pinning
  17. Anti-patterns and recovery
  18. Close the loop

All pages in Deliver changes →