Browse the reference ⌄
Implementation Guide published

Design and Operate the Model Gateway

Build the model gateway as the governed edge for every model call: per-client identity, per-request highest-class routing, tokenisation, pinned models, a conformant wire contract, a correlated evidence record, a tested concentration answer, and day-two cost, latency and reliability operations across two estates and three meters.

Version
0.2.0
Owner
DEIXEI
Last reviewed
Reading time
38 min

For Cto · Ai Architect · Platform Engineer · Security Engineer · Finops · Risk And Assurance · Internal Audit

Sources and details
Source works
Book 1 · Book 2 · Book 3 · Research Series · Supporting Tools
Source chapters
5, 6, 9, 10, 11, 13, 14, 18, 23, 25
Source appendices
04:D, 04:E, 04:J
Tool specifications
audit_evidence_schema@0.1.0, m7_gateway_emit
Refresh trigger
vendor release

Full source list ↓ Source index →

A model gateway is the governed path between every consumer of a model and every model route the institution uses. It authenticates the caller, decides which route the request may take, applies data-handling controls on the way out and the way back, enforces budgets, calls the institution’s policy decision service when an action needs one, and writes the network-edge evidence record. Book 3 calls it the place “where the audit evidence is born”; Book 1 Ch 13 treats it as one of three sub-planes of a single API boundary (identity-aware gateway, MCP broker, tokenisation gate). This page follows both: the gateway is one enforcement point with several jobs, and it is never the whole control plane.

The gateway is the enforcement point; the egress allowlist is what makes it the only path to the model.

At a glance: the gateway is the enforcement point, and the egress allowlist makes it the only path to the model. The detailed diagram is under Define the gateway contract. Source: Deploying Claude Safely (Book 3), Chapter 6, Figure 6.1.

Decisions this page supports

DecisionOwnerWhere on this page
What the gateway must enforce, independent of productPlatform architect with second lineThe gateway contract
How callers authenticateIdentity engineer, platformPer-client identity
Which route a request may takeData office with platformPer-request routing
Whether a field leaves in plain form, tokenised or not at allWrangler with privacy officeTokenisation
Which product implements the gatewayHead of platformConfigure or procure
How the gateway fails and recoversPlatform on-call, resilienceResilience and concentration
How cost, latency and reliability are run day to dayGateway product owner, FinOpsGateway operations

Define the gateway contract

Write the contract in terms of enforcement, observable behaviour and bypass resistance before looking at a product. Book 3 Ch 6 §4.1 names six jobs; the table below states each as a control.

Applications and coding agents reach multiple model routes through a gateway that applies identity, policy, data handling and evidence controls.

The gateway is the enforcement point; the egress allowlist is what makes it the only one. Source: Deploying Claude Safely (Book 3), Chapter 6, Figure 6.1.

JobControl the gateway must enforceFailure if missing
Caller identityEvery caller authenticates as itself (user token, managed or federated workload identity, or agent principal). Anonymous calls are rejected.Shared key: no per-caller scoping, billing or revocation.
RoutingRoute chosen from the request’s computed data class and the caller’s validated use-case class; explicit model identifiers per backend.A caller self-promotes to a broader route; an alias decides what model runs and what it costs.
Data handlingClassification on both the inbound prompt and the outbound response; block, tokenise or pass per policy.Response-only DLP misses prompt leaks; prompt-only DLP misses model leakage.
Budget controlPer-tenant and per-envelope caps with staged alerts and a hard stop.The first signal of a run-on is the invoice.
Policy checksCall the institution’s Policy Decision API for consequential actions and enforce its answer.Approvals live in a wiki, not on the wire.
EvidenceOne structured record per request with a gateway-issued correlation ID, forwarded to the SIEM and immutable store.An episode cannot be reconstructed across gateway, workstation and SIEM.

What the gateway is not, per Book 3 Ch 6 §4.2:

  • Not a model. It enforces deterministic policy and calls services (classifier, tokeniser, policy decision API) that are themselves versioned and auditable. A gateway that reasons about content has acquired a second, ungoverned model.
  • Not the sole evidence source. Workstation hooks and SDK hooks capture per-action events the gateway never sees (for example, agents calling internal MCP servers directly). The three streams join on the correlation ID; see Audit evidence.
  • Not the owner of policy. The wrangler, the agent control board and the change board decide; the gateway turns the bounded-agency envelope into model permissions, label thresholds and budgets.
  • Not the perimeter. It mediates only traffic that reaches it. Network egress policy makes it the only path (see Close bypass paths).

Two further limits are this page’s additions, drawn from Research Paper 9 rather than Book 3:

  • Not a guarantee about provider retention or residency. Those are properties of the contract and the exact service configuration, recorded per lane on Data classes and lanes.
  • Not an entitlement service. Retrieved data must be filtered by the caller’s rights at the data service; the gateway cannot fix an over-broad index.

Specify guardrail capabilities, not products

Runtime guardrails differ by vendor in what they actually do. Book 1 Ch 10 lists the categories that recur across platforms: content filters (including prompt-injection), denied topics, PII redaction, output validation against a schema and, on some platforms, formal-logic verification of factual output. Its anti-pattern 3 is treating two vendors’ policy engines as interchangeable because both are called “guardrails”.

  1. Write the required capability list into the gateway contract (and into any exit plan): which categories, on which direction, at what threshold, with what failure behaviour.
  2. Map each candidate implementation (Book 1 Ch 10’s examples: Copilot Studio guardrails with Purview DLP on Microsoft; Bedrock Guardrails with AgentCore Policy, applied model-agnostically through the ApplyGuardrail API, on AWS; semantic governance policies on Google Cloud) against the list, capability by capability.
  3. Re-run the map before switching engine or cloud. A swap that drops a capability is a control change, not a procurement detail.

Dated vendor note (Book 1 Ch 10 §4.3, as of May 2026): Bedrock Guardrails offered Automated Reasoning checks that the book found had no Microsoft equivalent at the same maturity. [FACT-CHECK: current guardrail capability parity across Azure, AWS and Google Cloud.]

Give every caller its own identity

The quickest way to stand up a gateway is one provider key in its configuration and one subscription key handed to every caller. Book 2 Ch 11 treats that as a shared identity in all but name: the gateway can no longer scope, bill or revoke one caller without touching the others.

Rules (Book 2 Ch 11 §4, §8; Book 3 Ch 5 PR-3 and Ch 6 §5.3):

  1. Per-client identity is the entry condition. Every application, orchestrator, agent and developer authenticates to the gateway as itself: a delegated user token, a managed or federated workload identity, or an agent principal. Benefits: least privilege per caller, showback per team or pod, and revocation with a blast radius of one.
  2. The gateway is the caller of last resort. Consumers authenticate to the gateway, never directly to the provider. The gateway authenticates onward with its own identity (managed identity to Foundry, federated or vaulted credential to the direct API, IAM role to Bedrock).
  3. Vendor-only keys are debt, not pattern. Where a provider offers nothing but a key, it lives in the vault tree, readable only by the gateway’s identity, retrieved by a separate policy so it never appears in request-handling code or logs. Track each one on the static-key sunset register (see Identity and secrets).
  4. Validate claims, do not trust headers. Any header that drives routing (Book 3 uses X-Use-Case-Class and X-Complexity) is checked against the caller’s token scope. A Class 1 tenant cannot self-promote to a Class 3 route or to the most expensive model by editing a header.
  5. Refuse on tier or identity mismatch. Book 3 Ch 14 makes the gateway (with the agent registry) the enforcer of the risk tier: it refuses traffic whose declared envelope tier does not match the registered controls, traffic from an agent whose envelope has drifted from its registered tier, and traffic for a use case on the prohibited-use list; for a Tier 2 use case (controlled operational assistance) it rejects calls bearing the wrong workload-identity and agent-identity pair. Log every refusal with the correlation ID. Tier definitions, the trigger list and re-tiering are on Risk tiers.
  6. One credential per developer on coding-agent routes, with the gateway base URL delivered by managed settings rather than a shell export. See Coding-agent workstation.

Recognise the shared-key anti-pattern when the token bill cannot be split by caller, or revoking one caller means rotating the key for everyone. Recover by moving callers to per-client identity one at a time and retiring the shared key when the last caller has left it.

Route every request by its highest data class

Per-application enforcement lets a product approved for confidential data process strictly confidential data the moment a user pastes some in. Book 2 Ch 23 §4.2, Ch 25 §5 and Research Paper 9 §5.2 all place the decision on the request, not the application.

The routing algorithm

For every request:

  1. Collect all inputs: the user’s text, retrieved passages, tool results, memory entries and attachments.
  2. Classify each input and take the highest class present, plus any category flags (for example special-category personal data, inside information, client-identifying data). There is no “mostly internal” prompt: one strictly confidential passage raises the whole request.
  3. Apply an approved down-classification service if the task allows it (redaction, pseudonymisation, aggregation). The service, not the agent’s prompt, sets the new class and records what it removed. An instruction such as “remove personal data before answering” is not a control.
  4. Look up the permitted lanes for the resulting class in the permission matrix (see Data classes and lanes and Research Paper 9 §5.1).
  5. Route to a permitted lane, or block and log. Blocking is a legitimate outcome and must reach the user as one, with a reason. A silent failure teaches people to route around the gateway.
  6. Stamp the computed class on the evidence record and on every derived artefact (traces, caches, prompt logs), which inherit the highest class of their sources.

The declared class on the application’s intake form is metadata, not a control. The control is the class the gateway computes.

Audit it with fifty requests

Book 2 Ch 23 §8 gives a quick test for the “permission matrix as a wish” anti-pattern: sample fifty real requests through the gateway and compare the class the gateway computed with the class the application declared. If the gateway computes no class, you have your answer. Run it at go-live and after any change to classifiers or use-case definitions. Composite example (Book 2 Ch 23 §6, Asset Manager X): six weeks after launch, 11% of requests to an “internal only” drafting assistant arrived carrying content the gateway classed as confidential; every one was routed correctly because enforcement was per request.

Classify the gateway itself

The gateway sees raw prompts and responses, so it is classified at the highest class it handles, and its logs inherit the retention and access rules of the most sensitive trace (Book 2 Ch 13 §4; Research Paper 9 §5.2). A team that stands it up as plumbing has built a strictly confidential system with plumbing-grade controls. The same applies to any response cache: a cache is a derived artefact and takes the class of its inputs.

Complexity routing inside the permitted set

Once the lane is fixed, cost and quality optimisation happens within it. Book 3 Ch 18 §5 (composite Bank A) routes on a validated X-Complexity header with an inline classifier as fallback: the smallest model for classification, extraction and formatting; the default model for drafting and bounded tool use; the largest model only where extended reasoning is documented as necessary. Over six months this moved about a third of would-be top-tier traffic to the mid-tier model without measurable quality complaints. The header is validated against the caller’s token like any routing claim. A cost preference must never override a data-class or risk decision.

Where inference actually runs

Residency has two parts that vendors document separately: where data is stored at rest, and where a prompt is processed. Research Paper 9 §6.2 (vendor pages accessed 10 November 2026) records:

PlatformWhat the documentation saysGateway consequence
Azure (Foundry deployment types)Data at rest stays in the chosen geography for every type; Global may process anywhere; Data Zone processes within the Microsoft-defined zone, which Microsoft may extend without notice; Standard and Regional Provisioned process in the chosen geography.Treat Global as cross-border processing without a fixed destination and Data Zone as cross-border within a named, vendor-variable area. Book 2 Ch 13 §4 calls the deployment type a regulatory configuration presented as a capacity choice.
AWS BedrockGeographic cross-Region inference profiles stay within the geography (US, EU, APAC); global profiles may use any commercial Region. Where retention applies, retained content is stored in the destination Region.Prefer geographic profiles where residency matters; record the profile per deployment.
Google CloudRegional or multi-regional endpoints keep the cache in-location; the global endpoint can serve from any region.Avoid the global endpoint where processing location is constrained.
Anthropic APIInference location global (default) or us; storage location us only.On the access date the direct API offered no EU inference or storage option; reaching the same models via a cloud regional endpoint is a different lane.

[FACT-CHECK: all four rows are current product claims from Research Paper 9 (accessed 10 November 2026) and Book 2 Ch 13 notes (accessed November 2026); re-verify before relying on them.] Record the assessed deployment type and its location answer in the model catalogue entry for each deployment, and let the gateway refuse to route a class to a deployment whose entry does not permit it. Article-level transfer obligations are on the Regulatory map.

Tokenise per invocation, not by default

Tokenisation replaces a sensitive value with a reversible token before the model sees it and restores it on the way back. It is the most operationally fragile part of the gateway (Book 3 Ch 6 §4.1, §6) and the one most often set as a blanket policy that quietly degrades the product (Book 1 Ch 13 §8).

Decide with seven questions

Book 1 Ch 13 §5.3 gives seven questions, taken in order on every model invocation and audited by the gateway. Its Monday-morning sequence (§7) starts with the five invocation types highest by volume in the model usage registry (volume, not importance, is where exposure compounds), records the answer as a one-page paper per type, and has the privacy officer countersign each. This page calls the set of those papers the tokenisation register.

#QuestionEffect on the decision
1Is the model in-tenant or external?External biases towards tokenisation.
2Does the data include special-category or criminal-offence data?Yes with an external model: tokenise or do not send.
3Does the task need entity resolution (customer 360, supplier reconciliation, lookalike detection)?Argues against tokenisation; if both apply, deterministic tokens are needed, and determinism itself leaks.
4Does the task need relationship awareness (family groups, corporate hierarchies, ownership chains)?Argues strongly against tokenising names; a relational graph alongside tokens works but is expensive.
5Does the use feed a solely automated decision with significant effect?The firm must be able to explain what the model saw, tokens or plain values.
6Is the endpoint in the approved jurisdiction?No adds transfer obligations; no plus special-category: tokenise or do not send.
7Does logging meet the retention rules for the original data?No forces tokenisation in the log.

Worked branches from the source: external + special-category + outside jurisdiction → do not send, redirect to an in-tenant model. External + no special category + summarisation + in-jurisdiction + compliant logging → tokenise direct identifiers, send the rest plain. In-tenant + relationship task + in-jurisdiction + compliant logging → send plain, log structurally with sensitivity labels. The output is a justified answer per invocation type, which is what an auditor reviews. The legal tests behind questions 2, 5 and 6 are on the Regulatory map; the class and lane rules are on Data classes and lanes.

Keep the terms distinct: tokenisation is reversible by the key holder; masking is irreversible; labelling attaches a sensitivity label without transforming data; pseudonymised data is still personal data; only anonymisation leaves the personal-data perimeter. Tokenisation is not anonymisation.

Mechanics and failure modes

Reference implementation (Book 3 Ch 6 §4.1 and §6, composite Bank A on Azure):

  1. Inbound body goes to the classifier, which returns a label and detected entity types.
  2. If the label is above the threshold (Bank A: Confidential) and the use-case class permits tokenisation, a tokenisation function replaces entities with stable tokens. The mapping is stored in a secret-backed cache keyed by the correlation ID.
  3. If the label is the highest class and tokenisation is not permitted, the gateway returns 403 and emits a denial event.
  4. A pre-prompt evidence record is written before the call reaches the model, whichever branch ran.
  5. Outbound, the response is classified again and the tokens are restored from the same keyed entry; a post-response record is written.
  6. tokens_in on the gateway record is counted after tokenisation; original counts sit in the tokenisation function’s own event stream, joined on the correlation ID.
FailureSignalRecovery
Token cache out of sync after a secret rotation; tokens no longer resolveAn explicit tokenisation-restore-failed event (Bank A paged on-call within four minutes, twice)Force a cache reload on every rotation; add a round-trip integration test (tokenise → call → restore) to the release regression pack.
Classifier mislabels a document and blocks it403 at the inbound gateThe denial event carries correlation ID, document hash and classifier result, so the wrangler escalates in an hour rather than days; correct the label the same day.
Sustained rise in tokenisation events for one tenantPolicy dashboard trendUsually drift in the source data’s classification; review it with the data owner.
”Always tokenise” as corporate policyUsers report the copilot was more useful in the pilotRetrofit the seven questions to production agents; per-task justification is the stronger compliance posture.

Pin models at the gateway

Every backend carries an explicit model identifier, and no route resolves an alias at call time (Book 3 Ch 6 §4.1, PR-4 row 11). The governance of pinning, update scope, retirement tracking and the upgrade lifecycle lives on Model governance; the gateway’s part is enforcement:

  • Explicit IDs per backend. On Foundry, deployment names are chosen by administrators rather than canonical model IDs, and (per Book 3 Ch 6 §6) Foundry does no start-up model check, so a wrong or retired name fails at first production prompt. Keep the deployment-name map as a change-controlled artefact, reviewed like the route table.
  • Switch off platform auto-upgrade. Book 2 Ch 13 §4 makes it absolute for production, minor versions included. As of November 2026 (Book 2 Ch 13 note): Azure OpenAI in Foundry Standard deployments take a versionUpgradeOption of OnceNewDefaultVersionAvailable, OnceCurrentVersionExpired or NoAutoUpgrade; a deployment with no value behaves as OnceCurrentVersionExpired, so an unconfigured deployment still upgrades itself at retirement. Only an explicit NoAutoUpgrade prevents it, and the deployment then stops working at retirement, so retirement dates must be tracked. [FACT-CHECK: versionUpgradeOption values and default behaviour against current Microsoft Learn.]
  • Alias drift and default changes are cost events. As of August 2026 (Book 3 Ch 18 §5): Claude Code deployments that do not pin a primary model are billed at the top-tier rate from v2.1.207, and the opus alias resolved differently on Bedrock and Agent Platform than on Foundry. [FACT-CHECK: Claude Code default model and per-provider alias resolution.]
  • Unknown IDs. As of August 2026 (Book 3 Ch 18 §5): the Claude apps gateway bills unknown model identifiers at a $5/$25 per million token default tier — sound anti-bypass design, and a surprise when a deployment is missing from the models map. [FACT-CHECK: apps-gateway default pricing tier for unknown models.]
  • Honour the release pin. The gateway serves the pinned combination recorded in the release manifest, so a canary never meets an untested model and prompt pair (Book 2 Ch 13 §4; see Release engineering).

Meet the wire contract

Book 3 Ch 6 §7 names the gateway that “helpfully” transforms traffic as an anti-pattern: standing house policies that are right for other APIs (normalised error envelopes, buffered streaming, request-body WAF inspection, allowlisted headers) let synthetic tests pass while real sessions fail. Run this checklist against any gateway, WAF or reverse proxy before it fronts a Claude workload. It restates Book 3 Ch 6 PR-4, which the author verified against Anthropic documentation on 3 August 2026. [FACT-CHECK: re-verify against current Anthropic LLM-gateway rollout documentation.]

#CheckFail signal
1Accepts the Messages API at POST /v1/messages404 or schema rejection on a minimal request per routed model name
2Streams server-sent events unbuffered end to endFirst tokens arrive in one burst at the end
3Forwards anthropic-beta, anthropic-version and the body unchanged in both directionsA feature works direct and fails through the gateway
4Returns upstream errors unmodified (the client’s recovery matches on error wording)Retries never recover; errors wrapped in a corporate envelope
5/v1/messages exempt from request-body WAF inspection, with a documented compensating control403 on long prompts, 200 on short synthetic tests (code and XML-like tags trip XSS rules)
6anthropic-* headers forwarded as an open listA feature disappears after a client upgrade
7One credential per developer; base URL set in managed settingsShared key in a wiki; a shell export reaches the model
8Not served behind a redirect; no conflicting forced login method with gateway credentialsStart-up authentication failure on recent clients
9Egress allowlist filed against the vendor’s published domain tableFirewall request lists inferred hostnames
10Closed-perimeter settings applied on third-party routes (see Coding-agent workstation)Third-party-route machines still resolve vendor hosts
11Model identifiers pinned per backend; no alias resolved at call timeTop-tier billing after a client upgrade; deployment name fails at first prompt
12Client session ID (x-claude-code-session-id) captured and joined to correlation_idGateway logs cannot be grouped into sessions

AWS-specific trap (Book 3 Ch 6 §5.4): Bedrock streams as application/vnd.amazon.eventstream; an intermediary that re-emits it as text/event-stream breaks the client. Add a streaming-latency assertion to the release regression pack whichever product you run.

Emit one correlated evidence record per request

The gateway issues the correlation ID at ingress if the caller has not supplied one, propagates it to the consumer in a response header and stamps it on every downstream record: classifier call, tokenisation entry, policy decision, SIEM event. Book 3 Ch 6 §7 calls it the most important field in the schema. The cross-stream contract and integrity rules are on Audit evidence.

Gateway record fields

Book 3 Appendix E §3.1 extends the seventeen fields of Ch 6 PR-3 to nineteen. This page uses the Appendix E names.

GroupFieldMeaning
Correlation and principalcorrelation_idUUID issued at gateway ingress; the spine
timestampISO-8601 with timezone, millisecond precision
principalUser, workload identity or agent principal
principal_typeuser, workload, agent
use_case_classFrom header, validated against the token
envelope_idBounded-agency envelope reference
Route, model, tokensdeployment_routeRoute taken (for example a named regional deployment)
model_idModel identifier as called
model_version_pinResolved version at call time
tokens_inPrompt tokens after tokenisation
tokens_outCompletion tokens
Policy and outcomedlp_classificationInbound / outbound labels
dlp_actionpass, tokenised, blocked
tokenisation_event_countEntity replacements at the gateway
policy_decisionallow, allow-with-evidence, allow-with-approval, deny
response_statusHTTP status returned to the consumer
latency_msEnd-to-end gateway latency
mcp_servers_invokedServer names visible at the wire
retention_tierRetention bucket driving immutable storage

Add cached_tokens as a metric on every call (Book 3 Ch 6 §4.1 and Ch 18 §5 emit it alongside prompt and completion tokens) so cache hit rate is reconstructable per envelope and prompt version.

Source differences, stated rather than blended: Ch 6 PR-3 names route and model where Appendix E uses deployment_route and model_id; Ch 6 §4.1 gives the policy outcomes as allow / allow-with-evidence / escalate-for-approval / deny where Appendix E uses allow-with-approval; and the retention buckets differ between Ch 6 PR-3 and Appendix E. The retention schedule is reconciled on Audit evidence, which this page defers to.

AES event kinds for a gateway emitter

The Supporting Tools M7 conformance spec (Draft 1, May 2026) asks a gateway that emits Audit Evidence Schema records on an agent’s behalf to produce five kinds:

KindWhenKey payload fields
gateway.requestStart of a model callgateway_id, gateway_version, model_id, model_endpoint_region, prompt_hash, tool_signatures, request_options, estimated_cost
gateway.responseResponse complete (recommended: stream completion for streamed responses)response_hash, model_version_resolved, finish_reason, duration_ms, cost
gateway.refusalThe model refusesrefusal_class (safety, policy, capability, other), refusal_text_hash
gateway.tool_useOne per tool call in a requesttool_name, tool_version, tool_invocation_id, parameters_hash, result_hash, duration_ms
gateway.safety_eventGateway safety classifier fires, whether or not it blocksclassifier_id, classifier_version, triggered_category, severity, action_taken

Rules from the spec: records name the agent served (executing_agent.agent_id_claim) with the gateway as a recorded intermediary, never the gateway speaking for itself. executing_agent.gateway_id is proposed as an extension field, not part of AES v0.1.0. Prompts are recorded as a hash; the spec recommends hashing the redacted prompt the model actually saw. Cost rides on payload.cost of gateway.response (token counts required, total_cost may be null when not computable) rather than a separate cost record. Conformance targets: all five kinds emitted, schema and signatures valid, payload.cost populated on every response, emit overhead ≤ 50 ms and emit availability ≥ 99.9%. See Schemas and tools.

Treat the gateway as a chosen concentration

The gateway is the most reused component in the estate. Book 2 Ch 13 §4 names three trade-offs:

  1. Single point of failure. Every model call depends on it. Give it a concentration entry (what depends on it, blast radius, how it is made resilient, how products operate through its failure) and a tested answer to what each product does when it is down. An internal shared capability is not a third-party arrangement; the concentration entry is the institution’s internal analogue, and the legal position is on the Regulatory map.
  2. Compliance scope. Classify it at the highest class it handles (above).
  3. Residency. The deployment type behind each route is a regulatory configuration (above).

Resilience parameters

Reference values from Book 3 Ch 6 PR-1 and §6 (composite Bank A); tune to your own risk appetite:

ParameterReference value
TopologyTwo gateway instances in paired regions behind a global front door or traffic manager
Backends per use-case classPrimary and fallback for any class where a route outage must not be a workload outage
Circuit breakerOpens on three consecutive 5xx; fifteen-second open state
Failover rehearsalAt least quarterly; without it the failover is theoretical
Service levelRun as a Tier-1 service; Bank A targeted a 20-minute MTTR for policy-decision failures that break model consumption
Gateway overheadUnder 300 ms inside the gateway, watched on its own line (Book 3 Ch 18 §4.3)

Failover changes the governing agreement

Failing over between cloud providers, or to a vendor’s direct API, changes which agreement, geography and terms govern the request (Book 3 Ch 6 §6, quoting Anthropic’s gateway documentation). Therefore:

  • notify the wrangler and the second line on every route change and on recovery;
  • record each failover in the outsourcing or third-party register with start time, duration and upstream;
  • allow failover only to routes already permitted for the request’s data class, so failover cannot widen the lane (this page’s recommendation, following the per-request routing rule above);
  • where routes include provisioned throughput, a 429 drains provisioned capacity before spilling to on-demand, so an on-demand spend spike is a capacity signal, not a demand signal (Book 3 Ch 18 §4.4).

Latency tiers, not latency exemptions

Book 3 Ch 6 PR-1 sets policy shape by latency budget:

BudgetPolicy pattern
Under 500 ms (real-time chat)Inline-only policy; no asynchronous DLP or tokenisation; 50 ms classifier timeouts; synchronous policy call only where the class requires it
500 ms – 2 s (most agent workflows)Inline DLP and tokenisation; asynchronous SIEM forwarding
Over 2 s (batch, long-running agents)Asynchronous tokenisation and write-evidence-on-completion, only for workloads that accepted that posture in design

If a high-risk call cannot be inspected or authorised inside its budget, fail safely or use an approved asynchronous flow; never skip the check. A latency complaint is not a bypass reason (Book 3 Ch 18 §7): a “temporary” direct path for performance becomes a production workload with no DLP, budget or evidence. Recognise it by an exception still live after a quarter, or spend that appears on an invoice but not in the gateway record stream. Close it with a hard deadline and tune the policy instead.

Configure or procure; do not build

Book 3 Ch 6 §5 treats the choice as configure-versus-procure. Every quarter spent maintaining a custom gateway is a quarter not spent on evidence, change tooling or the agent registry; the book reports two banks that built their own and retreated within eighteen months.

Dated product positions (as of August 2026, per Book 3 Ch 6 §5):

OptionRoleLimits to know
Azure API Management with custom policiesStrategic gateway for an Azure-first institution: Entra-native, inbound/backend/on-error/outbound hooks, named values from Key Vault, Sentinel forwarding. Treat the policy file (Bank A: about 280 lines of XML) as the gateway’s executable specification, version-controlled and regression-tested.DLP and tokenisation are services it calls, not features it has.
Claude apps gateway (shipped 29 June 2026)Self-hosted control plane for the Claude Code developer estate: OIDC sign-in, server-side model allowlist (non-granted model returns 400), per-user spend caps, failover across Claude routes, OTLP telemetry relayed to your collectors; sessions exit at start-up if it is unreachable.Fronts Claude Code only; no DLP, tokenisation or policy-decision callout; no CI service-token flow; no published Azure blueprint (Foundry is a supported upstream with use_azure_ad, and its models: block is mandatory). It complements, not replaces, the general gateway.
Third-party LLM gateways (for example LiteLLM, Kong AI Gateway, Cloudflare AI Gateway, Portkey)Tactical, for example upstream of the strategic gateway for engineering-internal flows with tight latency budgets.Anthropic neither endorses nor audits them; product claims must come from each vendor. Must pass the wire contract.
AWS: API Gateway with a Lambda authorizer, CloudTrail, KMS, MacieEquivalent pattern for Bedrock-routed flows.More custom code; Macie is less inline-callable than Purview.
Google Cloud: Apigee, or Cloud Run with Cloud Endpoints; Cloud DLP, Cloud KMS, Cloud Audit LogsEquivalent pattern; natural for an Apigee shop.A second product and team if adopted from a standing start.

[FACT-CHECK: Claude apps gateway capabilities and limits (Book 3 Ch 6 §5.2 and Ch 18 §5, accessed 3 August 2026); Research Paper 3 §4.2 notes the author had not evaluated it.] Book 2 Ch 25 §5 adds the architectural requirement whatever the product: a provider-agnostic router over three hosting routes (vendor API, private tenancy, controlled infrastructure), with a warm self-hosted floor that serves the most sensitive slice and doubles as the resilience fallback and credible exit.

For any candidate, run the wire-contract checklist, test WAF and DLP with realistic prompts (code, structured content, long sessions), confirm policy executes on every route, and confirm a safe upgrade path.

Close bypass paths

A gateway mediates only traffic that reaches it; closing other paths is a network decision. Anthropic says so of its own gateway (Book 3 Ch 6 §4.2).

  • Deny direct outbound access to provider endpoints from governed workloads; allow the gateway path and named dependencies only.
  • File the egress allowlist from the vendor’s published domain table, not from guesses.
  • Test developer workstations, CI jobs, SDK-hosted agents, local tools and application runtimes separately: they use different routes and credentials. Workstation specifics (managed settings, residual vendor traffic on third-party routes, provider restriction) are on Coding-agent workstation.
  • Inventory ungoverned arrival vectors (personal accounts, embedded SaaS features, keys bought on a card) as part of the landing zone; see Landing zone.
  • Keep provider credentials only in the approved secret path, held by the gateway’s identity. Never in source, committed environment files, prompts or tool arguments.

Run the gateway: day-two operations

The build organisation specifies what the gateway enforces; the run organisation operates it (Book 3 Ch 18). Six levers sit at or around the gateway and each needs a named owner: routing rules, model entitlements and pinned defaults, prompt-cache hit rate, spend caps, latency budgets and chargeback. A metric with no owner and no threshold is a number, not a control.

Two estates, three meters, one ledger

Do not run one cost surface. Book 3 Ch 18 §4.6 (as of August 2026) distinguishes:

Estate / meterWho meters, caps and alertsBlind spot
Vendor-metered seats (Claude Enterprise on claude.ai)The vendor: usage-and-cost dashboard by SCIM group, organisation-level caps, model entitlements; per-user spend report and an Analytics API (Primary Owner mints a read:analytics key)Thresholds and recipients are the vendor’s, not yours; it covers this estate only
Console / APIWorkspaces: per-workspace spend limits, independent rate limits, grouped keys; everything else is your gatewayConsole figures are labelled estimates
Cloud routes (Bedrock, Foundry, Agent Platform)Only the cloud’s budget alarms, which alert but do not enforce per userVendor analytics do not see this usage. The documented per-user stream is OpenTelemetry (claude_code.cost.usage, claude_code.token.usage) with OTEL_RESOURCE_ATTRIBUTES adding keys such as cost_center and team.id

Rules: every meter is an estimate and the invoice is the invoice. The monthly close carries a reconciliation line per estate (metered versus invoiced, plus the unexplained residual) with a named owner. If the OpenTelemetry stream is used for cost evidence, user.email in the collector is personal data for the processing record, and the stream needs mTLS or signed headers at the collector to be evidential. On the cloud routes, if you have not built the meter, you do not have one (Book 3 Ch 18 §6: a fleet upgraded past the default-model change with no pin and no telemetry; the first signal was the invoice). [FACT-CHECK: Enterprise admin cost controls, Analytics API scope and OTel metric names, as of August 2026.] Vendor “value” estimates (productivity, annual value) never enter the cost report as measured fact.

Spend caps on both sides of the boundary

LayerShapeAt the capTreat as
Your gateway (Book 3 Ch 6 §4.1, Ch 18 §5)Daily, weekly and monthly per tenant and envelope; alerts at 50%, 75%, 90%, escalating wrangler → FinOps → platform on-call (wrangler copied at 75% and 90%)429, the call fails. Failing is correct behaviour.Circuit breaker you designed
Claude Enterprise (as of August 2026)Organisation, group and member caps; admins alerted at 75% and 90%, users at 75% and 95%Not documentedAlerting control only until enforcement is confirmed in writing; no Tier 1–2 workload depends on it. [FACT-CHECK: hard stop versus soft breach at an Enterprise spend limit.]
Claude apps gateway (as of August 2026){scope, amount, period} over user, RBAC group or organisation; group and org caps are per-seat defaults, not shared pools429 with error.type: billing_error and x-should-retry: false; fails open by default unless enforcement.fail_closed_on_error: trueSet fail-closed in regulated use and record it as an availability trade-off; there is no approaching-cap alert, so poll the effective-caps endpoint and warn yourself. [FACT-CHECK: apps-gateway spend-limit semantics.]

Every alert above informational pages a named rotation, not a channel. Map vendor alerts into your own escalation explicitly, and test the rotation quarterly with a synthetic alert. The 50% alert buys investigation time; the 90% alert buys time to act.

Cost fallback and the cost-spike playbook

The envelope’s Mandate may carry a cost-fallback-permitted flag (Book 3 Ch 18 §5). When set, the gateway substitutes the next model tier down once projected end-of-day spend exceeds 90% of the cap, logs the substitution as a policy decision and tells the wrangler at day’s end. Where the wrangler has documented that answers are not interchangeable, the flag is off and the call fails instead.

Escalation playbook (Book 3 Ch 18 OD-2; Bank A’s SLAs, portable structure):

StepTimeOwnerAction
DetectT+0SOC day shiftAlert at the tenant’s 75% daily threshold, tagged with tenant, envelope, burn rate and projected end-of-day spend
AssessT+15 minOn-call FinOpsSix diagnostics: known onboarding; agent loop; cache miss; long session never cleared; scheduled task on an idle session; multi-agent team run
InformT+20 minWrangler (paged via rotation)Fifteen minutes to respond before platform on-call escalates
MitigateT+60 minWrangler decidesRate-limit, model fallback (if flagged) or suspend (429 for the rest of the day)
ResolveT+24 hWrangler files the changeTighten per-conversation cap, add cache warming, pin a default, or amend the envelope; FinOps owns the post-mortem entry

Composite incident (Book 3 Ch 18 §6): a search-and-summarise agent loop ran six hours to about 800,000 tokens (about $2,400) before the daily cap fired. Fixes: a per-conversation hard limit (120,000 tokens) in a pre-tool budget hook and the envelope, and tool-call depth bounded at six.

Prompt-cache discipline

  • Cache long, stable system prompts (Book 3 Ch 18 §5 uses one thousand tokens as the point where caching pays back within a handful of calls). Choose a longer cache write for continuous traffic and a short one for bursty traffic.

  • Track hit rate per envelope and prompt version from cached_tokens; the monthly report carries a volume-weighted average against a target line (Bank A: 85%).

  • Any change touching a cached prefix declares a cache-invalidation impact and a warming plan (synthetic calls at deploy time) in the change dossier. Composite incident: a standard-change system-prompt rewrite dropped hit rates from above 90% to zero for two days and roughly tripled cost per call.

  • Declare the cache window per agent in the envelope: Book 3 Appendix D’s Memory field prompt_cache_ttl takes 5m, 1h or disabled, with 1h reserved for system-prompt-heavy agents where the 2× write pays back within the hour.

  • Count tool definitions as fixed overhead. As of May 2026 (Book 3 Ch 9 §4.1): each tool adds roughly 313–346 tokens of system prompt, so a ten-tool agent spends about 3,300 tokens per session before any reasoning; caching offsets it, and trimming unused tools reduces it.

Pricing mechanics as of August 2026 (Book 3 Ch 18 §4.2; Ch 9 §4.1 gives the same multipliers as of May 2026): cache reads at 0.1× input price; 5-minute writes at 1.25× and 1-hour writes at 2×; cache lifetime five minutes by default on an API key or cloud route. Book 3’s glossary (Appendix J) lists prompt caching as ZDR-eligible. [FACT-CHECK: current Anthropic prompt-caching multipliers, default TTL per route, per-tool token overhead and ZDR eligibility.]

Batch for non-interactive work

Overnight classification, periodic screening and evaluation runs suit the Batch API (Book 3 Ch 18 §5: 50% discount against synchronous price, as of August 2026). Batch is not zero-data-retention eligible per the same source, so record that exclusion in the workload’s envelope and check it against the lane before routing any class there. [FACT-CHECK: Batch API discount, retention window and ZDR eligibility.]

Capacity sizing

Rate limits are a cost control read backwards: too high and a loop runs; too low and a rollout becomes a support queue. Size per-user limits from the vendor’s population guidance (it falls sharply as population grows), remember that organisation-level limits let individuals burst, and on Bedrock set the service tier per workload class rather than leaving the default (Book 3 Ch 18 §5). [FACT-CHECK: current per-user TPM/RPM sizing guidance.]

Five dashboards, each with an owner and a threshold

DashboardQuestionsDaily readerThresholds that page
CostTokens and spend per envelope today/week/month; burn against 30-day average and cap; breakdown by model and routeFinOps lead50/75/90% of cap; tenant >50% above prior month
UsageCalls per consumer, route and model; cache hit rateGateway product ownerCall volume jump (for example threefold overnight); hit-rate drop
Latencyp50 / p95 / p99 per route, model and envelope; gateway overhead on its own lineGateway product ownerp95 against the envelope’s budget (Bank A: 2 s coding assistants, 8 s research platform, 3 s customer services); overhead >300 ms
ReliabilityErrors by status, per-route availability vs SLO, failovers (start, duration, upstream)Platform on-callSLO breach; any failover
PolicyDLP blocks, tokenisation events, policy denials, hook decisionsSOC day shift; wrangler weekly; second line monthlyDLP-block spike (often misclassification); denial spike (policy out of step with behaviour); tokenisation climb (source drift)

Latency budgets are set by the agent control board and recorded in the envelope’s Mandate. The upstream provider’s SLA is contract material, not a public figure. [FACT-CHECK: vendor enterprise SLA in the executed agreement.] A panel that has never paged is reviewed for tuning or removed.

Operating cadence

CadenceForum and ownerOutput
Daily (15 min)SOC + FinOps + platform stand-up; gateway product ownerAction list owned by noon
Weekly (45 min)Cost-and-performance review; FinOps leadOne-page summary to the CIO; thresholds tuned; chargeback inputs frozen
Monthly (60 min)Chargeback meeting; business-owner chargeback leadChargeback report issued; budget adjustments approved or escalated
Quarterly (90 min)Steering committee cost-and-risk summary; CIORisk-appetite reaffirmed or amended; routing, entitlement and pinned-model review; capacity plan
Continuous24×7 platform on-callSev-1 for budget breach, route failover, p99 or reliability SLO breach

The quarterly review includes reading model release notes and checking the routing policy and entitlement list against the model usage registry. Composite incident: a model with a new tokeniser consumed materially more tokens for the same text and pushed one workflow about 26% over budget although list prices were unchanged.

Chargeback with authority

Chargeback that moves no budget is a wall poster (Book 3 Ch 18 §7). The CFO and CIO agree the model (full chargeback, allocation against a target, or hybrid) and cost centres see the line in their own accounts. Attribute cost through the same group structure that governs access (SCIM groups on the seat estate, cost_center on telemetry attributes on the API side) so one source serves both recertification and chargeback.

Monthly chargeback report (Book 3 Ch 18 OD-3), twelve rows: heading (month, run and FinOps contacts); total spend; variance against prior month; reconciliation by estate (seats, Console/API, cloud routes, residual and its owner); spend by envelope (tokens in, out, cached, total); spend by cost centre; per-envelope variance against cap (RAG); top three cost movers with one-sentence explanations; model mix with any model outside the entitlement list flagged; cache hit rate against target; reliability (failovers, downtime minutes, upstream); action register (three to five items with owners and dates). Two rules: every figure is an estimate until reconciliation closes, and no productivity or “annual value” estimate appears in it.

Never trim evidence to save cost

Proposals to cut hook payload retention or drop full-payload evidence for high-tier workloads to save SIEM ingestion go to the second-line evidence function. Retention is set by the use-case tier, not the cost dashboard. Tune storage tiers and archive instead; the schema does not move (Book 3 Ch 18 §7).

Ownership

ConcernAccountable owner
Gateway availability, policy file, route table, deployment-name mapPlatform (gateway product owner)
Route eligibility by data class; classifier thresholdsData office with second line
Tokenisation register and per-invocation decisionsWrangler, with privacy office sign-off
Model entitlements and pinsModel governance owner (see Model governance)
Spend thresholds, reconciliation line, chargebackFinOps lead; chargeback lead
Failover notifications and third-party register entriesPlatform on-call to second line
Evidence schema and retentionSecond-line evidence assurance

Failure modes and recovery

Anti-patternRecognise itRecover
Gateway that “helpfully” transforms traffic403s on long prompts only; retries that never recover; tokens in one burst; feature lost after client upgradeWrite the wire contract into the service contract; exempt the path from body inspection with a compensating control; add a streaming assertion to regression
Gateway as a proxy onlyPolicy file under ~50 lines; SIEM receives HTTP logs onlyAdd one job per sprint, one class at a time, each as a policy version with tests
One-sided DLPInbound body section does not call the classifierEnable inbound classification; tighten class definitions
Budgets that never alertTenant >50% above prior month; vertical burn on day two50/75/90% alerts to a rotation; 429 at cap
Gateway as single point of failureNo paired region; no quarterly rehearsalPaired regions behind a front door; backend failover; rehearse quarterly
Logging without a correlation IDEvidence trace fragments at the joinIssue at ingress, propagate, require on every emitter
Shared keysBill cannot be split; revocation hits everyonePer-client identity, one caller at a time
Performance bypassException live after a quarter; spend on invoice not in gateway recordsHard close date; tune policy within the latency tier
Vendor alerts assumed to be yoursOnly vendor emails fire; nobody on a rotaMap vendor alerts into your own escalation

Acceptance checks

  • An unauthorised workload cannot reach a model provider directly or through an unapproved route; egress tests run from workstations, CI, SDK hosts and runtimes separately.
  • Every caller presents its own identity; no shared subscription or provider key exists outside the gateway’s vault path, and each vendor-only key is on the sunset register.
  • Editing a routing header cannot obtain a different lane, region or model tier; a tier or registration mismatch is refused and logged.
  • A fifty-request sample shows the gateway computing a class per request, routing by the highest class present, and blocking with a user-visible reason where no lane is permitted.
  • The gateway, its logs and its caches are classified at the highest class handled.
  • Inbound and outbound classification both execute; tokenisation round-trip and restore-failed events are tested after a secret rotation.
  • Model identifiers are explicit per backend; no production deployment is on an automatic version policy.
  • All twelve wire-contract checks pass on each route.
  • Every request emits a record with the fields above and a gateway-issued correlation ID that joins to workstation, SDK and downstream records.
  • Failover has been rehearsed this quarter, and the last failover reached the wrangler, the second line and the third-party register.
  • Caps alert at 50/75/90% to a rotation and stop at 100%; vendor caps are documented as alerting-only where enforcement is unconfirmed; apps-gateway enforcement is fail-closed where used.
  • The monthly close reconciles three estates to invoice within an agreed tolerance, with a named owner for the residual.

Source status and differences

  • Layering of models. Book 1 Ch 13 describes one API boundary with three sub-planes (gateway, broker, tokenisation gate); Book 3 Ch 6 puts tokenisation inside the gateway’s jobs. This page follows Book 3 for placement and Book 1 for the tokenisation decision model.
  • Field names, policy outcomes and retention tiers differ between Book 3 Ch 6 PR-3 and Appendix E; this page uses Appendix E and defers retention to Audit evidence.
  • Foundry residency. Book 3 Ch 6 (August 2026) describes the Foundry EU data zone for Claude as undated; Research Paper 3 §4.1 (November 2026) reports no EU data zone for Claude on Foundry. Treat the later statement as current until verified. [FACT-CHECK: Claude on Foundry deployment types and data zones.]
  • Composites. Bank A, Asset Manager X and their incidents are composite examples, not real institutions. Thresholds quoted from them are reference values.
  • Vendor claims are dated to the source’s access date and marked for verification. Regulatory obligations and dates live on Regulatory map.

Source material

  • Book 1, Chapter 10, The Convergent Platform Pattern (§4.1–4.3 runtime guardrail categories; §8 anti-pattern 3, capability list before switching engines)
  • Book 1, Chapter 13, MCP Governance and API Management (§4.5 the AI gateway as tokenisation gate; §5.3 the seven-question tokenisation decision; §6 worked example)
  • Book 2, Chapter 11, Identity, Secrets and Configuration (§4 per-client identity at the model gateway; §8 shared keys at the model gateway)
  • Book 2, Chapter 13, Release Engineering and Change Resilience (§4 the model gateway is a chosen concentration; automatic upgrades disabled; §8 the model that upgraded itself)
  • Book 2, Chapter 23, The Ten Recipe Families (§4.2 per-request classification at the gateway; §8 the permission matrix as a wish)
  • Book 2, Chapter 25, Buy versus Build in the Factory Era (§5 the provider-agnostic router and three hosting routes)
  • Book 3, Chapter 5, Identity and Access (PR-3 the gateway as caller of last resort)
  • Book 3, Chapter 6, The LLM Gateway and Policy Decision Point (§4 what the gateway is and is not; §5 build versus buy, Claude apps gateway, wire contract; §6 Bank A worked example; §7 anti-patterns; PR-1 decision tree; PR-3 logging schema; PR-4 conformance checklist)
  • Book 3, Chapter 9, The Anthropic API and Agent SDK (§4.1 prompt caching and per-tool token overhead)
  • Book 3, Chapter 14, Risk Tiers and the Use-Case Taxonomy (gateway and registry enforce the tier; refusal on tier, envelope-drift or identity-pair mismatch)
  • Book 3, Chapter 18, Cost, Performance and Gateway Operations (§4 five dashboards and three meters; §5 seven routing and cost-control patterns; §6 three cost surprises; §7 anti-patterns; OD-1 cadence; OD-2 cost-spike playbook; OD-3 chargeback report)
  • Book 3, Appendix D, Bounded Agency Envelope Template (§2.3 Memory field prompt_cache_ttl)
  • Book 3, Appendix J, Glossary (prompt caching)
  • Book 3, Appendix E, Audit Evidence Schema for Claude Actions (§3.1 gateway record, nineteen fields)
  • Research Paper 9, Data Classes, Processing Lanes and Controller/Processor Roles for AI (§4 lanes; §5.2 routing by classification; §6.2 where inference runs; §8 vendor terms)
  • Research Paper 3, Securing Claude Code in the Enterprise (§4.1 approved routes; §4.2 rules)
  • Supporting Tools, M7 LLM Gateway emit conformance spec (five mandatory event kinds; payload.cost; conformance checks)

← All AI Delivery Factory guidance

On this page

  1. Decisions this page supports
  2. Define the gateway contract
    1. Specify guardrail capabilities, not products
  3. Give every caller its own identity
  4. Route every request by its highest data class
    1. The routing algorithm
    2. Audit it with fifty requests
    3. Classify the gateway itself
    4. Complexity routing inside the permitted set
    5. Where inference actually runs
  5. Tokenise per invocation, not by default
    1. Decide with seven questions
    2. Mechanics and failure modes
  6. Pin models at the gateway
  7. Meet the wire contract
  8. Emit one correlated evidence record per request
    1. Gateway record fields
    2. AES event kinds for a gateway emitter
  9. Treat the gateway as a chosen concentration
    1. Resilience parameters
    2. Failover changes the governing agreement
    3. Latency tiers, not latency exemptions
  10. Configure or procure; do not build
  11. Close bypass paths
  12. Run the gateway: day-two operations
    1. Two estates, three meters, one ledger
    2. Spend caps on both sides of the boundary
    3. Cost fallback and the cost-spike playbook
    4. Prompt-cache discipline
    5. Batch for non-interactive work
    6. Capacity sizing
    7. Five dashboards, each with an owner and a threshold
    8. Operating cadence
    9. Chargeback with authority
    10. Never trim evidence to save cost
  13. Ownership
  14. Failure modes and recovery
  15. Acceptance checks
  16. Source status and differences

All pages in Build the platform →