Architecture10 min read
One gateway for every model call
Keep provider keys, routes, limits and usage receipts in one gateway: pinned models never fall back silently, and calls retry only before output is committed.
Mercury Labs
Keep provider keys, routes, limits and usage receipts in one gateway: pinned models never fall back silently, and calls retry only before output is committed.
Provider credentials, routing rules, concurrency limits and usage records belong in one place: a model gateway that every model call passes through. In the Company Twin, the gateway is the only part of the system that holds a provider key. The executor that runs agents holds none, and neither does anything else. Each caller may use only the routes its class allows, a pinned model is the model that answers, and every call ends with a receipt.
A model gateway is a service that sits between an application and its model providers, holding the credentials and deciding, for each call, which model may serve it, how many may run at once, how long it may take and what it cost.
This article covers why we moved every caller behind one, what it enforces, the rule that decides when a failed call may be tried again, how our own self-hosted models fit behind the same contract, and what production taught us about running it.
When we audited it, our product code called model providers from about fifteen places: the executor's agent turns and tools, the tenant API's routers and background jobs, the build worker through a bridge, the runtime for our self-hosted models, and the app's dictation route. Each one held credentials, applied its own retry and timeout rules, and recorded usage, or did not. A key rotated in one place broke another. A retry policy fixed in one caller stayed broken in the rest.
Moving every caller behind one gateway changed the security question from "which of fifteen processes could leak a key" to "one role, which nothing else can read". It also gave us one place to apply policy and one stream of receipts. We moved the callers one family at a time, removing each caller's credentials in the same change.
| Caller class | May use |
|---|---|
| Executor | Interactive agent routes, the executor's tool routes and production self-hosted models |
| Tenant API | Routers, titles, organisation style, provenance, proposers and transcription |
| Build worker | The authoring route only |
| Evaluation | Maintenance and evaluation pools, and pinned evaluator routes |
Each class authenticates with its own credential, and any caller can ask the gateway which routes it may use.
upstream) or from the gateway itself, and what kind it was: unreachable, interrupted mid-stream, timed out, unavailable, rate-limited, refused credential, or a request the provider rejected. Only the first five are worth asking again. The gateway itself retries only before any output has reached the caller (below); a stream interrupted after that ends as a failure. The tenant may then run the whole turn again as a new attempt, discarding the partial answer, but only for a retryable provider failure, and never once the attempt has run a tool with side effects.The rule we hold hardest is that there is no silent fallback. If a pinned model is unavailable, the call fails and says why. A system that quietly answers with a different model has made a decision nobody took, and every evaluation of the pinned model is now measuring something else. This is the core of the argument in our guide to multi-model AI without lock-in: make the model a setting, then make sure the setting is what actually runs.
Streaming makes retries dangerous. Once the first words of an answer, or the start of a tool call, have reached the caller, trying again would produce a second, different answer stitched onto the first. So the gateway draws one line: the moment output is committed.
Before that line, a transport failure may be retried a bounded number of times with the same request identity, and usage is counted once. Where a route has a second credential path to the same model, switching to it is allowed only here, and the losing attempt is aborted before the other starts. After the line, nothing is retried or switched; a failure is reported as a failure, with its origin.
Every admitted call gets a gateway request ID, which the caller generates and reuses on its own retries, and a receipt: route, provider, model, attempts, timings and usage when the provider reports it. Calls made for a run also carry the tenant, organisation, run, attempt and the role of the call, so a receipt can be tied to the turn that caused it. Receipts never contain prompt or response content.
Usage has one form on every route, whichever provider served it. Input tokens exclude prompt-cache reads and writes, which are counted separately; output tokens include reasoning; and a separate context figure is all the input the model read. That separation matters. With caching on, the uncached input is often a small fraction of what the model saw, and adding the wrong fields together overstates or understates usage by multiples.
A resilience audit in October found that the gateway did not yet guarantee exactly one accurate receipt per call, and the fixes changed how we think about usage. A provider can return HTTP 200 and close the stream cleanly without the model having finished, so a clean end of stream no longer counts as completion. Early usage counts, such as those in a stream's opening event, are not final either. Usage is now final only after the provider's terminal event; anything else is recorded as uncertain and reconciled later, rather than charged on stale numbers. When a response is too large for the gateway to scan to the end, the receipt says its usage is truncated instead of reporting a partial count as the whole.
We serve open-weight models on GPUs we rent and control, under the name Project Venus, for reasons set out in why we serve our own open-weight models. To the caller, a Venus route is a route like any other: resolved, pinned, bounded and receipted by the gateway. A few rules keep it that way.
The gateway also has a residency setting. With it on, a server admits only in-region inference in London through Amazon Bedrock, and every other model route answers 403. Enforcing residency at the one place every call passes through is far easier to evidence than asking each caller to remember it.
Less than we feared. We measured one fake provider streamed directly, as callers used to, and through the gateway over loopback.
| Case | Direct, p50 (p95) | Through the gateway, p50 (p95) |
|---|---|---|
| First text, 64 chunks sent at once, 200 calls | 0.85 ms (2.39) | 2.82 ms (6.75) |
| Whole call, same | 1.40 ms (3.52) | 2.89 ms (7.46) |
| First text, 32 chunks 5 ms apart, 40 calls | 1.31 ms (2.59) | 2.91 ms (5.80) |
| Whole call, same | 178.2 ms (184.4) | 175.7 ms (181.0) |
| 16 concurrent calls, 32 chunks 5 ms apart | 173.1 ms | 185.3 ms |
Apple silicon, Node 25, September 2026. Real provider latency is hundreds of milliseconds to seconds and dominates these figures.
The gateway adds about 2 ms before the first text and relays text as the provider sends it, so a single paced stream takes no longer end to end. With sixteen paced streams at once, a round took about 12 ms longer at the median, around 7%.
One piece is still ahead of us: replacing the executor's class credential with per-run grants signed by the tenant API, so that every model call is bound to a claimed run rather than to the executor as a whole.
Most organisations we meet have model keys in several services, a fallback somebody added during an outage, and usage they reconstruct from invoices. A gateway is usually the highest-value piece of AI infrastructure to build first, because everything after it (evaluation, cost control, residency, switching models) depends on knowing exactly which model answered which call.
06 stages
Inventory every place that calls a model, and move them one family at a time
Watchpoint
A gateway that some callers bypass gives a false sense of control
Remove each caller's credentials in the same change that moves it
Watchpoint
Keys left behind are keys that leak
Name routes, pin them per run, and refuse rather than substitute
Watchpoint
A silent fallback invalidates every evaluation you ran
Retry only before output is committed, with the same request identity
Watchpoint
A retried stream stitches two answers together
Treat usage as final only after the provider's terminal event
Watchpoint
A clean end of stream is not proof the model finished
Keep prompt and response content out of receipts
Watchpoint
A usage log full of content is a second copy of your data
For the governance argument behind keeping the model layer separate, see LLM-agnostic by design.
Every model call in the Company Twin goes through one gateway, with a receipt that says exactly which model answered. See the Company Twin →