Models9 min read
Why we serve our own open-weight models
Residency is not the whole case: control of the weights fixes the serving setup, cache, training and capacity, behind one gateway with no silent fallback.
Mercury Labs
Residency is not the whole case: control of the weights fixes the serving setup, cache, training and capacity, behind one gateway with no silent fallback.
We serve our own open-weight models because control of the weights lets us do things a rented API cannot: keep the serving configuration fixed until we choose to change it, own the prompt cache and how requests are routed to it, train adapters on how work is actually done, and shape capacity for many agents working in parallel. Data residency matters too, and our private model already serves from dedicated UK or EU infrastructure, but residency alone is not the reason.
Serving open-weight models means running a model whose weights are published, such as Qwen or DeepSeek, on GPUs you choose and with an inference engine you configure, instead of sending prompts to a provider's API and accepting whatever model, settings and capacity sit behind it.
This is the reasoning behind Project Venus, the open-weight research programme that George Pullen leads for the Company Twin. It is also a reasoning with limits, and we set out where open weights still lose, measured on our own workloads, as plainly as where they win.
It is the reason most people give, and for some organisations it is a hard requirement: their prompts, files and business context must not leave a given jurisdiction. Our Project Venus research page describes where we are on that today. The private model serves from dedicated UK or EU infrastructure, and a fully UK-hosted Twin, with the server, storage and model inside one national boundary, is the goal rather than a standard deployment yet.
But residency is increasingly something hosted providers can offer through regional cloud deployments, so on its own it rarely decides the question. If a hosted model in the right region meets your rules, it is often the simplest answer. Our case for running the weights ourselves rests on what control lets us change.
A hosted model is a moving target. The provider can revise the checkpoint, the quantisation, the context handling or the tool-call format, and an agent that passed its tests last month can start failing this month without a line of your code changing. When we serve a model, the exact weights revision, engine build, image digest and every engine flag are pinned in one profile, and a change goes through the same gates as any other release. The one time our configuration did change without a decision, prefix caching switched off by an unrelated merge, the pin is how we found it.
Agent turns repeat a long prefix, so the key-value cache decides how fast they feel. With our own engine we choose whether it is on, how long entries live and which replica a session lands on. In our serving measurements, prefix caching took a 64,000-token agent turn from 104 seconds to 4.2 seconds to first token, and pinning sessions to replicas took it to 2.5. A hosted API gives you its cache policy, not yours.
With the weights you can train small adapters on how work is actually done and serve them beside the clean base model. Our research page describes the adapter we train on real task trajectories. We are careful about what it has proved so far: our adapters have reliably learned the Twin's action format, and we have not yet shown a gain in the accuracy of the answers.
An agent that fans out into parallel sub-tasks, or an evaluation sweep over hundreds of cases, wants throughput and low idle time rather than one fast conversation. On one eight-GPU node our bulk serving lane produced 8,687 output tokens per second across 768 concurrent requests. That is what lets Venus be offered without metering tokens, within concurrency limits, as the research page puts it.
| Hosted API only | Control of the weights | |
|---|---|---|
| Model and settings | The provider's, revised on their schedule | Pinned weights, engine and flags; changed through your own gates |
| Prompt cache | Their policy, lifetime and minimums | Your engine's cache, your lifetimes, your routing |
| Training | Fine-tuning where offered, on their terms | Adapters and training runs you control and can inspect |
| Capacity | Rate limits and per-token pricing | Throughput you can shape for parallel agent work |
| Time to first answer | Immediate | Minutes from cold unless capacity is kept warm |
| Operational load | Low | Qualification per model, per engine and per GPU type |
The Twin has exactly one place where model calls happen: a model gateway that holds every provider credential and enforces per-caller routes, concurrency, streaming, cancellation and usage receipts. No other part of the system holds a provider key. A self-hosted model is just another route behind it: an OpenAI-compatible /v1 endpoint with its own inference token, separate from the credentials that start and stop GPUs.
Three rules make that safe rather than merely tidy.
Because routing is per session and the governance sits outside the model, the same answer rules apply whichever model replies: sources shown, calculations explained, unsupported questions refused. That idea, separating governed products from model routing, is older than Venus; we described it in LLM-agnostic by design.
Open-weight releases arrive weekly, and every leaderboard flatters someone. We use three filters, in this order.
On our own measurements, in five places. None of them is a reason not to do it, but each is a reason not to oversell it.
The open models themselves are improving fast, and the gap is narrowing faster than most buyers assume; George's piece on open-weight architecture explains why. That is a reason to build the serving discipline now, not to assume the model will do the rest.
A choice of model per workload, under the same governance. Today a Twin can answer through an external provider configured for the organisation, or through Venus. Both run under the same rules for sources and access, and the private model already serves from dedicated UK or EU infrastructure; a fully UK-hosted Twin is the goal, not yet a standard deployment. A finance team might keep a hosted frontier model for open-ended analysis and route high-volume, repetitive agent work to the private model, where throughput and a fixed configuration matter more than the last few points on a benchmark. Nothing about who can see which record changes with that choice.
We would start by separating the decision about models from the decision about governance. Put one gateway in front of every model call, with credentials, routes and receipts in it, and make sure nothing falls back silently. Then build a frozen benchmark from your own tasks that grades the trace, not just the answer, and run every candidate, hosted or open, through your real harness.
With that in place, self-hosting becomes a measured choice per workload rather than a platform bet. You can start entirely on hosted models, as most organisations do, and move a workload to open weights when the evidence says the control is worth the operational load. Our guide to multi-model AI without lock-in and fine-tuning against RAG cover the next two decisions.
The Twin keeps your definitions, data and access rules in one governed place, whichever model, hosted or our own, answers the question. See the Company Twin →