Models11 min read
Serving open-weight MoEs: what we measured
Prefix caching, data and expert parallelism with pinned sessions, and contract-gated quantisation cut a 64K agent turn from 104 s to 2.5 s on eight PCIe GPUs.
Mercury Labs
Prefix caching, data and expert parallelism with pinned sessions, and contract-gated quantisation cut a 64K agent turn from 104 s to 2.5 s on eight PCIe GPUs.
For long-context agent work on open-weight mixture-of-experts models, three choices held up in our measurements: turn prefix caching on, run data and expert parallelism with each session pinned to one replica, and admit a quantisation only when it passes the agent's own contracts, not just its component tests. On eight PCIe GPUs that combination cut the time to first token for a 64,000-token agent turn from 104 seconds to 2.5 seconds. On raw, uncached long prompts the data-parallel layout also ran about 2.3 times faster than tensor parallelism, pinned or not.
A mixture-of-experts (MoE) model is a transformer whose feed-forward layers are split into many smaller "experts", of which a router activates only a few for each token, so the model holds far more parameters than it uses on any one step. That makes it cheap to run per token and awkward to serve: every expert still has to sit in GPU memory, and the way you spread them across GPUs decides how fast it goes.
These figures come from Project Venus, our research into serving open-weight models for the Company Twin. The main subject of these measurements is DeepSeek V4.1 Flash, whose checkpoint is about 475 GiB, served with vLLM on a node of eight NVIDIA RTX PRO 6000 Blackwell GPUs (768 GB in total, connected over PCIe rather than NVLink). We use DeepSeek as a teacher model that generates training text; the product model is the smaller Qwen3.8-27B, and where a lesson came from that work, we say so. Every result was measured against gates written before the run and a fixed correctness test we call the gold test; where a result missed its gate, we report the miss.
An agent turn is mostly repetition. Each request carries the same instructions and tool definitions, then the conversation and every tool result so far, and only at the end something new. Without a cache the engine rebuilds its key-value state for the whole prefix on every turn; with one, it reuses the blocks it has already computed and prefills only the new tail. Our guide to prompt caching covers the same mechanism on hosted APIs. When you run the engine yourself, you also own the switch.
We learned that the hard way. vLLM supports automatic prefix caching for DeepSeek V4.1 and turns it on by default, but our early launch scripts passed --no-enable-prefix-caching, a flag carried over from deterministic evaluation runs. With it removed, a 64K-token shared-prefix agent turn at 32 concurrent sessions went from a p90 time to first token of 104 seconds to 4.2 seconds on TP8, with a cache hit rate of 0.97. For this model the cache stores the compressed key-value and indexer blocks at 64-token granularity and replays the 128-token sliding window after a hit. The gold test passed with caching on, including a retrieval answered from a cache hit.
Time to first token, 64K-token agent turn at 32 concurrent sessions (p90)
The obvious worry is that a cache changes the answers. We tested that directly on the Qwen3.8-27B stack before turning it on in production: four arms on one GPU, every request sent twice, caching on and off, with and without the speculative draft model. With the draft model, 106 of 106 perplexity windows were bit-equal, 40 of 40 GSM8K outputs were identical, all six facts planted in a 51,753-token document were found, and the repeat time to first token on that document fell from 18.1 seconds to 0.46 seconds.
Tensor parallelism (TP8) splits every layer across all eight GPUs, so each layer ends in an all-reduce between them. On NVLink that is cheap. Over PCIe it is not: measured inside the CUDA graph, NCCL's all-reduce took 41 to 45 microseconds and FlashInfer's PCIe all-reduce 22 to 29, while a raw peer-to-peer hop takes about one microsecond. Data plus expert parallelism (DP8+EP8) gives each GPU its own copy of the attention layers and shares the experts between them, so the node behaves like eight independent prefill engines with no per-layer all-reduce for attention. It also gave us a key-value pool of 60.8 million tokens, against 9.5 million on TP8.
On raw, uncached long prompts, DP8+EP8 reached 44,200 tokens per second at 64K context and 128 concurrent requests, and 42,700 at 32K and 256. TP8 topped out at about 19,000. Our pre-registered bar was 40,000.
Raw long-prompt throughput, prefix cache cold (tokens per second)
Data parallelism has one catch, and it matters more than the parallelism itself. Each replica has its own cache, so a session whose turns land on different replicas pays for its prefix again and again. Unpinned, DP8 managed a first-turn hit rate of 0.76 and a p90 time to first token of 15 seconds, worse than TP8. Pinning each session to one replica, or routing by prefix, brought it to 2.5 seconds. That pinning was set up for the test. It is not yet built into our model gateway, which already knows the session behind every request and is where the routing rule belongs; today the gateway serves a single tensor-parallel replica.
Prompt speed is half of an agent turn; the rest is generation, and two workloads want different layouts. The untuned TP8 qualification peaked at 416 output tokens per second (2K-token prompts, 128 concurrent), and at 8K-token prompts the queue grew until the first token took up to 1,037 seconds. Prefill and inter-GPU communication were the bottleneck, not the experts.
| Bulk lane | Interactive lane | |
|---|---|---|
| Layout | DP8+EP8, the model's Engram memory tables (lookup tables of learned embeddings it reads alongside its experts) in host RAM, CUDA graphs captured up to batch 512 | TP8 with FlashInfer's PCIe all-reduce and DSpark speculative decoding, which proposes several tokens ahead for the model to check in one pass |
| Measured | 8,687 output tokens/s at 2K prompts and 768 concurrent; 4,013 at 8K and 512 | 224–251 tokens/s per user at concurrency 1; p90 time to first token 0.31–0.64 s |
| Use it for | Parallel agent work, batch extraction, evaluation sweeps | One person waiting on one answer |
| Watch for | 16K batched tokens failed the gold test; 1,024 concurrent is unqualified | DSpark doubles speed at concurrency 1 but costs about a third at 256, so gate it per lane |
DeepSeek V4.1 Flash, 1,024-token outputs with thinking on, 31 October to 1 November 2026. The bulk lane is about 21 times the untuned qualification's best output throughput.
For context, the interactive lane is about a quarter of SemiAnalysis's published per-user figure on an NVIDIA GB300 system with NVLink. Most of that gap is the price of PCIe.
We run one champion stack that all serving work assumes, and one challenger that has to reach parity before it can replace it. For a while we ran three engines: vLLM, SGLang and llama.cpp, which served our GGUF builds well on single streams. The cost showed up in October, when a round of cache research turned out to have targeted llama.cpp, the engine we were moving away from. Since 22 October, vLLM is the champion, SGLang the only challenger, and llama.cpp is retired from serving research.
The champion earned the role on numbers. On Qwen3.8-27B with the official FP8 checkpoint on an H100, stock vLLM decoded at 80.5 tokens per second, 129.4 with two-token multi-token prediction, and 155.2 with a DFlash2 draft model, a small companion model proposing seven tokens at a time for the main model to check: 93% faster, with no failures on our length canaries. On an H200 the same recipe went from 97.8 to 179.4.
Champion and challenger also give you a principled way to say no. In the same weeks we rejected:
Because an agent fails in ways a benchmark does not count. In September we ran 13 arms of Qwen3.8-27B checkpoints through 224 component tests that exercise the real agent loop: tool calls, streaming, cancellation and dependent answers over a sealed test warehouse. The BF16 and 5-bit references passed all 224. The faster 4-bit NVFP4 builds passed 223 and 222.
One or two failures sound small until you read them. They included extra prose where the loop expected a structured action, and a call to a tool the agent is not allowed to use. The first breaks a parser; the second breaks a safety boundary. Our gate allows zero degradation against the reference, so no quantisation was promoted, and the faster builds' speed counted for nothing.
| Component tests | What failed | Admitted | |
|---|---|---|---|
| BF16 reference | 224/224 | Nothing | Reference |
| 5-bit GGUF reference | 224/224 | Nothing | Reference |
| NVFP4 build A | 223/224 | One agent-loop test | No |
| NVFP4 build B | 222/224 | Extra prose; a forbidden tool call | No |
| Heavily quantised W4A16 | General benchmark parity | None on that benchmark; held back as too heavily quantised | Emergency tier only, then dropped |
Qwen3.8-27B, 7 and 30 September 2026, with all 2,072 attempts retained. We also excluded an NVFP4 key-value cache everywhere after a speculative-verify crash and correctness failures on the CUDA-graph path.
The last row is a judgement call we stand by. A W4A16 checkpoint matched the full model on a general benchmark, and we still took it out of production: we would rather tell a user that capacity is unavailable than give them a quietly worse model. The same rule applies to tuning. When 16K batched tokens produced a wrong SQL answer on the gold test, we did not retry until it passed; the setting stayed out.
Each study starts with its gates written down: the thresholds, the readings that count, and what happens if they hold or fail. The serving-levers study had a conditional second phase on larger B200 GPUs, to run only if the RTX PRO 6000 node missed its targets for long-context agent turns. It hit both, including the raw uncached reading, so the B200 phase never ran. The custom decode engine above stopped by its rule in the same way: a milestone missed its pre-set projection, and the all-reduce sprint that followed missed its gate. The discipline cuts the other way too: one cold-start recipe missed its startup gate by 55.6 seconds and was not admitted, and the recipe we use now was accepted only with an explicit, recorded latency waiver.
On 1 November we froze work specific to DeepSeek V4.1. At 475 GiB it needs all eight GPUs, so tensor and expert traffic crosses the CPU sockets over PCIe, which kept tensor-parallel prefill below about 19,000 tokens per second and limited long-context concurrency. A model of about 200 GB fits on the four GPUs behind one socket, giving two replicas per node and far less cross-socket traffic. New open-weight MoEs of that class were expected within days, so we built onboarding as a pipeline rather than another set of model-specific patches.
07 stages
Pin the assets: exact weights revision, engine build and image digest
Watchpoint
A moved tag or a deleted image digest invalidates every later measurement
Debug on a reduced model on CPU before renting GPUs
Watchpoint
Most launch failures are configuration, not capacity
Run the gold test and the agent contracts against a reference
Watchpoint
A component pass rate hides the failures that matter
Measure throughput per lane: bulk and interactive
Watchpoint
One layout rarely wins both
Turn prefix caching on and prove parity
Watchpoint
A leftover flag can switch it off without a trace
Measure a cold-start cohort on fresh machines
Watchpoint
A warm restart flatters every number
Write the report, including what missed its gate
Watchpoint
An unreported miss becomes next quarter's assumption
The pipeline has 45 tests and a full dry run of all eight stages for DeepSeek V4.1, and an offline safety check confirms it cannot reach a provider or the network while it runs locally. It runs CPU-first, and the GPU stages run per model when a release lands. The priors it starts from are the ones above: prefix caching on, data and expert parallelism with session pinning for agent work, and tensor parallelism with speculative decoding for one person waiting.
If you are weighing self-hosted models, measure your own workload before you choose hardware. Log a week of agent traffic and look at prefix length, turns per session and concurrency: that tells you whether prefix caching and session routing will dominate, as they did for us, or whether single-user latency matters more. Then pick one champion engine, write the gates before the runs, and test every quantisation through your real agent loop rather than a leaderboard.
For most organisations the right first step is still a hosted model behind a gateway that can later route to a self-hosted one, as the Twin does. Our guide to multi-model AI without lock-in covers the design, and the Project Venus research page describes the private model.
The Twin answers through a hosted provider or our private model under the same governance, so the choice of model never changes who can see what. See the Company Twin →