Models11 min read
Cold-starting a 475 GiB model in minutes
Removing work from the start path took a 475 GiB model's cold start from 33 minutes to 388 s at p90 on fresh machines. It still missed its 300 s gate.
Mercury Labs
Removing work from the start path took a 475 GiB model's cold start from 33 minutes to 388 s at p90 on fresh machines. It still missed its 300 s gate.
We bring a 475 GiB open-weight model from no machine at all to serving in about six and a half minutes at p90, about 3.7 times faster than the 23.6 minutes an untuned start took on the same hardware. We did it by taking work off the start path rather than doing it faster: a boot image with the engine already in it, weights on a filesystem in the same region, a replay store for kernel autotuning, persisted compiler caches and no profiling-time graph capture. We missed our 300-second target and accepted the result with a recorded waiver, because most of what remains is the cloud provider creating and booting the machine.
A cold start is the time from asking for a model to its first served token when nothing is running: placing and booting a GPU machine, fetching the inference engine, loading the weights, and preparing the engine's kernels and graphs. For a model that fills eight GPUs, each of those steps is measured in minutes.
The model is DeepSeek V4.1 Flash, an open-weight mixture-of-experts model, served with vLLM as part of Project Venus, our research into running private models for the Company Twin. Today that private model serves from dedicated UK or EU infrastructure, as our research page says. The studies below ran on two kinds of eight-GPU node in two regions, and we say which as we go.
Because an eight-GPU node waiting for questions costs the same as one answering them. Agent work arrives in bursts: a team runs an analysis, an evaluation sweep runs overnight, and then nothing happens for hours. Keeping the node warm is a legitimate choice, but it is an always-on cost decision, and we wanted to know how close a node started on demand could come before making it.
The product shape is simple. A person confirms that they want the private model, a node starts and the app shows its progress stage by stage. After 30 idle minutes the node is deleted.
The first end-to-end cold start ran on an eight-GPU NVIDIA B300 spot node from Nebius in a US region. From request to serving it took 1,978 seconds. The spot node was placed, booted and serving TLS in 4.9 minutes; it pulled the 22.8 GB engine image in 6.8; it read the checkpoint from a network filesystem in 7.2, at about 1.1 GB per second; and it spent 13.1 minutes compiling, autotuning kernels and capturing CUDA graphs before it could answer.
Where a cold start's time goes
Every one of those stages repeats work that was identical last time. The image does not change between starts, the weights do not change, and the kernel choices for a given model, engine build and GPU do not change either. That framed the campaign: first stop repeating work, then make what is left faster.
A boot image with the pinned engine image already pulled removed the image pull, which took 6.8 minutes on the first attempt and 524 seconds in another run; checking the image on the GPU node now takes about a second. The weights live on a filesystem in the same region as the GPUs, which removed a 20 to 25 minute download from the model hub, and a prefetching loader reads them in parallel. On the best B300 run the weights loaded in 93 seconds, 130 on the slowest of the four engine ranks.
The largest single stage was kernel autotuning. FlashInfer profiles candidate kernels for the shapes the MoE layers will see and picks the fastest, which is right for performance and wasteful when the answer is the same on every node. It already has a file cache for those choices. Yet when we loaded 152 saved configurations, tuning ran again anyway. Reading the source showed why: the MoE kernel asks for a cold-L2 profiling policy, and the cache did not apply to that policy, so every node re-profiled for about eight minutes.
We wrote a small replay store keyed by the profiling policy and installed it in every engine worker. In the lab it replayed 136 serving cases with zero profiling calls; without it, a capture-enabled start had made 29,918 MoE profiling calls per rank. The largest relative error between a replayed choice and the fallback was 0.002, against a gate of 0.02. Compiler caches for vLLM and FlashInfer moved to one directory on the same filesystem, so a later node reuses an earlier one's compilation.
On a fresh machine the qualified recipe reached native serving 430.6 seconds after its container started and 723.3 seconds after admission, with zero profiling calls on all four ranks. It passed native correctness and a retrieval at 127,998 tokens of context, and its throughput stayed within 4% of an unmodified engine. Our controller reproduced it in the product path: engine start to serving in 417 seconds, ready 797 seconds after the request, with a 543-millisecond first token on the verification call.
The pre-registered gate was 375 seconds from container to native. The recipe missed it by 55.6 seconds, so the confirmation runs it would have unlocked were not admitted. The recipe still went into the product because it was 2.5 times faster than what was there; it did not get to call itself qualified.
Three ideas that sounded right made things slower or proved nothing.
Spot capacity stopped holding. From 28 October, B300 spot nodes in the US region were reclaimed two to four minutes after creation, including when we bid the highest price the provider accepts; they were capacity reclaims, not outbids, so no bid could fix them. On-demand B300 capacity was rarely available. Eight-GPU RTX PRO 6000 Blackwell nodes in a UK region had high on-demand and spot availability, and their 768 GB holds the checkpoint across eight GPUs, so on 31 October we qualified the model there and moved the study.
The move reset the clock. Untuned, a fresh RTX PRO 6000 node took about 23.6 minutes from create to ready. Graph capture ran twice, once during the engine's memory-profiling pass (258 seconds) and again for serving (319 seconds), and 267 seconds were not attributed to any stage at all.
Two short campaigns on the new hardware produced a frozen recipe. Each change removes work or overlaps it with something that has to happen anyway.
| What it removes | |
|---|---|
| Explicit key-value cache size | The profiling pass and its graph capture: the engine is told how much memory to use |
| CUDA graphs captured up to batch 64 | Capture for batch sizes the interactive lane does not use |
| Local FlashInfer calibration store for this GPU | Re-profiling kernels on every start |
| JIT warm-up off; persistent TileLang kernel cache | Recompiling kernels that compiled last time |
| Baked image with precompiled bytecode | The image pull and Python's first-import compilation |
| Weight reads overlapped with imports and device set-up | Waiting for one before starting the other |
| The engine's API server launched directly | A wrapper process in the start path |
Five fresh on-demand starts, each after at least 30 minutes with no machine running: p50 381 s, p90 388 s, spread 19 s. The gold correctness test passed 5 of 5 and the throughput guard 15 of 15.
The breakdown explains where the floor is. Creating the machine and reaching its kernel took about 150 seconds, including about 55 seconds to create its boot disk; booting to our program took about 63; the engine pipeline took about 166 at p90. Even with a zero-second engine, the p90 would be about 223 seconds, so a three-minute start is not reachable on freshly created machines. We revised the gate to 300 seconds in the open, and missed that too, by about 88.
Restoring a checkpoint of the running engine does not rescue it either. A full native checkpoint is about 790 GB, and at the 6.3 GiB per second we measured from storage it takes at least 117 seconds to read, which does not beat a 166-second pipeline by enough to matter. One correctness check failed once during the campaign, and we did not relax it.
So the result was accepted with an explicit latency waiver. It is consistent, with a 19-second spread over five starts; every second of it is explained by a stage; and it is about 3.7 times faster than the untuned baseline on the same hardware. One caveat stays attached: the recipe was measured on the tensor-parallel interactive layout, and the data-parallel layout we use for bulk work, described in our serving measurements, needs its own cold-start check before anyone relies on it.
A fast start is only useful if the system around it is safe to leave alone. Each node is governed by a lease held in the Twin, and one narrow controller role is the only thing that holds the cloud credentials.
The edges found real bugs. On the first attempt through the app, the controller probed the new node too early, mistook it for a spot eviction and deleted it in the middle of being created. We fixed it before the next attempt. The lesson generalises: never let a health probe judge a machine that is still being created.
Three things, each a decision rather than an optimisation. A warm node removes the cold start, at the cost of idle time. Starting the node on intent, such as opening the new-session page, hides much of the wait while someone writes a question. And smaller models help most of all: a model of about 200 GB fits on the four GPUs behind one CPU socket, so it reads half the weights and needs no cross-socket communication. That is one reason we keep the 27-billion-parameter model as the one we adapt and serve by default, as we explained in what our LoRA adapters learned.
If you run models yourself, measure the cold start you actually have before choosing between warm capacity and scale-to-zero. Time it from the request on a fresh machine, attribute every stage, and look for work that repeats: image pulls, weight downloads, kernel autotuning and graph capture are usually most of it, and all four can be cached or removed. Then decide on evidence whether minutes of start time are acceptable for your users or whether warm capacity, pre-warming on intent or a smaller model is the better trade.
Whatever you choose, put the lifecycle in a lease with hard deadlines, exact clean-up and visible stages, and make the gateway refuse rather than reroute when the model is not there. Why we serve our own open-weight models sets out when that is worth the work.
The Twin can answer through a hosted provider or our private model, with the same sources and access rules, and shows you which one did. See the Company Twin →