Delivery11 min read
One customer per server: running our fleet
Each customer's Twin runs as a full stack on its own server, shipped as immutable images through one path: one CI run, staging, then production on one commit.
Mercury Labs
Each customer's Twin runs as a full stack on its own server, shipped as immutable images through one path: one CI run, staging, then production on one commit.
We run the Company Twin with one customer per server: each customer's Twin is a complete stack of its own, on a dedicated machine, with its own database. Every server is deployed the same way, from immutable release images built once from a clean commit, through one governed path: one CI run on the release tag, then staging, then production, all on the same commit. New releases swap in beside the old ones rather than overwrite them, so a bad release is a switch back, not a rebuild.
One customer per server means that every part of the platform that holds or processes a customer's data (the web app, the tenant API and its Postgres database, the agent runtime, the model gateway, the graph engine and the connectors) runs as a separate stack on a server that serves no one else.
It answers the first questions a security lead asks about an AI platform: where does it run, what else shares the machine, and who can change it? Here is how we run it for the Company Twin, and the part that still hurts.
Three reasons, in the order buyers raise them.
Not everything is per customer, and it is better to say so plainly. A central admin host runs sign-in (an OpenID Connect issuer that every customer's app trusts), the public API gateway that the CLI, SDK and MCP clients call, and deployment control. The gateway exchanges a caller's token for a signed subject token and relays the request to that customer's own server, which decides what the caller may read. Workspace data stays on the customer's server; the admin host holds sign-in accounts and the records that route each organisation to its server.
The cost is operational: many servers to keep current instead of one cluster. For the buyer's side of the same question, see four questions on control for finance teams.
A target is one customer's server described as code: a folder of versioned, non-secret configuration that says which services run there, where its release images come from, which backup check must pass before a migration, and which secrets project its settings render from. Secrets live in a secrets manager. At deploy time they are rendered into root-only files for that release, which are never edited in place on the server.
config.env committed, non-secret: services, image registry, backup checkruntime.env names the secrets project its settings render fromsecrets.env never committed, never copied to another targetEvery deployment script is pinned to a target, and there is no default. A command that does not name its customer fails before it reads anything, instead of deploying to whichever box was used last. We also keep apart three identities that are easy to conflate: the deployment target, the tenant's name in the product and the physical host. None is ever inferred from another: the right change on the wrong machine is a common incident, and this is the cheapest defence against it.
Each runtime component (the tenant API, the executor, the model gateway, the app, the public gateway, the integrations worker and so on) has one container recipe with two targets: a production image and a development image. The production image holds only the component's runtime dependencies and compiled output, and runs as an unprivileged user. Built for arm64 in September, they ranged from 385 MB for the public gateway to 1.23 GB for the executor, against 0.8 to 1.9 GB for the development-shaped images they replaced.
A release build has four properties we rely on.
git archive, so the code and the build tooling that run are both that commit's own. A dirty tree is refused.org.opencontainers.image.revision annotation) and with a SHA-256 digest of its inputs: its source files, its resolved dependencies and its recipe. A rebuild of the same commit recomputes the same digest.The manifest also records each role's hardening (a read-only root, dropped capabilities, no new privileges), and activation refuses any image reference that is not pinned by digest.
One path, for every release, on one commit.
Running CI once was deliberate: a second copy of the release tests on every push to main doubled the load on the same runners and told us nothing new. A later article in this series covers how many coding agents feed this one path without colliding.
Deployment itself has one door. A coding agent or an engineer submits intent, never commands, to a private service on the admin host. The request carries a signed assertion that lives for at most 60 seconds and is bound to the caller, the exact route and a hash of the request body. A separate database policy must grant that caller the exact action, targets and services; a valid signature without one is refused. Each request has an idempotency key, so a resubmission after a timeout returns the existing rollout, finished or not, instead of starting a second one.
A single runner then does the work under a five-minute lease renewed every 20 seconds. It prepares one clean snapshot of the admitted commit, gives each target an isolated copy holding only that target's own secrets, and rolls up to six targets at once (eight at most). If one fails, the runner starts no new targets, lets the active ones finish their cutover or rollback, and marks the rest as skipped, so the record is unambiguous. Workstations and customer servers are not deployment principals at all: only the runner can carry out a real deploy, and one started from a laptop fails before it connects to any server.
By swapping in beside it. Each role has a strategy that suits how it fails.
| Strategy | Roles | Why |
|---|---|---|
| Two slots behind nginx; the previous slot keeps running | Web app, tenant API, model gateway | A switch back is instant, and model streams outlive a short stop |
| Paired generations, each drained before it moves | Executor with its agent runtime and Python sandbox | An agent turn can run for up to 30 minutes; a single executor would claim nothing new while it drained |
| One instance, stop then start | Background worker, build worker, graph engine | Exactly one may run at a time |
| Rolling | Integration workers | Connector jobs retry; the first instance runs the scheduler |
| Once per release | Database migration | Runs explicitly before the new roles start; a tenant refuses to serve with pending migrations |
Roles switch in a fixed order under a per-target lock, after a preflight that checks the container runtime, disk and a recent backup. A role passes only when its container is healthy, its health response names the expected release and slot, and its image ID matches the manifest. Each phase has a compensating step, so a failure early in a rollout puts the server back on its previous release. Once the executor's previous generation has started to converge, we halt rather than unwind, because unwinding a half-drained executor is riskier than finishing forwards. Each server keeps its last three releases, so a rollback re-activates a known set of images, settings and digests together.
A started deploy is not a delivered one. Fleet status asks every server, in one parallel round trip, what each service is actually serving, reading the release from the running process rather than from symlinks, and compares it with the commit we expected.
| Rollout | Means |
|---|---|
| COMPLETE | Every selected service serves the expected commit, from a clean build |
| PARTIAL | Some services serve it |
| DRIFT | None does |
| DIRTY | A release marker says the build was not clean |
| UNKNOWN | A marker, active slot or running release could not be read |
| UNREACHABLE | The probe failed, with the reason in the row |
Health is reported separately, as one of four verdicts: healthy, or one of three that each call for a different action. Unhealthy means the serving path is broken: a required unit is down, the health probe fails or run claims are failing. Degraded means the serving path is up but something adjacent is not, such as a failed backup timer or a stale connector. Unknown means we could not reach the server, and it is never shown as healthy.
One more check came from an incident: a run left queued for more than three minutes while workers are active makes the server unhealthy, whatever its units say.
The graph engine, which computes every figure the Twin's agents quote, runs as a single instance per server and holds the organisation's model in memory. Each release stopped it, and while it reloaded, every graph read answered 503. Our telemetry showed the bursts lining up exactly with release changes. With frequent releases that became a visible share of failed answers, and the app gave up after six seconds.
The fix went in two places. The tenant API now waits out an engine that is restarting or still loading, for at most 20 seconds and always within the read's own time limit. On most servers the engine reopens from its snapshot in 3 to 20 seconds, so a release's restart now usually passes unseen. And the public gateway, which used to turn every upstream 5xx into a 502 that no client retries, now passes a 503 that carries Retry-After through as a 503, so the CLI, SDK and MCP clients can wait as the app does. A blue-green slot for the engine is the structural fix, and we have not built it yet: it costs a second copy of the model in memory on every server.
Many small, separate stacks make every singleton visible. Each needs a slot, a drain or a caller that knows how to wait.
Most organisations deploying AI near sensitive data do not need our fleet. They need the same properties at their own scale.
When we deliver a system for a client, we leave this operating model behind with it: the target configuration, the release path and the checks that prove what is serving.
Every customer's Twin runs on its own server, deployed through one governed path. See the Company Twin →