Delivery10 min read
Agents on the release train: how we ship
Many coding agents, one release path: a worktree per session, small commits to main, and one orchestrator shipping one commit through one CI run.
Mercury Labs
Many coding agents, one release path: a worktree per session, small commits to main, and one orchestrator shipping one commit through one CI run.
A small team can run many coding agents on a production platform without chaos if writing code and shipping it are kept apart. Each agent session works in its own git worktree and lands small, checked commits straight on main. Exactly one session, the release orchestrator, takes a commit to production: one path, one commit, one CI run. A shared board is where agents report progress and ask people for decisions, and a handful of cheap guardrails catch the mistakes agents make again and again.
The release train, as we use the term, is the single route from main to production: one orchestrator cuts a release tag, CI runs once on it, that exact commit goes to staging and is verified there, and then the same commit goes to production. Nothing reaches a customer any other way.
This is how we build and ship the Company Twin, where most commits are now written by agents. Quality rests on checks rather than trust: each session runs the typecheck and the affected tests before it pushes, the release's one CI run gates production, the author verifies its change on staging, and independent review is encouraged on top. It is the sequel to our piece on the last mile problem for coding agents, which covered keeping one agent on scope. This one is about keeping many of them out of each other's way.
By never sharing a working directory. Every agent session gets its own git worktree, cut from the tip of main, and the rules about other sessions' work are absolute:
This is trunk-based development, and agents make the case for it stronger. They produce change quickly and in parallel, so a branch that lives for a week accumulates conflicts with work nobody has read. Small commits on main keep conflicts small, make every change attributable, and let a bisect find a regression in minutes. Each commit follows Conventional Commits and carries a co-author trailer naming the agent that wrote it, so the history says who, or what, did what.
One shared resource needed its own rule. A local container instance that several sessions test against runs only pushed main, rebuilt by one owner from a clean checkout nobody edits. A session that rebuilt it from its own worktree would replace every other session's code under them, so none may.
Because every other arrangement we tried broke in a way that was hard to see. We started with an automatic release train, which we retired in favour of explicit releases. We built a coordination service that agents went through for release requests; it put another moving part between every agent and every release, and we removed it in September. We tried letting each session stage its own commits; their staging deployments replaced one another, and some commits reached staging without CI.
So one session, the release orchestrator, alone cuts release tags and deploys staging and production. Releasing is procedural rather than creative, which suits one patient session: check the range, cut the tag, wait for Test, stage, verify, roll out in waves. Everyone else pushes to main and hands over.
When a session's change is on main, it hands over without being asked. It files or updates a release record on the board with the commits, the services they deploy and what they need (a migration, a secret, a customer's go-ahead), and messages the orchestrator with the commit and what to verify. The orchestrator batches whatever is ready:
We used to run CI twice: a copy of the release tests on every push to main, then the real run on the tag. The same release went out either way, and the duplicate doubled the load on the same runners. Now the rule is one CI run per release, and precut is how a session gets early feedback before it pushes.
The last step belongs to the session that wrote the change. Once staging serves a release containing its commit, it verifies the affected user flow on staging, comments the proof on the record, and reports the URL, the commit actually serving and the outcome. A pushed commit, a filed request or a queued rollout is not delivery; only the verified flow is.
The board is the Tasks channel of our own Twin, and agents use it through the same CLI any customer would. Each piece of work has one record, not one per session or commit, with a kind and a reference that is never reused:
| Kind | Ref | Holds |
|---|---|---|
| Task or project | W | A piece of work, its owner and what done means |
| Decision | D | A choice a person must make, with the options |
| Release | R | The commits, what they deploy, what they need and the rollout state |
| Check | C | Something to try on a build |
| Incident | I | What broke, with a neutral title when the detail is sensitive |
Fields hold the current state: status and owner; Waiting on and Ask; the agent, its session and its latest update; commits, deploys, needs and rollout; done when, and the evidence. Comments hold the history: progress, findings and proof. Saved views give each reader their slice: what needs me, which agents are at work, the release train, open decisions and what is ready to test.
When an agent needs a person, it sets Waiting on to them and writes the Ask with its recommendation first and the options after, then mentions them, which notifies them in the app and in Slack. People answer by comment in the app or in chat by reference, "D106 a". Nobody has to watch an agent's terminal to know what it is waiting for.
Writes follow the same discipline as any shared database. Agents read before they write, write with the record's version (or an idempotency key for a new record) and read the result back. Even so, two agents once took the same next reference at the same moment; the fix was to retry "take the next number and add" until the board confirms the record was created.
Agents repeat mistakes with a consistency people do not, which is good news: a repeated mistake can be designed out. The most instructive one involved exit codes.
# Wrong: the pipeline's status is tail's, so a failing test still pushes.bun test src/feature | tail -3 && git push # Wrong: the ';' means nothing stops after the check.bun run check:lists > log; echo "exit=$?"; git push # Right: gate the push on the check's own exit code.bun run check:lists > log 2>&1 || { tail log; exit 1; }git pushOver four days, sessions pushed past a failing check this way seven times. Most made no difference. One did: a script carried on past a failed check, because the || exit 1 saw the status of tail. A related slip hid failures in the background: a command written as check > log; echo exit=$? >> log always finishes successfully, because the last command is the echo. And a fresh worktree without its dependencies installed fails every check in a fraction of a second, so a recheck has to run on a checkout that matches what is being pushed.
The guardrails that stuck:
06 stages
Gate every push on the check's own exit code, with pipefail in any chain of scripts
Watchpoint
A filtered or piped check reports the filter's success
Run the fast pre-push list checks before every push
Watchpoint
A new database table lands without the companion decisions its export policy needs
Run the advisory code-health delta before a non-trivial commit
Watchpoint
Raising a size budget to make a warning go away; that needs a person's approval
Start at most about four implementation sessions at once on one workstation
Watchpoint
Load makes timing-sensitive tests fail everywhere and work ends unpushed
Push the first small, checked commit early
Watchpoint
Everything batched behind one final check that never finishes
Let one helper run the full gates for a batch, then rebuild once
Watchpoint
Every helper rebuilding everything, through one queue
The parallel-session cap came from one evening when twelve sessions ran at once on a single workstation. The load average climbed past 20 and at times past 60, tests with five-second timeouts failed in nearly every session, and five sessions ended with correct work stranded as local commits. More sessions did not finish more work; they left more of it one step from main.
One thing is deliberately not a guardrail: asking a person to approve each tool call. The checks are automated and the boundaries are structural. No agent holds a production credential: no server key, no shell on a customer server, no deployment database. The orchestrator, the one session allowed to deploy, submits signed, short-lived requests; a separate runner checks each against a policy and carries it out. Other sessions do not deploy at all, and the governed door is the only way in. That is where safety comes from, not from a stream of approval prompts nobody reads.
Decisions, release timing and anything a customer has to agree to. The orchestrator ships on a person's go, not on its own schedule. A migration's safety review is signed by a person; an agent drafts it and proves it, but does not sign it. And every agent instruction file carries the decisions people have made, with their reasons, so the next session does not relitigate them. Skills and instructions are hypotheses to test, not prompts to tune, as our product team argues in a skill is a hypothesis.
If you are moving from one coding agent to several, the order matters:
We set this up with client engineering teams as part of delivery, alongside the evaluation discipline in evaluating AI systems at scale.
Many agents write the code; one path ships it, and every release is verified where users will see it. Learn about our method →