Architecture10 min read
Running authored Python workflows safely
Each step runs in a bubblewrap sandbox with no network or secrets, a pinned clock and seeded randomness. Accepted results replay; we say what is not yet proven.
Mercury Labs
Each step runs in a bubblewrap sandbox with no network or secrets, a pinned clock and seeded randomness. Accepted results replay; we say what is not yet proven.
When an agent or an analyst writes Python for a recurring business workflow, we run each step of it in its own bubblewrap sandbox with no network, no secrets and no writable system, under hard limits on time, memory and size. We pin the clock, reject randomness without a seed, and keep every accepted result so a replay reuses it instead of recomputing it. This article is about how we qualify that engine for unattended work in the Company Twin, including where it is not yet ready.
An authored workflow is a small program, written once and reviewed, that runs a recurring piece of business work step by step, with each step's inputs, outputs and effects recorded so it can be tested, replayed and audited.
We are direct about the status. A production-readiness audit on 25 October 2026 judged the engine not ready for unattended, production-critical business workflows, while finding the sandbox and the handling of secrets sound within stated bounds. Most of the work it called for has since landed; the final gate, a full product run with injected failures and a release transition, is still ahead of us. What follows is the contract the engine keeps today and the evidence behind it.
A finance team that wants supplier invoices matched every morning does not want a model to improvise the matching each day. It wants a procedure that does the same thing on the same inputs, that someone can read, and that can be tested on yesterday's documents before it touches today's. Models are useful inside that procedure, for reading a document or judging a near match. They should not be the procedure.
So a workflow in the Twin is Python plus a manifest: the steps, the sources they read, the effects they may have and the trigger that starts them. The Twin's agent can draft one from a sentence, but what runs is the code, saved as an immutable version. A workflow is switched on by its owner, never by the agent that wrote it. This is the idea we set out in skills, not prompts, taken to recurring work.
Every node runs as its own short-lived process, launched by the executor's sandbox role over a private socket. The process is started by bubblewrap with every namespace unshared, so it has no network, no view of other processes and no access to the host's files beyond a read-only system directory and a fresh /dev. Its environment is cleared, so no credential can leak in through a variable, and neither the executor's socket nor any workspace is mounted.
| Limit | Value | How it is enforced |
|---|---|---|
| CPU time | 2 seconds | Kernel resource limit inside the sandbox |
| Wall time | 5 seconds | The parent kills the process |
| Address space | 256 MiB | Kernel resource limit |
| New processes, file writes | None | Resource limits set to zero |
| Input | 512 KiB | Refused before the sandbox starts |
| Output | 256 KiB | The parent stops reading and kills the process |
| The role as a whole | 1 CPU, 1 GiB, 128 processes, no network | Declared on the container that runs the sandbox |
The last row is a declaration in our runtime configuration. The audit noted that a declaration is not a verification on a live host; checking it there is part of the remaining gate.
Before any code runs, the same harness reads it without executing it. Imports are limited to json, math, decimal, datetime, statistics and re. Private attributes and dunder names are refused, as are classes, async functions and globals. The top level may hold only functions, imports and docstrings, and the wiring between steps belongs in the manifest, not in decorators. A check returns the first broken rule with its line and the step it is in, so whoever wrote the code can fix it before saving.
from decimal import Decimalfrom datetime import datetime def match(invoice, purchase_orders): """Pair an invoice with the open purchase order it settles, or say why not.""" candidates = [po for po in purchase_orders if po["supplier"] == invoice["supplier"] and po["status"] == "open"] for po in candidates: if abs(Decimal(po["amount"]) - Decimal(invoice["amount"])) <= Decimal("0.01"): return {"po": po["id"], "matched_at": datetime.now().isoformat()} return {"po": None, "reason": f"{len(candidates)} open orders, none within a penny"}That datetime.now() does not read the host's clock. It returns the run's pinned evaluation time.
A replayed step should produce what it produced the first time. Python does not do that by default, so the harness removes the usual sources of variation:
datetime is replaced with a version whose now, utcnow and today return the evaluation time the run pinned. The inherited bounds, which are base-class objects that could reach the real clock, are replaced too. Clock access without a pinned time is an error.statistics.NormalDist().samples(3) and got different answers. Sampling now requires an explicit seed and is refused without one, while the deterministic statistical functions such as cdf and inv_cdf stay available.Each execution also records a compatibility identity for the runtime it ran on. A step from an older run is resumed only by a runtime that declares the same identity; anything else stops visibly before an effect, and needs a reviewed new activation to continue. There is no permanent fallback that quietly runs old work on new code.
It is easy to say a workflow engine is exactly-once. Ours is not, and we do not claim it. A step's Python, source read or model call runs first and is then accepted: its result is stored against the node, the stage, the effect key and a SHA-256 hash of the canonical request, before any later step can cause an effect. If the process dies after running a step but before its result is accepted, the step can run again. Once accepted, the result is immutable, and any replay with the same request returns it. A replay with a different request under the same identity fails.
| Boundary | What we promise | What we do not |
|---|---|---|
| Python, reads and model calls | An accepted result never changes and is reused on replay | That a step runs only once before acceptance, or that a model gives the same answer twice |
| Worker lease | A late worker cannot commit ordinary effects; owner, attempt and expiry are checked | That an orphaned item completes automatically |
| Local database writes | The write, its receipt and the authority check share one transaction; reusing an identity with other input fails | One execution per business document across separate activations |
| External creates (a draft bill, say) | One create attempt; uncertain outcomes are reconciled by a stable business key and a verified readback | That a lost response can be resolved without a readback |
| Source reads | Current access is checked at every start, read and memo retrieval | That every intermediate upstream revision between two polls is seen |
| Secrets | Provider credentials stay in host adapters; Python receives only declared inputs | That source data itself contains nothing sensitive |
Retries follow the same caution. Only typed transient failures (a 429, 502, 503 or 504) of pure reads or of commands that carry a receipt are retried, up to five attempts with backoff from a quarter of a second to four seconds. A permanent error is reported as one, rather than promised a retry that will never succeed.
A workflow reads through declared sources: a connected system, a channel's records, or the organisation's graph. Each source rechecks the current grants when the run starts, on every read and when a memoised result is retrieved, so access revoked yesterday is not honoured from a cache today. A graph source reads with the run's current warehouse grant, renewed as it nears expiry, and each answer carries the engine's certificate (described in certified answers from a Rust graph engine). The principle is the one in context is a security boundary: what a step may see is decided by policy, not by what the code asks for.
The 25 October audit traced the engine end to end through three independent read-only passes and ran isolated tests. Its rule was strict: an area is ready only with code evidence and no blocking defect, and the overall verdict stays "not ready" while any gap remains. It found ten issues and ranked nine slices of work.
| What it found | What we changed | Status, November 2026 |
|---|---|---|
| Replacing a workflow's active version could strand unfinished work | Every pending and running item is accounted for exactly once at hand-off | Landed |
| A pause or replacement could strand an external write with an unknown outcome | Uncertain postings keep a receipt or an explicit block, with one create attempt | Landed |
| An accepted upload had no recovery if dispatch failed | Dispatch intent is kept in the intake transaction | Landed |
| Temporary failures could become permanent manual work | Typed retry eligibility, and scheduler phases isolated from each other | Landed |
| Older failures could fall out of history and alert no one | Paginated history and one actionable notice to the owner | Landed |
| Replay had no release compatibility contract; sampling was not deterministic | A compatibility identity, refusal before effects, seeded sampling | Landed |
| Bounded capacity was not fairness between tenants | Tenant-aware sandbox admission and claim shares, measured under noisy load | Landed |
| One response could hold totals and rows that disagree | Rows and totals come from one observation | Landed |
| No full product run with crashes, revocation and a release transition | A disposable, exact-commit acceptance run with injected failures | Outstanding |
Two results are worth showing. For fairness, a test ran 100 rounds of 32 noisy jobs from one tenant, one quiet job from another and one cancelled waiter, through a single sandbox slot: in every round the quiet job waited behind none of the noisy ones. For consistency, the audit reproduced a single response whose two rows summed to 30 while its totals said 10, by committing a row between two statements; rows and totals are now produced from one observation. When the slices landed, the integration job ran 309 PostgreSQL and 96 sandbox and runtime readiness tests with none skipped.
The last slice is the one that matters for unattended use: an isolated stack on the exact commit, driven through the real product surfaces, with crashes injected before and after each durable step, a source revoked mid-run, a duplicate upload, and an upgrade and rollback while work is in flight. Until it passes, we do not describe the engine as ready for unattended, production-critical work, and we would not ask you to.
Most of our clients have a handful of recurring processes (matching, chasing, reconciling, reporting) that people do by hand because the last attempt to automate them was too brittle to trust. We would start with one, and qualify it the same way we qualify our own engine.
06 stages
Write the procedure as code with declared sources and effects
Watchpoint
A prompt that does the work is a procedure nobody can read
Run each step with no network, no secrets and hard limits
Watchpoint
Authored code with ambient credentials is a breach waiting for a typo
Pin time and seed randomness, and record the runtime's identity
Watchpoint
A replay that computes something different is not a replay
Accept and keep each result before the next step acts on it
Watchpoint
Recomputing on recovery can change an answer you already used
Create in outside systems once, then reconcile by readback
Watchpoint
A retried create after a lost response is a duplicate bill
Test the whole path with injected crashes before running it unattended
Watchpoint
Component tests do not prove recovery across a release
For finance teams, our earlier piece on agentic workflows on Xero shows the kind of work this is for, and evaluating AI systems at scale covers testing the model calls inside a workflow.
We automate recurring work as reviewed code with a stated contract, and tell you plainly what is and is not yet guaranteed. See how we work →