Architecture11 min read
Durable agent runs on Electric streams
Record each turn as a run, lease every attempt with a fenced heartbeat, keep agent state in Electric's durable streams, and never replay a tool call to recover.
Mercury Labs
Record each turn as a run, lease every attempt with a fenced heartbeat, keep agent state in Electric's durable streams, and never replay a tool call to recover.
An agent turn can run for many minutes and make dozens of model and tool calls, and we deploy far more often than that. To let turns survive restarts, deploys and dropped clients without losing or repeating work, the Company Twin records every turn as a run in Postgres, executes it under a lease that a heartbeat must keep renewing, and keeps the agent's own state in Electric's durable streams, so any process can resume from the log. Nothing that has already happened is done again to recover. (Electric is an open-source sync engine for Postgres; its Durable Streams project provides replayable HTTP streams, and its agents runtime builds on them.)
A durable agent run is one whose progress is recorded outside the process executing it, so that the process can stop at any moment and another can pick up the work, with a guarantee that the stopped process can no longer write.
This article walks through how a turn is admitted, claimed, executed and recovered; what Electric gives us and what it does not; how blue-green deploys carry running work across a release; and the failures that shaped each rule.
A chat completion is a request and a response. An agent turn is a loop: plan, call a tool, read the result, call the model again, perhaps for ten or twenty minutes while it reconciles a ledger or reads a mailbox. During that time the process running it might be restarted for a release, a host might reboot, the person who asked might close their laptop, and a network call might hang. Each of those is routine. None of them should lose the work, run it twice, or leave a half-written answer that looks finished.
The hard part is not saving progress. It is making sure that when a second process takes over, the first one cannot come back and write over it. That is the job of leases and fencing.
A session owns a task, and each turn in it is a task_run row in the control plane's Postgres database. Sending a message creates the next run on the server side, and returns 409 if the session already has one in progress, so two turns of one conversation can never run at once. Clients never create runs directly and never read the agent's internal streams; they read projections the control plane serves.
When a run is admitted, its limits are written to the run's own event log: a deadline (30 minutes for an ordinary turn) and a budget of tool calls (300). Because they are recorded on the run rather than read from configuration each time, a run admitted under one release keeps its limits if a later release changes the defaults.
Executors claim queued runs from the tenant API in batches. The claim is a single SQL statement that selects eligible rows with FOR NO KEY UPDATE SKIP LOCKED, so concurrent executors never wait on or take the same row, and orders them fairly across tenants so one busy organisation cannot starve a quiet one. Each claim sets the lease owner, a lease expiry 30 seconds ahead, and increments the run's attempt number. A run whose lease has expired is eligible to be claimed again, as a new attempt.
While it works, the executor sends a heartbeat every 10 seconds. The heartbeat locks the run row and checks four things: the run still exists and is not cancelled, the lease owner is this worker, the attempt number is this attempt, and the turn has not passed its deadline. Only then does it extend the lease. Any refusal, or a malformed reply, means this process no longer owns the attempt, and it fences itself: it stops and makes no further writes. Every event, heartbeat and terminal write carries the same identity, so a late write from an old attempt is refused even if the process does not notice.
This is the fencing token pattern from distributed systems, with the attempt number as the token. It does not depend on clocks agreeing or on a process knowing it was paused; the database decides, and the database has one answer.
The agent itself runs on Electric's agents runtime (version 0.6.3 at the time of writing), with an agents server that coordinates entities over durable streams. An entity is one addressable agent with its own stream; in the Twin, each session is one entity. A durable stream is an append-only HTTP log with catch-up reads and live tailing: a reader that falls behind or reconnects reads from its last offset rather than starting again. Each write carries a producer (the writer), an epoch (a generation number: a newer epoch makes the older writer's appends stale) and a sequence number, and the stream server deduplicates on that triple, so a retried append lands once. When new input arrives for an entity, the agents server wakes it: it asks the executor to run the entity's handler on the new input, and each wake starts a new epoch.
That gives us the property the Electric team describe as "the agent is the durable stream": the transcript, the audit trail and the context for the next step are all projections of one ordered log, not separate stores that can disagree. When an executor resumes an entity, it reads the log rather than trusting anything it held in memory.
Three failures taught us how to use it.
| What happened | Why | What we changed |
|---|---|---|
| A new message arriving mid-turn killed the running turn's final write | Waking the entity bumped its epoch, which invalidated the running handler's producer, so its last append was rejected as stale | One wake at a time per entity. New triggers wait until the running turn says it is done or its wake lease (90 s) lapses |
| A restart of the agents server stranded turns until their deadline | In-flight wakes lived only in memory, so nothing delivered them again | In-flight wakes are written to the registry and restored on start; a reconciler fires again any wake whose latest offset was never delivered |
| Two deploy slots could reserve the same epoch | Each slot counted epochs locally | Epochs are reserved in the shared durable store, where the producer triple makes the reservation atomic |
The pattern in all three is the same. Anything that decides ownership must live in shared, durable state, never in one process's memory, because the process is the thing that goes away.
Recovering a turn would be easy if we were willing to run things again. We are not, because a tool call may send an email, post a bill or write a record. So the rule is the opposite: recovery never replays a model call or a tool call.
The executor and its agents server run blue-green, as a pair of slots on each server. A release starts the new slot, waits for its health gate, then routes new work to it. Both resident slots must serve the same release before claims flow, because a run claimed by one release and continued by another is exactly the mixed-version case we want to avoid. The old slot then drains: it claims nothing new and finishes the runs it holds on the release they started on. Its agents server stops only after the drain completes. If the drain cannot be observed or misses its deadline, convergence stops and leaves the healthy new slot routed; it never kills the old slot's dependency to tidy up.
A liveness check is not enough to know a release works. On 1 September 2026 a mixed-version failure in the execution fence passed every health check, because an empty queue looks healthy. So every deploy that touches the control plane, the executor or the Electric runtime now runs a contract canary. It creates an isolated, authenticated task and session, and the executor must carry it through the claim, the claim-time context, the boundary between executor and runtime, entity creation, fence admission and normal completion. The runtime returns a fixed protocol marker before any model would be called, so the canary costs no model call. Only a terminal completion with that exact marker passes; claimed or running is not success. The rollout story across servers is in one customer per server.
Electric also offers Shapes, which sync subsets of Postgres to clients. We evaluated them for our desktop and iPhone apps and decided against them for now. Our clients do not receive tables. They receive server-built, client-safe projections of channels, records, sessions, inbox and files, each filtered by organisation, person and private-channel membership, then refetched through the API that enforces those rules. Shapes would have needed a new gateway that understands our signed subject tokens, our serialisers mapped into shape rows, and matching rules for deletions and schema changes. That is not a swap; it is a second sync engine. The native apps stay on SQLite with a cursor over our own workspace event feed, and we wrote down the conditions under which we would look again.
We also bypass the entity layer for one path. Turns on our self-hosted models, where time to first token matters most, are accepted and persisted in one transaction and dispatched straight to the runtime, which streams from the model directly. Authentication, scope, grants and lease checks all remain; the entity, inbox and stream setup do not. It is a deliberate exception for latency, with its own lease (30 seconds, renewed every two) and its own recovery scanner.
If you are building agents that run for more than a few seconds, durability is a design decision to make on day one. These are the options we would put in front of your team.
| Option | When to use | Trade-offs |
|---|---|---|
| Stateless retries | Short, read-only agents where repeating a call is harmless | Simple, but a retried write is a duplicate and a long turn starts from zero |
| Runs in your database with leases and fencing | Any agent that writes, or runs for minutes | Needs a heartbeat, an attempt number checked on every write, and a rule for unknown outcomes |
| Agent state in a durable log (Electric or similar) | Long, multi-step agents where resuming mid-turn matters | Excellent recovery and audit; ownership decisions must still live in shared state, not memory |
05 stages
Record each turn as a row before any work starts
Watchpoint
Work that exists only in a process dies with it
Lease every attempt and check the attempt number on every write
Watchpoint
Without a fencing token a paused process can overwrite its successor
Write limits and reservations onto the run itself
Watchpoint
Limits read from configuration change under a running turn
Never replay a tool call to recover; report it as unknown
Watchpoint
Replaying an email or a payment is worse than admitting uncertainty
Canary the full path after every deploy, with no model call
Watchpoint
Health checks pass on an empty queue
For how agents earn the right to act at all, see from chatbots to AI agents that do the work, and for why a message from another agent is never permission on its own, talk is not authority.
Every turn in the Company Twin is a durable run: recorded, leased, recoverable and never repeated. See the Company Twin →