Delivery10 min read
Monitoring AI agents with ClickStack
Log what explains any run without storing what was said: origin, release, model calls and refusals, as OpenTelemetry events. Missing data reads as unknown.
Mercury Labs
Log what explains any run without storing what was said: origin, release, model calls and refusals, as OpenTelemetry events. Missing data reads as unknown.
To understand an AI platform in production, log enough to explain any single run after the fact without storing what anyone said: which client started it, which release served it, which model calls it made and how many tokens they used, what it read and what it was refused. Send that from every server as structured OpenTelemetry events into one store, show missing data as unknown rather than healthy, and watch the monitor from somewhere else. We do this with ClickStack, and the first audit of a few days of production data found five problems no user had reported.
Agent observability is the operational record of what an AI system did (runs, model calls, tool reads, refusals, releases and failures), kept as structured, content-free events that can be correlated across services, so you can tell what broke, where and for whom.
This article describes how we monitor the Company Twin across a fleet in which every customer has its own server, the event contract that makes the data usable, what the data showed us, and the fields we wish we had logged from the start.
ClickStack is ClickHouse's open observability stack: the ClickHouse database for storage, HyperDX for search and dashboards, and the OpenTelemetry collector for shipping data. We run one instance on an existing server rather than buying another. The obvious objection, that monitoring on a shared box can fail with it, is answered by an independent observer on a different host, described below.
Every registered Linux host in the fleet sends four things: a heartbeat, resource metrics, bounded operational logs and an observation of which release it is serving. The collector on each host is capped at 128 MiB of memory and a fifth of a CPU, with a persistent queue on disk, so a store outage means a backlog that replays, not lost data. We tested that by stopping the store and watching the queues drain on recovery.
The store is fenced in the same way. It runs in a slice limited to four CPUs and 8 GiB, on a dedicated filesystem, so filling it stops ingestion instead of filling the host's root disk. Thirty days stay hot; complete days are exported to object storage as Parquet before they expire.
Two access rules matter more than the technology. The receiver gives each host its own mutual-TLS identity and strips any host or organisation claim a sender makes about itself, so one compromised server cannot write records that appear to come from another. And organisation-scoped export stays switched off until the mapping from authenticated context to organisation exists and negative tests prove one customer cannot see another's rows. Adding an organisation ID to a JSON field is not isolation.
Engineers and their coding agents query the store as themselves, through a read-only reader exposed as a CLI command and an MCP server. It accepts one statement at a time, each limited to 10 seconds, 512 MiB of memory and 10,000 rows, with four concurrent queries across everyone. Each call is journaled with the login and a hash of the SQL, never the text.
We use one versioned event contract for every service and every customer, built on the OpenTelemetry log data model rather than a format of our own. Customer-specific event names, formats or parsing branches are not allowed: a dashboard that works for one server works for all of them.
| Field | What it holds |
|---|---|
| Timestamp and ObservedTimestamp | When it happened, and when it was collected, kept apart so ingest lag and clock skew are measurable |
| EventName | A stable name from a registry, such as a connector job failing; never derived from a message or a tenant |
| Body | A registered safe template or event code; no free text |
| service.name and service.instance.id | The logical component, and the actual slot or process that emitted it |
| service.version | The immutable commit the process is actually serving |
| Correlation IDs | Request, run, job and diagnostic IDs where they apply, bounded and validated |
| Outcome, duration, error code | An outcome from a fixed list, milliseconds, and a classified failure code |
Each event family then adds typed fields. Requests carry the route template, method, status and duration, but no query strings, headers or bodies. Model gateway events carry the provider and model, latency, status, token usage and a classified error, but no prompts or responses. Database events carry a readiness outcome and a classified error, but no SQL, bind values or rows. Runs carry their lifecycle stage and correlation to the request that started them.
Two rules prevent a subtler kind of wrong data. Nothing is synthesised: a missing release, duration or organisation is left missing, never filled with a plausible default. And severity alone never pages anyone. Alert rules key on registered events, duration, repetition and recovery; the word "error" in a log line is not an incident. Our connector alert, for example, fires only after three consecutive distinct failed attempts of the same job, and a success clears it.
Because the most dangerous dashboard is a green one with no data behind it. A host whose collector has stopped looks exactly like a quiet, healthy host unless the system knows what it expected to hear. So every source declares its coverage, and a running collector alone does not count as coverage. A server excluded from background probes is stored as reachability unknown, health unknown and no claim about its release.
The store itself is checked from outside. Our fleet observer, on a separate host, runs a small root-owned check on the store's server: is the container healthy, is HyperDX ready, and is real heartbeat data from local and remote hosts still arriving? Missing data, a missing binary or a timeout all fail the check, which returns only fixed failure codes. Failures go to the alert channel by the observer's own route, independent of HyperDX. We tested it by stopping the store: the outage alert and the recovery both arrived.
About a week after enrolment we read four days of production data across the fleet. The events hold no message text, so some findings named a symptom without its cause, but every one was actionable.
The quiet results mattered too. No host had a heartbeat gap over 70 seconds, the parser rejected nothing, exporters dropped nothing and model calls almost never failed. A clean bill of health is only worth something when you can show the data behind it.
The audit was as useful for what it could not answer. These are the gaps it exposed, in the order we would fix them on a new system. Since the audit we have added run origin, the client kind on gateway rows, and each unit's memory against its own cap, with an alert; the others are still to do on ours.
09 stages
Run origin: app, API key, schedule, workflow or test
Watchpoint
Customer usage and your own test traffic look identical
Client kind on every gateway row: CLI, desktop, MCP or SDK
Watchpoint
Throttling bursts cannot be attributed
The serving release on every row, from every service
Watchpoint
You cannot line a regression up with the release that caused it
Per-unit memory and CPU, not just per host
Watchpoint
A process hits its own cap while the host looks fine
One chain from request to run to each model call
Watchpoint
Cost and latency stop at the gateway and cannot be tied to a user action
Collection time beside event time
Watchpoint
Ingest lag is invisible
Classified database and connector errors, not just a priority
Watchpoint
Thousands of warnings with nothing to say what they are
Lifecycle events for anything that loads: started, ready, how long
Watchpoint
A reloading engine raises no alert, only failed reads
Browser-side errors
Watchpoint
Client loops are found only from the server
Generic service dashboards (request rate, errors, latency) are necessary and nowhere near sufficient for an agent platform, where one user question can become a ten-minute run with sixty model calls. The views we use most are:
These are the operational counterpart of a per-answer receipt: telemetry says how the system behaved, and the receipt, described in every AI answer needs a receipt, says why one answer can be trusted. You need both, and you should not try to make one do the other's job.
Most teams putting agents into production already have a logging tool. What they usually lack is the contract and the discipline around it. On a client engagement we would:
This is the same thinking as our approach to evaluating AI systems: an AI system you cannot measure is one you cannot safely change.
Every Twin server reports what it serves and how its agents behave, without storing what anyone said. See the Company Twin →