Delivery10 min read
How we run AI research like a ledger
Every study, run and decision is a typed, linked record with a pass line set in advance. Rejections and failed runs are kept as carefully as successes.
George Pullen
Every study, run and decision is a typed, linked record with a pass line set in advance. Rejections and failed runs are kept as carefully as successes.
We run our model research as a ledger. Every programme, study, run and decision is a typed record in one channel of the Company Twin, each study states its hypothesis and pass line before it runs, and each decision cites the evidence behind it. A claim about a model can be traced from the decision that made it to the runs that produced the numbers, including the claims that failed.
A research ledger is an append-only, queryable record of experiments in which every result is linked to the hypothesis it tested, the runs that produced it and the decision taken on it, so that rejections and failures are kept as carefully as successes.
Between July and early November 2026 the ledger collected 13 programmes, 134 studies, 418 runs and 119 decisions for Project Venus, our open-weight model research, run by a small team and a larger number of coding agents. This article describes how it is structured, the rules that make it trustworthy, and what went wrong before those rules existed.
A research team that moves quickly produces results in many places: notebook cells, terminal logs, chat threads, an agent's session transcript, a provider's dashboard. Each is fine on the day. A month later nobody can say which configuration produced the number in the summary, whether the comparison used the same test set, or why an approach was dropped.
Two features of our work made this worse. Much of the experimentation is run by coding agents in long autonomous sessions, so far more runs happen than any person watches. And the questions overlap: a serving change affects an evaluation, which affects a training decision. Without one record, people remember the wins and the agents remember nothing.
The fix is not a better notebook. It is to treat research outcomes as business records, with the same properties we ask of a finance ledger: typed fields, links between entries, corrections that supersede rather than overwrite, and evidence attached to every claim.
The ledger is a channel in the Twin, the same product our clients use for their own records. It has four main data sets and one cross-cutting one.
| Record | What it is | Key fields |
|---|---|---|
| Programme | A durable research lane, such as memory or inference performance | Lane, objective, summary, next gate, status |
| Study | One bounded question, experiment or investigation | Hypothesis, method, result, verdict, data class, production effect, source audit, evidence references |
| Run | One execution: data preparation, training, inference, evaluation, deployment or clean-up | Kind, status, provider and hardware, model revision, dataset reference, configuration, metrics, monitoring, client-data flag, receipt hash, outcome |
| Decision | A ruling on a study or programme, with its reasons | Type, status (current or superseded), rationale, evidence references, the decision it supersedes |
| Sprint | A time-boxed push that cuts across programmes | Goal, dates, outcomes, next steps |
Because they are records rather than prose, the useful questions are views. The decision log lists every ruling by date. Active research shows the studies still running. The run ledger shows every execution with its status, and a run-incidents view shows only the ones that failed, stopped or were lost. Agents read the same views through the Twin's command-line interface before they start new work, so a session can see that an idea was tried and rejected last month.
twin channels --channel research --view decision-log --data \ --fields decision,decision_type,status,decision_date --jsonA decision record reads like a short, dated ruling. This one, abridged, closed the memory study described in can model weights replace RAG?:
{ "decision": "Reject parametric fast-weight memory as a RAG replacement at 0.8–2B; do not run the conditional 4B point", "decision_type": "Reject", "status": "Current", "decision_date": "2026-10-29", "study": "Fast Weight Memory Campaign", "rationale": "Every pre-registered hypothesis failed on sealed test splits at 2B and 0.8B ...", "evidence_refs": ["final report", "decision note", "13 run records", "experiment tracker run"]}A study is written before it runs. The hypothesis states what would count as success in numbers, and the method states how it will be measured: which splits, which comparators, which statistics. Protocols are versioned and frozen before each measurement, and test splits stay sealed until the final comparison. This is pre-registration, borrowed from clinical and social science, and it exists to stop the analysis bending towards the result.
The verdict comes from a fixed vocabulary. Adopt means it becomes the standard. Proceed means the next stage is justified. Pivot means the question changes. Hold means stop until a named condition is met. Reject means the hypothesis failed. Inconclusive means the evidence cannot decide. Decisions add two more: a safety control, which is a standing rule, and supersede.
A pass line is only useful if it is allowed to fail something that nearly passed:
If a team only records what worked, its ledger is a highlights reel. Ours is mostly the opposite.
Holds and rejections make up 45 of the 119 decisions, and pivots another 21. Adoptions are 28. For runs, about a third did not complete normally. Twenty-six never launched, usually because a provider had no capacity, a configuration failed before any machine was allocated, or a prerequisite was missing, and they are recorded because a run that silently never happened is how evidence goes missing. Superseded decisions are kept and marked, so the history of why we changed our minds is still there.
This is also how we judge whether the research process is healthy. A programme with only adoptions is either very lucky or not testing anything hard.
Several of the standing rules exist because something failed first. Each is recorded as a decision of type safety control, with the incident or study that prompted it as evidence.
| What prompted it | The standing rule |
|---|---|
| A provider's own deadline did not stop a rented GPU, which ran for 14.5 hours, and two finished adapters were deleted before anyone inspected them | Paid compute runs under an independent watchdog that tracks exact resource IDs, reserves the run's spend up front, rescues checkpoints before deletion and proves clean-up |
| Early training runs had no live experiment tracking, so their curves could not be checked later | No paid training starts until the run's tracking identity, address and one live metric are proven. Older runs stay labelled as unmonitored instead of being backfilled |
| Agent sessions finished planning, then sat idle for hours waiting for a budget approval | A launch reserves a finite budget and a guarded machine lease before the agent starts. A capacity failure is reported as a failure, not left as a wait |
| A feasibility study of a small local retrieval model raised the question of how much of the governed answer path a learned model could take over | No learned model may replace product binding, policy compilation, execution certificates, typed validation or tenant isolation |
Each rule is a safety-control decision in the ledger.
We record limitations alongside the rules. When launch-time budget binding was adopted, its regression tests passed but the hand-off from a paid machine to a running agent had not yet been proven, and the decision says so.
Pre-registration protects the analysis. Grading needs its own protection, because the easiest grader to reach for is the system that produced the answer.
Any vendor can say its system is accurate. The useful question is what happens when you ask how they know. With a ledger, a claim such as "this model is the default because it passed our agentic contract tests" points to a decision, the study it ruled on, the pass line set in advance and the runs that produced the numbers. It also shows the alternatives that were tried and why they lost.
It matters for the same reason inside a client project. Model and prompt changes are experiments whether or not anyone calls them that, and the decision to ship one should be as traceable as a change to a financial control. We wrote about the same idea for agent skills in a skill is a hypothesis.
You do not need our platform to run research this way, but you do need the records to be structured and shared. We would start with the decisions you are already making about AI (which model, which prompt, whether to fine-tune, whether a feature is ready) and give each one a record with its evidence. Then we would add studies and runs underneath as the work grows. Our guide to evaluating AI systems at scale covers the test sets these records point to.
07 stages
Record decisions first, each with a type, a date, a rationale and evidence links
Watchpoint
A decision without evidence is an opinion with a timestamp
Write the hypothesis and pass line before the study runs
Watchpoint
Thresholds chosen afterwards drift towards the result
Use a fixed verdict vocabulary that includes hold and reject
Watchpoint
If only success has a word, failures go unrecorded
Log every run, including failed, stopped and never-launched ones
Watchpoint
Missing runs are where cherry-picking hides
Grade with something other than the system being graded, blinded where you can
Watchpoint
A self-grader can overstate correctness several times over
Supersede decisions; never edit them
Watchpoint
Overwritten rulings erase the reason you changed course
Keep data and weights out of the ledger; keep hashes and references in it
Watchpoint
A research log that copies client data becomes a liability
The ledger is an ordinary channel in the Company Twin: typed records, views and a command line that people and agents share. See how the Company Twin keeps records like these →