Models9 min read
What our LoRA adapters learned, and didn't
Our LoRA adapters made a 27B model follow the action format without a prompt, 25 of 30 runs against none, but a blinded regrade found only 5 to 7 of 30 correct.
Mercury Labs
Our LoRA adapters made a 27B model follow the action format without a prompt, 25 of 30 runs against none, but a blinded regrade found only 5 to 7 of 30 correct.
Our LoRA adapters taught a 27-billion-parameter open model the Twin's action format, reliably: with no system prompt at all, the adapted model produced usable actions on 25 of 30 held-out real tasks, where the base model managed none. They did not teach it to get the answers right. An independent, blinded regrade found only 5 to 7 of those 30 fully correct, against the 17 to 19 our first grader reported. The adapters learned the protocol, not the job, and telling those apart was the most useful thing we learned.
A LoRA adapter is a small set of low-rank weight matrices trained on top of a frozen base model, so one base can carry several learned behaviours that are attached and detached per request rather than baked into a new copy of the model.
This is part of Project Venus, our research into serving open-weight models for the Company Twin. The research page describes the adapter as trained on real task trajectories. This article is the evidence behind that sentence: two training cycles, the grading mistake that nearly misled us, how the adapter is served, and what we changed.
When the Twin's agent answers a question from company data, it works in small steps: look at which tables exist, describe one, run a read-only query, look at the result, decide what to do next. A frontier model does this from a long system prompt that explains the tools, the rules and the format. A smaller open model, Qwen3.8-27B in our case, struggles with that prompt and pays for it on every call.
The bet was that a LoRA adapter could carry the format and the habits instead, so the model needs no long prompt and spends its context on the evidence. We were explicit about the two things that could improve: format (does it produce a valid action every time?) and substance (does it reach the right answer more often?). The first is much easier to get and much easier to mistake for the second. Our guide to fine-tuning against RAG makes the general case that fine-tuning teaches behaviour, not facts; this is what that looked like in practice.
The first cycle trained on 90 fictional tasks derived from the shape of historical requests, then tested on 29 held-out cases.
| With instructions | No system prompt | |
|---|---|---|
| Base model | 24 | 0 |
| Base model with adapter | 27 | 25 |
Qwen3.8-27B, September 2026. After removing two penalties for harmless extra prose, the instructed scores were 26 and 27.
The prompt-free adapter used 54% fewer input tokens on the 21 cases both arms got right, with no instructions at all. It also regressed in two places: both adapters mishandled a sum where every value was unknown, and each failed one replay where we changed the underlying data and checked that the query still followed it.
Then a review of the test itself changed the reading. The 29 cases used fictional schemas and narrowed versions of real questions, and only one training demonstration involved more than one successful query. The evaluation was far easier than real work, so the result showed that the model had learned the protocol, and nothing about whether it retrieved better. We decided not to scale the same curriculum to 500 tasks and to rebuild on real trajectories instead.
The second cycle drew on real task trajectories. Getting from raw history to a training set took more filtering than training: 7,391 recorded runs gave 774 distinct questions, of which 128 could be supported from the source data, split into 98 for training and 30 held out. A stronger teacher model then worked each training question, and every step was reviewed: the frozen release had 98 trajectories and 650 approved actions. Loss was applied to 115,971 of its 1,039,486 tokens, the model's own actions and decision notes; the rest was context it read but was not trained to write.
The training itself was small: a rank-16 LoRA on 496 projection matrices (116.7 million trainable parameters), three epochs, 39 optimiser updates on one H200, final loss 0.79. Padding was 0.037% of tokens, so sequence packing would have saved almost nothing. We served it as a 5-bit build and ran the 30 held-out questions.
Cycle 2 on 30 held-out real tasks
The first grade came from the model that ran the evaluation harness, judging outputs it had helped produce, and it was generous about answers that completed and looked plausible. The regrade used a different model, blinded to which arm produced each answer, and judged the full intent of each question. It found 16 errors in the keys used to identify records: answers that read well and pointed at the wrong things.
The same review found problems in the comparison itself: a proven harness defect in how cases were admitted, and 13 malformed responses from other models in the comparison whose cause we could not establish, because the harness had discarded the raw response bodies. We did not repair or replace those attempts. They stayed in the record as unknown, and the fix was to capture rejected responses in future rather than guess. A regrade that halves your result is uncomfortable; one you cannot reproduce is worse. Our guide to evaluating AI systems at scale covers the grading practices in more depth.
We keep the adapter's job narrow. Our trial deployment serves an earlier action adapter, trained in August on a different base: an open mixture-of-experts model in FP8, served with SGLang, with the adapter loaded under an internal alias that no user can pick. For each step, the request and the evidence gathered so far go straight to the adapter, which emits exactly one action from a frozen contract. A broker runs it read-only and appends the result. When the adapter returns control, it is detached and the pinned base model writes the answer.
Three design choices follow from what the cycles taught us.
That serving path works. The adapter it was built for did not pass: it loads and runs, but it failed its frozen production behaviour gate, so it remains an authorised trial and not a production model. We keep that status explicit wherever the route appears. Infrastructure that works is not evidence that the model behind it should be admitted.
An independent architecture review looked at what the harness had been trained and tested on. At least 78.6% of the 7,311 runs in the catalogue it examined, a separate count from the 7,391 recorded runs we trained from, turned out to be developer or evaluation traffic rather than real use. Of 50 sampled turns from real use, 28 were follow-ups, and much of their work involved documents, spreadsheets, corrections and dashboards that the narrow evaluation harness excluded by construction. We had been teaching the model a slice of the job that people rarely did on its own.
The recommendation was a shared harness that handles continuations and documents, versioned business context, a shared procedural adapter as the baseline, and organisation-specific adapters only where they show value on top. We have since moved the student onto a minimal adaptation of an existing general-purpose agent loop, as a candidate rather than a proven winner, because the purpose-built harness had a mismatch between its training and serving formats and only three data tools.
The third cycle is a dataset, not a training run: 1,000 examples, each written by one model, approved by a separate one and gated before release, with training a separate decision. Its first source selection approved none of the 1,000: 957 needed correction, 43 were rejected, and a privacy-authority check failed on every one. The corpus stays rejected until that is fixed. It is the gate working, not failing.
We also settled the teacher question. DeepSeek V4.1 Flash is a strong teacher, so it generates training text, but distilling its output probabilities into Qwen3.8-27B is out because the two use different tokenisers. Making the large model the student instead would need an eight-GPU node whose cold start, when we decided this in late October, was about a quarter of an hour, which defeats scaling to zero. A later campaign on different hardware brought that start to about six and a half minutes at p90, as described in cold-starting a large model, which is still far from instant. Qwen3.8-27B stays the model we adapt.
06 stages
Decide in advance whether you are measuring format or substance
Watchpoint
Format gains arrive first and look like everything improving
Test on real tasks, not tidied analogues of them
Watchpoint
Fictional schemas and narrowed questions make any adapter look good
Grade independently and blind, against the question's full intent
Watchpoint
The harness's own grader rewards completion
Keep every failed attempt and its raw response
Watchpoint
Discarded bodies turn defects into permanent unknowns
Audit exposure as well as loss in the training data
Watchpoint
Masked tokens are still read, and can still leak
Hold the model when it fails its behaviour gate, even if serving works
Watchpoint
A working route invites quiet promotion
Before training anything, we would build the evaluation: 30 to 100 real questions from your own work, each with an answer you can verify from source, graded blind by something other than the system being tested. Most fine-tuning projects skip this, and then cannot tell a format gain from a real one. Then we would try the cheaper levers first: better tool design, versioned definitions and retrieval. Our earlier piece on what a Twin should learn from a mistake argues for evaluated, reversible context changes before weight changes.
If an adapter still earns its place, keep its job narrow, serve it beside a clean base model, and treat it like any other release: it ships only through its gates. That is the discipline behind why we serve our own open-weight models, and the Project Venus research page shows where it is heading.
In the Twin, live figures stay in your governed data and models learn only how the work is done, so every answer still shows its sources. See the Company Twin →