Models11 min read
Can model weights replace RAG? Not yet
Not yet. In a pre-registered test, facts written into a 2B model's weights lost to BM25 retrieval on every hypothesis. Retrieval stays the default.
George Pullen
Not yet. In a pre-registered test, facts written into a 2B model's weights lost to BM25 retrieval on every hypothesis. Retrieval stays the default.
Not yet. In a pre-registered study on Qwen3.5 at 0.8 billion and 2 billion parameters, facts written into the model's weights at test time lost to plain BM25 retrieval on every hypothesis we set before looking at the test data. At 2B it was 18.7 points behind on recalling an updated fact, 67.6 points behind on keeping the latest of several conflicting facts, and it cost the model some of its general ability. Retrieval stays the default for the Twin's memory.
Fast-weight memory (also called test-time training or in-weight memory) stores new information by changing a small part of a model's weights while it reads, so that later questions are answered from the weights instead of from retrieved text.
The idea comes up whenever someone suggests that a company should "just train the model on our data". It is also an active research area, because the newest open models mix attention with recurrent layers that already update a fast internal state as they read. This article describes the study we ran in Project Venus, our open-weight model research, to find out whether that state, or a deliberate write into the weights, can replace a retrieval index for facts that change. It covers the design, what worked, what failed, and what we decided.
Retrieval works, but it has moving parts. You chunk and index documents, choose a search method, tune how many passages to fetch, and pay for every retrieved token in the context window on every question. A model that simply knew the latest state of the business would skip all of that.
There are good reasons to think it might be possible. Work on knowledge editing, starting with ROME and MEMIT, showed that individual facts in a transformer can be located in its feed-forward (MLP) layers and rewritten with a targeted update. Test-time training treats a layer's weights as a memory that is updated by gradient steps while the model reads. And recurrent layers such as Gated DeltaNet, which Qwen3.5 uses alongside attention, run a delta-rule update on a fast state at every token: in effect a small learned memory built into the architecture.
For a company the interesting case is not a static encyclopaedia. It is a stream of updates: a supplier's terms change, a client changes owner, a project's status moves on. The question we set was whether a model could absorb such a stream and answer current-value questions better than a system that retrieves the relevant records at question time.
Memory experiments are easy to flatter. A method tuned on the same questions it is scored on, or compared against a weak baseline, will look good. The campaign, led by George Pullen, was set up to make that hard.
| Hypothesis | What it tests | Pass line |
|---|---|---|
| H1 overwrite | After updates, the model gives the latest value | At least 10 points above recency-aware retrieval |
| H2 propagation | After updates, multi-hop questions use the new fact | At least 10 points above vector retrieval, tested against hybrid (keyword plus vector) retrieval |
| H3 locality | Writing facts does not damage what the model already knew | General multiple-choice drop of 2 points or less; nearby-fact drop of 5 points or less |
The pass lines were deliberately demanding: a memory that only matches retrieval is not worth the operational cost of writing to weights.
Before comparing anything with retrieval, the campaign asked whether the mechanisms worked at all. The answers were more positive than the final verdict suggests.
The recurrent state is working memory, not storage. We wrote a reference implementation of the Gated DeltaNet layer from scratch and matched the library's Qwen3.5 implementation to within 1e-7, so that the state could be reasoned about exactly. Under one-hot keys it holds exactly 128 slots per head, and it decays. That is useful within a long document, but it is not a store for a company's facts.
Facts can be addressed in the MLP layers. A capacity map of candidate write locations showed that keys built from the subject's tokens in an early MLP layer pick out the right fact among 1,024 with top-1 accuracy of at least 0.99, at both model sizes. Finding where a fact lives is not the problem.
Training-free writes failed. Closed-form ridge and Hebbian writes into the MLP's down-projection, in the style of recent in-place test-time training methods, added at most 2.8 points of exact recall over no memory.
A trained write worked. The method we call TTT-MEMIT takes each session's text, runs 25 optimiser steps on its own next-token loss to find value vectors at high-surprisal positions in an early layer's MLP output, then writes them into the down-projection with a whitened delta-rule update and MEMIT's covariance prior, which protects directions the model already uses. On the development split it scored 61 to 79 points above no memory, and it preferred the latest value over the stale one. This is real in-weight memory.
A learned writer did not. A small network trained end to end through a differentiable write (3,000 steps on about 10 million tokens at 0.8B) added 0.6 and 1.9 points on its gate. So TTT-MEMIT went forward alone.
On the sealed test splits it failed every pre-registered hypothesis, at both sizes. The figure shows the 2B differences against the lines they had to clear.
The full 2B results put those differences in context:
| Task | No memory | TTT-MEMIT | Full context | BM25 RAG | Hybrid RAG |
|---|---|---|---|---|---|
| CounterFact exact recall (7,168 questions) | 0.013 | 0.687 | 0.861 | 0.874 | 0.882 |
| CounterFact paraphrased questions | 0.011 | 0.271 | 0.753 | 0.816 | 0.798 |
| CounterFact nearby facts (locality) | 0.254 | 0.178 | 0.316 | 0.368 | 0.375 |
| FactConsolidation, latest value (148 questions) | 0.142 | 0.034 | 0.320 (6k only) | 0.709 | 0.595 |
| MQuAKE multi-hop after edits (5,952 questions) | 0.019 | 0.044 | 0.109 | 0.075 | 0.118 |
Every arm uses the same Qwen3.5 2B reader. Retrieval arms are recency-aware. Scores here are rounded to three places; the differences quoted in the text and figure are paired means from the unrounded scores, so they can differ by 0.1 point from subtracting these figures (67.6 rather than 67.5 on FactConsolidation, 7.5 rather than 7.4 on MQuAKE). At 0.8B the picture is the same: CounterFact exact 0.607 against 0.899 for BM25, FactConsolidation 0.000 against 0.703, MQuAKE 0.036 against 0.126 for hybrid retrieval.
Three patterns stand out.
Every fact written into the weights shares the same parameters with every other fact. Each new write disturbs earlier ones a little, and the disturbance adds up. A retrieval index has no such interference: adding the thousandth record does not change the first.
The trend is the strongest argument against the approach for company data. A real organisation has far more than 1,024 facts, and they change daily. A memory whose recall falls from 0.69 to 0.47 between 256 and 1,024 facts on the sealed test, before any of them are updated, would need a very different mechanism to be usable.
One system did, on one front. As an exploratory arm, outside the pre-registered hypotheses, we fed the same 2B reader from Tencent's Hy-Memory, a learned writer that turns sessions into discrete memory entries which are then retrieved. On multi-hop questions after updates it beat hybrid retrieval by 8.8 points (95% interval 7.6 to 10.0). It lost on paraphrased questions (−6.4), on keeping the latest value (−10.8) and on single-hop questions (−11.0).
Because it was exploratory it cannot change the verdict, but it is instructive. The one method that helped with multi-hop reasoning was not storing facts in the reader's weights at all. It was writing better notes and retrieving them, which is a form of retrieval with more structure.
We rejected parametric fast-weight memory as a replacement for retrieval at 0.8B and 2B, and did not run the conditional 4B point, because the brief released it only on a passing learned write path. Retrieval remains the default for memory in the Twin and in Project Venus.
A result that would reopen the question is specific. A write path would need to generalise from the written text to paraphrased and question forms, keep the latest value through repeated updates at realistic numbers of facts, stay within the locality allowances, and beat recency-aware retrieval by the pre-registered margin on sealed splits. The candidate follow-ups are written down (writes trained on self-generated questions, a band of MEMIT writes across several layers with AlphaEdit's null-space protection, and a hybrid that retrieves first) and none has been started without a new case for it.
The research did change its question. A later review kept the rejection and asked something narrower: not whether weights can replace a store of facts, but whether accumulated experience can improve what an agent chooses to investigate, retrieve, check and prioritise. That work uses a native test-time-training reader, which passed its first parity gate in November, and it is judged against notes plus retrieval, not against no memory. It is in progress and has made no claim yet.
The study supports a split we already make. Facts that change (balances, owners, statuses, dates) live as records in the Company Twin, with their sources, versions and access rules, and the model reads them at question time. What a model can usefully learn is how the business works: its terminology, its policies and when it should say it cannot answer. We explain that distinction in who remembers the company.
There is also a governance reason, separate from accuracy. A fact in a record can be cited, corrected, restricted to the people allowed to see it, and deleted. A fact spread across a layer's weights can do none of those things: you cannot show its source, apply one person's access rules to it, or prove it has gone. Even a much better in-weight memory would need an answer to that before it held company facts.
When someone proposes training a model on your data so that it knows your facts, we would start by asking which questions it should answer, then measure a simple retrieval baseline first. BM25 over well-kept records is cheap, explainable and, as this study shows, hard to beat. If the gap is about behaviour (format, tone, policy, refusing when evidence is missing) rather than knowledge, fine-tuning may be the right tool, and our guide to fine-tuning and RAG covers how to choose.
06 stages
Write the pass lines down before you measure
Watchpoint
A threshold chosen after seeing results is a description, not a test
Compare against strong retrieval with the same reader model
Watchpoint
Beating no memory proves almost nothing
Test paraphrased questions and repeated updates, not only the stored wording
Watchpoint
Exact recall flattered the in-weight method by more than 40 points
Measure what the method damages: nearby facts and general ability
Watchpoint
Writes into weights have side effects that retrieval does not
Grow the number of facts until something breaks
Watchpoint
A memory that recalls 0.69 at 256 facts can fall to 0.47 by 1,024
Ask how a fact will be cited, restricted and deleted
Watchpoint
Accuracy is not the only requirement for company data
If you are weighing retrieval designs themselves, our comparison of RAG and agentic retrieval and our note on embeddings and keyword search pick up from here.
The Twin keeps your company's facts as governed records with sources, and lets any model read them under your access rules. See how the Company Twin keeps facts current →