On the CounterFact benchmark, which tests deliberately changed facts, EngramEdit ranked a supplied replacement answer above the original on 97.0% of held-out reworded prompts after 2,000 sequential edits to one model, its authors report. The score records which answer gets a higher model score after accounting for answer length; it does not check which answer is true.
LongCat-Flash-Lite, the model in this study, uses conditional memory: a lookup table of learned vectors retrieved from short groups of adjacent tokens called n-grams. A reworded question can activate different entries, while a frequently reused entry can also affect unrelated prompts.
EngramEdit asks the model to generate four alternative phrasings for each requested change. It then solves for memory changes across the original prompt and those phrases, jointly adjusting the n-gram vectors they share while leaving the Transformer and other model parameters fixed. The method applies larger penalties to shorter or more frequently reused entries to limit spillover.
After 2,000 sequential edits, EngramEdit scored 99.5% on the original edit prompts and 97.0% on held-out paraphrases. MoEEdit scored 99.1% and 67.1%, respectively. These results come from the paper’s single-model experiment, not a comparison across model families.
On CounterFact, a prompt succeeds when the edited model gives the supplied answer a higher score than the original after accounting for answer length. This pairwise test does not measure unrestricted generated responses. Because the benchmark includes intentionally counterfactual changes, the result does not show that EngramEdit can verify real-world updates. The paper is a preprint tested on one model; cross-model and live-fact tests remain open.
Sources
- EngramEdit: Decoupled Knowledge Updates in LLMs through Conditional Memory (original paper)
- Authors’ EngramEdit code and method figure
- Illustration: authors’ method diagram from the repository, used under its MIT License.