Researchers have introduced a way to combine task-specific Low-Rank Adaptation (LoRA) adapters—small weight updates that tune a language model for separate jobs—inside one model, avoiding the need to choose among separate adapters each time it runs. The arXiv preprint calls it READ (Read-only Expansion of Adapter Deltas) and reports benchmark results.
Separately trained LoRA updates can interfere when merged directly. Keeping each adapter separate avoids that merge, but the serving system must choose which one to use for a given task.
READ puts each adapter’s two-part update into a common form without changing what the update does on its own. One factor captures patterns in the model’s input; the other turns those patterns into changes to the model. When a task is added, READ lets it reuse earlier input patterns only to make its own task-specific changes. It does not let the new task change the outputs of earlier adapters, so their learned functions remain fixed.
Only the new adapter’s connections are trained. Each addition still takes one pass through data from both old and new tasks, so the method requires access to prior task data and is not data-free. After training, the authors fold the combined update into the model weights, removing the need to choose among adapters at inference.
The tests used Llama-3.2-3B-Instruct and Qwen3-4B on four classification-focused suites: GLUE, SuperGLUE, BBH and Domain. Across 32 test lineages—combinations of model, benchmark suite, random seed and task order—READ beat the strongest of 14 foldable alternatives built from the same task adapters in 24. All eight losses occurred on BBH.
The result is limited to those two model families, four suites and groups of up to six tasks. The authors identify open-ended generation and larger adapter groups as tests still to come.
Sources
- “New LoRA Skills Should Read but Never Write,” arXiv:2609.31600
- Image: Generic neural-network diagram by QuantuMechaniX8, released under CC0 1.0 via Wikimedia Commons.
