Researchers from Cambridge and Google trained a 200-million-parameter language model to run with 20 different numbers of layers. Choosing fewer layers uses less computation but made less accurate predictions in their test, giving operators a fixed way to trade compute for quality. The study did not test production-scale models.
Teams serving different compute budgets often train separate models or compress a large model after training. Nested models can offer several sizes in one system, but fixed-exit methods prepare only selected stopping points. In the researchers’ reproduced baseline, prefixes stopped between those points were not trained and often performed poorly.
The new method, called a Telescopic Language Model, keeps the nested Transformer architecture and changes how it is trained. Each step randomly selects an initial set of layers, trains that prefix to predict the next token, then trains the full model on the same batch as an anchor. Repeating this process gives every depth a training signal while preserving a pass that targets the full model’s performance. At inference, the user selects a depth before generation; the method adds no serving-time component.
On a 200-million-parameter proxy trained on 20 billion FineWeb-Edu tokens, using the same data stream for every method, uniform depth sampling produced usable models at 20 layer prefixes. Across those depths, the authors’ area-under-loss-budget score—which aggregates prediction loss across the tested range—fell to 3.28 from 5.73–5.90 across four fixed-exit versions; lower is better. At full depth, validation perplexity was 14.99. Perplexity measures how well a model predicts held-out text, and lower scores are better; the result sits within the fixed-exit versions’ 14.96–15.42 range but above a separately trained 200-million-parameter model’s 13.98.
The result is broader coverage across one model’s depth options, not evidence that it beats a dedicated model at its best size. The study has not tested models at 8 billion parameters or larger, most configurations used one training seed, and code was not public when the paper appeared. It also evaluates a fixed depth chosen before generation, not a system that changes its compute budget mid-answer.
Sources
- Telescopic Language Models, arXiv:2609.35769v1, September 28, 2026. The preprint reports the method, test setup, results, and limitations.
- Matryoshka Language Model Suites, the fixed-exit comparison method discussed in the new paper.
- Server-rack photograph by CSIRO, licensed under CC BY 3.0; illustrative, not part of the study.
