An 8.9-billion-parameter model learned to predict compressed concepts alongside the next token.
It reached OLMo-3-7B's final Stage 1 loss after using 51.3% of the training tokens on the same data.
The figure measures token convergence, not elapsed training time or computing cost.
Researchers from Shanghai AI Laboratory and Shanghai Jiao Tong University have released NCP-ArchPreview, a language model that learns at two levels. According to a technical report posted on September 9, the model predicts the next token while also predicting a compressed representation built from a group of tokens.
A conventional language model works with tokens, which can be whole words or fragments of words. NCP keeps that process but adds a second path. Its encoder first reads the tokens, then averages every four token representations into a larger unit. Learned codebooks convert that unit into discrete codes, and an eight-layer module predicts which coded concept should come next. A decoder combines the concept sequence with the original token information to generate ordinary text.
The team trained the 8.9-billion-parameter model on 5.73 trillion tokens from the Dolma 3 corpus. It was compared with OLMo-3-7B Stage 1, which was trained on the same data. NCP reached the comparison model's final training loss after processing 51.3% as many tokens. The authors describe this as a 1.95-fold improvement in convergence measured by token budget.
Because NCP performs additional work for the concept path, the token figure does not establish an equivalent reduction in training time or cost. The report provides no matched-hardware comparison of elapsed training time or energy use. The measured result is that NCP's loss curve reached the reference point earlier in the stream of training data.
At the end of Stage 1, NCP averaged 49.04 across the reported evaluations, compared with 46.59 for OLMo-3-7B. Its GSM8K score was 45.26 against 39.27, while HumanEval rose from 27.10 to 31.38. After a later mid-training phase, however, the overall advantage narrowed to 0.59 percentage points and differed across benchmarks.
The concept module also produced a smaller gain during generation. In a speculative decoding test, the mean number of draft tokens accepted at each step increased from 5.933 to 6.180, a relative improvement of 4.17%. The authors report negligible extra overhead in that experiment.
The model weights are available under the Apache 2.0 licence, together with evaluation code and checkpoints. The training code is still listed as forthcoming. The report is a first-version preprint, has not been peer reviewed, and the preview model was not trained for long-context use.
NCP-ArchPreview gives researchers a working model for learning from individual tokens and compressed concepts at the same time. Its practical value will depend on matched-hardware measurements of training time and energy, long-context tests, and independent reproduction. Those results will show whether the lower token requirement becomes a real saving beyond the training-loss curve.
Sources
NCP-ArchPreview technical report
Full research paper
NCP-ArchPreview model card and weights