Researchers at the University of Illinois Urbana-Champaign have developed a method that learns when language-model forecasts add useful information to an existing market, crowd or statistical prediction. Across 2,357 resolved yes-or-no questions, their competence gate lowered the main external forecast's Brier score from 0.0771 to 0.0732. The Brier score measures the squared gap between a predicted probability and the eventual outcome; lower values are better. The September 10 paper is an unreviewed preprint.
A forecasting service may already have a probability from specialists or observed market prices before it asks a language model. Adding the model improves the final number only when its disagreement tends to correct errors in that existing forecast. A model can score well alone and still duplicate information already present. Its own confidence describes certainty in its answer; the gate needs evidence that this answer improves on the available alternative.
For each new question, the system receives one external probability and independent probabilities from language models. It groups the question by source domain, then applies source weights learned from resolved training questions in that domain. Estimates from smaller domains are pulled toward the global average. The system combines the weighted numbers, clips the output to the zero-to-one probability range and recalibrates the result. The external forecast is withheld from the language models to preserve an independent signal.
The evaluation used questions dated from July 2024 to April 2026 from two prediction markets, two forecasting communities and four structured-data sources. The five models were Qwen2.5 at 7, 14 and 32 billion parameters, Qwen3-8B and Gemini-2.5-flash. The researchers used five-fold cross-validation, so every question was scored in a held-out fold. They retained actual market and crowd probabilities; structured sources received base rates calculated only from each training fold. A separate test covered 1,294 questions with genuine market probabilities.
In the held-out folds, one global two-source mixture scored 0.0767 and a global multi-source pool scored 0.0759; neither improvement over 0.0771 was statistically significant. The domain-specific gate reached 0.0732, with p=0.001 against the external baseline and p=0.010 against the global pool. On 2,103 questions selected to reduce training-data leakage risk, it lowered the score from 0.0805 to 0.0767. On the 1,294-question live-market subset, its gain was not significant and it assigned little weight to the models. Verbal confidence from four Qwen models produced area-under-the-curve scores of 0.389 to 0.501 for identifying relative advantage.
The study used historical questions, excluded retrieval and represented each forecasting source as a broad domain. It did not measure latency, token use or provider cost, while several structured sources had too few questions for separate conclusions. The gate also needs resolved examples before it can estimate a domain's weights. A prospective trial should freeze those weights before new questions arrive, then report Brier score, deferral rate, recalibration, inference cost and performance on previously unseen domains.
Sources
- Aditi Tiwari, Aashrith Bandaru and Heng Ji, arXiv, September 10, 2026. Abstract, authorship, submission date and headline results.
- Full paper. Gating sequence, data sources, evaluation protocol, results and limitations.
- Lead image: Figure 2 from Tiwari and colleagues, reproduced from the paper under CC BY 4.0.