Six of 12 AI agent systems built models that scored above a published reference in a new test of climate modeling. The study used El Niño–Southern Oscillation (ENSO), the irregular warming and cooling of the tropical Pacific that affects weather worldwide. It is an early result from one benchmark, not a real-time climate forecast.
In SciExam for ENSO, each system paired a language model with software for reading data and writing code. The systems had up to six hours of active work to turn observations into a compact, probabilistic model of five linked ocean-and-atmosphere measures. They wrote diagnostic programs to check their own work, then froze those checks before building the model.
Hidden graders scored each final model on three things: whether long simulations reproduced ENSO’s statistical patterns, whether the model could reconstruct variables withheld from it, and whether it predicted temperatures in 2015–2024 better than a simple “no change” forecast. The last test used lead times of one to 12 months. The composite gave the three tests weights of 30%, 30% and 40%.
The leading final model scored 0.515, compared with 0.378 for the published reference; scores are benchmark-specific, not percentages. The reference reproduced long-run climate statistics best, while most AI-built models scored higher on variable reconstruction and the held-out forecasts.
The comparison has limits. The agent systems saw observations from 1980–2014, while the reference had been developed using observations spanning all three evaluation periods. Each of the 12 model-and-software systems was run once, so the result does not measure how consistently they would perform across repeated attempts. And the 2015–2024 forecast test replays historical data; it does not show that the models can issue dependable operational forecasts. The arXiv paper is a preprint, and its results need replication across runs and climate problems.
Sources
- Original paper: “SciExam for ENSO: Can AI Agents Build Climate Models?”
- Authors’ code and released benchmark data
- Illustration: NOAA image of the December 1997 El Niño. NOAA says nearly all its content, including images, is public domain unless otherwise noted; NOAA image-use guidance.