Ai2 launches a shared benchmark for AI climate models

The Allen Institute for AI and a group of climate and machine learning labs have set up AIMIP, a shared benchmark for AI weather and climate models. The motivation is straightforward. AI models can produce a forecast far faster than a physics simulation, but the field had no common way to check whether those forecasts are actually accurate and reliable. Older intercomparison projects were built for conventional climate models and do not ask the questions an AI model raises, such as how well it holds up outside the conditions it was trained on.

Phase 1 fixes the rules so results are comparable. Every model forecasts global atmospheric conditions from 1979 to 2024 after training only on ERA5 observations through 2014, and reports temperature, humidity, and winds at seven atmospheric levels plus surface variables in the standard CMIP format. Ocean and sea ice are prescribed from observations to keep the first round focused on the atmosphere. Six groups, including Ai2, NVIDIA, Google Research, and three universities, submitted eight runs. The headline result is split. The best AI models cut the time-averaged error in fields like near-surface air temperature by about half, but they were uneven on long-term warming trends and on scenarios they had not seen, such as strong ocean warming. Ai2 released the dataset through Germany's DKRZ computing center and posted a preprint, described on the Ai2 blog.

Why it matters

If you train or rely on AI weather models, AIMIP gives you a shared yardstick and a clear warning: these models match the historical record well but get shakier the moment conditions move outside their training data.