A farmer in Iowa looks at the calendar: plant now or wait? In a month, a heat wave might strike, or rain might set in. Forecasters throw up their hands: the two-week forecast still holds, the seasonal one is out of reach, and in between lies a gap. The atmosphere changes too quickly to remember yesterday, and the ocean is too slow to come to the rescue. This stretch from two weeks to two months is called the "predictability desert."
In a desert, as everyone knows, survival goes to those who can find water. Researchers in the United States set out to test which machine learning methods could squeeze even a drop of signal from this arid strip. They took 40 years of data on temperature, soil moisture, ocean temperature, and atmospheric pressure, and tried to predict the average temperature for the third and fourth weeks ahead. Ten models, from simple linear regression to neural networks, competed on the same task.
XGBoost gradient boosting came out on top, a method that builds many weak decision trees and has them correct each other's errors. Its average cosine similarity (a measure of how well the forecast matches reality) was 0.304, compared with 0.201 for the climate norm. That's like getting a map where warm and cold patches lie closer to the truth, instead of the answer "it's usually 20 degrees this month." Interestingly, a linear model with L1 regularization, which automatically selects important features, came close to the complex algorithms.
Why is that? Moist soil acts like a sponge: it soaks up heat by day and releases it at night, smoothing out temperature swings. El Niño, a warm current in the Pacific, changes weather across the whole continent, but it does so unhurriedly, over weeks. The atmosphere, by contrast, is too fidgety: today's cyclone is far away tomorrow, and its trace in the data quickly fades. So models that look for patterns gradually learn to ignore fast noise and listen to the slow hum of ocean and land.
But the desert is not uniform. Forecasts are more accurate in coastal and southern parts of the United States, while in Montana, Wyoming, and Idaho, boosting falls short. There, a neural network with an LSTM encoder, which remembers long sequences, did better. It's like having two guides: one knows every trail in his valley, the other can navigate by the stars in unfamiliar terrain. Together, they can cross the desert more confidently.
The study shows that the predictability desert has oases: soil moisture and oceanic indices. Those planning planting, managing reservoirs, or preparing drought relief can count on more accurate maps a month ahead. And the farmer in Iowa may soon get a forecast that helps him decide when to sow.
The maximal information coefficient between the temperature of the past two weeks and the forecast for the following two weeks was below 0.1 at most locations in the United States. In other words, recent weather says almost nothing about what will happen a month out.