Every month, satellites and ships record the surface temperature of the ocean, and these maps build up a history of the climate. In the tropical Pacific, this history is sometimes interrupted by a warm patch, El Niño, which alters weather across the planet. Spotting the patch on a map is easy, but teaching a computer to do it has proven tricky: conventional neural networks require lots of data and long training times. Echo state networks, which resemble a reservoir with memory, do it quickly and with almost no tuning. The catch: until now, no one could explain what exactly the reservoir remembers.
An echo state network works like a pond into which you drop pebbles. Each pebble is a column of the temperature map, and the ripples on the water are the network's response. Training only touches the output layer, so there are few parameters: 301 instead of nearly 88,000 for a conventional network. But how does the pond recognize El Niño? The authors applied a method previously used for deep networks: they traced the answer backward through all 180 steps and looked at which pebbles caused the strongest ripples.
Next, the researchers ran a simple experiment: they shuffled the columns of the map randomly. The network still found the right region, even though the spatial structure was gone. This means that an echo state network does not rely on pixel positions; it extracts features from any order of data. But the reservoir has a parameter that determines how long it remembers the past: the leaking rate. When they increased it from 0.01 to 0.4, accuracy dropped to 58%, and the relevance maps showed that the network had forgotten early columns and was only looking at the end of the map.
Echo state networks are no longer a black box. Climatologists and oceanographers can see what the model is looking at and trust its conclusions. And since such networks train in seconds and require little data, they can be applied to rare events where deep networks overfit. The reservoir with memory no longer hides its secrets: it highlights what matters on its own.
An echo-state network with 300 reservoir neurons required only 301 trainable parameters, whereas a conventional neural network for the same task needed almost 88,000. Yet the accuracy in recognizing El Niño was the same.