How Language Models Spot Repeats in Proteins express
Proteins are chains of amino acids, and they often contain repeated chunks. Sometimes the repeats are exact, like the same phrase said twice. Sometimes they come with typos: a couple of letters swapped, but the meaning is the same. These repeats matter: they help the protein fold into the right shape and do its job.
Scientists already knew that language models trained on millions of protein sequences can find these repeats. But exactly how they do it remained a mystery. In the new study, the authors looked inside the model and traced its work step by step. It turns out that the mechanism for inexact repeats is more general: it includes the mechanism for exact repeats, as if the ability to read text with typos automatically gives you the ability to read text without them.
After that, so-called 'induction heads' of attention find identical or similar chunks in different parts of the chain and transfer information between them. This helps the model predict which amino acid should fill a gap. In this way, the language model combines a general pattern-finding mechanism with biological knowledge about amino acid properties.
This finding is a stepping stone toward understanding how models can capture more complex evolutionary processes written in protein sequences. And that means, in the future, they will be better at predicting protein structure and function, which is important for drug development and biotechnology.
Repeats in proteins can be very long — for example, titin, the protein that gives muscles their elasticity, contains hundreds of repeating domains that resemble chain links.