Modern neutrino telescopes, such as IceCube, reconstruct particle properties from sparse Cherenkov light flashes in Antarctic ice. Each event is a variable set of photomultiplier pulses, and direction reconstruction is influenced not only by the light pattern but also by detector geometry, ice properties, noise, and trigger conditions. Foundation models trained on broad detector data create a general representation that different tasks read in different ways. This work sits at the intersection of particle physics and machine learning interpretability: nearby studies have linked learned representations to observables in jets and have used attribution-based explanation methods, but they do not uncover internal concepts. This work is the first attempt to apply sparse autoencoders to a neutrino foundation model, to see which physical structures are encoded in the general representation and which are actually used by a specific head.
The researchers took PolarBERT, an eight-block transformer model pretrained on IceCube pulse sequences and fine-tuned for direction reconstruction. They froze the representation and trained a sparse autoencoder (SAE) with a 4x expansion and a budget of 16 active latents per event on one million events. The validation protocol included three stages: association (finding candidates by label), selectivity (checking on an independent sample with confounding variable control), and causal relevance (interventions: deletion, overwriting, and scaling of latents). To test robustness, the dictionary was retrained with different random seeds and architectures. Additionally, they trained an uncertainty head, a single-layer perceptron predicting the log angular error, and checked which latents are causal for it.
In the final model layer, a validated atlas of physical concepts was found. The strongest feature is bright-clean, which activates on events dominated by standard pulses and high charge; it separates bright-clean events from auxiliary-dominated ones with an AUROC of 0.911. There is also a secondary clean feature, an auxiliary activity feature, and a dense brightness-support feature. At layer 2, an almost monosemantic detector depth feature was found with an AUROC of 0.986. However, the direction head barely uses this atlas: deleting or overwriting features changes the predicted direction by less than 0.1°, comparable to control. Instead, one stable sensitivity axis dominates, without a simple physical name. In contrast, the uncertainty head causally depends on bright-clean features: deleting one latent changes the predicted error by 0.31 on a log scale, corresponding to multiplying the error by 1.36. At 20% selection efficiency, the median angular resolution improves from 20.2° (best detector feature) to 3.2°.
This work shows that 'represented but not used' is a relationship between the general representation and a specific task, not a property of the features themselves. Unlike neighboring works that either explain via attribution or seek interpretable features without causality checks, here a rigorous validation protocol with confounding control and interventions is applied to a neutrino foundation model for the first time. This shifts the focus from 'what the model knows' to 'what the model uses' and provides a tool for discovering underused physics. For foundation models in particle physics, this means the possibility of designing new tasks that explicitly exploit available structure rather than relying on implicit correlations.
The method could extend to other foundation models in particle physics and to real data where true labels are incomplete. Monitoring the preservation of physical concepts during fine-tuning is possible, to avoid their erosion. The uncertainty head could be integrated into a full likelihood analysis to improve event selection. Also promising is training dictionaries oriented toward preserving the behavior of specific heads, and exploring multi-layer atlases to understand where detector geometry is stored.
It will affect interpretability of foundation models in particle physics, event selection in neutrino telescopes, and practices of representation fine-tuning.
Transfer the uncertainty estimate into a full likelihood analysis, repeat the study with models trained on event morphology labels, and extend the protocol to other foundation models.
For neutrino physics, an open question remains: how to extract maximum information from sparse Cherenkov events given incomplete knowledge of detector systematics. This work connects that problem to the model's internal representation: if the model encodes physics but does not use it for a specific task, it points to a potential reserve of accuracy. The problem of interpreting distributed representations, where information is not localized in a single latent, as in the case of event morphology, remains unsolved.
This section is addressed to the author of the paper and their colleagues. Neighbouring works were found by meaning-based search across our archive, not by word overlap. We do not review: we show the field around the work.
What the work does
This work applies sparse autoencoders (SAEs) to the foundational neutrino model PolarBERT for the first time. A dictionary of 1024 latent directions was trained, and a rigorous validation protocol revealed physical concepts: brightness, purity, auxiliary activity, and detector depth. It is shown that the direction head barely uses these features, but an uncertainty head trained on them improves resolution from 20.2° to 3.2° at 20% selection efficiency.
Where its strength is
The strong point is the rigorous validation methodology for interpretable features: association, selectivity, causal interventions, and robustness checks. This provides a reliable atlas of physical concepts and allows quantitative assessment of their use by different heads.
Where it could go next
Worth trying
Why this may matter
Understanding which physical features are encoded but unused opens the way to more efficient architectures: one can either explicitly feed these features to the heads or fine-tune the model to use them. This could improve resolution and event selection in neutrino astronomy, as well as provide a tool for controlling systematic uncertainties.
Nearby works in our archive
The archive is sparse around this work: the closest review is only 0.58, against a usual 0.64. It looks like few have drilled nearby.
The bright-clean latent fires on only 14% of events, yet it turned out to be causal for the uncertainty head, not for the direction head.