main concepts scientists sections
SimplePopularAdvanced

Multi-view Self-distillation of Light Curves with Astrophysical Informedness ✛ ✦

Original: "Domain-Informed Multi-View Self-Distillation for Astronomical Light-Curve Representation Learning with JEPA"
· Yicheng Rui
arXiv:2606.28446 · 2026-06-26 · CC BY 4.0 · 4 min
Instrumentation Stellar Artificial Intelligence
A method for representing irregular astronomical time series is proposed, combining C-RoPE time encoding, measurement uncertainty handling, and LeJEPA self-distillation across three domain projections.

Context

Modern astronomy is entering the era of petabyte-scale data streams: photometric surveys such as the Legacy Survey of Space and Time at the Vera C. Rubin Observatory will record billions of light curves nightly from variable stars, transients, and supernovae. Classifying these objects is fundamental for studying stellar evolution and cosmological measurements via redshift. Hand-crafted features, created since the time of Cecilia Payne-Gaposchkin, remained the standard for decades, but their extraction requires expert knowledge and doesn't scale. Existing deep learning models for time series struggle with extremely irregular sampling, complex noise, and the vast range of timescales inherent in astronomical observations. New approaches are needed that incorporate astrophysical inductive biases directly into the architecture.

Methods

The authors proposed a domain-informed architecture based on machine learning, combining three complementary representations of each light curve: 1) the original irregular series; 2) the generalized Lomb–Scargle periodogram spectrum, revealing periodicities; 3) the phase-folded curve, stacked according to the best period. Each representation uses a separate transformer encoder with two key innovations. Continuous Rotary Position Embedding (C-RoPE) sets the rotation phase of attention vectors proportional to the actual timestamps $t_i$ rather than the ordinal position, encoding the temporal translational invariance of astronomical signals. Error-Aware Numeric Embedding (EANE) treats the measured brightness not as a point but as a Gaussian distribution N($m_{t_i}$, $e_{t_i}^2$) and averages embeddings over neighboring quantized bins with weights proportional to probability, naturally accounting for heteroscedastic noise. The model is trained unsupervised using the LeJEPA loss function, which simultaneously aligns latent representations of different projections (invariant part) and regularizes their distribution to a standard normal (sigma regularization), optimizing the space for linear probing. Pre-training was conducted on the LEAVES dataset (~980k variable stars from ASAS-SN, Gaia, and ZTF), and testing on the StarEmbed benchmark (~40k curves from Catalina Surveys).

Results

The full model outperformed hand-extracted features on 15 of 16 StarEmbed classification metrics using various classifiers (MLP, k-NN, logistic regression, random forest). The gap is especially pronounced in few-shot mode: with one labeled example per class, macro-averaged F1 reaches 42.56 ± 7.21% vs. 30.28% for hand-crafted features; with 100 examples—63.58 ± 1.20% vs. 62.39%. Ablation experiments confirmed the importance of each component: removing C-RoPE or EANE reduces performance; excluding any representation (especially the periodogram) also degrades results. Interestingly, a model with a symmetric CLIP loss underperforms LeJEPA regularization. UMAP visualization of the latent space reveals compact clusters corresponding to variable star classes without using labels. In detecting photometric zero-point drift, the model reliably detects brightness jumps even at low amplitudes of 0.05 magnitudes, with EANE contributing significantly at small drifts. When adapted to 12 irregular PYRREGULAR datasets from other domains, the model achieves parity or superiority on 5 of them, whereas the best predecessor led on only 3.

Implications

The work shows for the first time that a machine learning model without a teacher can outperform expert features in classifying astronomical light curves. This marks a shift from artisanal feature engineering to automated extraction of physically meaningful representations, crucial for handling mega-survey data. The proposed C-RoPE and EANE components can be reused in other tasks with irregular time series and heteroscedastic errors, and the three astrophysical projection strategy highlights the universal need for domain-specific inductive biases instead of a silver-bullet general architecture. This shifts the paradigm from general-purpose transformers to purpose-built encoders for specific domains.

Future development

The method can be extended to multimodal data, e.g., simultaneous photometry and spectroscopy (future missions like JWST or upcoming telescopes), as well as alert streams requiring incremental learning. Replacing hard-coded transformations (periodogram, phase folding) with differentiable learnable modules remains an open problem, promising to eliminate manual feature crafting completely. Integration with physical models predicting light curves (e.g., for exoplanets or supernovae) will enable interpretable models and solve inverse problems of numerical simulation at big data scales.

Impact

The technology will directly impact variable star science, accelerating classification and discovery of rare classes; cosmology, by improving distance estimation via Cepheids and RR Lyrae; and stellar physics, providing a tool for statistical studies of stellar evolution and star formation. Outside astronomy, the adapted architecture is promising for financial time series, medical signals, and geophysical data.

Next steps

Needed are: research on differentiable learnable analogs of periodogram and phase folding to remove manual steps; scaling pre-training to full LSST data; and creating a generalized benchmark for irregular time series spanning multiple scientific domains.

Key open problems

The paper is tangentially linked to the quest for a unified theory of everything via the nature of dark matter and dark energy: precise standardization of candles (Cepheids, supernovae), relying on quality classification of variable objects, remains key in the current Hubble tension.

The knowledge machine's view

This section is addressed to the author of the paper and their colleagues. Neighbouring works were found by meaning-based search across our archive, not by word overlap. We do not review: we show the field around the work.

What the work does

The authors proposed an architecture for self-supervised learning of light curve representations that combines three projections—the unevenly sampled time series, the Lomb–Scargle periodogram, and the phase-folded curve—through transformer encoders with new components, C-RoPE and EANE. The model was pretrained on the LEAVES dataset and tested on StarEmbed, where it outperformed handcrafted features in most metrics, especially in the few-shot regime.

Where its strength is

A strong point is the explicit incorporation of astrophysical inductive biases—uneven sampling, heteroscedastic noise, and periodicity—into the architecture, which enables efficient learning without labels and transfer of representations to new tasks.

Where it could go next

Worth trying

  • Test whether adding augmentations that mimic systematic errors (e.g., trends, outliers) improves the robustness of representations when transferring to other surveys.
  • Evaluate how the representations separate classes of variable stars with similar periods but different physics (e.g., contact binaries and pulsators), using latent space visualization methods.
  • Try fine-tuning the model on a small set of labeled transients to see if the few-shot learning advantage holds for rare classes.

Why this may matter

Further development of such approaches could accelerate the scientific use of large photometric surveys by automatically extracting informative representations without manual labeling and quickly adapting to new object types, which is crucial for discoveries in the LSST era.

Nearby works in our archive

The closest work in the archive sits at 0.63 by meaning — the usual level for our corpus.

Original abstract

as written by the authors arXiv:2606.28446

Light curves describe temporal variations in the brightness of celestial objects. Learning robust representations of light curves is essential for large-scale automatic discovery in the dynamic universe, but existing time-series foundation models often struggle with the uneven sampling, complex noise, and wide range of physical timescales that characterize astronomical observations. We propose a domain-informed representation learning framework for irregular astronomical time series with Joint-Embedding predictive architecture (JEPA), combining semantics-preserving views, uncertainty-aware tokenization, and multi-view self-distillation. The encoders are trained with multi-view self-distillation using LeJEPA regularization on the LEAVES dataset and evaluated on the StarEmbed classification benchmark. On StarEmbed, our model outperforms hand-crafted features on 15 of 16 classification metrics. In few-shot linear probing, it achieves macro-F1 scores of 42.56 $$\pm$$ 7.21 with one sample per class and 63.58 $$\pm$$ 1.20 with 100 samples per class, consistently improving over hand-crafted features. Beyond variable-star classification, the learned representation supports similarity search, parameter estimation, and photometric zero-point drift detection. We further evaluate cross-domain adaptation on 12 heterogeneous irregular time-series datasets from PYRREGULAR, where the adapted variant matches or exceeds previous state-of-the-art performance on 5 datasets, compared with at most 3 wins by any single prior baseline. These results demonstrate that domain-informed multi-view self-distillation is an effective strategy for learning representations of irregular time series, while also highlighting that successful time-series representation learning requires domain-specific inductive biases rather than a universally optimal architecture.

Whole graph →

Variable stars have been known since antiquity: Mira Ceti was recorded by Han dynasty astronomers in 134 BCE, but only in 1908 did Henrietta Leavitt discover the period-luminosity relation, enabling the measurement of the Universe.

L_{LeJEPA} = \lambda L_{SIGReg} + (1-\lambda) L_{inv}
Combination of a mismatch penalty for projections and regularization of the latent vector distribution to a Gaussian.
\omega_d = \omega_0^{-\frac{2(d-1)}{D}}
Frequency dependence on dimension d, ensuring sensitivity to different timescales.
z_{t_i} = \frac{\sum_{k=-R}^R \exp\left(-\frac{1}{2}\left(\frac{c_{Q(m_{t_i})+k} - m_{t_i}}{e_{t_i}}\right)^2\right) \text{Emb}(Q(m_{t_i})+k)}{\sum_{k=-R}^R \exp\left(-\frac{1}{2}\left(\frac{c_{Q(m_{t_i})+k} - m_{t_i}}{e_{t_i}}\right)^2\right)}
Expected embedding when treating a measurement as a normal distribution considering the error.

Key numbers

  • fraction of outperformed metrics: 15 out of 16
  • macro-F1 with one example per class: 42.56 ± 7.21%
  • macro-F1 with 100 examples per class: 63.58 ± 1.20%
  • size of pre-training dataset LEAVES: ~980,000 light curves
  • number of projections: 3
Scientists
Andrey Markov
Concepts
self-supervised learning light curve labeled data active learning stellar age Contrastive Learning supernova light curves