main concepts scientists sections
SimplePopularAdvanced

How to Focus on Small Data ✎

Original: "Gauge-optimal approximate learning for small data classification problems"
· Edoardo Vecchi, Davide Bassetti, Fabio Graziato, Lukas Pospisil, Illia Horenko
arXiv:2310.19066 · 2023-10-29 · CC BY-NC-SA 4.0 · independent analysis · 2 min
Machine Learning Artificial Intelligence Statistical ML
The GOAL algorithm rotates and compresses feature space to classify data with fewer samples than dimensions.

An optician turns the lens barrel, catches the focus, and a blurry blob becomes a sharp image. The GOAL algorithm works much the same way: it rotates the data cloud, compresses redundant dimensions, and brings into focus what truly matters.

When there are more features than observations, standard methods feel like trying to see a face through a fogged-up window: noise drowns out the signal. GOAL takes a different approach. It simultaneously rotates the coordinate system to discard empty directions, partitions the space into cells, and fits a classifier so that predictions match the real labels. Each step follows closed-form formulas, with no lengthy tuning, and convergence is proven mathematically.

In forecasting the warm phase of El Niño from 295 200 features and only 720 observations, GOAL achieved an accuracy of about 0.9 (AUC) for a 24-month horizon. That is like guessing a melody from 720 notes out of a 295 200-line score.

The secret is that the algorithm does not try to grasp the ungraspable. The rotation is computed via singular value decomposition, the same trick used to compress photos. Cell membership is assigned by nearest center with a label correction, and label probabilities are updated by frequencies. All this forms a single loop where each iteration sharpens the picture a bit more.

On reconstructing gene activity networks from 182 genes, GOAL came out on top in 82.7% of 910 scenarios, taking 0.019 seconds where random forest took 9.84 seconds and LSTM took 39.88. This speed advantage means analyses can run on a regular laptop rather than a cluster.

Before GOAL, no single method could simultaneously reduce dimensionality, partition space, and classify small data without overfitting. Now such a tool exists, and it has already proven itself in climatology and bioinformatics.

The optician puts the lens back in place: now you see not just outlines but details. GOAL does the same for data that are few but noisy. And where such tasks once demanded expensive computation, now the focus is achieved in hundredths of a second.

Whole graph →

For the El Niño forecast, 295,200 features were used with only 720 observations, which is like trying to guess a melody from 720 notes when the score has 295,200 lines.

L_{GOAL} = \frac{1}{T}\sum_{d=1}^D\sum_{t=1}^T (X_{d,t} - \{RS\Gamma\}_{d,t})^2 - \frac{\epsilon_{CL}}{TM}\sum_{m=1}^M\sum_{t=1}^T \Pi_{m,t}\log\left(\sum_{k=1}^K \Lambda_{m,k}\Gamma_{k,t}\right)
The first term accounts for the quality of rotation and partitioning, the second for the match between predicted label probabilities and true ones.
R^* = U \tilde{I} V^\top, \quad X\Gamma^\top S^\top = U\Sigma V^\top
The rotation into a lower-dimensional space is computed analytically from the singular value decomposition, which speeds up iterations.
\arg\min_{\Gamma\in\Omega_\Gamma} \frac{1}{2}\mathrm{vec}(\Gamma)^\top (I_T \otimes S^\top S)\mathrm{vec}(\Gamma) - \mathrm{vec}\left(S^\top R^\top X - \frac{\alpha}{2}\log(\Lambda)^\top \Pi\right)^\top \mathrm{vec}(\Gamma)
This allows approximating the nonlinear partitioning step by solving a quadratic program, simplifying implementation.
Concepts
genetic algorithm optimization problem least squares fitting least squares xgboost generalization k means clustering Generative Adversarial Networks image classification labeled data generative model self-supervised learning Weakly Supervised Learning optimization Gause's competitive exclusion principle finite element method convergence training clustering information criterion