main concepts scientists sections
SimplePopularAdvanced

The Common Language of Order: A Neural Network for Mixed Responses ✛ ✎ ✦

Original: "Deep Multitask Learning for Mixed-Type Outcomes with Shared Sparsity"
· Huichao Li, Tong Wang, Sanguo Zhang, Shuangge Ma
arXiv:2607.00995 · 2026-07-01 · arXiv non-exclusive · independent analysis · 4 min
Statistical ML Machine Learning
A new multitask learning approach uses a rank criterion to merge heterogeneous data without manually tuning weights.

Context

Genomic studies often yield tasks where multiple responses of different natures are measured on the same subjects: continuous gene expression level, binary disease status, categorical subtype. All of these can depend on a common set of genetic variants. For example, in cancer genomics, RNA expression profiles are used to predict both inflammatory markers and diagnosis. However, combining such data in a single model is hindered by the incommensurability of loss functions: mean squared error for continuous and cross-entropy for binary — like degrees Celsius and kilometers. Traditionally, the problem is solved with a weighted sum, but the choice of weights is arbitrary. There is a need for a common language that describes only the relative order within each task, ignoring units of measurement. This paper proposes such a language — one of order‑based machine learning.

Methods

Researchers from China and the USA (Li, Wang, Zhang, Ma) built a multitask transformation model: $Y_t$ = $D_t$( $f_t$(X) + ε ), where $D_t$ is an unknown monotone transformation. To avoid estimating $D_t$, they switched to a rank criterion: maximizing the probability that, for any pair of objects, the order of predicted values $f_t$(X) matches the order of observed responses $Y_t$. This criterion is unchanged by any monotone transformation of the scale — that is the “common language of order.” For smooth optimization, the hard indicator I(y1>y2) was replaced with a sigmoid. The model is implemented as a deep neural network with a shared first layer for all tasks and a group penalty on that layer’s weights, which zeroes out the contributions of uninformative features for all tasks simultaneously. Training used mini-batches with feature reuse from the previous epoch to reduce computational complexity. Architecture: one shared layer, then for each task several fully connected layers with SiLU activation. All experiments were carried out in numerical simulation on synthetic and real data.

Results

Theoretical identifiability is proved: the population rank criterion maximum uniquely defines the target functions $f_t$, and smoothing does not bias the objective as the parameter tends to zero. Non-asymptotic bounds on the excess risk are established: when p = O(log n), the convergence rate reaches about $n^{-β/(2β+ps)}$ (β — smoothness of the functions, ps — number of important features), and when p = O($n^c$) — somewhat slower. Consistency of variable selection is also proved: the method asymptotically correctly identifies the subset of common relevant predictors. In simulations with heavy-tailed noise and corrupted labels, the proposed method gave lower regression errors (MAE, RMSE) and better F1 score compared to alternatives: GradNorm, dynamic weighting, uncertainty weighting, and single-task learning. The advantage is especially noticeable in mixed scenarios where one task is regression and the other classification. On real bioinformatics data: mouse genetics (60 individuals, predicting enzymes of genes SCD1 and PEPCK), breast cancer METABRIC (1980 samples, subtype classification), and lung cancer (234 samples, mixed prognosis of status and CRP) — the proposed approach consistently led in accuracy. For example, in the lung dataset, classification accuracy rose to 0.968 (vs 0.951 for single-task), and CRP prediction error dropped from 22.96 to 18.80 MAE. The selected genes are biologically meaningful: Scd1 — a key regulator of lipid metabolism, E2F1 — a transcription factor in breast cancer, RPLP2 — a marker of lung adenocarcinoma.

Implications

The work paves the way for objective joint analysis of heterogeneous biomedical data. Instead of manually tuning weights for each task, researchers obtain a single criterion that automatically aligns multi-scale responses. This should enhance reproducibility and enable the discovery of shared molecular mechanisms, for example, those linking immunity and oncogenesis.

Future development

In the future, the method could be extended to ordinal, count, and censored data, where handling ties and dependencies will be needed. Another direction is to relax the strict assumption of shared sparsity, allowing partially overlapping feature sets, which would make the model more realistic. Efficiency could also be improved via a semiparametric approach.

Impact

The method will impact biostatistics, computational genomics, and clinical research, where diverse data types on the same patients are routinely collected.

Next steps

Immediate next steps are asymptotic efficiency analysis and adaptation to longitudinal data, where correlation across observations of the same subject will need to be incorporated into the rank criterion.

Key open problems

The study contributes to solving the fundamental problem of high-dimensional nonparametric feature selection with mixed response types — a task at the intersection of statistical learning and applied genomics.

The knowledge machine's view

This section is addressed to the author of the paper and their colleagues. Neighbouring works were found by meaning-based search across our archive, not by word overlap. We do not review: we show the field around the work.

What the work does

The authors developed a multitask model based on a rank criterion that combines responses of different natures (continuous, binary, categorical) without tuning weights for loss functions. They proved identifiability and derived non-asymptotic error bounds, demonstrating the method's advantages in simulations and on real genetic data.

Where its strength is

The strong point is the theoretically grounded shift to ordinal statistics, which makes the model invariant to monotonic transformations of scales, and the proven consistency of common feature selection.

Where it could go next

Worth trying

  • Apply the method to cell atlas data, where responses include continuous expression and categorical cell type.
  • Add regularization based on biological networks to improve selection of shared genes.
  • Use anti-forgetting techniques to stabilize training of the shared layer.

Why this may matter

The work offers a principled way to integrate heterogeneous biomedical data without arbitrary weights, which could improve prediction reliability in genomics and other fields with multitask measurements.

Nearby works in our archive

The closest work in the archive sits at 0.62 by meaning — the usual level for our corpus.

Whole graph →

The idea of rank correlation dates back to Han’s maximum rank correlation criterion (1987), which here is combined with deep neural networks for multitask learning for the first time. And the sigmoid smoothing trick comes from a 2007 paper by Song et al., originally proposed for single-response regression.

The concept of a universal language for incommensurable data evokes the “universal translator” from science fiction — a device that allows beings with different languages and perceptual systems to communicate.

Y_{t} = D_{t}(f_{t}(X) + \epsilon_{t})
The observed response Y_t results from the target function f_t(X) with noise, passing through an unknown monotone transformation D_t.
H(f_t) = \mathbb{E}_{X,\tilde{X},Y,\tilde{Y}} [ I(Y > \tilde{Y}) I(f_t(X) > f_t(\tilde{X})) ]
The probability that the order of observed responses matches the order of predictions of the target function.
s(\zeta; a_n) = \frac{1}{1+\exp(-\zeta/a_n)}
A smooth approximation of the indicator function; the parameter a_n controls the steepness.

Key numbers

  • sample size in mouse genetics: 60
  • METABRIC breast cancer samples: 1980
  • lung cancer samples: 234
  • number of important predictors (ps): 30
  • dimensionality of predictors p (up to): 800
Scientists
Andrey Markov
Concepts
transfer learning training training data predictive learning neural networks genetic algorithm random forest Large Language Models decision making boosted decision tree generalization xgboost probabilistic model regularization methods prediction error gibbs sampling metropolis hastings algorithm overfitting weighted mean bayesian hierarchical model jackknife resampling