ICML 2026 Papers

Vision Transformers increasingly incorporate extra tokens beyond patch tokens—from class tokens for aggregation to register tokens for artifact mitigation. While effective for their intended purposes, these tokens typically lack semantic structure. We ask a more ambitious question: Can we design regularization constraints that transform extra tokens into disentangled representations, enabling them to decompose images into semantic parts (e.g., heads, bodies, legs) without explicit supervision? We propose XTRA, an intuitive yet powerful framework that augments Vision Transformers with dedicated ``factor tokens'' and enforces disentanglement via a novel Minimum Volume Constraint (MVC). A multi-stage aggregation process further enforces these factor tokens into semantically pure components, preventing token collapse that often occurs when training with MVC alone. On ImageNet-1K, XTRA achieves superior disentanglement (8.4× improvement in SEPIN@1 over DINOv2) while simultaneously improving representation quality: KNN accuracy improves by 5.8\% and linear-probe accuracy by 2.3\%.

Early exit neural networks, which adapt computation to input complexity, have proven effective in supervised learning but remain largely unexplored in deep reinforcement learning (DRL). In this paper, we propose Budgeted EXit Actor (BEXA), a novel actor-critic architecture that integrates early exit branches into the actor network. To ensure a mathematically principled trade-off between performance and inference expenditure, the exit decision is formulated as a constrained linear program during training, the solution of which is subsequently amortized to enable highly efficient runtime execution. BEXA is general, governed by an interpretable budget parameter, and compatible with any off-policy actor-critic method. We evaluate BEXA by integrating it with standard off-policy algorithms like SAC and TD3 on a suite of MuJoCo tasks. Our results demonstrate a substantial improvement in inference efficiency with minimal or no loss in performance. These findings highlight early exits as a promising direction for improving computational efficiency in DRL.

The effectiveness of unlabeled data in Semi/Self-Supervised Learning (SSL) depends on appropriate assumptions for specific scenarios, thereby enabling the selection of beneficial unsupervised pretext tasks. However, existing research has paid limited attention to assumptions in SSL, resulting in practical situations where the compatibility between the unsupervised pretext tasks and the target scenarios can only be assessed after training and validation. This paper centers on the assumptions underlying unsupervised pretext tasks and explores the feasibility of preemptively estimating the impact of unsupervised pretext tasks at low cost. Through rigorous derivation, we show that the impact of unsupervised pretext tasks on target performance depends on three factors: assumption learnability with respect to the model, assumption reliability with respect to the data, and assumption completeness with respect to the target. Building on this theory, we propose a low-cost estimation method that can quantitatively estimate the actual target performance. We build a benchmark of over one hundred pretext tasks and demonstrate that our estimated performance strongly correlates with the actual performance obtained through large-scale training and validation.

Obtaining high-quality labels is costly, whereas unlabeled covariates are often abundant, motivating semi-supervised inference methods with reliable uncertainty quantification. Prediction-powered inference (PPI) leverages a machine-learning predictor trained on a small labeled sample to improve efficiency, but it can lose efficiency under model misspecification and suffer from coverage distortions due to label reuse. We introduce Machine‑Learning‑Assisted Generalized Entropy Calibration (MEC), a cross‑fitted, calibration‑weighted variant of PPI. MEC improves efficiency by reweighting labeled samples to better align with the target population, using a principled calibration framework based on Bregman projections. This yields robustness to affine transformations of the predictor and relaxes requirements for validity by replacing conditions on raw prediction error with weaker projection‑error conditions. As a result, MEC attains the semiparametric efficiency bound under weaker assumptions than existing PPI variants. Across simulations and a real‑data application, MEC achieves near‑nominal coverage and tighter confidence intervals than CF‑PPI and vanilla PPI.

Transfer learning aims to facilitate the learning of a target domain by transferring knowledge from a source domain. The source domain typically contains semantically meaningful samples (*e.g.*, images) to facilitate effective knowledge transfer. However, a recent study observes that the noise domain constructed from simple distributions (*e.g.*, Gaussian distributions) can serve as a surrogate source domain in the semi-supervised setting, where only a small proportion of target samples are labeled while most remain unlabeled. Based on this surprising observation, we formulate a novel problem termed *Semi-Supervised Noise Adaptation* (SSNA), which aims to leverage a synthetic noise domain to improve the generalization of the target domain. To address this problem, we first establish a generalization bound characterizing the effect of the noise domain on generalization, based on which we propose a Noise Adaptation Framework (NAF). Extensive experiments demonstrate that NAF effectively leverages the noise domain to tighten the generalization bound of the target domain, leading to improved performance. The codes are available at https://github.com/AIResearch-Group/SSNA.

Many stochastic physical systems evolve smoothly over time in the sense that the distribution of states changes regularly with time. The precise transition from current to next state is often modeled as the interplay of a smooth map and an explicit source of randomness. Stochastic Lifting leverages this premise by attaching an independent, high-dimensional random label to each state transition in the training data and fitting a transition map from the current state and label to the next state using a standard regression loss. The labels act as auxiliary coordinates that let the model represent multiple plausible outcomes for similar current states, avoiding collapse to a mean prediction in the finite-sample size regime. At inference, drawing fresh labels and rolling the map forward generates diverse trajectories with a single network evaluation per time step, with the smoothness bias of the learned map supporting accurate sampling in practice.

Neural operators have emerged as powerful surrogates for the solution of partial differential equations (PDEs), yet their ability to handle general, highly variable boundary conditions (BCs) remains limited. Existing approaches often fail when the solution operator exhibits strong sensitivity to boundary forcings. We propose a general framework for conditioning neural operators on complex non-homogeneous BCs through function extensions. Our key idea is to map boundary data to latent pseudo-extensions defined over the entire spatial domain, enabling any standard operator learning architecture to consume boundary information. The resulting operator, coupled with an arbitrary domain-to-domain neural operator, can learn rich dependencies on complex BCs and input domain functions at the same time. To benchmark this setting, we construct 18 challenging datasets spanning Poisson, linear elasticity, and hyperelasticity problems, with highly variable, mixed-type, component-wise, and multi-segment BCs on diverse geometries. Our approach achieves state-of-the-art accuracy, outperforming baselines by large margins, while requiring no hyperparameter tuning across datasets. Overall, our results demonstrate that learning boundary-to-domain extensions is an effective and practical strategy for imposing complex BCs in existing neural operator frameworks, enabling accurate and robust scientific machine learning models for a broader range of PDE-governed problems.

In many modern machine learning pipelines, abundant pretrained representations serve as noisy proxy covariates, while task-specific labels remain scarce. We study semi-supervised regression in this setting, and propose a simple two stage estimator that learns kernel eigenfeatures from all proxy covariates and fits a ridge predictor on labeled data. We derive finite sample bounds showing that fast labeled sample rates are recovered when proxy perturbation is controlled and unlabeled proxy covariates are sufficiently abundant. We also show that distribution regression is a direct special case, with analogous guarantees when the finite bag size is large enough. Experiments show consistent gains over supervised and semi-supervised baselines, especially in low label regimes.

Learning to rank can be cast as a probabilistic modeling problem over permutations, where the goal is to estimate the likelihood of an observed total ordering of items. This formulation naturally involves full-order probabilities of the form $\mathbb{P}(z_1 \leq \cdots \leq z_n)$, whose exact computation and optimization are intractable due to the factorial growth of the permutation space with respect to the list size. In this work, we introduce the *Full-Order Bound* (FOB), a tractable lower bound on the probability of an observed ordering, constructed from a subset of ordering constraints that factorizes across items while preserving full-order structure and order-reversal invariance. Under log-concave latent densities, the bound induces a convex inner tightening problem over latent cut points, which we solve efficiently during training using a *safe-region gradient ascent* (SRGA) procedure. Experiments on synthetic ranking tasks and large-scale learning-to-rank benchmarks show that FOB improves full-list ordering metrics and remains competitive on NDCG, while an optional metric-aligned variant recovers NDCG gains. Our code is available at https://github.com/tyxaaron/FOB.

Despite their dominance in vision and language, deep neural networks often underperform relative to tree-based models on tabular data. To bridge this gap, we incorporate five key inductive biases into deep tabular learning: robustness to irrelevant features, axis alignment, localized irregularities, feature heterogeneity, and training stability. We propose *LassoFlexNet*, an architecture that evaluates the linear and nonlinear marginal contribution of each input via Per-Feature Embeddings, and sparsely selects relevant variables using a Tied Group Lasso mechanism. Because these components introduce optimization challenges that destabilize standard proximal methods, we analyze stochastic hierarchical proximal dynamics and develop a *Sequential Hierarchical Proximal Adaptive Gradient optimizer with exponential moving averages (EMA)* designed to stabilize training in practice. Across 52 datasets from three benchmarks, LassoFlexNet matches or outperforms leading tree-based models, achieving up to a $10$\% relative gain, while maintaining Lasso-like interpretability. We substantiate these empirical results with ablation studies and theoretical proofs confirming the architecture's enhanced expressivity and structural breaking of undesired rotational invariance.

Auto-regressive models have emerged as powerful tools for sequential data, from language to video. Understanding how and why these models learn latent representations remains an open theoretical question. In this work, we demonstrate that when trained by empirical risk minimization on data from partially observed linear dynamical systems, two-layer linear auto-regressive models naturally learn to approximate Kalman filtering. In particular, we show that the learned hidden representation coincides, up to a similarity transformation, with the state estimates produced by the optimal (Kalman) filter, even though the model has no explicit knowledge of the underlying dynamics or state. The result follows from three main insights. First, we establish that the Kalman filter is well approximated by an auto-regressive model with bounded truncation error. Second, we show that despite non-convexity, the two-layer optimization landscape is benign, i.e., all stationary points are either strict saddles or global minima. Finally, as our main contributions, we provide finite-sample guarantees on prediction error, parameter estimation error, and latent state recovery. Numerical simulations support the theoretical results and demonstrate that the latent representations of auto-regressive models recover state estimates.

Semi‑supervised statistical inference plays a key role in biomedical research, where labeled data often have higher quality but are limited due to costly clinical annotation. Yet, existing semi‑supervised statistical inference methods rely heavily on structured variables and strictly matched covariates between labeled and unlabeled datasets -- limitations ill‑suited for the heterogeneity and unstructured nature of real-world biomedical data. Modern biomedical studies increasingly collect unstructured data (clinical notes, patient audio and video recordings), with inconsistent protocols across datasets causing covariate misalignment (for instance, detailed medication histories may be recorded in one study but not another). Recent advances in pre‑trained multimodal large language models (LLMs), which excel at handling unstructured data, present an attractive potential solution. To transform this potential into rigorous semi-supervised statistical inference methods for biomedical research, two key challenges must be addressed: (1) How can we reliably integrate LLMs to enhance semi-supervised inference efficiency without compromising statistical validity? (2) How can those efficiency gains persist despite mismatched covariates between labeled and unlabeled datasets? In this paper, we tackle these challenges by systematically *calibrating pseudo-labels* provided LLMs with a novel *prediction-invariance identification* strategy. Our resulting semi‑supervised inference framework improves parameter estimation efficiency while maintaining full statistical validity, as demonstrated through our theoretical results and illustrated in a case study for identifying key biomarkers in Alzheimer’s disease detection with speech data.

Multi-objective optimization (MOO) has emerged as a powerful approach to solving complex optimization problems involving multiple objectives. In many practical scenarios, function evaluations are unavailable or prohibitively expensive, necessitating optimization solely based on a fixed offline dataset. In this setting, known as offline MOO, the goal is to find out the Pareto set without access to the true objective functions. This setting suffers from the out-of-distribution (OOD) issue, where the surrogate model is not accurate for unseen designs. Due to the OOD issue, surrogate errors may cause the optimizer to select solutions that do not lie on the true Pareto front and are biased toward its extremes. To address this, this paper proposes Diversity-driven Offline Multi-Objective Optimization (DOMOO), which aims to find out a diverse and high-quality set of solutions. First, DOMOO incorporates an accumulative risk control module that estimates the potential risk of candidate solutions and alleviates the OOD issue between the training data and the generated solutions. In addition, a nested Pareto set learning (PSL) strategy is proposed to jointly learn preference and PSL parameters, then optimize them, enabling adaptation to diverse Pareto front geometries. To further enhance solution quality, we design a diversity-driven selection strategy that extracts a representative and well-distributed set of final solutions. To achieve this diversity-driven selection strategy, we propose $\text{IGD}_\text{offline}$, a tailored indicator for the offline setting that considers both diversity and convergence, and avoids the bias of hypervolume indicator. Extensive experiments on synthetic and real-world benchmarks show that DOMOO achieves the best average rank across tasks in both convergence and diversity among the compared methods.

Recent class-incremental learning (CIL) methods built on large pre-trained vision models have shown that strong performance can be retained even under strict data access constraints. This raises a fundamental question: which properties of pre-trained representations make such recovery possible in the class-incremental setting? In this work, we show that class-level feature statistics play a central role in enabling effective CIL under strong pre-training. When the visual backbone is frozen, maintaining simple class-wise statistical estimators of features can recover a substantial fraction of the performance achieved by static joint training across diverse benchmarks. We make this observation explicit through deliberately minimal reference points built on frozen CLIP representations. In particular, we demonstrate that competitive performance can be achieved without continual parameter updates, by performing class-incremental inference based solely on class-level statistical estimators instantiated from frozen features. Our findings suggest that class-level statistics constitute an important and previously underemphasized component of recent PTM-based CIL approaches, offering a complementary perspective for understanding their strong empirical performance. Our code is available at https://github.com/HdTgon/baseCIL.

Evidential Deep Learning (EDL) enables trustworthy multi-view classification, yet suffers from a critical vulnerability: the Scale Mismatch Problem. We theoretically demonstrate that existing evidential fusion rules erroneously equate logit magnitude with semantic confidence, rendering them susceptible to semantic hijacking by inflated but uninformative views. To resolve this, we propose Scale-Invariant Evidential Fusion (SAEF), a framework utilizing instance-wise standardization to strictly decouple confidence from scale. Instead of relying on magnitude dominance, SAEF aggregates views based on statistical consensus. Theoretically, SAEF guarantees invariance to global scaling and robustness to asymmetric dominance. Experiments on four diverse datasets confirm that SAEF outperforms state-of-the-art baselines in accuracy and robustness to semantic conflicts and noise, ensuring stability against severe scale perturbations.

Controlling partial differential equations (PDEs) with learning-based policies remains fundamentally limited by fixed-dimensional representations: policies trained for a specific sensor, actuator, or agent configuration typically fail when the configuration changes. This limitation is particularly severe in multi-agent PDE control, where policies do not scale across population sizes without retraining. We address this challenge by introducing **C**ardinality **I**nvariant **N**eural **O**perator **C**ontrol (**CINOC**), reformulating PDE control as an operator learning problem that maps state fields to continuous control functions and trains them end-to-end through differentiable PDE solvers, yielding policies that naturally adapt to varying sensor and actuator configurations. Remarkably, CINOC policies trained on small swarms exhibit cardinality invariance, allowing for zero-shot transfer to significantly larger populations as well as robustness to partial agent failure. This scalability arises from agents sharing a common policy and coordinating through their physical environment, which produces an emergent self-normalization effect. To explain this phenomenon, we provide a theorem grounded in mean-field theory demonstrating that policy gradients computed from finite-agent systems converge to those of a continuous control limit. Empirically, we validate CINOC on tracking, stabilization, and density transport across linear, nonlinear, chaotic, and turbulent PDEs.

Long-tailed recognition suffers from extreme class imbalance, where scarce tail data leads to biased and fragile feature distributions that exacerbate confusion with semantically or visually similar classes. Prior feature-space reconstruction methods transfer head-class structure or train conditional generators to synthesize tail features, yet the resulting \emph{statistical} updates are often heuristic and can degrade multi-class separability when tail estimates are unreliable. Given this issue, we propose \emph{Risk-Bounded Distribution Reconstruction} (RBDR), an offline statistic calibration framework for the two-stage long-tailed pipeline, grounded in an analysis of rival-induced discriminative directions. RBDR performs (i) risk-aware mean calibration by softly projecting any candidate update onto a supportive set such that a surrogate discriminative-risk upper bound does not increase, and (ii) covariance control by shrinking dispersion in a rival subspace while preserving diversity in orthogonal components. In the frozen-feature reconstruction setting, these calibration components convert heuristic reconstruction cues into controllable distributional updates, improving both performance and stability across multiple long-tailed benchmarks.

We introduce Euler--Poincar\'e Neural Dynamics (EPND), a geometric-mechanics framework that casts evolution-operator learning as Lie-group flows for long-horizon dynamical modeling. Unlike conventional operator-learning approaches that treat temporal propagation as an unconstrained black-box map, EPND places geometric mechanics at the core of its architecture, playing a role of the mathematical engine. This foundation enables a principled treatment of curvature, symmetry, and conservation, with the learned evolution expressed in geometric terms. Building on this foundation, we develop the Euler--Poincar\'e Parallel Scan, a parallel algorithm that leverages the associative algebra of Lie-group compositions to overcome the inefficiencies of sequential computation. By unifying geometric structure with scalable computation, EPND achieves high accuracy, strong stability, and significant parallel acceleration in modeling long-horizon dynamics in versatile scientific simulations.

We study compute reduction in neural networks through a unified partial versus full computation view, captured by one-shot magnitude pruning in the static regime and early exit in the adaptive regime. In an asymptotic single-neuron model, we prove a concentration theorem for one-shot magnitude pruning with explicit rates. We also introduce the conditional perceptron for early exit and show that its excess generalization error decays as a power of the compute gap, with an exponent that grows to infinity as the alignment between partial and full computations tends to one. We then extend the analysis to deep networks, characterizing how pruning-induced distortions accumulate with depth and deriving a corresponding compute–accuracy tradeoff for frozen-backbone early exit under a neural network Gaussian process model. Numerical simulations corroborate the predicted scaling laws.