ICML 2026 Papers

*Model merging* offers an appealing route to multi-task learning by composing independently fine-tuned checkpoints without centralized data or retraining. However, this convenience can come with a hidden cost. Model merging may *amplify* performance disparities across subgroups, raising fairness concerns even when average accuracy remains competitive. To explain this phenomenon, we develop a sensitivity-based theoretical analysis that upper bounds the fairness gap induced by model merging. Theoretical analysis with empirical verifications reveals that the resulting fairness gap is governed by two coupled factors, a *merging magnitude* term that measures how far the merged parameters move from the target model and *global sensitivity* terms that determine how unevenly the perturbation affects subgroup losses. Guided by these insights, we propose *FairMerging*, a two-stage merging framework that first reduces the sensitivity of the target model and then performs fairness-aware coefficient optimization with orthogonally normalized task vectors. Experiments across multiple datasets, backbones, and merging baselines demonstrate that FairMerging substantially mitigates unfairness while retaining competitive multi-task performance.

Self-supervised learning (SSL) has emerged as a powerful paradigm for learning meaningful representations from unlabeled data. However, the standard protocol for evaluating these representations, linear probing, is computationally expensive, sensitive to hyperparameters, and provides limited insight into the geometric structure of the representation space. In this work, motivated by connections between neural network generalization and intrinsic dimension (ID) we propose IdEst, a method for estimating the ID of SSL representations via the Minimum Spanning Tree dimension estimator ($\mathrm{dim}_\mathrm{MST}$). Across diverse datasets, architectures, and SSL pretraining objectives, we show that IdEst strongly correlates with downstream linear probe performances. Furthermore, we demonstrate that IdEst enables efficient hyperparameter selection, significantly reducing the computational cost compared to supervised alternatives. Our results highlight intrinsic dimensionality as a principled geometric proxy for assessing SSL representations, complementing standard supervised probing protocols.

Symbolic regression (SR) traditionally balances accuracy and complexity, implicitly assuming that simpler formulas are structurally more rational. We argue that this assumption is insufficient: existing algorithms often exploit this metric to discover accurate and compact but structurally irrational formulas that are numerically ill-conditioned and physically inexplicable. Inspired by the structural stability of real physical laws, we propose the Effective Information Criterion (EIC) to quantify formula rationality. EIC models formulas as information channels and measures the amplification of inherent rounding noise during recursive calculation, effectively distinguishing physically plausible structures from pathological ones without relying on ground truth. Our analysis reveals a stark structural stability gap between human-derived equations and SR-discovered results. By integrating EIC into SR workflows, we provide explicit structural guidance: for heuristic search, EIC steers algorithms toward stable regions to yield superior Pareto frontiers; for generative models, EIC-based filtering improves pre-training sample efficiency by 2–4 times and boosts generalization $R^2$ by 22.4\%. Finally, an extensive study with 108 human experts shows that EIC aligns with human preferences in 70\% of cases, validating structural stability as a critical prerequisite for human-perceived interpretability. We release our code at https://github.com/tsinghua-fib-lab/EIC

Clustering in a big data setting is an intensively studied problem, with coresets emerging as one of the important paradigms in this line of work. Given a cost function $\text{cost}(P,S)$ mapping input points $P$ and a solution $S$ to an objective value, a coreset is a typically weighted sketch $\Omega\subseteq P$ such that $\text{cost}(\Omega,S)\approx \text{cost}(P,S)$. In practice, coreset sizes much smaller than those suggested by theoretical guarantees are often found to be sufficient. In this paper, we offer an explanation for this phenomenon. Smaller coreset sizes suffice if we only wish to preserve the costs of \emph{good} solutions, i.e., solutions with low cost. We define and devise \emph{approximation-preserving coresets}, which provide a weaker guarantee than strong coresets, which apply to all solutions, while providing stronger guarantees than weak coresets, which apply only to the optimum solution. We complement this result by showing that even a very small distortion in the approximation factor cannot admit coresets of this size.

Imbalanced Unreliable Partial Label Learning (I-UPLL) is a challenging weakly supervised learning setting in which severe class imbalance and unreliable candidate labels jointly degrade model performance. By revisiting existing approaches for imbalanced learning, we observe that most of them fundamentally rely on estimating the class prior to guide balancing operations, such as re-sampling, pseudo-label generation, or logit adjustment. However, under I-UPLL, obtaining stable and accurate prior estimates at the early stage of training is often unrealistic due to the ambiguity and unreliability of partial labels, thereby leading the model to rapidly converge to a suboptimal solution. To address this issue, we propose CLAPOR, a novel CLAss-PriOr perturbation-Robust regularization framework that fundamentally avoids dependence on accurate prior estimation. Specifically, the proposed regularization trains the model under deliberately perturbed class priors, sampled from a Dirichlet distribution that deviates from the current estimated prior. This design encourages consistent performance under prior uncertainty and naturally preserves attention to minority classes. Extensive experiments on benchmark datasets demonstrate the effectiveness of CLAPOR across various settings of I-UPLL.

The Platonic Representation Hypothesis posits that neural networks trained on different modalities converge toward a shared statistical model of the world. Recent work exploits this convergence by aligning frozen pretrained vision and language models with lightweight alignment layers, but typically relies on contrastive losses and millions of paired samples. In this work, we ask whether meaningful alignment can be achieved with substantially less supervision. We introduce a semi-supervised setting in which pretrained unimodal encoders are aligned using a small number of image-text pairs together with large amounts of unpaired data. To address this challenge, we propose SOTAlign, a two-stage framework that first recovers a coarse shared geometry from limited paired data using a linear teacher, and then refines the alignment on unpaired samples via an optimal-transport-based divergence that transfers relational structure without overconstraining the target space. Unlike existing semi-supervised methods, SOTAlign effectively leverages unpaired images and text, learning robust joint embeddings across datasets and encoder pairs, and significantly outperforming supervised and semi-supervised baselines.

Semi-supervised imitation learning (SSIL) consists in learning a policy from a small dataset of action-labeled trajectories and a much larger dataset of action-free trajectories. Some SSIL methods learn an inverse dynamics model (IDM) to predict the action from the current state and the next state. An IDM can act as a policy when paired with a video model (VM-IDM) or as a label generator to perform behavior cloning on action-free data (IDM labeling). In this work, we first show that VM-IDM and IDM labeling learn the same policy in a limit case, which we call the IDM-based policy. We then argue that the previously observed advantage of IDM-based policies over behavior cloning is due to the superior sample efficiency of IDM learning, which we attribute to two causes: (i) the ground-truth IDM tends to be contained in a lower complexity hypothesis class relative to the expert policy, and (ii) the ground-truth IDM is often less stochastic than the expert policy. We argue these claims based on insights from statistical learning theory and novel experiments, including a study of IDM-based policies using recent architectures for unified video-action prediction (UVA). Motivated by these insights, we finally propose an improved version of the existing LAPO algorithm for latent action policy learning. We experiment on the Procgen, Push-T and LIBERO benchmarks.

This paper studies the Minimal Embeddable Dimension (MED): the least dimension in which there exists a configuration of $m$ object vectors so that every subset of size at most $k$ is exactly retrieved by score comparison. Our result shows MED is $\Theta(k)$, independent of $m$, for inner product, Euclidean distance, and cosine similarity. We then consider Robust MED (RMED), where all vectors are unit normed and an $\epsilon$ gap of scores is required. We derive the $m$-dependent feasibility ceiling $\epsilon_\star(m,k)=m/\sqrt{k(m-1)(m-k)}$, which approaches $1/\sqrt{k}$ when $m\gg k$, and a Gaussian centroid construction gives a robust witness upper bound in the feasible margin regime. Numerical simulation on synthetic top-$2$ retrieval with cyclic polytope and centroid query optimization confirmed our theoretical claims. Experiments on LIMIT and LIMIT-small datasets also show that simple embedding-based retrieval baselines can overfit and outperform the reported single-vector LLM embedding baseline. Both theoretical and empirical findings rule out the lack of exact geometric capacity as the obstruction.

In this paper, we study the rent-or-buy problem (also called the Bahncard problem) in the learning-augmented setting. In this problem, a traveler must complete a sequence of trips that are revealed online over time, each of which has an associated cost with it. The traveler has the option of buying a discount card at a fixed cost that gives a discount on trip costs for a fixed time after buying the card. The goal is to minimize the overall cost of all the trips, including the money spent on buying discount cards. For this problem, it is well-known that the best deterministic algorithm has a competitive ratio of 2. In this paper, we ask whether we can do better if the traveler has a *sample* of trips available offline, e.g., obtained from an ML model based on historical data. We show that even a sparse sample of the input can significantly improve the competitive ratio of the algorithm from 2 to 3/2, and further to close to 1 under some additional conditions. We also verify our theoretical bounds via numerical simulations, which reveal that our proposed algorithm obtains nearly optimal solutions for a variety of natural input classes.

Weak-to-strong generalization studies how to improve a strong student using supervision from a weaker teacher when reliable labels are scarce. We view this primarily as a data selection problem, where the key challenge is to identify which weak labels are reliable enough to serve as a training signal. To address this, we introduce trust functions that assign each weak label a scalar trust score and use these scores to filter weak supervision. Across several domains, including world knowledge, quantitative reasoning, decision making, trust filtering yields students that match and sometimes surpass ground-truth supervision, achieving near-lossless weak-to-strong generalization. Moreover, trust functions enable an iterative weak-to-strong chain that compounds gains by training a student and reusing it as the next teacher, producing the strongest final model. Our analyses suggest that neural trust functions improve learning through more than label error reduction. They induce an implicit easy-first curriculum, recover near-optimal alternatives where ground truth labels are incomplete, and produce more coherent gradient updates, offering a mechanistic account of the stability and efficiency of trust-filtered weak-to-strong generalization.

Decentralized Unlearning (DU) aims to remove the influence of specific clients from a collaboratively trained global model. However, existing methods suffer from strong reliance on static, problem-specific hyperparameters or restrictive convexity assumptions, limiting their general applicability. To overcome these limitations, we propose **TRA**jectory-aware **CE**rtified **D**ecentralized **U**nlearning (**TRACE-DU**), a generic unlearning framework for decentralized training. **TRACE-DU** introduces a fine-grained sensitivity analysis that leverages local SGD updates and decentralized training dynamics, thereby eliminating the need for convexity assumptions and reducing dependence on manually tuned parameters. By integrating strategic checkpoint selection with calibrated noise perturbation, the proposed framework enables efficient certified unlearning. Moreover, we exploit historical model trajectories to extend this framework, enabling it to naturally support sequential unlearning requests from an arbitrary number of clients. We provide theoretical guarantees for certified unlearning and derive sensitivity bounds under both convex and non-convex loss functions. Experimental results demonstrate that our framework outperforms state-of-the-art baselines across diverse metrics.

Anomaly detection (AD) on real-world tabular data is challenged by diverse anomaly types, scarce labels, and high sensitivity to data-specific hyperparameter tuning. A central difficulty is that different anomaly types favor opposing representations: *compactness* - tightening the normal class to expose pointwise deviations - and *structure preservation* - retaining cluster and cross-feature relationships. Existing one-class detectors capture only one of these or balance them via hyperparameters, which is problematic without validation labels. We propose **Discriminant Vector Machine for Anomaly Detection (DVM-AD)**, a closed-form one-class method derived from discriminant analysis that captures both behaviors under a single fixed configuration. From a deterministic reference point built from the training data, DVM-AD derives a bounded ratio whose two ends correspond to compressive and structure-preserving directions. This bound enables selecting directions from both ends simultaneously without dataset-specific tuning. In addition, a Moore-Penrose pseudo-inverse keeps the method well-posed under high-dimensional or rank-deficient settings, and test samples are scored by nearest-neighbor distance in the discriminant space, normalized for thresholding-ready use. Across 47 ADBench tabular datasets and 10 NLP/CV embedding benchmarks against 28 baselines, DVM-AD achieves the best average AUROC (89.65\%, average rank 2.98) on tabular datasets and remains top-ranked across four anomaly types and on embedding tasks (average rank 1.60, AUROC 72.68\%).

Cross-subject emotion recognition aims to improve a model's generalization to previously unseen subjects. Existing methods are mainly built upon domain generalization or data augmentation, but suffer from two major limitations: 1) heavy dependence on modality-specific feature designs—almost exclusively tailored to EEG signals—resulting in limited generalizability; and 2) the widespread assumption of independently and identically distributed data, which restricts the diversity of generated samples. To address these challenges, we systematically analyze the heterogeneous distribution characteristics of emotion data and propose **MixEmo**, a framework that integrates heterogeneous distribution augmentation and collaborative learning. Specifically, a well-trained backbone is used to extract representations and partition them into multiple single-distribution subsets as distribution prototypes. These prototypes are randomly combined to synthesize unseen distributions, thereby enhancing distributional diversity. Finally, heterogeneous distribution collaborative learning jointly optimizes the model across subsets. Extensive experiments demonstrate that **MixEmo** substantially improves generalization performance in cross-subject emotion recognition.

Semi-supervised referring expression segmentation (SS-RES) aims to achieve precise pixel-level language grounding under limited annotation, yet suffers from limited supervision and unreliable pseudo-labels when exploiting unlabeled image–text pairs. In this work, we propose Learning to Label, a reinforced self-evolving framework (L2L) that casts pseudo-label construction as a learnable decision-making process. To build foundational understanding, we leverage a multimodal large language model to extract semantic–spatial priors, which are instantiated as initial soft segmentation proposals and elevated—together with textual cues—into learnable guidance signals that condition a hierarchical segmentation network. To ensure stable learning, a reinforced pseudo-label selection is further formulated as an exploratory decision process that adaptively rewards high-utility pixel-level supervision based on multimodal priors and model predictions. This reinforced self-evolving loop enables joint optimization of the segmentation model and pseudo-labels, progressively enhancing label reliability under sparse supervision. Extensive experiments on RefCOCO, RefCOCO+, and RefCOCOg datasets demonstrate improvements over existing methods, validating its effectiveness and generalization.

Sharpness-Aware Minimization (SAM) improves generalization by minimizing the worst-case loss within a fixed parameter-space radius neighborhood. SAM and its variants mainly rely on a first-order linearized surrogate, while flat minima are inherently a second-order (curvature) notion. We revisit this mismatch and propose Loss-Equated SAM (LE-SAM), which inverts the traditional SAM mechanism that replaces the fixed perturbation radius with a fixed loss-space budget, effectively removing gradient-norm–dominated learning signals and shifting optimization toward curvature-dominated terms. Extensive experiments across diverse benchmarks and tasks demonstrate the strong generalization ability of LE-SAM that consistently outperforms SAM and even its variants, achieving state-of-the-art performance.

LLM serving frameworks are quickly evolving with a complex software stack and a vast number of optimizations. The rapid development process can introduce silent errors where output quality silently degrades without any explicit error signals. Diagnosing silent errors is notoriously difficult due to the substantial semantic gap between the high-level symptoms and the low-level root causes. We observe that diagnosis of silent errors can be effectively framed as a differential debugging problem by leveraging the existence of semantically correct reference implementations. We propose Ekka, an automated diagnosis system that identifies root causes by systematically aligning and comparing intermediate execution states between a target and a reference framework. We constructed a benchmark of real-world silent errors from popular serving frameworks, where Ekka shows 80\% pass@$1$ diagnosis accuracy and 88\% pass@$5$ diagnosis accuracy, outperforming state-of-the-art systems. Ekka also diagnoses 4 new silent errors from serving frameworks, all of which have been confirmed by the developers.

We introduce ECSEL, an explainable classification method that learns formal expressions in the form of signomial equations, motivated by the observation that many symbolic regression benchmarks admit compact signomial structure. ECSEL directly constructs a structural, closed-form expression that serves as both a classifier and an explanation. On standard symbolic regression benchmarks, our method recovers a larger fraction of target equations than competing state-of-the-art approaches while requiring substantially less computation. Leveraging this efficiency, ECSEL achieves classification accuracy competitive with established machine learning models without sacrificing interpretability. Further, we show that ECSEL satisfies some desirable properties regarding global feature behaviour, decision-boundary analysis, and local feature attributions. Experiments on benchmark datasets and two real-world case studies i.e., e-commerce and fraud detection, demonstrate that the learned equations expose dataset biases, support counterfactual reasoning, and yield actionable insights.

Although transformers are most commonly optimized as statistical sequence models, it is unclear to what extent they can implement and learn exact algorithmic computations. Here, we specify a transformer implementation from first principles that executes a fundamental and widely used method for $k$-means clustering: Lloyd's algorithm. We theoretically prove and empirically demonstrate that this implementation of a transformer architecture, which we term the _$k$-means transformer_, exactly implements Lloyd's algorithm for $k$-means clustering using the standard circuit mechanisms of modern transformers: attention block, residual connections, and feed-forward block. In learning experiments, we find that training this base architecture on $k$-means clustering yields a generalizable clustering algorithm that surpasses Lloyd's algorithm in terms of clustering quality. Finally, we demonstrate that interpretable alterations (e.g., inclusion of layer normalizations) to this architecture yields diverse and novel variants of clustering algorithms, including soft $k$-means, spherical $k$-means, trimmed $k$-means. Overall, our results show that transformer circuit mechanisms can instantiate exact algorithmic routines for clustering, while simultaneously providing an effective learnable model.

Gradient-based optimization in class-incremental learning (CIL) often faces the plasticity–stability dilemma, since continuous weight updates can distort decision boundaries learned from earlier tasks. We revisit this problem from the viewpoint of stochastic geometric memory allocation and propose BTSP-CAM, a gradient-free memory system that instantiates theoretical insights from the hippocampal simpleBTSP model into a practical algorithm. Rather than fine-tuning a frozen encoder by backpropagation, BTSP-CAM externalizes plasticity into a binary synaptic matrix that evolves through local stochastic bit-flip updates. A trace-gated plateau process, driven by eligibility traces together with familiarity and collision signals, modulates when and where synapses are rewritten and suppresses cross-class interference in Hamming space. The resulting geometric memory states are mapped to semantic logits through a CA1-like competitive layer and a closed-form ridge readout, enabling fast consolidation after each task. Empirically, BTSP-CAM rivals gradient-based methods in a strictly exemplar-free setting and consistently boosts SOTA baselines as a lightweight plugin. Mechanistic analysis validates our geometric theory, confirming that stochastic repulsion actively bounds class overlap and stabilizes decision margins.