ICML 2026 Papers

Diffusion Language Models (DLMs) have recently achieved significant success due to their any-order generation capabilities. However, existing inference methods typically rely on local, immediate-step metrics—such as confidence or entropy—which inherently lack a more reliable perspective, leading to sub-optimal generation quality. To address this, we propose **C**oherent **C**ontextual **D**ecoding (**CCD**), a novel inference framework built upon two core innovations. First, CCD bypasses the potential bias of the single context to leverage historical contexts for approximating the marginal distribution of token prediction, leading to better sequence coherence and the early rejection of sub-optimal paths. More importantly, we demonstrate that this mechanism is theoretically equivalent to modeling the consistency of historical steps via the conditional mutual information between contexts and token predictions. Finally, CCD achieves significantly milder performance degradation under highly parallel decoding scenarios compared to baselines. Empirically, our method achieves a simultaneous enhancement in both inference speed and performance across diverse benchmarks on Dream and LLaDA.

Understanding gradient descent dynamics is key to explaining the success of over-parameterized models, where implicit bias manifests through conservation laws in gradient flow. While such laws are well understood for linear and ReLU networks, they remain largely unexplored for modern architectures. This work develops a unified framework to characterize conservation laws for contemporary models, including feedforward networks with GELU, SiLU, and SwiGLU activations, multihead attention with sinusoidal and rotary positional encodings, and Mixture-of-Experts architectures under diverse gating designs. Our theoretical findings are supported by experiments that validate the predicted invariants.

Vision-Language Models (VLMs) inherit social biases from large-scale pretraining data, and these biases can amplify in downstream tasks, leading to systematic performance disparities across sensitive groups. Due to the high training cost and the risk of catastrophic forgetting, recent research has focused more on lightweight \emph{test-time} debiasing, aiming to obtain an ideal fair embedding for each query. However, such point-based corrections are often unstable and become notably weaker in multi-class settings, where group structure cannot be adequately captured by a single point. Therefore, we propose W4D, a distributional debiasing framework that reframes fairness as aligning query embedding distributions to group reference distributions under the Wasserstein distance, which provides a geometry-aware notion of discrepancy beyond mean shifts. To make this alignment practical at test-time, W4D introduces probabilistic prompts that induce controlled distributional perturbations and optimizes a Wasserstein-based objective to reduce cross-group disparity while preserving task-relevant semantics. This distributional perspective improves robustness in multi-class debiasing and yields a stronger fairness--utility trade-off across diverse VLM downstream evaluations. Our code is available at https://github.com/QDRhhhh/W4D.

Clustering-based features are widely used in machine learning, but most methods must choose a resolution---a choice that is global, fixed, and ad hoc. Recent work shows that varying the resolution parameter produces only a finite set of structurally stable partitions, known as configurations. Based on this, we introduce Configuration-Mixed Prediction (CMP), a setting where models learn to adaptively weight these configurations per sample for downstream prediction. We propose MixConfig, a plug-and-play feature augmentation module that extracts configurations from any embedding and learns energy-aware mixing weights via a novel selector that jointly reasons about sample context, cluster assignments, and stability statistics. Experiments across tabular, molecular, vision, and text domains demonstrate consistent improvements over single-resolution and static baselines across diverse predictor architectures, with gains particularly pronounced in low-data regimes.

Multivariate time series (MTS) forecasting critically depends on modeling inter-variable dependencies, yet existing paradigms face a trade-off: channel-isolation strategies can suffer from information fragmentation in strongly coupled systems, whereas channel-interaction methods often introduce spurious interactions among irrelevant variables. To address this challenge, we propose Coherent Resonance Interaction with Spectral Priors (Crisp). Crisp adopts the principle that effective information exchange should occur only between variables with compatible oscillatory patterns. Concretely, we derive spectral priors in the frequency domain to construct dynamic resonance topologies. With a differentiable, adaptive, and strictly sparse blocking mechanism, Crisp forces attention weights for spectrally inconsistent neighbors to be exactly zero. In addition, we introduce a spectral-gated feature filtering module to refine variable representations using intrinsic spectral characteristics. Extensive experiments demonstrate that Crisp significantly outperforms 20+ baselines. Our code is available at Anonymous GitHub.

Test-time scaling (TTS) can boost LLM reasoning through long chain-of-thought (CoT), but the linear KV-cache growth amplifies the memory-bound bottleneck of LLM decoding. Query-aware sparse decoding methods can achieve state-of-the-art performance under constrained FLOP budget, but are mainly constrained by both sequential-dependent page filtering and coarse-grained token selection, hampering the serving efficiency and model performance on TTS tasks under high concurrency and long CoT scenarios, where token selection can even occupy higher runtime than the forward pipeline itself. In this paper, we first find that the query state of the current decoding token can be approximated in a unified manner from a short sliding window of recent queries, enabling training-free query-aware sparsity without sequential dependency in the decoding loop. Based on the findings, we propose $\texttt{\textbf{AsyncSpade}}$, an asynchronous framework for efficient TTS, built on two core components: $\textbf{(1) a novel light-weight temporal-regressive module}$ that predicts the next-token query state, and $\textbf{(2) an asynchronous disaggregated framework}$ that decouples the KV cache selection from the auto-regressive decoding loop, overlapping the token-level KV selection with the forward inference computation through asynchronism, thereby eliminating the sequential dependency without sacrificing model performance. We validate the effectiveness of $\texttt{AsyncSpade}$ on common LLM serving setups with an A100 node, where $\texttt{AsyncSpade}$ can fully overlap KV-cache operations with the inference pipeline within a certain workload range, $\textbf{achieving theoretical optimal time-per-output-token~(TPOT)}$. Specifically, $\texttt{AsyncSpade}$ delivers over 20% reduction on TPOT compared to SoTA baseline ($\textit{i.e.}$ Quest) and at least 50% TPOT reduction compared to full attention on Qwen3-8B and Qwen3-32B models, while matching or surpassing their accuracy on various TTS benchmarks (AIME-24/25, GPQA-Diamond, MATH-500). Our code is available through https://github.com/UNITES-Lab/AsyncSpade.

Graph Masked Auto-Encoder (GMAE) has emerged as a prevalent self-supervised paradigm, showing superior performance in graph learning. However, existing methods mainly emphasize reconstruction outcomes and give limited specification to how neighborhood context is used for reconstruction. Our experimental investigation shows that, even when reconstruction outcomes are similar, the ways of using neighborhood context differ substantially, resulting in performance shifts. To address this issue, we propose Consistency- and Coverage-aware Graph Masked Auto-Encoder (C2-GMAE), which encourages more consistent use of neighborhood context and promotes broader training coverage in the graph. Specifically, C2-GMAE leverages positional encoding as an observable structural reference, introduces density-partitioned masking to improve coverage across regions, and amplifies heterophilic edges to reduce the attenuation of discriminative relational information during reconstruction. Extensive experiments on multiple benchmarks demonstrate that C2-GMAE improves downstream performance against GMAE baselines.

Supervised fine-tuning (SFT) is computationally efficient but often yields inferior generalization compared to reinforcement learning (RL). This gap is primarily driven by RL’s use of on-policy data. We propose a framework to bridge this chasm by enabling On-Policy SFT. We first present ***Distribution Discriminant Theory (DDT)***, which explains and quantifies the alignment between data and the model-induced distribution. Leveraging DDT, we introduce two complementary techniques: (i) ***In-Distribution Finetuning (IDFT)***, a loss-level method to enhance generalization ability of SFT, and (ii) ***Hinted Decoding***, a data-level technique that can re-align the training corpus to the model’s distribution. Extensive experiments demonstrate that our framework achieves generalization performance on par with prominent offline RL algorithms, including DPO and SimPO, while maintaining the efficiency of an SFT pipeline. The proposed framework thus offers a practical alternative in domains where RL is infeasible. We will open-source the code and data on GitHub.

We introduce a new context-enriched, multimodal time series forecasting benchmark TimesX. TimesX contains a wide selection of high-quality real-world time series with diverse domains and textual contexts obtained from an automated data generation pipeline, which helps address three main issues of existing multimodal forecasting benchmarks: (1) poor generalization due to the small scale and synthetic nature of benchmark data, (2) very limited types of textual contexts in the benchmarks, and (3) an inability to mitigate data leakage in evaluation. We conduct a thorough empirical study of zero-shot multimodal forecasting approaches on TimesX. Our results suggest that many approaches that perform well on existing benchmarks may fail on TimesX. In contrast, simple ensemble methods that leverage rich textual context accompanying time-series can outperform strong baselines on the TimesX benchmark.

Foundation models often generate unreliable answers, while heuristic uncertainty estimators fail to fully distinguish correct from incorrect outputs, causing users to accept erroneous answers without any statistical guarantee. We address this problem through selection-conditioned risk control, aiming to ensure that an accepted prediction has an error probability no larger than a user-specified risk level. To this end, we propose LEC, a principled framework that reframes selective prediction as a decision problem governed by a *linear expectation constraint* over selection and error indicators. This formulation directly controls the ratio between the expected number of accepted errors and the expected number of accepted predictions, which corresponds to the marginal error probability conditioned on selection. Under exchangeability, we derive a *finite-sample sufficient condition* that relies only on a held-out calibration set, enabling the computation of a risk-constrained, retention-maximizing threshold. Furthermore, we extend LEC to two-model routing systems: if the primary model's uncertainty exceeds its calibrated threshold, the input is delegated to a subsequent model, while maintaining system-level selection-conditioned error control. Experiments on both closed-ended and open-ended question answering (QA) and vision question answering (VQA) demonstrate that LEC maintains the prescribed risk level in accepted predictions and substantially improves sample retention compared to baselines.

Speculative decoding has emerged as a widely adopted paradigm for accelerating large language model inference, where a lightweight draft model rapidly generates candidate tokens that are then verified in parallel by a larger target model. However, due to limited model capacity, drafts often struggle to approximate the target distribution, resulting in shorter acceptance lengths and diminished speedup. A key yet under-explored observation is that speculative decoding inherently provides *verification feedback* that quantifies the deviation between the draft and target models at no additional cost. This process naturally forms an iterative "draft commits--feedback provides--draft adapts" evolving loop, which precisely matches the *online learning* paradigm. Motivated by this connection, we propose OnlineSPEC, a unified framework that systematically leverages interactive feedback to continuously evolve draft models. Grounded in *dynamic regret minimization*, we establish a formal link between online learning performance and speculative system's acceleration rate, and develop novel algorithms via modern online learning techniques, including optimistic online learning that adaptively reuses historical gradients as predictive update hints, and online ensemble learning that dynamically maintains multiple draft models. Our algorithms are equipped with theoretical justifications and improved acceleration rates, achieving up to 24% speedup over seven benchmarks and five foundation models.

Multimodal large language models (MLLMs) struggle with numerical regression under longtailed target distributions. Token-level supervised fine-tuning (SFT) and point-wise regression rewards bias learning toward high-density regions, leading to regression-to-the-mean behavior and poor tail performance. We identify the lack of cross-sample relational supervision as a key limitation of existing MLLM training paradigms. To address it, we propose a distribution-aware reinforcement learning framework based on Group Relative Policy Optimization, which introduces batch-level comparison-based supervision via the Concordance Correlation Coefficient-based reward to align predicted and ground-truth distributions in terms of correlation, scale, and mean. The framework is plug-and-play, requiring no architectural modification. Experiments on a unified suite of long-tailed regression benchmarks show consistent improvements over SFT and existing MLLM regression methods, with particularly strong gains in medium- and few-shot regimes.

Higher-order structures are powerful relational modeling tools, yet existing spectral operators decompose topology into separate ranks, leaving practitioners to fuse information back to vertices through ad-hoc choices. We introduce _Collapsed Effective Operators_, which condense higher-order degrees of freedom into a single vertex-level operator via Schur complementation of a graded Laplacian. This yields a (generally dense) operator that encodes long-range interactions mediated by topology and is applicable to arbitrary higher-order constructs. We show it preserves positive semi-definiteness with a strict spectral upper bound relative to the rank-0 Hodge Laplacian, effectively lowering system energy under higher-order connectivity. Empirically, our operator improves spectral clustering, signal smoothing and enables the inclusion of topological features in neural network architectures via positional encoding.

The adversarial robustness of attributions is a fundamental requirement for reliable explainability in deep learning, yet existing approaches typically rely on computationally expensive explicit regularization. In this work, we show that attribution robustness can arise implicitly from the learning dynamics of standard stochastic gradient descent. We theoretically motivate this effect through connections between parameter-space and input-space curvature, and validate it across architectures, datasets, and attribution methods, with negligible computational overhead. In contrast, we prove that such robustness gains often does not transfer to attention-based attribution under softmax normalization, due to inherent entropy constraints, and we validate this limitation experimentally. Finally, we show that replacing softmax attention with kernel-based attention restores the robustness gains in transformer models. Our results highlight learning dynamics as a principled and practical mechanism for robust explainability, and reveal fundamental limitations of attention-based attribution under normalization.

Modern robotic policies increasingly rely on action chunking to execute complex tasks in the physical world. While action chunking improves temporal consistency at moderate action frequencies, it becomes insufficient when the action frequency is further increased (e.g., to 60~Hz). At such high frequencies, policies often fail to generate actions that are both temporally smooth and spatially consistent. We address this challenge by shifting high-frequency action learning from the action space to a latent space with variational autoencoder (VAE). This formulation significantly improves both temporal and spatial consistency of high-frequency control. To enable smooth real-time execution, we further introduce Reuse-then-Refine, a chunk-level refine strategy that improves continuity between adjacent action chunks under asynchronous inference. As a result, robots controlled by our policy can execute complex contact-rich tasks continuously, with less pauses and jerky motions. Experiments on three real-world contact-rich robotic tasks show that our approach consistently completes tasks with smooth motions. Our code and data are available at https://github.com/tars-robotics/RTR.

Autonomous image-editing agents powered by multimodal large language models (MLLMs) improve transparency and controllability by translating high-level instructions into tool-mediated edit sequences, but training such agents with reinforcement learning often relies on dense proxy rewards (e.g., incremental image-quality score gains) to compensate for sparse human feedback. When these proxies overvalue small local changes, the resulting optimization signal can be dominated by numerically measurable yet perceptually negligible edits, biasing policy gradients toward proxy artifacts rather than meaningful progress. We propose B-Spar, a reward-centric Reinforcement Learning framework for perceptually aligned image retouching under sparse feedback that combines prior-guided trajectory sampling to reduce inefficient exploration, Bayesian reward modeling to densify sparse binary feedback into a stable training signal, and anchor-regularized policy optimization to steer updates toward high-reward regions while preventing early mode collapse. Experiments on public benchmarks demonstrate that B-Spar improves perceptual quality and metric alignment with stable training and competitive inference efficiency over strong prompt-based and training-based baselines. Notably, it outperforms AIGC-based baselines by over 95\% in perceptual quality, achieving an improvement of approximately 33.5\% over the state-of-the-art.

MLLM-powered embodied agents deployed in real-world environments encounter physical hazards. However, existing approaches lack explicit mechanisms for identifying hazards and reasoning about action-conditioned risks, leading agents to either miss risky interactions or over-identify risks. To address this, we propose EMBGuard, the first MLLM-based safety guardrail for embodied agents designed to decouple physical risk reasoning from agent policy. By evaluating a (visual observation, action) pair, EMBGuard identifies hazardous configurations and provides natural language explanations of potential risks. Alongside EMBGuard, we contribute EMBHazard, a training dataset of 15.1K action-conditioned pairs, and EMBGuardTest, a benchmark of 329 manually curated real-world scenarios spanning seven physical risk categories. Through compositional variation of hazards and actions, we generate diverse risky and benign scenarios that agents may encounter during planning. Despite its compact size (2B, 4B), EMBGuard achieves performance competitive with proprietary MLLMs (e.g., GPT-5.1, Gemini-2.5-Pro) while significantly reducing the false-positive rates that hinder real-time deployment. We make the code, data, and models publicly available at https://github.com/dongwxxkchoi/EMBGuard.

While CLIP has achieved strong performance across vision–language tasks, fine-grained image–text alignment remains challenging. Recent efforts improve textual granularity by leveraging long, detailed descriptions and replacing CLIP’s text encoder with LLM, but often overlook the visual-side bottleneck: achieving finer alignment requires region- and pixel-level visual grounding. To address it, we propose PixCLIP, a framework that jointly enhances both sides by accommodating visual prompt regions and long-form text within a unified training objective. Firstly, to support training at this granularity, we develop an automated annotation pipeline that produces long-form descriptions with pixel-level localization, and use it to construct LongGRIT, a large-scale dataset with 1.5M samples. Furthermore, we introduce a three-branch pixel–text alignment framework that aligns image regions with corresponding textual descriptions across multiple granularities. Experiments show that PixCLIP achieves state-of-the-art performance on pixel- and region-level alignment tasks while preserving strong results on standard global image–text retrieval benchmarks, even with arbitrarily shaped region prompts and long texts. Our code is available at https://github.com/StuHude/PixCLIP.

Despite rapid progress in Grounded Visual Reasoning (GVR) with MLLMs and RL-style fine-tuning, existing approaches often lack effective learning signals for intermediate grounding decisions and are prone to shortcut solutions. In this work, we explicitly decompose GVR into Evidence Generation followed by Counterfactual Answer Reasoning, and formalize this structure as a Causal Grounding Graph (CGG) in which the generated evidence acts as a causal mediator. Building on this formulation, we propose CofactGVR, which estimates the mediator’s utility via a matched counterfactual intervention that perturbs the predicted region while keeping the original image–question context fixed. The factual–counterfactual reward gap yields a principled intermediate bonus, selectively assigned to high-quality factual rollouts to promote evidence-faithful reasoning. To further stabilize and efficiently exploit this causal training signal, we incorporate a Quantile-filtered Prioritized Advantage Sampling (Q-PAS) strategy that preferentially updates on trajectories with high-magnitude advantages while filtering low-signal samples. Extensive experiments across GVR benchmarks show consistent improvements, indicating that CofactGVR strengthens reliance on informative visual evidence under controlled interventions. Our source code will be released at https://github.com/zhangy0822/CofactGVR.