ICML 2026 论文列表

Automatic Prompt Optimization (APO) enables Large Language Models (LLMs) to adapt to specific tasks while minimizing manual engineering costs. However, since existing APO approaches either rely solely on multi-round iterative procedures or use model-specific generators tailored to optimizing prompts for a single model and objective, they are not readily applicable to auto-routing scenarios, which require operating over diverse LLMs and juggling multiple, often competing, trade-offs. To address this issue, we propose TAMPO, a novel task- and model-aware APO framework for auto-routing in LLM-based systems. Specifically, to capture performance variation across a broad range of tasks and models, we construct a comprehensive, heterogeneity-aware dataset to train an uncertainty-aware reward model. Serving as an offline proxy, this reward model can greatly mitigate reward hacking, allowing TAMPO to learn an optimal multi-objective conditional policy for robust prompt generation. Based on the user requirements encoded in our defined preference vector, this policy enables flexible control over prompt generation and supports a cost-effective deployment strategy. Extensive experiments across 86 tasks demonstrate that TAMPO effectively maintains performance stability across diverse tasks and models, providing a robust, controllable solution for auto-routing in various LLM-based systems.

While multimodal large language models (MLLMs) have made substantial progress in single-image spatial reasoning, multi-image spatial reasoning, which requires integration of information from multiple viewpoints, remains challenging. Cognitive studies suggest that humans address such tasks through two mechanisms: *cross-view correspondence*, which identifies regions across different views that correspond to the same physical locations, and *stepwise viewpoint transformation*, which composes relative viewpoint changes sequentially. However, existing studies incorporate these mechanisms only partially and often implicitly, without explicit supervision for both. We propose Human-Aware Training for Cross-view correspondence and viewpoint cHange (HATCH), a training framework with two complementary objectives: (1) Patch-Level Spatial Alignment, which encourages patch representations to align across views for spatially corresponding regions, and (2) Action-then-Answer Reasoning, which requires the model to generate explicit viewpoint transition actions before predicting the final answer. Experiments on three benchmarks demonstrate that HATCH consistently outperforms baselines of comparable size by a clear margin and achieves competitive results against much larger models, while preserving single-image reasoning capabilities.

Generative video diffusion models have shown strong robustness to large motion and occlusions for video frame interpolation (VFI). However, their inference efficiency lags significantly behind learning-based methods due to the structural redundancy of pairwise inference and the procedural latency of multi-step iterative denoising. To address these limitations, we propose SpeedVFI, a task-specific one-step diffusion formulation that recasts generative VFI as unified sequence interpolation. SpeedVFI achieves dual efficiency improvements by interpolating the entire video sequence in a single forward pass to eliminate pairwise overhead, and by distilling the generation trajectory into a one-step denoising process to bypass iterative latency. To make this formulation effective for VFI, we introduce temporal RoPE alignment for temporally consistent conditioning and noise-centric partial attention to reduce computational overhead while preserving global context. Extensive experiments demonstrate that SpeedVFI accelerates diffusion-based VFI by orders of magnitude while maintaining competitive quantitative and visual quality.

Biological and neuromorphic recurrent neural networks (RNNs) are subject to spatial and temporal locality constraints on the information that can plausibly be used during learning. A common strategy to satisfy these constraints is to modify gradient descent by neglecting non-local terms to varying degrees, as in random feedback local online (RFLO) learning and truncated backpropagation through time (tBPTT). However, the learning dynamics of these algorithms, and how they compare with BPTT, remain poorly understood. We apply dynamical systems theory to data-aligned linear RNNs---whose dynamics can be separated into orthogonal modes---to compare stationary solutions, stability properties, and convergence rates, finding qualitatively distinct behaviour for RFLO versus BPTT and one-step tBPTT. We further observe that the solutions learned by RFLO are restricted to low-rank perturbations of initial parameters, a result which holds beyond the data-aligned setting. Our work provides analytical insight into how locality constraints shape learning dynamics, with implications for neuroscientific models of learning and alternative optimization approaches for RNNs.

The unification of generative details and discriminative semantics presents a structural paradox in *diffusion-based representation learning*. Early approaches decouple semantics from generation, inevitably compromising representational completeness (i.e., *information split*). While recent bridge-based methods achieve unification via a tightly coupled mapping, they suffer from *information overload*. This is because unconstrained reconstruction objectives incentivize the encoder to entangle high-frequency stochastic noise into the latent bottleneck. To solve this, we introduce *asymmetric rectified contrastive diffusion autoencoder* (ArcDAE), which rebuilds the diffusion bridge as a *dynamic sifter*. Through imposing a *timestep-aware rectification constraint* that orthogonalizes the semantic manifold from the stochastic noise space, ArcDAE compels the bottleneck to distill discriminative features while actively shedding high-frequency redundancy. Consequently, our approach eliminates the overload trap without reverting to decoupling. Extensive experiments validate the superiority of our FFHQ-trained ArcDAE, surpassing state-of-the-art methods by up to 6.4% in downstream semantics regression and 9.7% in reconstruction fidelity.

Classifier-Free Guidance (CFG) is a cornerstone of flow-matching models, significantly enhancing visual quality and prompt adherence. However, high guidance scales inherently violate the optimal transport dynamics, leading to visual artifacts and mode collapse. In this paper, we investigate the mechanisms of this failure through the lens of velocity moment decomposition. Our analysis reveals that the distributional shift induced by CFG decouples into two geometric components: a Linear Barycentric Drift that shifts the global distribution center, and a Quadratic Energetic Instability that injects surplus kinetic energy, disrupting the transport cost and triggering variance explosion. To mitigate these issues, we introduce MIST (Moment-aligned Invariant Stability Transform), a training-free method designed to confine the sampling trajectory to the learned data manifold. MIST comprises two hierarchical stages: (1) Invariant Alignment (IA), a global statistical rectifier that restores structural integrity by removing the linear drift and realigning the energy profile; and (2) Stability Thresholding (ST), a local dynamical regulator that enforces Lipschitz-like smoothness via temporal decay and spatial suppression. MIST enables robust, high-fidelity generation across a wide range of guidance scales while consistently improving performance at moderate scales. Extensive experiments on diverse text-to-image and text-to-video benchmarks demonstrate that MIST outperforms standard CFG and state-of-the-art corrections, establishing a new benchmark for robust guidance in flow-based generative models.

Reconstruction-based methods are a dominant paradigm in time series anomaly detection (TSAD), however, their near-universal reliance on Mean Squared Error (MSE) loss results in statistically flawed reconstruction residuals. This fundamental weakness leads to noisy, unstable anomaly scores, hindering reliable detection. To address this, we propose Constrained Gaussian-Noise Optimization and Smoothing (COGNOS), a universal, model-agnostic enhancement framework that tackles this issue at its source. COGNOS introduces a novel Gaussian-White Noise Regularization strategy during training, which directly constrains the model's output residuals to conform to a Gaussian white noise distribution. This engineered statistical property creates the ideal precondition for our second contribution: Adaptive Residual Kalman Smoother that provably operates as a statistically robust estimator to denoise the raw anomaly scores. Extensive experiments on multiple benchmarks demonstrate that COGNOS consistently enhances the performance of state-of-the-art backbones significantly, validating the efficacy of coupling statistical regularization with adaptive filtering.

As Large Language Models transition to autonomous agents, user inputs frequently violate cooperative assumptions (e.g., implicit intent, missing parameters, false presuppositions, or ambiguous expressions), creating execution risks that text-only evaluations do not capture. Existing benchmarks typically assume well-specified instructions or restrict evaluation to text-only, single-turn clarification, and thus do not measure multi-turn disambiguation under grounded execution risk. We introduce DRIFT-BENCH, the first diagnostic benchmark that evaluates agentic pragmatics under input faults through multi-turn clarification across state-oriented and service-oriented execution environments. Grounded in classical theories of communication, DRIFT-BENCH provides a unified taxonomy of cooperative breakdowns and employs a persona-driven user simulator with the Rise evaluation protocol. Experiments show substantial performance drops under these faults, with clarification effectiveness varying across user personas and fault types. DRIFT-BENCH connects clarification studies with agent benchmarking, providing a framework to diagnose failures arising from faulty user inputs.

Cooperative reasoning under incomplete information remains complex for both humans and multi-agent AI, requiring agents to transcend individual logic in favor of recursive Theory-of-Mind (ToM) and strategic coordination. To investigate these challenges, we conduct a large-scale evaluation of 17 state-of-the-art LLMs (4B–600B+) on Hanabi card game across 2–5 players. To examine their limitations, we analyze the impact of context engineering and scaffold robustness, ranging from minimal prompts (Watson setting) to Bayesian-motivated scaffolding (Sherlock setting) and multi-turn working memory (Mycroft setting). Our findings reveal that: (1) top-performing models can autonomously track game states via internal working memory, although not reliably, and (2) cross-play performance scales smoothly with model capability. However, even the best models (scoring ≈ 15/25) trail specialist human experts (> 20/25). We introduce and release two novel datasets: HanabiLogs (1,520 annotated trajectories) and HanabiRewards (560 games with dense move-level utilities). By fine-tuning a 4B open-weight model (Qwen3-Instruct) on our datasets, we achieve performance gains of up to 156%, bringing performance to within 3 points of a strong proprietary reasoning model (o4-mini) and surpassing the best non-reasoning model (GPT-4.1) by 52%. Crucially, our HanabiRewards RL-finetuned model further generalizes beyond Hanabi, improving performance on a cooperative group-guessing benchmark by 11 percentage points, temporal reasoning on EventQA by 6.4 points, instruction-following on IFBench by 1.7 Pass@10, and matching AIME 2025 mathematical reasoning Pass@10. Code and datasets are available at https://app.primeintellect.ai/dashboard/environments/mahesh-ramesh/hanabi.

Informal mathematics has been central to modern large language model (LLM) reasoning, offering flexibility and efficient construction of arguments. However, purely informal reasoning is prone to logical gaps and subtle errors that are difficult to detect and correct. In contrast, formal theorem proving provides rigorous, verifiable mathematical reasoning, where each inference step is checked by a trusted compiler, but lacks the exploratory freedom of informal problem-solving. This mismatch leaves current LLM-based math agents without a principled way to combine the strengths of both paradigms. In this work, we introduce Hermes, the first tool-assisted agent that explicitly interleaves informal reasoning with formally verified proofs in Lean. The framework performs intermediate formal checking to prevent reasoning drift and a memory module for proof continuity across multi-step reasoning chains, enabling both exploration and verification. We evaluate Hermes on four challenging mathematical reasoning benchmarks using LLMs of varying parameter scales, from small models to state-of-the-art systems. Across all settings, Hermes reliably improves the reasoning accuracy of base models while substantially reducing reasoning token usage and computational cost compared to reward-based approaches. On difficult datasets such as AIME and HARDMath2, Hermes@1 achieves up to a 40% accuracy improvement while using 80% fewer total inference FLOPs. When scaled at test time, Hermes@5 boosts accuracy further by 20%.

Effective time series representation is critical for revealing temporal dynamics in many fields. However, existing approaches encounter fundamental limitations. Discrete-time representations struggle with irregular sampling and the tradeoff of fidelity and efficiency, while traditional implicit neural representations suffer from spectral bias and frequency entanglement. To address these challenges, we conceptualize time series as the superposition of continuous trends and discrete events from a continuous-time perspective and propose DualTimesField, a framework that utilizes dual implicit neural fields. Its Continuous Time Field captures smooth trends through bandwidth-limited parameterization, while a Discrete Geometric Field models transient events using learnable Gabor atoms, gated sparsity, and coarse-to-fine scale annealing. This explicit field separation effectively overcomes both limitations. Experiments on nine real-world benchmarks demonstrate substantial improvements in representation fidelity, achieving 51.2% average MSE reduction over discrete-time baselines and competitive interpolation on irregular data. Code is available at https://github.com/WisdomTogether/DualTimesField.

Graph Neural Networks (GNNs) are bottlenecked by sparse, irregular memory access. Popular frameworks such as DGL and PyTorch Geometric support general message passing, but complex layers often materialize edge-wise intermediates, increasing memory traffic and limiting scalability on large graphs. We take an I/O- and arithmetic-intensity--centric view and show that widely used layers fall into three kernel families: SpMM-based convolutions, reduction-based aggregations, and attention-based layers (GATv2/Graph Transformer). For each family, we develop GPU kernels that reduce data movement, improve locality, and remain robust across realistic graphs. We also study graph reordering and find that its impact depends on the kernel mapping: it benefits neighbor-parallel (gather-dominated) kernels more consistently than feature-parallel designs. Empirically, our fused attention kernels reach up to **3.9**$\times$ speedup for Graph Transformer (median **1.6**$\times$), with Tensor Core (block-sparse) variants up to **7.3**$\times$ on locally dense graphs; for GATv2 we reach up to **8.5**$\times$ speedup (median **2.0**$\times$) while reducing peak memory by up to **76**$\times$ (median **6**$\times$). Our degree-aware reduction kernels achieve up to **10**$\times$ speedup (median **2.6**$\times$). For SpMM-based layers, properly cached cuSPARSE achieves up to **8**$\times$ speedup over DGL and outperforms evaluated custom baselines in the majority of evaluations. We release our implementations as drop-in replacements in our [GitHub repository](https://github.com/yandex-research/On-Efficient-Scaling-Of-GNNs) to support reproducible, hardware-aware GNN acceleration.

Embedding geometry plays a fundamental role in retrieval quality, yet dense retrievers for retrieval-augmented generation (RAG) remain largely confined to Euclidean space. However, natural language exhibits hierarchical structure from broad topics to specific entities that Euclidean embeddings fail to preserve, causing semantically distant documents to appear spuriously similar and increasing hallucination risk. To address these limitations, we introduce hyperbolic dense retrieval, developing two model variants in the Lorentz model of hyperbolic space: HyTE-FH, a fully hyperbolic transformer, and HyTE-H, a hybrid architecture projecting pre-trained Euclidean embeddings into hyperbolic space. To prevent representational collapse during sequence aggregation, we introduce the Outward Einstein Midpoint, a geometry-aware pooling operator that provably preserves hierarchical structure. On MTEB, HyTE-FH outperforms equivalent Euclidean baselines, while on RAGBench, HyTE-H achieves up to 29\% gains over Euclidean baselines in context relevance and answer relevance using substantially smaller models than current state-of-the-art retrievers. Our analysis also reveals that hyperbolic representations encode document specificity through norm-based separation—with over 20\% radial increase from general to specific concepts—a property absent in Euclidean embeddings, underscoring the critical role of geometric inductive bias in faithful RAG systems. The code is available at: https://github.com/Graph-and-Geometric-Learning/HypRAG

Recently, the physics reasoning capabilities of (M)LLMs have attracted growing attention. However, existing physics benchmarks lack systematic coverage of recent physics Olympiads and direct comparison with human contestants. We present **HiPhO**, the first benchmark dedicated to high school physics Olympiads with human-aligned evaluation. HiPhO highlights three key innovations. **(1) Comprehensive data:** it compiles 13 latest Olympiads from 2024--2025, covering international and regional competitions and spanning mixed modalities from text-only to diagram-based problems. **(2) Professional evaluation:** it adopts official rubrics for fine-grained answer- and step-level grading aligned with human examiners. **(3) Human-level comparison:** it assigns gold, silver, and bronze medals to models based on official medal scores, enabling direct comparison with human contestants. Evaluating 30 (M)LLMs across 13 exams, we find that most open-source MLLMs remain at or below the bronze level, open-source LLMs demonstrate notable progress with multiple gold medals, and closed-source MLLMs achieve 6-13 gold medals, while most models still fall well short of full marks. These results underscore the substantial gap between current (M)LLMs and top human contestants, as well as the room for further improvement. The dataset and leaderboard are available at https://huggingface.co/datasets/SciYu/HiPhO and https://phyarena.github.io, respectively.

The human brain is a biologically instantiated on-device neural system that integrates both learning and inference in a unified architecture, which enables rapid and flexible learning on-the-fly. This extraordinary online learning ability is realized through biological learning mechanisms operating on a well-initialized innate model. This work considers the on-chip edge learning upon pretrained models with zeroth-order (ZO) methods. ZO optimization methods, which resemble bio-plausible perturbation-based learning, offer a promising approach that enables learning with only forward passes and hence can significantly reduce the complexity of on-chip hardware implementation. However, in this work we show that applying ZO methods to spiking neural networks (SNNs) is non-trivial due to the step-function nature of spiking activation. We analyze the challenges posed by the spiking activation, and reveal a variance amplification effect of it. Based on this insight, we propose a subspace-based ZO (SZO) method that leverages the intrinsic low-dimensional structure of the SNN optimization trajectory. By learning in a low-dimensional subspace, SZO substantially enhances ZO learning efficacy, achieving accuracy comparable to first-order (FO) methods with faster learning speed than full-space BP. We evaluate SZO on model training from scratch, continual training, and unsupervised adaptation. Experimental results demonstrate that SZO closely approaches FO training performance for the first time while offering fast learning speed.

Recent advances in diffusion models show promising potential to accelerate nonconvex problem solving by leveraging their multimodality. However, most existing diffusion-based optimization approaches rely on supervised learning and lack a mechanism to enforce constraint satisfaction, which is required in real-world applications. In that case, we investigate and theoretically analyze the inherent problem of supervised diffusion solvers and identify the distributional misalignment problem, i.e., the generated solution distribution often exhibits low probability mass on the feasible region. To resolve this issue, we propose DiOpt, a new diffusion-based learning framework for constrained nonconvex optimization, which effectively learns the mapping from noise to the constraint region. Specifically, this framework operates in two distinct phases: an initial warm-start phase, implemented via supervised learning, followed by a bootstrapping training phase. This dual-phase architecture is designed to iteratively refine solutions, thereby improving the objective function with high constraint satisfaction. Finally, we also employ a solution selection technique in inference for better optimality. Notably, DiOpt is the first successful integration of the diffusion solver in constrained nonconvex optimization. Evaluations on diverse nonconvex tasks demonstrate the superiority of DiOpt in both optimality and constraint satisfaction. Our official page is released at \url{https://dingsht.tech/diopt-webpage}.

Determining the complexity, or Intrinsic Dimension (ID), of data is fundamental to efficient and interpretable representation learning. This is particularly challenging in multi-modal settings when trying to learn disentangled representations for shared and private information. Existing techniques leave a critical gap: they are often static, uni-modal, or in the case of contrastive methods, adapt only to the shared ID implicitly. We introduce Fidelity-Guided Rank Optimization (FiGuRO), a framework for approximating the ID of uni- and multi-modal data under constraints of model capacity and hyperparameters. FiGuRO learns the dimensions of low-rank projections using truncated singular value decomposition and an algorithm that determines when to reduce or increase dimension and in which latent space. Disentanglement of shared and private information arises as an emergent property of this optimization, eliminating the need for complex auxiliary loss functions. We demonstrate that FiGuRO outperforms existing ID estimation techniques and is more robust to hyperparameter changes. Across simulations and real-world data, FiGuRO captures distinct ID scales and varying subspace ratios, and decomposes shared and private information successfully. Furthermore, we show that FiGuRO can be applied to modern uni-modal pretrained models, enabling efficient, post-hoc disentanglement of multi-modal representations.

Quadratic Discriminant Analysis (QDA) assumes Gaussian class-conditional distributions, causing systematic misclassification when data exhibit heavy tails. We propose Stable-QDA, which replaces the Gaussian likelihood with a symmetric $\alpha$-stable likelihood that decays polynomially rather than exponentially in Mahalanobis distance. Crucially, we find that correcting likelihood misspecification yields larger gains than robustifying parameter estimation: standard estimators (sample mean, Ledoit--Wolf covariance) often outperform robust alternatives when class heteroscedasticity is discriminative. We provide consistency guarantees under infinite-variance regimes, data-driven diagnostics for estimator selection, and demonstrate 15--53\% error reduction on real-world heavy-tailed benchmarks.

End-to-end deep learning has achieved impressive results but often relies on large labeled datasets, exhibits limited generalization to unseen scenarios, and incurs substantial computational cost. Classical optimization methods, in contrast, are more data-efficient and lightweight but frequently suffer from slow convergence. Learned optimizers aim to bridge this gap, yet existing approaches have focused primarily on first-order methods, while learned second-order optimization has received much less attention. We introduce L-SR1, a learned second-order optimizer inspired by the classical Symmetric Rank-One (SR1) method. At its core, L-SR1 employs a Projection-Guided Secant Mechanism (PGSM) that generates positive semi-definite preconditioners and biases meta-training toward the quasi-Newton secant relation. Through controlled analytic benchmarks, we study stability, generalization across problem dimensions, and search direction quality, and further evaluate L-SR1 on Monocular Human Mesh Recovery (HMR), where it outperforms both classical and learned optimization-based baselines. With a compact model and no reliance on task-specific fine-tuning or annotated data, L-SR1 demonstrates strong generalization and can be integrated into a broad range of iterative optimization problems to accelerate convergence and reduce the required number of iterations.