AI conference papers

AI conference paper library

Search papers from 30+ top conferences, including NeurIPS, ICLR, ICML, CVPR, ACL and AAAI. Filter by keyword, then open any paper for its abstract, authors and links.

ICML 2026

RobuQ: Pushing DiTs to W1.58A2 via Robust Activation Quantization

Kaicheng Yang, Xun Zhang, Haotong Qin, Yucheng Lin, Kaisen Yang, Xianglong Yan, Yulun Zhang

Diffusion Transformers (DiTs) have emerged as a powerful backbone for image generation, offering superior scalability over U-Nets. However, their practical deployment is hindered by significant computational costs. While Quantization-Aware Training (QAT) shows promise, its application to DiTs is challenged by the high sensitivity and complex distributions of activations. Identifying activation quantization as the primary bottleneck for low-bit settings, we propose **RobuQ**, a systematic QAT framework. We first establish a strong ternary weight (W1.58A4) baseline. Building on this, we introduce **RobustQuantizer**, which utilizes the Hadamard transform to convert unknown per-token distributions into normal distributions. Furthermore, we propose **AMPN**, the first **A**ctivation-only **M**ixed-**P**recision **N**etwork pipeline, applying ternary weights globally while allocating layer-specific activation precisions to eliminate information bottlenecks. Extensive experiments demonstrate that **RobuQ** achieves state-of-the-art performance on **ImageNet-1K**, representing the first stable image generation with activations quantized to an average of 2 bits. Code is available at https://github.com/racoonykc/RobuQ.

ICML 2026

Granularity-Aware Adaptive Classifier Expansion via Zero-Shot Learning

Xiangyu Wang, Yanze Gao, Changxin Rong, Lyuzhou Chen, Xiren Zhou, Huanhuan Chen

Zero-shot classifier expansion aims to recognize unseen classes by learning a shared mechanism to map semantics of all classes to classifier weights without access to images. However, existing methods rely on a shared mapping, which is difficult to classify in scenarios containing a mixture of distinct and similar classes, especially with the continuous expansion of classes. Since this mapping prioritizes general attributes for distinct classes while neglecting subtle attributes for similar ones, this granularity mismatch, compounded by sensitivity to noise, induces optimization interference where gradients from distinct classes dominate the learning process. To overcome this limitation, a granularity-aware adaptive framework with interventions is introduced to balance them. Specifically, this method first generates multi-source semantics by intervening on non-causal noise, then discovers latent class structure to separate distinct classes, and finally refine similar classes to synthesize weights with invariance to non-causal noise. The effectiveness is demonstrated through theoretical and empirical analysis in multiple aspects.

ICML 2026

One LR Doesn’t Fit All: Heavy-Tail Guided Layerwise Learning Rates for LLMs

Di He, Songjun Tu, Keyu Wang, Lu Yin, Shiwei Liu

Learning rate configuration is a fundamental aspect of modern deep learning. The prevailing practice of applying a uniform learning rate across all layers overlooks the structural heterogeneity of Transformers, potentially limiting their effectiveness as the backbone of Large Language Models (LLMs). In this paper, we introduce Layerwise Learning Rate (LLR), an adaptive scheme that assigns distinct learning rates to individual Transformer layers. Our method is grounded in Heavy-Tailed Self-Regularization (HT-SR) theory, which characterizes the empirical spectral density (ESD) of weight correlation matrices to quantify heavy-tailedness. Layers with weaker heavy-tailedness are assigned larger learning rates to accelerate their training, while layers with stronger heavy-tailedness receive smaller learning rates. By tailoring learning rates in this manner, LLR promotes balanced training across layers, leading to faster convergence and improved generalization. Extensive experiments across architectures (from LLaMa to GPT-nano), optimizers (AdamW and Muon), and parameter scales (60M–3B, up to 100B tokens) demonstrate that LLR achieves up to 1.5× training speedup and outperforms baselines, notably raising average zero-shot accuracy from 47.09% to 49.02% for 1B models and from 48.58% to 50.61% for 3B models. A key advantage of LLR is its low tuning overhead: it transfers nearly optimal LR settings directly from the uniform baseline. Code is available at https://github.com/hed-ucas/Layer-wise-Learning-Rate.

ICML 2026

Next-Token Prediction and Regret Minimization

Mehryar Mohri, Clayton Sanford, Jon Schneider, Kiran Vodrahalli, Yifan Wu

We study the robustification of next-token prediction models in adversarial online decision-making environments. Suppose a next-token predictor is trained on a distribution over sequences of opponent actions, and a decision maker acts by approximately best responding to the model’s predictions. If this induced decision-making algorithm achieves low adversarial regret, we call such distributions low-regret distributions. For models with unbounded context, we show that every distribution is exponentially close in total variation distance to a low-regret distribution. Thus, sublinear regret can always be achieved while changing the original next-token prediction distribution by only a negligible amount. By contrast, for models with bounded context windows, as in standard transformer architectures, we prove an impossibility result: some distributions are $\Theta(1)$-far from every low-regret distribution. We complement these results with two positive findings. First, allowing a modest increase in context length restores robustification with near-optimal regret guarantees. Second, the unbounded-context robustification procedure can be implemented by adding a constant number of layers to a standard transformer architecture. Finally, we provide empirical evidence that small transformer models can be efficiently trained to represent these robustified low-regret distributions.

ICML 2026

How Hard Can It Be? Hardness-Aware Multi-Objective Unlearning

Jiangwei Chen, Xinyuan Niu, Rachael Hwee Ling Sim, Zhengyuan Liu, Nancy F. Chen, Bryan Kian Hsiang Low

Machine unlearning aims to remove the influence of specific forget training data due to privacy, copyright or bias concerns while maintaining the model performance on the remaining retain data. Existing unlearning algorithms, such as optimizing a weighted combination of losses, have tried to achieve these objectives of improving forget quality and maintaining retain utility. However, they do not guarantee that these objectives can be improved by a specified extent for all forget and retain data. In this work, we address this limitation with a novel and theoretically-grounded approach from a constrained optimization perspective. Firstly, we identify that the *hardness* of reconciling both objectives can be quantified by the similarity between the forget data and the retain data. Next, we derive an unlearning algorithm (HAMU) with the overall goal of guaranteeing a specified improvement in forget quality while minimizing the retain utility cost/degradation by updating the model weights based on our hardness measure. Our hardness measure also informs users when retain utility degradation is unavoidable, i.e., both objectives cannot be improved simultaneously, and stopping should be considered. Our algorithm is applicable to non-convex models and is easily parallelizable, making it readily deployable in real-world scenarios. We empirically demonstrate HAMU's superior performance over baselines on both image and text datasets using large models. Our code is available at https://github.com/aoi3142/HAMU.

ICML 2026

Overcoming PINNs Failure Modes In High Dimension With Low-Rank Fourier Sum

Natan Kaminsky, Daniel Freedman, Kira Radinsky

Physics-informed neural networks (PINNs) can be unreliable on PDEs with oscillatory, multiscale, stiff, or long-time solutions, and these difficulties worsen in high dimensions where collocation-based training yields large numerical integration error and high-variance gradients. We propose Low-Rank Fourier Sums (LoRFS), representing the solution as a low-rank sum of separable Fourier expansions (products of one-dimensional Fourier series across coordinates). This makes high-frequency structure explicit and enables closed-form evaluation of common physics-based objectives and their gradients (e.g., $L^2$ residual and variational losses), replacing sampling-based collocation estimates with analytic loss evaluation and eliminating sampling noise. We further provide theoretical results that clarify why LoRFS is particularly well suited to high-dimensional regimes. Across canonical PINN failure-mode benchmarks and their high-dimensional extensions, LoRFS consistently outperforms strong PINN baselines and remains stable in regimes where competing methods degrade.

ICML 2026

GOTabPFN: From Feature Ordering to Compact Tokenization for Tabular Foundation Models on High-Dimensional Data

Al Zadid Sultan Bin Habib, Md Younus Ahamed, Prashnna Kumar Gyawali, Gianfranco Doretto, Donald Adjeroh

We investigate how to make small tabular foundation models effective for High-Dimensional, Low-Sample Size (HDLSS) tabular prediction without retraining large backbones. We introduce Graph-guided Ordering with Local Refinement (GO-LR), show its equivalence to weighted Minimum Linear Arrangement, and interpret the practical solver as a TSP-path-style surrogate. We propose GOTabPFN,which builds on GO-LR, and a Neuro-Inspired Subunit Compression (NSC) unit to pool locally adjacent ordered features into meta-features, yielding a compact representation that makes TabPFN-style prediction practical in HDLSS regimes. Across tabular benchmarks, GOTabPFN improves stability and accuracy under tight token budgets.

ICML 2026

Transporting Task Vectors across Different Architectures without Training

Filippo Rinaldi, Aniello Panariello, Giacomo Salici, Angelo Porrello, Simone Calderara

Adapting large pre-trained models to downstream tasks often produces task-specific parameter updates that are expensive to relearn for every model variant. While recent work has shown that such updates can be transferred between models with identical architectures, transferring them across models of different widths remains unexplored. In this work, we introduce Theseus, a training-free method for transporting task updates across heterogeneous-width models. Rather than matching parameters, we characterize a task update by the functional effect it induces on intermediate representations. We formalize task-vector transport as a functional matching problem on observed activations and show that, after aligning representation spaces via orthogonal Procrustes analysis, it admits a stable closed-form solution that preserves the geometry of the update. We evaluate Theseus on vision and language models across different widths, showing consistent improvements over baselines without additional training or backpropagation. Our results show that task updates can be meaningfully transferred across architectures when task identity is defined functionally rather than parametrically. Code is available at https://github.com/apanariello4/merge-and-rebase.

ICML 2026

Pseudo-Mallows for Efficient Probabilistic Preference Learning

Sylvia Qinghua Liu, Valeria Vitelli, Carlo Mannino, Arnoldo Frigessi, Ida Scheel

We propose the Pseudo-Mallows distribution over the set of all permutations of $n$ items, to approximate the posterior distribution of the Bayesian Mallows model. The Bayesian Mallows model has been successfully used for recommender systems to learn personal preferences from highly incomplete users data. However current inference algorithms do not scale, preventing its use in real-time applications. The Pseudo-Mallows distribution is a product of univariate discrete Mallows-like distributions, where the quality of the approximation depends on the order of the $n$ items in the factorization sequence. In a variational setting, we optimize the variational order parameter by minimising a marginalized KL-divergence, conjecturing a certain form of the optimal variational order that depends on the data, and proposing an approximation algorithm for this discrete optimization. Empirical evidence and some theory support our conjecture. We demonstrate on clicking data that variational inference via the Pseudo-Mallows distribution allows much faster probabilistic preference learning compared to alternative MCMC-based options.

ICML 2026

ManifoldKV: Training-Free KV Cache Compression via Euclidean Outlier Detection

Debajyoti Datta, Trishala Neeraj, Bibek Paudel, Vyom Sharma, Subhabrata Mukherjee

Long-context inference is constrained by KV-cache memory, which grows linearly with sequence length; KV-cache compression therefore hinges on reliably selecting which past tokens to retain. Most geometry-based eviction methods score keys by cosine similarity to a global centroid, but cosine is scale-invariant and can discard magnitude cues that distinguish semantically salient tokens. We propose ManifoldKV, a training-free scorer that ranks tokens by Euclidean distance to the key centroid, capturing both angular and radial deviations. On the RULER benchmark, ManifoldKV achieves **95.7%** accuracy at 4K–16K contexts with 20% compression, matching the best geometric baseline overall while *decisively* outperforming it in two regimes where magnitude information is critical. First, on multi-key retrieval, ManifoldKV reduces directional collisions, achieving **92.4%** vs KeyDiff's 77.0% (+15.4 points) on 3-key NIAH at 50% compression. Second, to address dilution and performance collapse of global centroids at 64K context, we introduce WindowedManifoldKV, which restores accuracy to 84.3% at 25% compression, a 49-point recovery over global L2 and +3.2 points over KeyDiff. Beyond RULER, we validate on real-world benchmarks: on LongBench, ManifoldKV outperforms KeyDiff by +2.80 points on Qwen3-8B (winning 12 of 14 tasks) and +0.49 on Phi-4; on HELMET, ManifoldKV achieves +6.5 EM on RAG and WindowedManifoldKV reaches +42 points on multi-key recall at 131K; and on InfiniteBench at 100K+ context, WindowedManifoldKV wins by +7.16 on Phi-4. Cross-architecture evaluation across six models reveals that the optimal distance metric depends on key-norm geometry, providing the first systematic guidelines for metric selection in geometric KV cache compression. The method requires only 3 lines of code and works across diverse architectures without tuning.

ICML 2026

Beyond Static Endpoints: Tool Programs as an Interface for Flexible Agentic Web Services

Mugeng Liu, Shuoqi Li, Yixuan Zhang, Yun Ma

In the agentic web era, LLM-based agents increasingly invoke web services as tools, yet most interfaces remain static endpoints that poorly express long-horizon workflows with loops, conditionals, joins, and retries. We present ToolPro, which represents an agent's tool intent as an executable tool program that compactly encodes multi-step service interactions with explicit effect types. ToolPro combines constraint-guided program construction, effect-aware replay for exactly-once state-modifying calls, and a profile-driven policy that decides when program execution outperforms stepwise calling. We instantiate ToolPro over MCP-style services with WebAssembly sandboxing and evaluate it on diverse workflows of real-world applications. ToolPro reduces end-to-end latency by up to 53.4% and client-side traffic by up to 96.1%, with larger gains under higher network latency and workflow complexity.

ICML 2026

Bring My Cup! Personalizing Vision-Language-Action Models with Visual Attentive Prompting

Sangoh Lee, Sangwoo Mo, Wook-Shin Han

While Vision-Language-Action (VLA) models generalize well to generic instructions, they struggle with personalized commands such as "bring *my* cup," where the robot must act on one specific instance among visually similar objects. We study this setting of manipulating personal objects, in which a VLA must identify and control a user-specific object unseen during training using only a few reference images. To address this challenge, we propose **Visual Attentive Prompting (VAP)**, a simple-yet-effective training-free perceptual adapter that equips frozen VLAs with top-down selective attention. VAP treats the reference images as a non-parametric visual memory, grounds the personal object in the scene through open-vocabulary detection and embedding-based matching, and then injects this grounding as a visual prompt by highlighting the object and rewriting the instruction. We construct two simulation benchmarks, Personalized-SIMPLER and Personalized-VLABench, and a real-world tabletop benchmark to evaluate personalized manipulation across multiple robots and tasks. Experiments show that VAP consistently outperforms generic policies and token-learning baselines in both success rate and correct-object manipulation, helping to bridge the gap between semantic understanding and instance-level control.

ICML 2026

Overclocking Electrostatic Generative Models

Daniil Shlenskii, Alexander Korotin

Electrostatic generative models such as PFGM++ have recently emerged as a powerful framework, achieving competitive performance in image synthesis. PFGM++ operates in an extended data space with auxiliary dimensionality $D$, recovering the diffusion model framework as $D\to\infty$, while yielding superior empirical results for finite $D$. Like diffusion models, PFGM++ relies on expensive ODE simulations to generate samples, making it computationally costly. To address this, we propose Inverse Poisson Flow Matching (IPFM), a principled distillation framework that accelerates electrostatic generative models across all values of $D$. Our IPFM reformulates distillation as an inverse problem: learning a generator whose induced electrostatic field matches that of the teacher. We derive a tractable training objective for this problem and show that, as $D\to\infty$, our IPFM closely recovers Score Identity Distillation (SiD), a recent method for distilling diffusion models. Empirically, our IPFM produces distilled generators that achieve near-teacher or even superior sample quality using only a few function evaluations. Moreover, we find that one-step generator distillation converges faster at finite $D$ than in the $D\to\infty$ diffusion limit, aligning with prior evidence that finite-$D$ PFGM++ models offer more favorable optimization and sampling behavior.

ICML 2026

Search Space Synthesis for Parametric Functions

Felix Laarmann, Andreas Pauly, Sebastian Buschjäger, Andrea Bommert, Jakob Rehof

We present a general framework for synthesizing search spaces of parametric functions, along with strategies for traversing these spaces to find optima. We formalize an algebraic theory for the categorical model of parametric functions in finite combinatory logic with predicates (FCLP). Based on a component-oriented synthesis framework for FCLP we automate composition from given components and search for parametric functions. Components are language-agnostic and may be instantiated as any implementation of parametric functions, e.g., as PyTorch modules. A proof-of-concept implementation demonstrates how to represent more specific concepts, such as neural architecture search and hyperparameter optimization, within the framework.

ICML 2026

PS-PPO : Prefix-Sampling PPO for Critic-Free RLHF

Doo Hwan Hwang, Kee-Eung Kim, Kee-Eung Kim

Reinforcement Learning from Human Feedback (RLHF) for Large Language Models increasingly relies on critic-free methods as a practical alternative to actor--critic training. Despite their simplicity, existing critic-free approaches propagate a trajectory-level learning signal uniformly across all tokens in a trajectory. This requires full-trajectory policy updates for every rollout, leading to substantial optimization cost for long reasoning traces, even though intermediate prefixes often contain enough information to largely determine the final outcome. We propose Prefix-Sampling Proximal Policy Optimization (PS-PPO), a compute-efficient critic-free method for RLHF that exploits this temporal redundancy. PS-PPO introduces a prompt-conditioned cutoff distribution and samples a cutoff timestep for each trajectory. During the update pass, PS-PPO backpropagates only through the sampled prefix of each trajectory and applies an importance-weighting correction so that the resulting truncated gradient estimator remains unbiased with respect to the full-trajectory objective. Experiments on mathematical reasoning and RLHF benchmarks show that PS-PPO achieves large reductions in training compute and peak GPU memory, while maintaining accuracy comparable to strong critic-free baselines.

ICML 2026

Transformers learn factored representations

Adam Shai, Loren Amdahl-Culleton, Casper L. Christensen, Henry R Bigelow, Fernando Rosas, Alexander Boyd, Eric A. Alt, Kyle J Ray, Paul M. Riechers

Transformers pretrained via next token prediction learn to factor their world into parts, representing these factors in orthogonal subspaces of the residual stream. We formalize two representational hypotheses: (1) a representation in the product space of all factors, whose dimension grows exponentially with the number of parts, or (2) a factored representation in orthogonal subspaces, whose dimension grows linearly. Both track context-induced uncertainty over the latent parts, but the factored representation sacrifices fidelity when factors are not conditionally independent. We derive precise predictions about the geometric structure of activations for each, including the number of subspaces, their dimensionality, and the arrangement of context embeddings within them. We test between these hypotheses on transformers trained on synthetic processes with known latent structure. When factors are conditionally independent, models learn factored representations; when noise or dependencies break this structure, models gradually expand their effective dimensionality over training to recover fidelity. This provides a principled explanation for why transformers decompose the world into parts, and suggests that interpretable low dimensional structure may persist even in models trained on complex data.

ICML 2026

DLLMQuant: A Post-Training Quantization Framework Tailored for Diffusion-Based Large Language Models

Chen Xu, Zhixuan Chen, Dawei Yang

Diffusion-based large language models (DLLMs) have shown promise for non-autoregressive text generation, but their deployment is constrained by large model sizes and heavy computational costs. Post-training quantization (PTQ), a widely used method for compressing and accelerating Large Language Models (LLMs), suffers from severe accuracy degradation and reduced generalization performance when directly applied to DLLMs (e.g., AWQ suffers a 16% accuracy drop on LLADA under W4A4). This paper explores how the unique mechanisms of Dynamic Language Models (DLLMs) conflict with quantization, identifying three core issues: 1) During the iterative generation process of DLLMs, dynamic masking ratios are inherently involved, leading to notable differences in token distributions across decoding steps. Unfortunately, these distinct distributions are not sufficiently captured by current PTQ calibration approaches; 2) Quantization errors propogate and accumalte progressively during iterations in DLLMs, leading to a gradual decline in the performance of quantized models as decoding steps advance; 3) The stability of unmasked tokens, combined with the probabilistic nature of masked tokens, gives rise to an overall feature distribution that is uncoordinated and unsuitable for PTQ. To address these issues, we propose DLLMQuant, a PTQ framework tailored for DLLMs, which incorporates three novel techniques: 1) Temporal-Mask Adaptive Sampling (TMAS), a calibration method that accounts for both time and mask factors, with the capacity to capture distributions across timesteps. 2) Interaction-Aware Activation Quantization (IA-AQ), which utilizes bidirectional attention scores to identify important tokens, and prioritizes these tokens when minimizing quantization error. 3) Certainty-Guided Quantization (CGQ) incorporates mask status and token scores as core weighting criteria for error compensation, enabling PTQ to better align with the unique weight distribution of DLLMs. Experiments show that DLLMQuant achieves significant performance gains (e.g., over 10-point accuracy improvement on GSM8K for LLADA under 4-bit quantization) while enhancing efficiency.

ICML 2026

Factor-Wise Homogeneity of Slot-Attention for Continual Object-Centric Learning

Ilmin Kang, Hoyong Kim, Seungju Bang, Minwoo Kang, Kangil Kim

While Object-Centric Learning has shown great promise in modular perception, its extension to Continual Learning remains underexplored. In this work, we observe that Slot Attention exhibits a distinctive behavior: it organizes latent representations into small and separated regions, each of which preserves identical factor states, crucially emerging not only in the current task but also across sequential tasks with novel factors. This *inter-task separation* offers significant advantages in continual learning, which typically suffers from severe object-wise forgetting. We refer to this phenomenon as *Factor-Wise Homogeneity*, and show that this intrinsic inter-task separation is crucial, serving as a key mechanism to prevent catastrophic forgetting in Continual Object-Centric Learning. However, despite its strong robustness, factor-wise homogeneity alone is insufficient due to the bottleneck in exploiting this separation at the decoder. To overcome this limitation and demonstrate the significance of our findings, we show that a minimal strategy *Decoder-only Post-Replay*, which freezes the factor-wise homogeneous representations and employs decoder-only fine-tuning, is sufficient. This work serves as a fundamental basis for understanding and leveraging the intrinsic dynamics of Slot Attention, offering essential insights for advancing object-centric systems.

ICML 2026

Privacy-Aware Video Anomaly Detection through Orthogonal Subspace Projection

Lei Wang, Wenxiang Diao, Andrew Busch, JUN ZHOU, Yongsheng Gao

Video anomaly detection (VAD) systems often prioritize accuracy while overlooking privacy concerns, limiting their suitability for real-world deployment. We propose the Orthogonal Projection Layer (OPL), a lightweight module that removes task-irrelevant variations to produce representations focused on anomaly-relevant cues. To address privacy risks in human-centered scenarios, we introduce Guided OPL (G-OPL), which suppresses facial attributes using weak supervision from face-presence signals while preserving non-identifying features such as pose and motion. A cosine alignment objective enforces consistent capture and removal of facial information without identity labels or adversarial training. We further present a privacy-aware evaluation framework that jointly assesses detection performance and privacy preservation, and enables analysis of how sensitive information is filtered. Experiments show that embedding privacy constraints into model design reduces sensitive information while maintaining or improving detection accuracy, supporting projection-based architectures as a principled approach for privacy-aware VAD.

ICML 2026

miniF2F-Dafny: LLM-Guided Mathematical Theorem Proving via Auto-Active Verification

Mantas Baksys, Stefan Zetzsche, Olivier Bouissou, Sean B. Holden

LLMs excel at reasoning, but validating their steps remains challenging. Formal verification offers a solution through mechanically checkable proofs. Interactive theorem provers (ITPs) dominate mathematical reasoning but require detailed low-level proof steps, while auto-active verifiers offer automation but focus on software verification. Recent work has begun bridging this divide by evaluating LLMs for software verification in ITPs, but the complementary direction—LLMs for mathematical theorem proving in auto-active verifiers—remains unexplored. We present miniF2F-Dafny, the first translation of the widely-used mathematical benchmark miniF2F to an auto-active verifier: Dafny. We find that Dafny's automation alone solves 39-44% of problems with empty proofs, whereas many require substantial proof guidance in ITPs. We evaluate 8 off-the-shelf LLMs on proof generation, with the best model (Claude Opus 4.6) achieving 62.7% cumulative pass@4 on the full test set, improving over the 38.9% empty-proof baseline by 23.8 percentage points. These results show that auto-active verification offers a complementary empirical setting for AI-assisted mathematical reasoning, where LLMs provide high-level guidance while SMT automation handles low-level details. Our benchmark and evaluation infrastructure are publicly available on GitHub.

Keep browsing

More papers from top AI conferences

30+ top conferences, by conference, year, and topic.