AI conference papers

AI conference paper library

Search papers from 30+ top conferences, including NeurIPS, ICLR, ICML, CVPR, ACL and AAAI. Filter by keyword, then open any paper for its abstract, authors and links.

ICML 2026

Context-Aware Reasoner: Enhancing Contextual Reasoning in Multimodal Large Language Models

Zhe Zheng, Wenqi Zhang, Xiaohe Zhou, Guiyang Hou, Xinyu Mei, Tianyang Mao, Anni PIAO, Shengpei Jiang, Yongliang Shen, Yueting Zhuang, Weiming Lu

Multimodal large language models (MLLMs) have demonstrated remarkable reasoning capabilities over internalized knowledge. However, current research overlooks contextual reasoning, the ability to reason based on the relevant information present in the context. To investigate this issue, we construct the Visual Contextual Reasoning Benchmark (ContextReasonV-Bench), and our analysis reveals two predominant failure modes: *context neglect*, where models rely on pre-trained knowledge instead of contextual information, and *superficial pattern matching*, where models exploit shallow correlations rather than genuine patterns. To address these limitations, we propose a two-stage approach that progressively establishes and reinforces contextual pattern acquisition. The first stage establishes an "analyze-then-solve" reasoning paradigm through supervised fine-tuning (SFT). We then employ a context-aware reinforcement learning (RL) framework that integrates context-aware reward modeling with hierarchical advantage estimation to encourage the model to identify genuine contextual patterns. This approach yields Context-Aware Reasoner (CAR), a model that achieves 38.14\% accuracy on ContextReasonV-Bench, a 22.09\% improvement over the base model, and generalizes effectively to unseen tasks.

ICML 2026

Spatially-Regularized Entropy for Discriminative Token Merging in Fine-Grained Re-Identification

Shangze Li, Yifan Xu, Jingmiao Liang, Yongfei Zhang, Yuzhuo Ma, Yingbo Qu

While Vision Transformers (ViTs) offer strong global modeling, their quadratic computational cost limits utility in latency-sensitive applications like person re-identification (ReID). Existing compression strategies, such as token pruning or generic merging, typically rely on coarse-grained criteria tailored for image classification. In fine-grained retrieval, these approaches often discard or smooth out subtle but discriminative local details. To resolve this, we propose SRE-Merge, a training-free framework designed for discriminative token compression. SRE-Merge injects spatial priors into the merging process through three mechanisms: (i) Spatial-Entropy Saliency Assessment (SES-Assess), which quantifies token importance as Spatial-Entropic Mass (SE-Mass) by coupling spatial structure with local attention entropy; (ii) Hybrid Context-Affinity Matching (HCA-Match), which guides precise pair selection by combining feature similarity with mass-derived context; and (iii) Energy-Preserving Weighted Fusion (EPW-Fuse), which incorporates SE-Mass weighting to counteract feature variance reduction. Extensive experiments on standard benchmarks show that SRE-Merge reduces GFLOPs of the base ViT model by about 24\% while retaining competitive retrieval accuracy, establishing a superior accuracy-efficiency trade-off.

ICML 2026

Absorbing Quantization Error by Deformable Noise Scheduler for Diffusion Models

Mingrui Yang, Wei Huang, Hao Nan SHENG, Donglin Yang, Jichang Yang, Xin Yu, Huining Yu, Yuzhong Jiao, Zhongrui Wang, Xiaojuan Qi

Diffusion models deliver state-of-the-art image quality but are expensive to deploy. Post-training quantization (PTQ) can shrink models and speed up inference, yet residual quantization errors distort the diffusion distribution (the timestep-wise marginal over $\boldsymbol{x}_t$), degrading sample quality. We propose a distribution-preserving framework that absorbs quantization error into the generative process without changing architecture or adding steps. Deformable Noise Scheduler (DNS) reinterprets quantization as a principled timestep shift, mapping the quantized prediction distribution $\boldsymbol{x}_t$ back onto the original diffusion distribution so that the target marginal is preserved. Unlike trajectory-preserving or noise-injection methods limited to stochastic samplers, our approach preserves the distribution under both stochastic and deterministic samplers and extends to flow-matching with Gaussian conditional paths. It is plug-and-play and complements existing PTQ schemes. Empirically, our method consistently enhances generation quality across diverse backbones and existing PTQ baselines. Notably, when further quantizing the FP16 LoRA branch of SVDQuant to enable fully integer inference, our approach effectively mitigates the performance drop, reducing FID from 27.16 to 26.22. Code is available at https://github.com/ZephyrYoung-eYuan/DNS_AQE

ICML 2026

AnomSeer: Reinforcing Multimodal LLMs to Reason for Time-Series Anomaly Detection

Junru Zhang, Lang Feng, Haoran Shi, Xu Guo, Han Yu, Yabo Dong, Duanqing Xu

Time-series anomaly detection (TSAD) with multimodal large language models (MLLMs) is an emerging area, yet a persistent challenge remains: MLLMs rely on coarse time-series heuristics but struggle with multi-dimensional, detailed reasoning, which is vital for understanding complex time-series data. We present AnomSeer to address this by reinforcing the model to ground its reasoning in precise, structural details of time series, unifying anomaly classification, localization, and explanation. At its core, an expert chain-of-thought trace is generated to provide verifiable, fine-grained reasoning from classical analyses (e.g., statistical measures, frequency transforms). Building on this, we propose a novel time-series grounded policy optimization (TimerPO) that incorporates two additional components beyond standard reinforcement learning: a time-series grounded advantage based on optimal transport and an orthogonal projection to ensure this auxiliary granular signal does not interfere with the primary detection objective. Across diverse anomaly scenarios, AnomSeer, with Qwen2.5-VL-3B/7B-Instruct, outperforms larger commercial baselines in classification and localization accuracy, particularly on point- and frequency-driven exceptions. Moreover, it produces plausible reasoning traces that support its conclusions.

ICML 2026

Diffusion Controller: Framework, Algorithms and Parameterization

Tong Yang, Moonkyung Ryu, Chih-wei Hsu, Guy Tennenholtz, Yuejie Chi, Craig Boutilier, Bo Dai

Controllable diffusion generation often relies on various heuristics that are seemingly disconnected without a unified understanding. We bridge this gap with Diffusion Controller (DiffCon), a unified control-theoretic view that casts reverse diffusion sampling as state-only stochastic control within (generalized) linearly-solvable Markov Decision Processes (LS-MDPs). Under this framework, control acts by reweighting the pretrained reverse-time transition kernels, balancing terminal objectives against an $f$-divergence cost. From the resulting optimality conditions, we derive practical reinforcement learning methods for diffusion fine-tuning: (i) $f$-divergence-regularized policy-gradient updates, including a PPO-style rule, and (ii) a regularizer-determined reward-weighted regression objective with a minimizer-preservation guarantee under the Kullback–Leibler (KL) divergence. The LS-MDP framework further implies a principled model form: the optimal score decomposes into a fixed pretrained baseline plus a lightweight control correction, motivating a side-network parameterization conditioned on exposed intermediate denoising outputs, enabling effective *gray-box adaptation* with a frozen backbone. Experiments on Stable Diffusion v1.4 across supervised and reward-driven finetuning show consistent gains in preference-alignment win rates and improved quality–efficiency trade-offs versus gray-box baselines and even the parameter-efficient white-box adapter LoRA.

ICML 2026

Off-Policy Evaluation with Strategic Agents via Local Disclosure

Kiet Q. H. Vo, Abbavaram Gowtham Reddy, Julian Rodemann, Siu Lun Chau, Krikamol Muandet

We study off-policy evaluation (OPE) under strategic behavior where decision subjects (or agents) respond to a decision maker's policy by strategically modifying their covariates. Such behavior induces a policy-dependent covariate shift, breaking the standard assumption in existing methods that covariates are exogenous to the policy. Related work addresses this challenge by imposing strong assumptions such as repeated interactions or full knowledge of agents’ response behavior, substantially limiting its applicability to OPE. In contrast, we consider a one-shot OPE setting where the decision maker has only partial knowledge of the agents' response behavior. Our key insight is that disclosing local information through post-hoc explanations reveals agents’ pre-strategic covariates prior to adaptation, mitigating the information loss induced by strategic behavior. Leveraging this structure, we estimate a statistical model for the agents’ responses and construct a doubly robust estimator for policy value. By assuming that the agents' cost sensitivity follows a conditional log-normal distribution, we establish consistency of the proposed estimator and validate our approach empirically. More broadly, our results highlight how interaction design can mitigate information asymmetry by revealing otherwise hidden structure in agents' strategic responses.

ICML 2026

Do Activation Verbalization Methods Convey Privileged Information?

Millicent Li, Alberto Mario Ceballos Arroyo, Giordano Rogers, Naomi Saphra, Byron C Wallace

Recent interpretability methods have proposed to translate LLM internal representations into natural language descriptions using a second *verbalizer* LLM. This is intended to illuminate how the target model represents and operates on inputs. But do such *activation verbalization* approaches actually provide *privileged* knowledge about the internal workings of the target model, or do they merely convey information about the inputs provided to it? We critically evaluate popular verbalization methods and datasets used in prior work and find that one can perform well on such benchmarks without access to target model internals, suggesting that these datasets are not ideal for evaluating verbalization methods. We then run controlled experiments which reveal that verbalizations often reflect the parametric knowledge of the verbalizer LLM that generated them, rather than the knowledge of the target LLM whose activations are decoded. Taken together, our results indicate a need for targeted benchmarks and experimental controls to rigorously assess whether verbalization methods provide meaningful insights into the operations of LLMs.

ICML 2026

Multimodal Fact-Level Attribution for Verifiable Reasoning

David Wan, Han Wang, Ziyang Wang, Elias Stengel-Eskin, Hyunji Lee, Mohit Bansal

Multimodal large language models (MLLMs) are increasingly used for real-world tasks involving multi-step reasoning and long-form generation, where reliability requires grounding model outputs in heterogeneous input sources and verifying individual factual claims. However, existing multimodal grounding benchmarks and evaluation methods focus on simplified, observation-based scenarios or limited modalities and fail to assess attribution in complex multimodal reasoning. We introduce MuRGAt (Multimodal Reasoning with Grounded Attribution), a benchmark for evaluating fact-level multimodal attribution in settings that require reasoning beyond direct observation. Given inputs spanning video, audio, and other modalities, MuRGAt requires models to generate answers with explicit reasoning and precise citations, where each citation specifies both modality and temporal segments. To enable reliable assessment, we introduce an automatic evaluation framework that strongly correlates with human judgments. Benchmarking with human and automated scores reveals that even strong MLLMs frequently hallucinate citations despite correct reasoning. Moreover, we observe a key trade-off: increasing reasoning depth or enforcing structured grounding often degrades accuracy, highlighting a significant gap between internal reasoning and verifiable attribution.

ICML 2026

Bridging the Grounding Gap in VideoQA via Typed Memory for Language-based Belief-State Reasoning

Saman Forouzandeh, Wei Peng, Xinghuo Yu, Mahdi Jalili

VideoQA models can be accurate yet often fail to align answers with the correct video segments (the \emph{grounding gap}). We introduce \textbf{LINGUA} (\textbf{L}anguage-based \textbf{IN}ference for \textbf{G}rounded Video \textbf{U}nderstanding \textbf{A}gent), a memory-based agent that performs grounded VideoQA by reasoning in an explicit \emph{linguistic belief state}. LINGUA uses five mechanisms: (1) event-driven perception (retains 8--12\% of frames while preserving 94\% of question-relevant events); (2) typed memory for episodic narratives, semantic affordances, and procedural scripts; (3) Belief-Action-Verification loops with postcondition and temporal checks; (4) meta reflection with contrastive refinement; and (5) Bayesian reliability tracking for continual learning without gradient updates. Built with Gemma3-4B (Ollama, 4-bit), LINGUA outperforms strong baselines on five VideoQA benchmarks, reaching 82.4\% on NExT-QA and 42.3\% Acc@GQA on NExT-GQA (answer + IoU$\geq$0.5 temporal localization), while running 2.6$\times$ faster than dense-frame methods. In continual learning over 100 videos, accuracy rises from 45.2\% (first 10) to 61.8\% (last 10) without catastrophic forgetting, indicating online adaptation via memory refinement.

ICML 2026

Active Attacks: Red-teaming LLMs via Adaptive Environments

Taeyoung Yun, Pierre-Luc St-Charles, Jinkyoo Park, Yoshua Bengio, Minsu Kim

We address the challenge of automatically generating diverse attack prompts for large language models (LLMs) that elicit harmful behaviors (e.g., insults, sexual content) and are used for safety fine-tuning. While several prior approaches train LLMs with reinforcement learning (RL) to generate such prompts using only a toxicity classifier as a reward, existing diversity-seeking RL methods often collapse to limited modes: once high-reward prompts are found, exploration of new regions is discouraged. Inspired by the active learning paradigm that encourages adaptive exploration, we introduce \textbf{Active Attacks}, a novel RL-based red-teaming algorithm that adapts its attacks as the victim evolves. By periodically safety fine-tuning the victim LLM with collected attack prompts, we naturally induce an \emph{easy-to-hard exploration curriculum}, where the attacker progresses beyond easy modes toward increasingly difficult ones. We observe that this simple plug-and-play module, which seamlessly integrates into existing RL objectives, unexpectedly outperformed prior RL-based methods, improving cross-attack success rates against GFlowNets, the previous state-of-the-art, from 0.07\% to 31.28\% (a relative gain of more than 400×) with only a 6\% increase in computation.

ICML 2026

Physics-Informed Pre-training on Efficient Electron-Density Images for Organic Material Property Prediction

Zhixiang Cheng, hongxin xiang, Mingquan Liu, Tengfei Ma, Yingzhuo Tu, Wenjie Du, Bosheng Song, Yiping Liu, xiangxiang Zeng

Precise property prediction of organic materials is pivotal for next-generation electronic and energy devices. In density functional theory (DFT), the electron density (ED) serves as the fundamental determinant of material properties. Yet, establishing it as an input modality for material property prediction has been impeded by two practical barriers: scarce large-scale ED data and the enormous computational complexity of ED representation. To bridge these gaps, we introduce VisionED, an efficient physics-informed model pre-trained on electron-density images. We curate a dataset of 2 million molecules and represent ED as multi-shot images that efficiently encode both geometric and electronic structure. VisionED is then pre-trained on 12 million multi-shot ED images via cross-scale, physics-informed pretext tasks. Empirical evaluations on photovoltaic and organic chromophore datasets show that VisionED outperforms state-of-the-art baselines by up to 27.0\%, exhibiting superior robustness under distribution shifts and data scarcity. Notably, the model generalizes to unseen device-scale applications, successfully recovering experimental trends and mixing-ratio effects in ternary blends with an average accuracy of 92.77\%. Moreover, relative to the previous ED point cloud, the ED image improves performance by 26.2\% with 2.6$\times$ fewer memory and 4.6$\times$ lower time. The code and data are available at https://github.com/ZhixiangCheng/VisionED.

ICML 2026

Unifying Dataset Pruning and Distillation for Efficient Large-scale Compression

Lingao Xiao, Songhua Liu, Yang He, Xinchao Wang

Dataset pruning (DP) and dataset distillation (DD) fundamentally differ in their outputs: DP selects original image subsets, while DD generates synthetic images. Recently, DD's increasing reliance on original images suggests a convergence of the two directions. To investigate this convergence trend, we propose a unified dataset compression (DC) benchmark. This benchmark reveals an interesting trade-off for soft-label-DD: while soft labels provide valuable information, they can make the distillation process less essential, as distilled images may not always outperform random subsets. In addition, the benchmark reveals that in current stages, dataset pruning outperforms dataset distillation at small dataset sizes. Given these observations, we explore hard-label-DC as a complementary approach that emphasizes image quality while offering substantial storage efficiency. Our PCA (Prune, Combine, and Augment) is the first framework that does not rely on soft labels but instead focuses on image quality. (1) "P'' means selecting easy samples based on dataset pruning metrics, (2) "C'' indicates combining these samples effectively, and (3) "A'' is to apply constrained image augmentation during training.

ICML 2026

Beyond Continuity: Simulation-free Reconstruction of Discrete Branching Dynamics from Single-cell Snapshots

Junda Ying, Yuxuan Wang, Bowen Yang, Peijie Zhou, Lei Zhang

Inferring cellular trajectories from destructive snapshots is complicated by the challenges of stochasticity and non-conservative mass dynamics such as cell proliferation and apoptosis. Existing unbalanced Optimal Transport (OT) methods treat mass as a continuous fluid, performing inference at the population level. However, this macroscopic view often fails to capture the discrete, jump-like nature of birth-death events at single-cell resolution, which is essential for understanding lineage branching and fate decisions. We present **Unbalanced Schrödinger Bridge (USB)**, a simulation-free framework for learning underlying dynamics that effectively integrates both stochastic and unbalanced effects which also models the discrete, jump-like birth–death dynamics at single-cell resolution. Theoretically, USB provides a tractable solution to the Branching Schrödinger Bridge (BSB) problem, offering a rigorous microscopic interpretation where individual cells undergo both Brownian motion and discrete birth-death jumps. Technically, the method implements an efficient solver by introducing a simulation-free training objective that effectively scales to high-dimensional omics data. Empirically, we demonstrate on both simulated and real-world datasets that USB not only achieves trajectory reconstruction performance better than or comparable to deterministic baselines but also uniquely enables realistic discrete simulation of birth-death dynamics at single-cell resolution.

ICML 2026

GradPower: Powering Gradients for Faster Language Model Pre-Training

Jinbo Wang, Mingze Wang, Jiaqi Zhang, Wei Wang, Peng Pei, Xunliang Cai, Weinan E, Lei Wu

We propose **GradPower**, a lightweight gradient-transformation technique for accelerating language model pre-training. Given a gradient vector $\boldsymbol{g}=(g_ {i})_ {i}$, GradPower first applies the elementwise `sign-power` transformation: $ \varphi_ p(\boldsymbol{g}) = \left({\rm sign}(g_ i)|g_ i|^p\right)_ {i} $ for a fixed $p>0$, and then feeds the transformed gradient into a base optimizer. Notably, GradPower requires only a **single-line code change** and no modifications to the base optimizer’s internal logic, including the hyperparameters. When applied to AdamW (termed **AdamWPower**), GradPower consistently achieves lower terminal loss across diverse architectures (LLaMA, Qwen2MoE), parameter scales (66M to 2B), datasets (C4, OpenWebText), and learning-rate schedules (cosine, warmup-stable-decay). The most pronounced gains are observed when training modern mixture-of-experts models with warmup-stable-decay schedules. GradPower also integrates seamlessly with other state-of-the-art optimizers, such as Muon, yielding further improvements. Finally, we provide theoretical analyses that reveal the underlying mechanism of GradPower and highlight the influence of gradient noise.

ICML 2026

Adaptive Bandit Algorithms for Contextual Matching Markets

Shiyun Lin, Simon Mauras, Vianney Perchet, Nadav Merlis

We study bandit learning in matching markets, where players and arms constitute the two market sides, and the players' utilities are linear in the arm contexts. In each round, new arms arrive with observable contexts. Then, the algorithm matches them to players, aiming to minimize each player's regret against a *stable matching benchmark*. This contextual structure creates significant complexity: subtle context shifts can slightly alter one player's utility while completely reconfiguring the underlying benchmark, causing large regret spikes for others. We address this in two settings: *stochastic* contexts, drawn from a latent distribution, and *adversarial* contexts, which may be arbitrary. For the stochastic case, we introduce a novel minimum preference gap to capture learning difficulty and provide a fully adaptive algorithm with an instance-dependent poly-logarithmic regret upper bound. We also establish matching instance-independent regret upper and lower bounds under a mild distributional assumption. For the adversarial setting, we propose a tractable regret notion that remains valid under arbitrary contexts and achieves an instance-independent sublinear regret bound via an adaptive algorithm.

ICML 2026

MeshTok: Efficient Multi-Scale Tokenization for Scalable PDE Transformers

Yanshun Zhao, Xiaoyu Peng, Jiamin Jiang, Congcong Zhu, Jingrun Chen

Conventional patchified Transformers operate on uniform spatial partitions, distributing computational effort evenly across the domain irrespective of local features. This inflexible tokenization scheme is inherently limited in its ability to efficiently represent and process solutions to complex PDEs. To address this, we propose MeshTok, an adaptive mesh refinement (AMR)-inspired tokenization and sequence modeling framework. This method selectively refines spatial regions exhibiting sharp gradients, transient features, or multiscale structures, generating a heterogeneous set of multiscale tokens defined on a fixed simulation grid. These tokens are processed within a unified Transformer sequence, enabling the model to simultaneously capture coarse-grained global context and fine-grained local details without requiring specialized architectural components. Although adaptive refinement moderately increases token count, it promotes a more targeted allocation of computational resources to physically informative regions, which we view as a practical inductive bias rather than a formal optimality guarantee. Experimental evaluations across multiple PDE families and benchmark datasets demonstrate that MeshTok consistently improves the efficiency-accuracy trade-off compared to uniform-grid baselines. This suggests adaptive multiscale tokenization as a scalable and generalizable design principle for neural PDE modeling. Code is available at https://github.com/SCAILab-USTC/MeshTok.

ICML 2026

Conditional Quantile Adjusted Conformal Prediction for Time Series

Cheng Yu, Zhoufan Zhu, Ke Zhu

Conformal prediction is challenging for time series with time-varying conditional distributions. Existing sequential conformal methods can yield volatile, non-nested prediction intervals due to noisy tail conditional quantile estimation and quantile crossing issue. To overcome these challenges, we construct prediction intervals for time series via a novel method called Conditional Quantile Adjusted Conformal Prediction (CQACP), which stabilizes sequential conformal calibration by modeling the conditional quantile curve of nonconformity score. At each time step, CQACP evaluates a base conditional quantile learner on a grid of quantile levels and fits a Cornish--Fisher approximation parameterized by conditional moments of nonconformity score with monotonicity constraints. Asymptotically, we prove the conditional validity of the prediction interval under serial dependence and show improved conditional quantile estimation accuracy. Experiments on multiple real-world datasets demonstrate that CQACP maintains accurate coverage and produces smooth, narrow, and nested prediction intervals across different significance levels and prediction models.

ICML 2026

SSDCN: Spatial-Spectral Dual-Clustering-based Network for Hyperspectral Image Super-resolution

Yong Yang, Xuran Zhang, Shuying Huang, Xiaozheng Wang, Weiguo Wan, Hangyuan Lu

Hyperspectral Image Single Image Super-Resolution (HSI-SISR) faces a conflict between computational efficiency and global non-local modeling. Existing Transformers suffer from quadratic complexity, while window-based methods compromise global capture. To address this, we propose the Spatial-Spectral Dual-Clustering-based Network (SSDCN). Our method introduces three innovations. First, we design a Spatial-Spectral Dual-Cluster Block (SSDCB). Replacing expensive point-to-point attention, it uses content-driven clustering to learn low-rank structural bases, achieving global modeling with linear complexity $\mathcal{O}(KN)$. Second, we propose a pyramid progressive hierarchical architecture with a Feature Reuse Reconstruction Block (FRRB). It reuses the core tensor and spectral factors from coarse levels, updating only spatial factors to minimize redundancy. Third, we propose a Pyramid Hierarchical Reconstruction Joint Loss to supervise intermediate levels, ensuring structural accuracy and preventing error accumulation. Experiments demonstrate that SSDCN surpasses SOTA methods in metrics and visual quality with significantly fewer parameters and FLOPs, achieving an optimal efficiency-performance balance.

ICML 2026

Domain Transfer Becomes Identifiable via a Single Alignment

Sagar Shrestha, Subash Timilsina, Hoang-Son Nguyen, Xiao Fu

Domain transfer (DT) maps source to target distributions and supports tasks such as unsupervised image-to-image translation, single-cell analysis, and cross-platform medical imaging. However, DT is fundamentally ill-posed: push-forward mappings are generally non-identifiable, as measure-preserving automorphisms (MPAs) preserve marginals while altering cross-domain correspondences, leading to content-misaligned translation. Recent work shows that MPAs can be eliminated by jointly transferring multiple corresponding source/target conditional distributions, but supervision signals labeling such conditionals are not always available in practice. We develop an alternative route to DT identifiability. Under a structural sparsity condition on the Jacobian support pattern, we show that distribution matching together with a single paired anchor sample suffices to identify the ground-truth transfer---requiring substantially less supervision than prior approaches. To enable practical high-dimensional learning, we further propose an efficient Jacobian sparsity regularizer based on randomized masked finite differences, yielding a scalable surrogate without explicit Jacobian evaluation. Empirical results on synthetic and real-world DT tasks validate the theory.

ICML 2026

Robust-U1: Can MLLMs Self-Recover Corrupted Visual Content for Robust Understanding?

Jiaqi Tang, Jianmin Chen, Youyang Zhai, Wei Wei, Runtao Liu, Mengjie Zhao, Xiangyu Wu, Qingfa Xiao, Qifeng Chen

Multimodal Large Language Models (MLLMs) have demonstrated remarkable success in visual understanding, yet their performance degrades significantly under real-world visual corruptions. While existing robustness enhancement approaches exist, they are limited: black-box feature alignment lacks interpretability, and white-box text-based reasoning cannot restore lost pixel-level details. This work investigates a fundamental research question: Can MLLMs recover corrupted visual content by themselves? To address this, we propose Robust-U1, a novel framework that equips MLLMs with explicit visual self-recovery capability for robust understanding. The approach comprises three core stages: supervised fine-tuning for initial reconstruction, reinforcement learning with dual rewards (pixel-level SSIM and semantic-level CLIP similarity) for aligning high visual quality, and multimodal reasoning that jointly considers both the corrupted input and the recovered image. Extensive experiments demonstrate that Robust-U1 achieves state-of-the-art robustness on the real-world corruption benchmark and maintains superior performance under adversarial corruptions on general VQA benchmarks. Analysis confirms that high-quality visual recovery directly enhances reasoning performance, establishing self-recovery as a critical mechanism for robust visual understanding. The source code is available at https://github.com/jqtangust/Robust-U1.

Keep browsing

More papers from top AI conferences

30+ top conferences, by conference, year, and topic.