NeurIPS 2026 Papers

Vision-language models (VLMs) have rapidly evolved into general-purpose multimodal reasoners with strong zero-shot generalization. In this context, VLMs could greatly benefit the analysis of human gaze and attention, a central task in human behavior understanding that requires reasoning about the physical scene as well as the activity, interactions, and social context. However, the extent to which VLMs can reliably understand human gaze and related attentional behaviors remains largely unexplored. In this work, we present EyeVLM, a systematic evaluation framework for gaze understanding in VLMs across two complementary dimensions: tasks and models. To assess gaze understanding capabilities, we focus on two core tasks. The first, gaze following, i.e., predicting the 2D location where a person is looking, has a geometric and visual processing focus, requiring a precise understanding of the human face, attention direction, 3D scene structure, and spatial grounding of attended targets. The second, social gaze prediction, requires social and relational reasoning over multi-person interactions (e.g., mutual gaze and shared attention), and may benefit more from the LLM semantic reasoning capabilities within VLMs. Regarding models, EyeVLM evaluates these tasks in two ways: a zero-shot setting with a diverse set of state-of-the-art open- and closed-source VLMs, exploring different prompting strategies; and a fine-tuning approach based on task-specific QA pairs, studying the impact of model scale and data scale. As benchmarks, we rely on existing gaze understanding datasets and perform a systematic comparison with state-of-the-art purely visual models. Overall, our results show that current VLMs lack precise gaze understanding capabilities. While standard training helps reduce the gap with visual models, significant improvements are still needed.

Cognitive Load Theory (CLT) shows that human problem-solving is affected by both intrinsic load, arising from task difficulty, and extraneous load, arising from how task information is presented. LLMs show analogous effects, where presentation alone can affect model performance, but these effects have not yet been systematically isolated from task difficulty. To address this gap, we introduce CogLoadBench, a controlled benchmark that varies five CLT-inspired presentation factors across three reasoning task families. Across models, we find that higher extraneous load conditions often reduce accuracy by either adding competition or complicating the reasoning path. To support diagnosis and mitigation of load-related effects, we propose the Extraneous Load Score (ELS), a prompt-time metric computed from activations that estimates a model's extraneous load, and show that it generalizes across reasoning task types and load levels. We further show that shifting model representations toward lower-ELS directions at inference time can significantly improve model performance without changing prompt content.

Electrocardiogram (ECG) foundation models pretrained on typical diagnostic 10-second ECG segments, have demonstrated strong transferability across a range of clinical applications. However, many real-world applications produce recordings that are typically longer, and are varied in duration during inference time. These 10-second models have no built-in way to combine information across time. Extending them to longer horizons introduces two challenges: _structural_ incompatibilities arising from input-length disparities, and _semantic_ challenges that limit meaningful temporal aggregation. We propose a parameter-efficient framework that extends pretrained ECG foundation models to longer and variable-length ECGs without retraining the backbone. Guided by a frozen pretrained 10-second model, we introduce a lightweight plug-in module that extends the model in two complementary ways: (i) structurally compatible long-sequence processing and (ii) semantically informed temporal modeling. Experiments on multiple long-horizon ECG tasks, datasets, and foundation model backbones demonstrate that our method enables robust long-horizon extension from pretrained snapshot models, consistently outperforming sliding-window and pooling-based baselines with strong parameter efficiency.

Multimodal Retrieval-Augmented Generation (RAG) helps mitigate hallucinations in Vision-Language Models (VLMs) by grounding generations in external knowledge bases. However, this externalized memory also introduces privacy risks, as external queries may reveal signals about sensitive visual records in the retrieval corpus, such as medical images, scanned contracts, and proprietary business documents. This exposes a retrieval-corpus privacy risk: private records may be detectable through interactions with multimodal RAG systems. To study this risk, we propose the Semantic Degradation Attack (SDA), a two-query method for exposing private corpus leakage in multimodal RAG by testing how strongly generated responses depend on retrieved evidence. SDA constructs transferable retrieval-disrupting perturbations using local surrogate visual encoders. By measuring the drop in the semantic similarity of the VLM's generated descriptions before and after perturbation, SDA can distinguish whether a target image-caption record exists in the private retrieval corpus. Extensive experiments demonstrate that SDA consistently detects private corpus leakage more effectively than existing baselines on two image-caption datasets across five popular VLMs.

Vision Language Action (VLA) models can execute isolated atomic skills, but practical long horizon manipulation requires recombining these skills into new compositions without demonstrations for every skill combination. The direct full instruction route treats skill chaining as long horizon imitation. In our controlled study, full instruction training on limited skill combination sequences reaches 40.0% ordered success on covered combinations but only 7.7% on recombination, suggesting memorization of sequence templates rather than robust skill recombination. We therefore introduce Hierarchical Subtask Execution (HSE), a diagnostic execution framework that exposes skill recombination through a shared modular interface over the planner, trigger, and policy, enabling long horizon failures to be decomposed into separable modes. To make this diagnosis measurable, we introduce RoboCombine, a benchmark for skill recombination without combined skill training trajectories, with metrics for ordered success and transition conversion across subtask boundaries. RoboCombine reveals that atomic competence alone does not ensure ordered execution. Failures concentrate at transitions where downstream subtasks must launch from handoff states produced by preceding subtasks rather than canonical atomic starts. We identify this mismatch as the start state gap. To address it, we introduce Pose Robust Task Initialization (PRTI), a lightweight atomic task post training strategy that broadens start state coverage beyond canonical starts. Under oracle timed HSE on RoboCombine, adding PRTI raises ordered success from 2.11% to 26.32% and transition conversion from 2.63% to 60.53%, without combined skill training trajectories. Together, these results show that combinatorial generalization requires handoff robust atomic skills that convert first stage progress into reliable downstream execution, not merely decomposition into known subtasks. We will release RoboCombine and HSE code to enable reproducible diagnosis of atomic skill recombination failures.

Light Field (LF) deraining relies on leveraging redundant information between views to restore regions occluded by rains. Existing LF deraining methods typically rely on predicted depth maps to guide cross-view information interaction, while learning a brute-force mapping to separate rain from background content. However, the semi-transparent nature of rain streaks often leads to inaccurate depth estimation, which undermines geometric alignment and degrades restoration quality. More critically, the blending of rain and background during restoration makes brute-force mappings ineffective, resulting in visible rain artifacts in the output. In this paper, we first construct a consistency-constrained LF rain model by considering the consistency and the accumulation effects of rain layers. Then, we propose a novel Consistency-Guided Recovery Network (CGRNet) specifically designed for LF deraining. Inspired by the fact that background regions occluded by rain streaks often exhibit inconsistencies across all views, we introduce consistency maps to perceive degradation by aggregating pixel-wise consistency information along epipolar lines. Guided by consistency maps, the proposed model adaptively samples and aggregates consistent and redundant features from clean background regions to restore the details of degraded areas. Experiments validate our method's superiority over state-of-the-art approaches, and show that our dataset effectively enhances practical performance under real-world conditions.

Cross-view object correspondence (CVOC) aims to establish associations of the same target between query and search videos captured from different viewpoints. Existing methods often rely on visual cues for CVOC. Despite recent progress, these methods struggle in complicated scenarios with substantial viewpoint changes and appearance variations, due to lacking sufficient target information. Addressing this, we introduce TeSCo, a novel framework that exploits rich Textual Semantics of an object, in addition to its visual cues, for cross-view object Correspondence. Our key insight is, textual description of a target, by capturing diverse attributes such as category, texture, and color, provides viewpoint- and appearance-invariant semantics, which are complementary to visual cues and can hence enhance correspondence robustness in complicated scenes. Inspired by this, TeSCo first generates a textual expression of the target from the given query view using a vision-language model, and then extracts textual feature to enhance visual features for localization in the search view. To realize this, we introduce a simple yet effective text-conditioned cross-modal fusion (TCF) module that incorporates the textual feature into visual features with text-guided modulation, producing more robust multimodal target representation for object correspondence. Since not all textual cues are equally compatible with the target in current search-view frame, we propose an iterative textual feature refinement (ITFR) module that progressively adjusts textual feature using search-view information before the TCF module, enabling search-view-aware textual semantics to be fused into visual features and leading to better performance. In extensive experiments on Ego-Exo4D and HANDAL-X, TeSCo demonstrates state-of-the-art results and largely outperforms other models, validating its efficacy. Our code and model as well as results will be released.

Vision-language-action(VLA) models have emerged as a promising interface for autonomous driving. However, existing VLA models still suffer from a fundamental mismatch: driving actions require precise 3D geometric cues, while visual-language understanding and reasoning are largely conducted in a 2D semantic space. In this paper, we propose GeoCoTDrive, an explicit geometric chain-of-thought framework that grounds geometry in a planning-oriented manner. GeoCoTDrive follows a ``think with 2D first, drive with dedicated 3D priors'' paradigm: it first identifies sparse 2D regions corresponding to decision-critical cues, and then retrieves localized 3D priors by sampling features from a geometric foundation model within the grounded regions. These localized geometric features are interleaved into the autoregressive context to support trajectory generation. To supervise this process, we introduce planning-relevant grounding, a new region-level grounding task that focuses on local spatial cues directly affecting ego planning, and construct the PlanningGrounding dataset to endow VLM models with planning-oriented grounding ability. Experiments across multiple end-to-end autonomous driving benchmarks show that GeoCoTDrive consistently improves safety-critical planning performance, demonstrating the effectiveness of explicit geometric chain-of-thought reasoning for VLA-based planning.

Current evaluations of LLM safety predominantly rely on *severity-based taxonomies* to assess the harmfulness of models' responses to malicious queries. We argue that this formulation requires re-examination as it implicitly equates model output with realized harm while neglecting *Execution Likelihood*—the conditional probability of a threat being realized by a real-world actor given a model's response. In this work, we introduce **Expected Harm**, a metric that weights the severity of a jailbreak by its execution likelihood, modeled as a function of *execution cost*. Through empirical analysis of state-of-the-art models, we reveal a systematic *Inverse Risk Calibration*: models disproportionately exhibit stronger refusal behaviors for low-likelihood (high-cost) threats while remaining vulnerable to high-likelihood (low-cost) queries. Mechanistically, we first provide an analysis using linear probing, revealing that while refusal mechanisms linearly separate queries by severity, they do not use execution likelihood as an indicator for refusal. Then, we use Olmo-3's open-source staged checkpoints to trace the root cause of miscalibration to the Supervised Fine Tuning (SFT) training data distribution (82.8\% of safety examples at cost levels 0–1) and show that cost representations are weak from the SFT stage onward. Finally, we demonstrate that this miscalibration creates a vulnerability: by exploiting this property, we increase the attack success rate of existing jailbreaks by up to $2\times$.

Text-to-image (T2I) diffusion models often synthesize policy-violating content, necessitating robust safeguards and rigorous evaluation frameworks. However, current approaches remain largely monolithic in both execution and evaluation, limited to coarse-grained categorization within broad unsafe concepts. Such approaches fail to account for the specific constituent factors of a violation, resulting in weak learning signals and allowing model collapse to bypass safety checks. To overcome these limitations, we propose Example-Based Spatial Guidance (EBSG), a training-free method that utilizes user-editable exemplar packs to provide granular spatial guidance for precise concept steering. Unlike existing methods that coarsely steer the entire image away from a broad concept, EBSG decomposes these categories into specific sub-concepts via exemplar text-image pairs, providing explicit, localized steering signals for more robust safety control. Furthermore, we introduce a vision language model (VLM)-based evaluation protocol that provides a fine-grained assessment, avoiding the pitfall of conventional binary evaluators that permit model collapse to bypass safety checks. Empirically, EBSG achieves state-of-the-art erasure performance across diverse safety datasets and concept categories, spanning four nudity benchmarks, seven I2P harmful-concept slices, and four MJA metaphor categories, while supporting multi-concept removal and transferring to a modern backbone, SD3.

Video temporal grounding (VTG) takes an untrimmed video and a natural-language query as input and localizes the temporal moment that best matches the query. Existing methods rely on large, task-specific datasets requiring costly manual annotation. We introduce EvoGround, a framework of two coupled self-evolving agents, a proposer and a solver, that learn temporal grounding from raw videos without any human-labeled data. The proposer generates query--moment pairs from raw videos, while the solver learns to ground them and feeds back signals that improve the proposer in return. Through this self-reinforcing reinforcement-learning loop, the two agents are initialized from the same backbone and mutually improve across iterations. Trained on 2.5K unlabeled videos, EvoGround matches or surpasses fully supervised models across multiple VTG benchmarks, while emerging as a state-of-the-art fine-grained video captioner without manual labels.

High-fidelity diagram creation requires coordinated visual-spatial decisions over semantic topology, visual styling, and geometric layout. Existing methods face a representation gap: pixel-based generation offers limited object-level control, while code-based synthesis provides executable structure at the cost of intuitive manipulation. We introduce EvoDiagram, an agentic framework for editable diagram creation through an intermediate canvas schema that is both machine-actionable and directly manipulable by users. EvoDiagram first constructs a diagram manifest through coordinated structure, style, and layout agents, and then renders the manifest in a diagnostic verification environment. This environment produces objective defect signals and VLM-based critique, enabling localized refinement of the current diagram while also providing evidence for long-term design expertise evolution. The evolution mechanism distills refinement traces into a three-tier design expertise memory, where candidate strategies are linked to the verification evidence that supports them and are promoted into broader guidelines and principles only through repeated cross-context support. We further define CanvasBench, a canvas-recoverable benchmark and evaluation protocol with content, visual, and cognitive dimensions. The current manuscript focuses on the framework, benchmark design, and reporting protocol; empirical results should be added only when the corresponding artifacts, judge traces, and analysis scripts are available for verification. Our code is available at \href{https://anonymous.4open.science/r/evo-diagram/}{https://anonymous.4open.science/r/evo-diagram/}.

Vision-Language-Action (VLA) models have emerged as a promising paradigm for robotic manipulation by unifying perception, language grounding, and action generation. However, they often struggle in scenarios requiring precise spatial understanding, as current VLA models primarily rely on 2D visual representations that lack depth information and detailed spatial relationships. While recent approaches incorporate explicit 3D inputs such as depth maps or point clouds to address this issue, they often increase system complexity, require additional sensors, and remain vulnerable to sensing noise and reconstruction errors. Another line of work explores implicit 3D-aware spatial modeling directly from RGB observations without extra sensors, but it often relies on large geometry foundation models, resulting in higher training and deployment costs. To address these challenges, we propose \textbf{Evo-Depth}, a lightweight depth-enhanced VLA framework that enhances spatially grounded manipulation without relying on additional sensing hardware or compromising deployment efficiency. Evo-Depth employs a lightweight \textit{Implicit Depth Encoding Module} (IDEM) to extract compact depth features from multi-view RGB images. These features are incorporated into vision-language representations through a \textit{Spatial Enhancement Module} (SEM) via depth-aware modulation, enabling efficient spatial-semantic enhancement. A \textit{Progressive Alignment Training} strategy is further introduced to align the resulting depth-enhanced representations with downstream action learning. Extensive experiments in simulation and real-world settings demonstrate the effectiveness of Evo-Depth. With only \textbf{0.9B} parameters, Evo-Depth achieves state-of-the-art-level performance across four simulation benchmarks. In real-world experiments, Evo-Depth attains the highest average success rate while also exhibiting the smallest model size, lowest GPU memory usage, and highest inference frequency among compared methods. These results demonstrate that lightweight implicit depth enhancement is an effective and practical solution for spatially grounded robotic manipulation. We release code and models to facilitate future research on lightweight depth-enhanced VLA models.

Referring Remote Sensing Image Segmentation (RRSIS) aims to segment target objects from remote sensing images based on natural language descriptions, requiring precise vision–language alignment under significant ambiguity. Despite the strong generalization ability of Segment Anything Model 3 (SAM3), the large spatial extent and varying scales of targets in remote sensing images intensify the ambiguity of referring expressions. This poses a major challenge for fine-grained segmentation. To address the challenge, we propose **EviSAM3**, an evidence-driven parameter-efficient adaptation framework that mimics human decision-making by dynamically accumulating partial evidence. Specifically, EviSAM3 progressively integrates cross-modal evidence through an evidence memory bank, where each incoming piece is aggregated via momentum-based updates. As evidence accumulates, the model performs iterative evidence rectification to resolve ambiguity under incomplete observations. Meanwhile, the contributions of different evidence are adaptively modulated based on their diagnostic relevance, allowing more informative evidence to dominate the reasoning process and guide the final prediction. Extensive experiments demonstrate that EviSAM3 consistently outperforms state-of-the-art methods, particularly in challenging scenarios with high ambiguity and significant scale variation. The code is available in the supplementary material.

Creating images from noise is image generation; reconstructing fine details from coarse inputs is super-resolution. Despite their practical differences, both can be understood as reversing information loss across scales. We introduce **SKILD**, a **S**cale-invariant **K**-Space **I**mage **L**earning **D**iffusion model that unifies generation and continuous super-resolution within a single unconditional framework. Both natural images and critical physical systems exhibit scale invariance, and we leverage it to design a forward process that attenuates image content from fine to coarse scales while injecting spectrum-matched Gaussian noise, making scale an explicit coordinate of the diffusion dynamics. The same trained reverse process performs generation and continuous super-resolution by varying only the starting timestep: *no task-specific architecture, no conditioning branch, no classifier-free guidance, no retraining per scale factor*. Empirically, SKILD reaches FID $2.65$ and Inception Score $9.63$ on unconditional CIFAR-10, performs $2\times$–$8\times$ super-resolution on ImageNet from a single unconditional checkpoint while outperforming conditional models across perceptual metrics, and reconstructs critical Ising models whose connected four-point correlations closely track the ground truth.

First-person dynamic spatial reasoning requires models to track continuous motion and precise geometric structure, but the quadratic attention cost of Transformer-based Video-LLMs makes dense visual tokens computationally expensive. Existing token pruning paradigms predominantly rely on discrete static snapshots, failing to preserve the physical dynamics essential for reasoning. We propose Event Cascade Pruning (ECP), to our knowledge the first training-free framework that leverages the high-frequency motion cues from event cameras as a continuous event-guided motion prior to guide token selection. ECP combines three stages: Event-Triggered Causal Sampling to anchor motion-informative keyframes, Event-guided Motion Saliency Filtering to remove static or low-motion regions, and Event-Attention Ranking Fusion to calibrate spatial attention with motion salient dynamics. With 80\% visual token reduction, ECP outperforms the full-token baseline (37.62\% vs. 36.31\%) while achieving 1.89× inference speedup and 52\% GFLOPs reduction. We further introduce ESR-Real, the first real-world RGB-event benchmark for first-person spatial reasoning, where ECP improves accuracy by 2.68\% over full-token baselines.

Vision-Language-Action (VLA) policies translate language and visual inputs into robot actions, where their hidden representations directly shape closed-loop behavior. However, mechanistic interpretability tools from language and vision-language models do not transfer cleanly to VLAs: outputs are robot actions rather than human-readable tokens, and validating internal changes requires expensive closed-loop rollouts. We propose an event-grounded interpretability pipeline that anchors SAE feature analysis to behavioral events rather than text contexts. End-effector keyframes are clustered within each task using visual, state, and temporal cues, linking SAE features to behaviorally salient events and, via optional VLM annotations, to semantic context. To our knowledge, our pipeline is among the first to ground SAE-based VLA analysis in closed-loop behavioral events. Across two simulation architectures and a real-robot study, event-grounded ranking yields the strongest causal effects on OpenVLA and transfers to the continuous action chunks of $\pi_{0.5}$. SAE is a sparse but imperfect intervention basis: usability varies with architecture and intervention site, and aggressive intervention reveals safety and interpretability limits. Overall, event-grounded SAE analysis emerges as a practical starting point for behavior-anchored VLA interpretability, motivating future work on SAE features beyond action-aligned coordinates, finer-grained closed-loop evaluation, and safe interventions for high-stakes VLA deployments.

Vision-Language Models (VLMs) such as CLIP have demonstrated strong performance across tasks yet remain highly vulnerable to adversarial attacks. While adversarial fine-tuning has been demonstrated to be effective against such malicious inputs, its multi-step adversary generation scheme during fine-tuning further induces prohibitive computational costs for VLM backbones. Previous works in unimodal models have introduced single-step strategies to reduce this burden. However, we find naive single-step strategies fail on VLMs due to more severe catastrophic overfitting. In addition, we identify a novel failure mode termed *Perturbation Radius Overfitting*, where VLMs overfit to the specific training attack budget while paradoxically becoming fragile to weaker perturbations. We trace these failures to growing angular misalignment and magnitude surge of the local gradients, which compromise the local linearity essential for accurate single-step approximation. Guided by this analysis, we introduce a joint adversarial optimization scheme that actively rectifies the local geometry by suppressing both angular misalignment and magnitude surge along the perturbation path. Our approach consistently outperforms existing single-step methods across various datasets and VLM architectures, achieving state-of-the-art robustness while preserving efficiency. To our knowledge, this is the first work to systematically examine single-step adversarial fine-tuning on VLMs and establish the geometric foundations of their robustness.

Vision-Language Models (VLMs) have achieved strong performance on static visual understanding benchmarks, yet their ability to act in dynamic environments remains poorly understood. We introduce AtariBench, a controlled benchmark that evaluates VLMs across 20 Atari 2600 games, requiring models to make sequential decisions from high-temporal-resolution gameplay streams. By decoupling fast in-game dynamics from wall-clock inference speed, AtariBench focuses on whether models can perceive rapid visual state changes, reason over spatiotemporal dynamics, and ground actions in the current observation. Our experiments reveal substantial limitations in state-of-the-art VLMs: models often fail to infer object motion and direction, exhibit brittle sequential decision-making, and increasingly over-rely on textual interaction history rather than newly appended visual feedback. We further introduce a separability analysis to identify which in-game scores provide reliable signals of model competence, and use the benchmark to study how memory design and prompting strategies affect performance. Overall, AtariBench exposes a persistent gap between passive visual perception and active decision-making, providing a lightweight and rigorous testbed for developing VLMs that can reason and act in dynamic environments.