NeurIPS 2026 Papers

Vision-language models (VLMs) learn strong statistical regularities during training, which can make them fail to perceive visual evidence when input images violate those regularities. Such failures are often treated as missing visual capability, but they may instead arise because the model has access to the relevant visual evidence and yet selects an answer dominated by learned training priors. In this work, we use controlled visual in-context learning to show that VLMs can overcome learned priors for visual tasks and use relevant visual evidence and capabilities. Specifically, we show that across five vision-centric tasks, demonstrations yield large gains on prior-conflicting counterfactual examples ($+27.2$% for Qwen3-VL 235B and $+18.1$% for Qwen3-VL 32B) while leaving real accuracy nearly unchanged. These findings also generalize across five VLM families with performance increases from $+9.7$% to $+30.3$%. To explain this effect, we analyze visual representations and find that demonstrations redirect attention toward grounding cues while reducing reliance on cues that support the learned prior. Finally, we find that in-context learning is limited when the underlying visual capability is weak or absent. Together, these results show that current VLMs possess visual grounding capabilities that standard evaluations may overlook, underscoring the need for controlled evaluation design in diagnosing and improving model capability.

Active perception—the ability of a robot to proactively select viewpoints to acquire task-relevant information—is essential for robust operation in real-world environments. However, existing approaches are typically limited to fixed objectives or constrained settings, and struggle to generalize to open-ended perception intents specified in natural language. We propose I-Perceive, a foundation model for language-conditioned active perception in large-scale indoor environments. Given a query image, a set of context images, and a natural language instruction, I-Perceive predicts a 6D camera pose that fulfills the specified perception intent. The model integrates a vision-language pathway for semantic grounding with a geometric reasoning pathway for multi-view 3D understanding, connected via multi-layer semantic fusion to enable language-conditioned geometric reasoning. To support scalable training, we construct a large-scale dataset of language-viewpoint pairs from both real-world scene-scanning data and simulated environments using an automated pipeline. Extensive experiments demonstrate that I-Perceive significantly outperforms strong baselines on prediction accuracy, viewpoint feasibility, and instructions alignment. The model exhibits strong zero-shot generalization to unseen scenes and instructions, and enables closed-loop active perception, progressively refining viewpoints over sequential interactions.

Temporal action detection in long untrimmed videos still suffers from inaccurate temporal localization and low-quality proposals, especially under sparse temporal annotations and ambiguous event boundaries. In the context of open-ended video understanding and LLM-driven event analysis, reliable temporal action proposals have emerged as a critical interface bridging temporal localization, event annotation, and semantic understanding. However, conventional temporal action detection (TAD) pipelines lack explicit modeling of proposal localization quality, cross-representation support, and inter-proposal relations, making it difficult to consistently provide high-quality proposals. We propose Hybrid Rank-Calibration of Action Proposals (HyrCap), a proposal-centric query-based TAD framework for temporal event understanding. In HyrCap, Hybrid denotes the use of complementary video features that provide different temporal localization cues, while Rank-Calibration refers to calibrating the ranking confidence of dense action proposals so that it better reflects proposal localization reliability and candidate competition. HyrCap treats dense query-based predictions as an over-complete temporal event hypothesis space and improves the ranking reliability, localization quality, and candidate discrimination of action proposals through modules such as Cross-Representation Consensus Calibration, Localization Quality Estimation, and Proposal Duplication Suppression. Furthermore, inspired by the strong capability of LLMs in fine-grained semantic recognition, event description, and reasoning, we introduce HyrCap for Semantic Action Understanding: calibrated proposals can be used for standard TAD detection and as inputs for LLM-based semantic understanding. We further define a staged evaluation protocol to separately measure event localization ability, segment-level semantic understanding, and their combined event understanding performance. Experiments on public benchmarks show that HyrCap achieves state-of-the-art performance across multiple metrics and provides a more fine-grained paradigm and evaluation protocol for query-based event segmentation and annotation.

Modern data-driven decision-making methods, such as imitation learning (IL) and reinforcement learning (RL), have achieved great success in solving many complex tasks. However, these methods often suffer from serious control instability and robustness issues when applied in real-world applications such as robotics and autonomous driving, posing notable challenges for their practical deployment. We argue that this instability issue stems largely from their limitations on solely supervising and optimizing zeroth-order actions (i.e., the action labels), failing to account for higher-order action dynamics and temporal consistency. In this paper, we show that simultaneously supervising both zeroth- and first-order actions can dramatically enhance policies' performance and control robustness. To achieve this, we introduce a novel and elegant loss scheme supported by formal theoretical guarantees that can equip any off-the-shelf policy model (e.g., deterministic, stochastic, or flow policies) with the capability for higher-order action supervision, without requiring any structural modifications. Moreover, our proposed method can serve as a lightweight plug-and-play module that seamlessly integrates with a broad spectrum of existing offline RL frameworks. Extensive evaluations on OGBench and D4RL demonstrate that our approach yields substantial performance and robustness improvements across a wide range of continuous control environments. Notably, our method can also enhance policies' out-of-distribution (OOD) generalization capability in the challenging low-data regime, making it an ideal tool in tackling many real-world control problems.

Vision-Language-Action (VLA) models enable robots to follow natural language instructions and generalize across diverse tasks, but they remain vulnerable to execution failures that compromise reliability in real-world deployment. Detecting such failures during execution is therefore critical for the robust deployment of embodied systems. Existing failure detection methods either rely on expensive action resampling or external models, while alternatives propagate trajectory-level labels uniformly across every timestep, obscuring localized failure signals. In this paper, we propose \textbf{Hide-and-Seek}, a framework that formulates VLA failure detection as a coarsely supervised learning problem. By combining inter-trajectory and intra-trajectory contrastive objectives, Hide-and-Seek localizes failure-indicative actions and induces temporally structured failure signals from trajectory-level supervision alone, without any step-level annotation. We evaluate Hide-and-Seek on LIBERO, VLABench, and a real-world robotic platform across three representative VLA policies: OpenVLA, $\pi_0$, and $\pi_{0.5}$. Our method achieves state-of-the-art multi-task failure detection performance with a practical accuracy--timeliness trade-off under conformal prediction, and generalizes well to both seen and unseen tasks.

Distilling vision-language models into faster hybrid architectures, such as 3:1 Mamba-2/attention mixes, is now standard practice for making inference efficient. Aggregate benchmarks suggest that this works but they hide selective failures. When we distill Qwen3-VL-8B-Instruct into a 3:1 Mamba-2/attention hybrid, student model stays within 2 points of the teacher across visual reasoning benchmarks like MMStar, MMBench, and MMMU-Pro, while dropping 13 points on optical-character-recognition and document tasks. The student can still understand the scene but loses the fine-grained text needed to answer. We localize much of the failure to a specific kind of position. In a high-resolution image, most patches are sky, wall, or smooth texture, while a small fraction carries text, edges, object boundaries, or other local details. In a token-level diagnostic, the top 10\% highest-density patches have 3.6$\times$ larger residual drift than the bottom 10\% lowest-density patches and 3.5$\times$ larger teacher-masking answer contribution. Uniform weighting devotes many loss terms to low-information background patches, whereas sparse answer-bearing patches receive no special protection. The required intervention is minimal: we replace uniform residual alignment with density-weighted residual alignment, using patch self-dissimilarity as a training-free proxy for position importance. We call this HEED. Compared with normal end-to-end distillation, HEED increases performance by 8.7 points on OCRBench v2 and 5.13 points on a 10-benchmark average. The gain is realized on different teacher models and hybrid architectures. After standard post-training, the student reaches teacher-level performance on the 10-benchmark average with a 4.12$\times$ throughput and a 68\% memory saving at 128k context, with no additional parameters and no inference-time cost.

Vision-language models (VLMs) are increasingly used as perception and reasoning modules for embodied intelligence, yet they remain brittle when understanding fine-grained physical processes in videos. While existing models often recognize the overall task, they frequently fail to recover the intermediate actions and result states that determine how an embodied interaction actually unfolds. We argue that this limitation stems in part from a supervision bottleneck: conventional video-language data is typically constructed through post-hoc annotation, where captions are written after video collection and often provide only coarse, loosely aligned descriptions of dynamic physical events. We propose **Gen4Understand**, a generation-for-understanding framework that reverses this data flow. Instead of captioning videos after they are observed, Gen4Understand first represents embodied interactions as structured action scripts composed of atomic events and state deltas, then synthesizes videos conditioned on these scripts and real start-end frame anchors. A verifier-calibrator scores each generated candidate using its conditioning script, endpoint anchors, and video content, filtering for semantic consistency, endpoint fidelity, physical plausibility, temporal coherence, and visual quality. The retained atomic clips are further merged into action-, subtask-, and task-level supervision and converted into captioning and question-answering data for VLM adaptation. Experiments across public embodied video splits, distractor-rich hallucination tests, simulated VLA benchmarks, and real-robot tasks show that Gen4Understand consistently improves fine-grained action understanding, result-state recognition, temporal ordering, hallucination resistance, and downstream embodied control. These results suggest that controllable video generation can serve not merely as data augmentation, but as a mechanism for creating calibrated process-level supervision for embodied intelligence.

Vision-language models (VLMs) for radiology have emerged as a scalable paradigm by leveraging image-report pairs naturally produced in clinical workflows. However, this pairing reveals a mismatch in scale: each finding occupies only a small region of the image, yet supervision is provided only at the global image-report level. This poses a central challenge: prior approaches spread weight densely across all patches rather than concentrating on the sparse subset relevant to a given query. To address this, we present GLINT (Gated Language-Image alignmeNT), a framework that explicitly models this sparse correspondence. On the alignment side, we introduce Sparsely Gated Alignment, a novel architecture in which a sigmoid gate over a separate gate embedding space activates only the patches relevant to each textual query, enforcing explicit sparsity. On the representation side, we add Dense Feature Regularization, which anchors the trainable encoder's intermediate features to a frozen self-supervised learning (SSL) teacher, preserving the fine-grained patch features that the gate relies on. The same recipe applies to both 2D chest X-ray (CXR) and 3D chest computed tomography (CT), built with DINOv3 and V-JEPA 2.1, respectively. GLINT enables zero-shot classification, grounding, and segmentation from free-text queries, and to our knowledge is the first to demonstrate zero-shot segmentation on 3D CT volumes without mask supervision. Notably, the most pronounced gains arise on zero-shot grounding and segmentation, where sparse, query-specific localization is required, consistent with our design intent. In downstream evaluation, GLINT outperforms both SSL encoders and medical VLMs on classification, report generation, and segmentation.

Vision-language models (VLMs) have achieved remarkable progress in 2D visual understanding tasks, yet the passive view-centric perception paradigm limits their capacity for 3D spatial reasoning. Such reasoning often requires actively acquiring spatial evidence beyond the currently visible views, such as camera poses and cross-view spatial relationships, to infer 3D relationships that are not explicitly represented in 2D images. However, existing methods either lack active exploration or merely provide additional views during inference, without explicitly acquiring such spatial evidence beyond 2D images. In this paper, we propose View-to-World (V2W), an active exploration framework that enables VLMs to actively explore 3D worlds and lifts VLMs from passive view-centric perception to active world-centric spatial reasoning. Given a spatial reasoning task with multi-view images, V2W first reconstructs an explicit 3D world with geometry, camera poses, and language-grounded semantics, and exposes it to VLMs through visual and linguistic interfaces. Through multi-turn interaction with these interfaces, VLMs iteratively acquire spatial evidence from the constructed 3D world, enabling active world-centric spatial reasoning. V2W can be integrated with existing VLMs in a training-free manner and can be further enhanced by optimizing the exploration policy via reinforcement learning. Experiments on spatial mental modeling benchmarks demonstrate that V2W substantially improves VLM baselines in the training-free setting and achieves state-of-the-art performance with the learned exploration policy, highlighting the importance of active exploration for spatial intelligence. The code will be available upon acceptance.

Although Large Vision-Language Models (LVLMs) have achieved remarkable progress, complex visual reasoning remains challenging. Existing approaches suffer from fundamental limitations: *Thinking about Image* mainly performs reasoning in text space, *Thinking with Image* relies on external tools to introduce additional visual evidence, and *Latent Visual Reasoning*, despite moving reasoning into latent space, still focuses largely on static visual modeling and lacks explicit characterization of temporal dynamics. We therefore propose **Dynamic Latent Visual Reasoning (DLVR)**, a training framework that enables LVLMs to reason about dynamics in continuous latent space from only a static image. We introduce Mahalanobis Novelty Token Selection and Novelty-Adaptive Temporal Quantization to construct dynamic latent supervision from open-source datasets, building DLVR-SFT-100K and DLVR-RL-4K. We further develop a two-stage SFT pipeline that first builds temporal grounding over explicit dynamic processes and then teaches the model to encode dynamic semantics and temporal structure into latent tokens. Finally, we propose Contrastive Dynamic Latent Policy Optimization, which encourages latent trajectories to align with real dynamics while moving away from counterfactual ones. DLVR empirically improves both dynamic-centric and general visual reasoning, achieving an average gain of 10.82% on BabyVision and a 10.50% improvement on HRBench4K, demonstrating the promise of dynamic latent reasoning for LVLMs.

Adapting pretrained Vision-Language-Action (VLA) policies to new deployment environments typically requires collecting expert demonstrations in each target domain, which is costly and difficult to scale. In this paper, we present FreeAct, a framework for demonstration-free robot adaptation that converts generated videos into physically grounded supervision. However, a key challenge is that generated manipulation videos, while visually plausible, often contain embodiment artifacts and kinematic inconsistencies that make direct inverse-dynamics labeling unreliable. FreeAct addresses this challenge by learning a discrete latent action space from large-scale multi-lab robot videos and aligning it with camera-frame end-effector SE(3) motion and gripper-state changes. The resulting latent action tokenizer captures task-relevant motion while reducing reliance on embodiment-specific visual artifacts, enabling generated target-domain videos to be pseudo-labeled with reliable latent action tokens. These tokens are first injected into VLA pretraining and then used, together with source-domain action supervision, to adapt the policy to unseen environments. Empirically, FreeAct improves success from 2\% to 57\% under target-domain shift using generated videos alone, and further to 74\% through MLLM-guided self-evolution on the policy's own rollouts, closing the adaptation loop without expert demonstrations. Further analysis shows that the learned latent action space forms a kinematically structured and transferable representation, establishing generated videos as a scalable source of supervision for robot manipulation.

Large vision-language-action (VLA) policies are increasingly trained as conditional generative models over action chunks, yet they typically improve by imitating the data they are given. This is limiting in deployment, where robots collect mixed-quality experience containing demonstrations, partial completions, recoverable mistakes, and failures. Full behavior cloning imitates failures, filtered behavior cloning discards useful sub-trajectories, and offline RL usually requires a large separate critic. We introduce ForesightFlow, a self-guided flow policy that augments each generated action chunk with a learned success-potential vector. The same flow therefore proposes candidate actions and scores them, enabling best-of-K inference without an external critic. The key challenge is that policy improvement and value calibration require different supervision: advantage weighting should suppress low-quality actions, but applying the same weights to the potential dimension removes the failure gradients needed for calibration. We address this with decoupled advantage-weighted flow matching, applying exponentiated advantage weights only to action velocities while training potential velocities uniformly. We also derive a single-step boundary estimator for conditional flow matching with independent endpoint sampling, allowing advantage computation with one stop-gradient forward pass. Across five BEHAVIOR-1K simulation tasks and five real-world bimanual tasks, ForesightFlow improves over imitation baselines, matches the strongest separate-critic baseline in average simulation success, improves real-world success, and reduces training compute by 38\%. Ablations show that decoupling prevents value hallucination, the one-step estimator preserves candidate-ranking fidelity, and self-guided sampling improves long-horizon performance.

Vision-Language-Action (VLA) models are becoming an important approach for robotic manipulation, but their long visual token sequences make inference expensive. Visual token pruning is a practical way to reduce this cost, with many pruning methods rely on attention scores to decide which tokens to remove. However, we show that this reliance can be exploited: an attacker can hijack attention scores and turn the visual token pruner into a vulnerability. Based on this insight, we propose FocusVLA, a backdoor attack against attention-based pruning. FocusVLA inserts trigger-controlled attention patterns into a few low-impact attention heads. These poisoned heads have little effect on normal manipulation. Without pruning, the poisoned model still behaves normally when the trigger appears. With pruning enabled, clean performance remains intact. But inputs with the trigger shift attention to task-irrelevant peripheral regions. As a result, the pruner removes task-critical tokens, leading to task failure. We evaluate FocusVLA on OpenVLA-OFT and $\pi_{0.5}$ across LIBERO tasks with ten attention-based pruning methods. For example, on OpenVLA-OFT with ADP pruning, the success rate on LIBERO-10 drops from 90.6\% to 31.0\% under triggered inputs. These results reveal a broader risk: attention is widely used as an importance signal, but it can be manipulated and should not be blindly trusted.

Vision-and-Language Navigation(VLN) requires agents to sustain task-directed spatial perception to focus on navigation-critical regions while translating linguistic intent into low-level actions. Prevalent hierarchical planning methods rely on a high-level policy to select waypoints or subgoals for low-level control, but they often fail to remain focused, causing inaccurate or redundant subgoals that directly mislead execution. Moreover, their separation between visual perception and action execution neglects fine-grained spatial reasoning in low-level movements and limits real-world deployability. To address these two issues, we propose FocusNav, a grounding-driven end-to-end monocular framework that learns task-directed spatial perception through action-aware future reconstruction. Concretely, FocusNav first boosts the agent’s visual attention on task-relevant regions with a Grounding-driven Visual Reconstruction module that reconstructs instruction-relevant targets and traversable areas via a semantic-guided latent diffusion process. Subsequently, to bridge the perception–execution gap, we incorporate physical action priors via Action-aware Visual Conditioning and enforce Intent-Effect Alignment between anticipated visual changes and observed action effects. These effectively enhance the agent’s task-directed spatial perception to direct attention to navigation-critical regions, and tighten the perception–execution loop. Experiments on R2R-CE and RxR-CE benchmarks show that FocusNav achieves substantially improved state-of-the-art performance with a simple monocular low-level policy, while providing faster inference than recent methods. Furthermore, real-world robot experiments validate our FocusNav’ s effectiveness, ease of deployment, and lightweight design.

Large Vision-Language Models (LVLMs) have achieved impressive progress in multimodal reasoning, yet they remain prone to object hallucinations, generating descriptions of objects that are not present in the input image. In this work, we investigate hallucination from the perspective of attention-value dynamics inside LVLM vision encoders. We identify a consistent three-phase structure of visual processing---diffusion, focus, and rediffusion---and show that the focus phase is where attention most clearly separates strongly and weakly supported visual tokens. However, low attention does not necessarily imply negligible downstream influence: low-attention tokens in the focus phase can still exert non-negligible value-side influence on the attention output relative to their small attention mass. Through controlled phase-wise value interventions, we find that hallucination behavior is particularly sensitive to the value content of low-attention tokens during this phase. Replacing or neutralizing these values reduces hallucination metrics while largely preserving grounded object evidence. A token-level teacher-forcing analysis further shows that the intervention reduces the probability of hallucinated object tokens with much smaller effects on ground-truth object tokens. In addition, Visual Attention Ratio (VAR) analysis shows that focus-phase intervention is accompanied by increased attention to visual tokens during decoding. Based on these observations, we instantiate a simple training-free inference-time intervention that replaces focus-phase low-attention values with an image-level mean value vector using statistics from a single forward pass. Experiments across multiple LVLM backbones demonstrate that this analysis-derived intervention reduces object hallucination with negligible additional runtime and remains compatible with existing inference-time mitigation methods. Our project page is available at: https://anonymous.4open.science/w/FocusMatters-7B32/

Multimodal driving planning faces a long-standing tension between two paradigms: scoring-based methods benefit from dense reward supervision but are confined to a fixed action vocabulary, while anchor-based methods generate proposals dynamically yet suffer from sparse supervision constrained to a single ground-truth trajectory. In this work, we propose FlowR2A, which resolves this tension by reframing simulation-based rewards from discriminative targets into generative conditions. By learning the reward-conditioned action distribution from dense trajectory-reward pairs with a flow-matching decoder, FlowR2A unifies the dense supervision of scoring-based methods with the proposal generation of anchor-based methods in a single generative model, forcing the model to internalize the correlation between an action and its outcomes in safety, progress, comfort, and rule compliance. To balance hard safety constraints against soft progress objectives, we introduce fine-grained per-timestep reward conditioning and reward noise augmentation. The generative formulation naturally supports controllable test-time sampling via reward guidance and anchored sampling, producing high-quality proposals. FlowR2A achieves state-of-the-art results on the NAVSIM v1 and v2 benchmarks, with multimodal proposals of substantially higher quality than prior methods.

Vision-Language-Action (VLA) models have advanced rapidly by leveraging large-scale vision-language priors and robot trajectory data, yet they remain weak at following fine-grained human instructions. We argue that this limitation stems from a fundamental mismatch between language and action supervision in existing robot datasets. Most open-source datasets annotate each trajectory with a goal-oriented task description, while leaving the underlying execution process largely unspecified, including motion trajectories, contact and approach patterns, object state transitions, recovery behavior, and final configurations. Since different choices along these dimensions can lead to substantially different action sequences, coarse task descriptions induce a many-to-one mapping from action trajectories to language, preventing precise action-instruction alignment. To address this problem, we propose FineVLA, a framework for constructing and leveraging fine-grained embodied action instructions for VLA learning. FineVLA includes FineVLA-Tool, a scalable pipeline for data cleaning, temporal alignment, and fine-grained annotation; RoboFine-VLM, a vision-language model for fine-grained robotic action understanding and annotation; RoboFine-Bench, a benchmark for evaluating fine-grained robotic action understanding via VQA and captioning; and FineVLA-Policy, a VLA policy trained with fine-grained instructions on open-source ALOHA robot data. Across benchmark evaluation and policy learning experiments, FineVLA demonstrates that high-density, action-aligned language supervision leads to more controllable and instruction-sensitive robot policies. Our results highlight fine-grained instruction alignment as a critical step toward VLA systems that not only complete tasks, but complete them in the way humans specify.

World Action Models (WAMs) have emerged as a promising alternative to Vision-Language-Action (VLA) models for embodied control because they explicitly model how visual observations may evolve under action. Most existing WAMs follow an imagine-then-execute paradigm, incurring substantial test-time latency from iterative video denoising, yet it remains unclear whether explicit future imagination is actually necessary for strong action performance. In this paper, we ask whether WAMs need explicit future imagination at test time, or whether their benefit comes primarily from video modeling during training. We disentangle the role of video modeling during training from explicit future generation during inference by proposing **Fast-WAM**, a WAM architecture that retains video co-training during training but skips future prediction at test time. We further instantiate several Fast-WAM variants to enable a controlled comparison of these two factors. Across these variants, we find that Fast-WAM remains competitive with imagine-then-execute variants, while removing video co-training causes a much larger performance drop. Empirically, Fast-WAM achieves competitive results with state-of-the-art methods both on simulation benchmarks (LIBERO and RoboTwin) and real-world tasks, without embodied pretraining. It runs in real time with 110 ms latency, over 6$\times$ faster than existing imagine-then-execute WAMs. These results suggest that the main value of video prediction in WAMs may lie in improving world representations during training rather than generating future observations at test time.

Vision Language Action (VLA) models can execute isolated atomic skills, but practical long horizon manipulation requires recombining these skills into new compositions without demonstrations for every skill combination. The direct full instruction route treats skill chaining as long horizon imitation. In our controlled study, full instruction training on limited skill combination sequences reaches 40.0% ordered success on covered combinations but only 7.7% on recombination, suggesting memorization of sequence templates rather than robust skill recombination. We therefore introduce Hierarchical Subtask Execution (HSE), a diagnostic execution framework that exposes skill recombination through a shared modular interface over the planner, trigger, and policy, enabling long horizon failures to be decomposed into separable modes. To make this diagnosis measurable, we introduce RoboCombine, a benchmark for skill recombination without combined skill training trajectories, with metrics for ordered success and transition conversion across subtask boundaries. RoboCombine reveals that atomic competence alone does not ensure ordered execution. Failures concentrate at transitions where downstream subtasks must launch from handoff states produced by preceding subtasks rather than canonical atomic starts. We identify this mismatch as the start state gap. To address it, we introduce Pose Robust Task Initialization (PRTI), a lightweight atomic task post training strategy that broadens start state coverage beyond canonical starts. Under oracle timed HSE on RoboCombine, adding PRTI raises ordered success from 2.11% to 26.32% and transition conversion from 2.63% to 60.53%, without combined skill training trajectories. Together, these results show that combinatorial generalization requires handoff robust atomic skills that convert first stage progress into reliable downstream execution, not merely decomposition into known subtasks. We will release RoboCombine and HSE code to enable reproducible diagnosis of atomic skill recombination failures.