NeurIPS 2026 Papers

Model inversion is a widely adopted technique in data-free learning that reconstructs inputs from a pretrained model through iterative optimization, without access to original data. However, its application to Vision Transformers (ViTs) incurs high computational cost due to expensive self-attention mechanisms. To address this, $\textit{Sparse Model Inversion}$ (SMI) was proposed to improve efficiency by gradually pruning seemingly unimportant patches, even claiming they are obstacles to knowledge transfer. However, our empirical findings suggest the opposite: even randomly selected patches can eventually acquire transferable knowledge over the inversion process. In fact, we further observe that removing prematurely inverted patches hinders the extraction of class-agnostic features essential for knowledge transfer, as well as class-specific features. In this paper, we propose $\textit{Patch Rebirth Inversion}$ (PRI), a novel approach that constructs multiple sparse images within a single inversion process by incrementally detaching informative patches, instead of removing unimportant ones. This strategy not only improves efficiency, but also encourages initially less informative patches to gradually accumulate more class-relevant knowledge, a phenomenon we refer to as the $\textit{Re-Birth}$ effect, thereby effectively balancing class-agnostic and class-specific knowledge. Experimental results show that PRI achieves up to 10$\times$ faster inversion than standard $\textit{Dense Model Inversion}$ (DMI) and 2$\times$ faster than SMI, while consistently outperforming SMI in accuracy and matching the performance of DMI.

Vision-Language-Action (VLA) models promise a single language-conditioned policy adaptable to many manipulation behaviors, yet deployment additionally requires skill retention and long-horizon composition: a policy must retain previously acquired skills and reuse them inside composite tasks whose behavior changes across phases. The standard recipe absorbs every new task into one shared update, which provides neither a parameter-level locus for reusable skills nor a mechanism for invoking each skill along the trajectory. We observe that this missing structure is latent rather than absent: per-skill low-rank updates of a pretrained VLA occupy a robustly distinguishable, low-dimensional subspace, and a single composite-task LoRA concentrates roughly half of its Frobenius energy in this subspace, against a near-zero matched random-rotation control. Building on this geometry, we introduce PRISM, which represents reusable skills as parameter-level primitives: each skill family is stored as a dedicated low-rank action-expert adapter in a primitive bank, primitives are composed at inference by a per-chunk VLM-conditioned router that reads the prefix representation to track task progress, and the router is supervised by per-chunk labels from a deterministic, action-derived oracle. During composite training, only the vision encoder, the language LoRA, and the router are updated; every primitive is held frozen, so the per-skill action-expert parameters are preserved by construction. On RoboCasa365, PRISM preserves atomic-skill success through composite training and reaches 20% on long-horizon composite tasks, while a standard $\pi_{0.5}$-LoRA two-stage fine-tune instead loses roughly 25 atomic points and reaches only 11% composite. We additionally validate PRISM on an AgileX PIPER real-robot setup.

Vision-Language-Action (VLA) models show promising ability in language-guided robotic tasks. However, making VLA policies reliable remains challenging, because a manipulation task is completed through closed-loop interaction, where each action affects subsequent execution. To analyze this problem, we revisit VLA policy during execution and argue that a VLA policy acts both as a planner, which makes task-oriented decisions that change the direction of execution, and as an executor, which realizes these decisions through dense continuous actions. This view suggests that improving VLA reliability requires particular attention to planning actions. Existing optimization methods can imitate actions or improve complete trajectories, but they usually do not explicitly identify planning actions or measure their importance for task success. To address this issue, we propose Planning-Aware Policy Optimization for VLA models (PAPO-VLA). PAPO-VLA first identifies planning actions by jointly considering action variation and trajectory outcome, then estimates their importance through causal sufficiency and causal necessity, and finally incorporates this importance into GRPO advantage estimation. In this way, more important planning actions receive stronger optimization emphasis, while the whole trajectory is still optimized by trajectory-level feedback. Experiments on multiple benchmarks demonstrate the effectiveness of PAPO-VLA.

Examining existing benchmarks for grounding in 3D VLMs reveals a pervasive "semantic leakage" in the evaluation and prompting protocols: a heavy reliance on bounding boxes as spatial prompts or representation primitives. Because bounding boxes are highly informative of object categories, they introduce a shortcut for the models to neglect visual information during 3D VLM training. We demonstrate this with a "blind" LLM that, prompted only with the box coordinates, can perform surprisingly well. To further analyze this shortcut, we propose point-based grounding prompting - querying models with a single 3D point instead of a box. Under this protocol, the accuracy of the blind LLM drops from $\textbf{36.4}$% to $\textbf{18.0}$% on ScanNet's Nr3D, and similarly degrades for 3D VLMs (e.g. LL3DA). Building on this insight, we introduce $P^3$-VLM, a novel tri-pathway 3D VLM that supports point-based prompts and achieves $\textbf{52.1}$% state-of-the-art accuracy on location-grounded captioning under this setting. To achieve this, we redesign the architecture while keeping it detector-free, scene-centric, and suitable for 3D GS inputs. $P^3$-VLM utilizes three complementary pathways (location-, task-, and context-aware), enabling strong reasoning about scenes without any bounding box dependencies. The key benefit of point-based learning is a consistent improvement across settings: $P^3$-VLM achieves state-of-the-art performance under point-based grounding while also improving robustness, OOD generalization, and performance on ungrounded scene-centric 3D VQA benchmarks, reflecting its reduced reliance on bounding-box shortcuts and resulting stronger visual-spatial reasoning. Our source code and models will be made publicly available.

The goal of this paper is to strengthen the reasoning of Omnimodal Large Language Models (Omni-LLMs) at inference time, without additional training. These models jointly process video, audio, and text, and given the large number of tokens they consume, how attention is routed across them is central to their behaviour. We focus specifically on attention sinks, tokens that absorb a disproportionate share of attention mass regardless of their semantic content, to understand how this routing unfolds. To this end, we conduct a systematic analysis of sink behaviour in Omni-LLMs. Our analysis yields two key findings: (i) high sink attention does not solely indicate head redundancy, suggesting that sink value representations play additional functional roles; (ii) the sink value vector acts as a shared bias added to every token's output, serving as a global signal that organises the representation as a whole. Building on this, we propose OutRo, which correspondingly aligns non-sink token representations with the sink in feature space, and relaxes the causal mask for sink tokens at an early layer to sharpen this bias before the rest of decoding proceeds. This design enhances the reasoning process without requiring additional forward passes or access to attention maps. Based on extensive experiments, OutRo consistently improves performance on seven video QA benchmarks and demonstrates strong generalisation, while incurring only a 1.1\x decoding overhead.

Understanding human gaze behavior is essential for complex scene comprehension and human-computer interaction. Traditional gaze following models are typically restricted to pure spatial localization, lacking the high-level capacity to reason about semantic targets or complex social contexts. Furthermore, these models often process individuals sequentially, requiring redundant computations over the same scene image for multi-person inference. While recent Vision-Language Models (VLMs) offer the exceptional semantic reasoning needed to address gaze-related semantic tasks, their reliance on discrete text generation inherently limits precision in continuous spatial tasks like gaze localization. To bridge this gap, we propose OmniGF, a unified vision-language framework that adapts foundational VLMs for highly scalable multi-person gaze reasoning. The model adopts a dual-branch decoding strategy: a structured language branch generates discrete reasoning states, while a continuous spatial branch directly taps into the VLM's dense hidden states. Supervising these extracted representations with high-resolution gaze target heatmaps effectively overcomes the spatial bottleneck of text-only coordinate generation. Furthermore, to explicitly ground the model in multi-person scenes, we augment the input with head embeddings encoded from cropped head images, providing fine-grained appearance and orientation cues for all individuals simultaneously. By modeling all individuals and leveraging the strong semantic capability of VLMs, OmniGF seamlessly integrates precise spatial gaze target estimation, semantic gaze prediction, and complex social gaze reasoning. Extensive experiments demonstrate that our framework establishes new state-of-the-art performance across multiple standard benchmarks.

Large vision-language models (LVLMs) achieve strong multimodal reasoning performance but remain prone to object hallucination, where generated objects or existence judgments are not faithfully grounded in the image. Existing mitigation methods either require additional training or rely on external visual experts, which limits plug-and-play usability and increases inference overhead. In this paper, we propose SIGMA, a training-free framework for hallucination mitigation based on Self-vision Information-theoretic Gating and dual-Mask Assembly. SIGMA operates directly on the LVLM's own vision encoder, constructing complementary support and complement masks from image-driven concept relevance, question-aware entity cues, and CLS--patch shortcut suppression. These masks form global, support-masked, and complement-masked visual streams for query-conditioned evidence separation. During decoding, SIGMA activates correction only under high uncertainty, using full-vocabulary and candidate-answer entropy to perform support-first refinement and complement-on-demand suppression. The resulting streams are combined through logit-level assembly with lightweight positive calibration and a plausibility constraint. Experiments across multiple LVLM backbones and hallucination-oriented benchmarks show that SIGMA effectively reduces object hallucination while maintaining competitive perception performance and a selective low-overhead inference path.

Multi-view 3D vision-language-action (VLA) policies are brittle to small sensor perturbations: depth jitter, partial occlusion, calibration drift and per-view dropout can move the predicted action chunk by far more than the perturbation itself. We trace the brittleness to the *fusion stage*: concatenating per-view tokens lets a small geometric shift flip arbitrary token indices, turning a continuous-mass-transport problem into a discrete indexing problem. We propose **a single substitution**: replace concat-then-attention fusion with a *differentiable Wasserstein barycentric tokenizer* that fuses $K$ view measures into $M$ barycentric tokens by transport. We prove a chain of Lipschitz-style stability bounds that hold conditional on a declared OT perturbation set, calibrated Lipschitz constants, and bounded Sinkhorn-solver residuals. Empirically, the barycenter is the single dominant mechanism behind the gain: (i) the proposed model has the lowest mean clean $\ell_2$ on synthetic, LIBERO, RLBench and CALVIN ABC→D (3-of-4 with non-overlapping 95% CIs); (ii) a controlled ablation on synthetic regresses by $46\times$ on the corruption-stability metric when the barycenter is removed and by $<1.5\times$ when any other component is; (iii) the same mechanism transfers to LIBERO with a $+37\%$ clean $\ell_2$ regression and $+62\%$ action-OT regression on real data; (iv) in synthetic closed-loop the concat-fusion baseline collapses from 85% clean to 0% under per-view dropout while the barycenter keeps a 92% worst-case across all five conditions (3 seeds, narrowest CI of all four methods).

Vision-language-action (VLA) models have shown strong promise in robotic manipulation, yet their post-training still predominantly relies on trajectory-level supervised imitation over robot demonstrations. While effective for learning feasible actions, this paradigm supervises policies to reproduce expert actions at each step, biasing them toward demonstration-specific action paths and providing limited guidance for recovering toward task-progress states once execution deviates from demonstrated trajectories. To address this limitation, we propose OGPO, an Offline Goal-conditioned Policy Optimization framework for recoverable VLA post-training. Instead of treating offline demonstrations merely as step-wise action labels, OGPO adaptively relabels trajectories with semantic key states and constructs goal-conditioned process rewards to optimize the policy for reaching these task-progress states. In this way, OGPO shifts VLA post-training from trajectory-level action imitation to key-state reachability learning, enabling policies to acquire a more robust ability to complete goal states from feasible off-demonstration states without requiring additional online interaction. Experiments on LIBERO and MetaWorld show that OGPO consistently improves VLA policy performance over standard supervised fine-tuning. Moreover, evaluations on LIBERO-Plus and our proposed LIBERO-ReAct, a mid-execution perturbation benchmark, further demonstrate that OGPO achieves stronger robustness and goal-directed recoverability under distribution shifts and recoverable off-demonstration states.

Continual tuning is essential for adapting multimodal large language models to evolving tasks and domains, yet existing methods often assume that every image-present example provides valid supervision for the visual pathway. We address the resulting gap: how to learn from all incoming multimodal instructions while preventing visually unnecessary supervision from drifting the visual interface. We propose Visual-Necessity-Gated Continual Tuning (VNG-CT), a path-aware framework that estimates sample-level dependence on visual evidence by comparing target likelihoods under the original image and a counterfactual null image. The resulting gate routes gradients to different parameter paths: all examples update the language path, visually necessary examples update the visual path, and low-necessity examples are absorbed by a lightweight calibration path that promotes invariance to irrelevant visual evidence. Across CoIN, MLLM-CL, and UCIT-style continual instruction streams, VNG-CT improves final and average performance while reducing visual forgetting. These results suggest that visual necessity offers a practical principle for stabilizing continual multimodal adaptation without discarding useful language-side supervision.

Robot fine-tuning can improve manipulation success while degrading the semantic structure inherited from large vision-language pretraining. This trade-off limits the reuse of vision-language-action (VLA) policies in settings that require compositional instructions, camera changes, and coordination with higher-level agents. We present \textbf{NoTVLA}, a semantics-preserving robot adaptation framework that replaces dense low-level action supervision with a sparse narrative action interface. NoTVLA converts demonstrations into sparse, semantically meaningful waypoints, grounds each decision with a task-relevant visual anchor and depth value, and reconstructs executable motion through a deterministic detokenizer. The resulting interface keeps the autoregressive vision-language backbone close to its pretrained prediction format while delegating high-frequency control to a transparent motion-rendering stage. We evaluate this design through matched and backbone-family comparisons, semantic retention probes, semantic out-of-distribution manipulation tasks, camera and depth perturbations, and deployment-oriented efficiency analysis. The results suggest that robot adaptation should be judged not only by task success, but also by how much task-relevant semantic competence is preserved during fine-tuning.

Delayed hits occur when requests arrive while a previous miss for the same item is still being fetched. They break the equivalence between hit ratio and latency because caching decisions depend on frequency, burst structure, and fetch latency. We develop a regret framework for delayed-hit caching, with static regret against long-term placement and switching regret against piecewise-changing workloads. We show that classical deterministic policies and randomized Marker fail to achieve no-regret guarantees, both suffering linear static regret. For stochastic arrivals, we introduce delayed-hit shadow values and propose UCB-DH, which achieves $O(\log T)$ static regret on fixed instances with positive gaps, where $T$ is the horizon. A matching gap-dependent lower bound shows that the delayed-hit latency-amplification factor is statistically unavoidable, and restarted UCB-DH gives sublinear regret for piecewise-changing workloads. For oblivious adversarial arrivals, we formulate an action-independent shadow loss and propose LA-FTPL with latency-scaled perturbations and completion-time admission. LA-FTPL achieves $O(\sqrt{KT\log N\,(Z^2+C_BZ)})$ shadow static regret, where $K$ is the cache capacity, $N$ the item universe size, $Z$ the fetch latency, and $C_B$ the maximum one-window delayed-hit penalty. With periodic restarts, it achieves $O(ZT^{2/3}B^{1/3}(K\log N)^{1/3})$ switching regret against comparators with at most $B$ cache changes. Experiments on synthetic and real-world traces show that our algorithms achieve sublinear regret and improve latency over the state-of-the-art baselines.

Chain-of-thought reasoning has driven striking advances in language model capability, yet every reasoning step grows the KV cache, creating a bottleneck to scaling this paradigm further. Current approaches manage these constraints on the model’s behalf using hand-designed criteria. A more scalable approach would let end-to-end learning subsume this design choice entirely, following a broader pattern in deep learning. After all, if a model can learn to reason, why can't it learn to forget? We introduce \textbf{Neural Garbage Collection (NGC)}, in which a language model learns to forget while learning to reason, trained \emph{end-to-end} from outcome-based task reward alone. As the model reasons, it periodically pauses, decides which KV cache entries to evict, and continues to reason conditioned on the remaining cache. By treating tokens in a chain-of-thought and cache eviction decisions as discrete actions sampled from the language model, we can use reinforcement learning to jointly optimize how the model reasons and how it manages its own memory: what the model evicts shapes what it remembers, what it remembers shapes its reasoning, and the correctness of that reasoning determines its reward. Crucially, the model learns this behavior entirely from a \emph{single learning signal} — the outcome-based task reward — without supervised fine-tuning or proxy objectives. On Countdown, AMC, and AIME tasks, NGC maintains strong accuracy relative to the full-cache upper bound at a 2–3x compression in peak KV cache size and substantially outperforms eviction baselines. Our results are a first step towards a broader vision where end-to-end optimization drives both capability and efficiency in language models.

Vision-Language-Action (VLA) models promise generalist robotic agents, yet consolidating skills learned across heterogeneous tasks, environments, and embodiments remains difficult: training data is fragmented across platforms, joint multi-task training suffers from interference, and model merging tends to introduce parameter conflicts or require architectural changes. We instead advocate a paradigm in which skills are first learned independently and then consolidated post hoc. To this end, we propose **NestedVLA**, which replaces parameter aggregation with **parameter generation**: a nested hypernetwork, conditioned on the target task, synthesizes task-adaptive parameters that compose with a frozen base VLA. This formulation enables flexible task-conditioned skill reuse while mitigating interference. Across three RL benchmarks (MetaWorld, ManiSkill, CALVIN), NestedVLA outperforms the strongest model-merging baseline by **41 pp on average**, matches per-task fine-tuning **within 1 pp on ManiSkill**, and substantially closes the gap on CALVIN, pointing toward a scalable route to skill consolidation.

We study a structured stochastic multi-armed bandit problem in which each decision simultaneously determines the learner's reward and the feedback available for learning. At each round, the learner selects a task and an option. Each task is associated with an unknown parameter that governs its reward, while each option has a known reward function of this parameter and determines how information about that reward is observed. Some options may provide fine-grained feedback at the cost of lower immediate reward, whereas others may yield higher reward but return only coarse feedback. This creates an intrinsic trade-off between reward maximization and information acquisition. We formalize this setting as parametric bandits with action-dependent coarsened feedback. For stationary tasks, we propose ACF-UCB, an optimistic algorithm that constructs task-level confidence sets by aggregating information from all atoms of the option-dependent feedback distributions. For piecewise-stationary tasks, we develop SW-ACF-UCB, a sliding-window variant that adapts to temporal changes. We prove high-probability regret bounds for stationary environments and dynamic regret guarantees for piecewise-stationary environments. We also establish lower bounds showing that the dependence on the number of tasks and the horizon is unavoidable, while methods that ignore the shared parametric structure suffer substantially larger regret.

Vision-Language-Action (VLA) models are formulated to ground instructions in visual context and generate action sequences for robotic manipulation. Despite recent progress, VLA models still face challenges in learning related and reusable primitives, reducing reliance on large-scale data and complex architectures, and enabling exploration beyond demonstrations. To address these challenges, we propose a novel \textbf{N}euro-\textbf{S}ymbolic \textbf{V}ision-\textbf{L}anguage-\textbf{A}ction (NS-VLA) framework via online reinforcement learning (RL). It introduces a symbolic encoder to embedding vision and language features and extract structured primitives, utilizes a symbolic solver for data-efficient action sequencing, and leverages online RL to optimize generation via expansive exploration. Experiments on robotic manipulation benchmarks demonstrate that NS-VLA outperforms previous methods in both one-shot training and data-perturbed settings, while simultaneously exhibiting superior zero-shot generalizability, high data efficiency, and an expanded exploration space. Our code is available at https://anonymous.4open.science/r/NS-VLA/.

We study multi-armed bandits (MABs) augmented with best-action queries, in which the learner may additionally query an oracle that reveals the best arm in the current round. This setting was recently characterized by Russo et al. [2024] in the full-feedback model, where the learner observes the rewards of all arms after each round. They show that, in both stochastic and adversarial environments, $k$ best-action queries reduce the optimal $\widetilde{\mathcal{O}}(\sqrt{T})$ regret to $\widetilde{\mathcal{O}}(\min\{T/k,\sqrt{T}\})$. Whether this improvement extends to the more realistic bandit-feedback model---where the learner observes only the reward of the played arm---was left as an open problem. We fully resolve this question. When rewards are stochastic but correlated among arms, we show that the full-feedback result does not carry over: any algorithm must incur regret at least $\Omega(\sqrt{T-k})$. This lower bound directly extends to adversarial environments. On the positive side, we show that $\widetilde{\mathcal{O}}(\min\{T/k,\sqrt{T-k}\})$ regret is still achievable when rewards are stochastic and i.i.d., and establish a matching lower bound, up to logarithmic factors. Together, these results provide a complete characterization of the benefits of best-action queries in the bandit-feedback model.

When learning new concepts sequentially, Vision-Language Models (VLMs) face forgetting that degrades both previously acquired tasks and inherent zero-shot capabilities. This challenge is particularly pronounced in the Cross-domain Task-Agnostic Incremental Learning (X-TAIL) setting, where the absence of explicit task identifiers forces diverse concepts to coexist within a unified and crowded multi-modal space. In this context, we identify an underexplored vulnerability termed Asymmetric Boundary Encroachment (ABE). Even when historical feature spaces remain stable, unconstrained new text features actively encroach upon these established sub-spaces during training. To systematically counteract ABE, we propose a unified framework. First, we introduce Analytic Decision Boundary (ADB), which constructs a geometric defense by enforcing an analytic margin derived from cumulative statistics to secure historical multi-modal boundaries. Furthermore, we integrate Orthogonal Visual Adaptation (OVA) and Analytic Ridge Classifier (ARC) to safely evolve the visual backbone and enhance joint inference. Experiments under X-TAIL setting demonstrate that our framework effectively mitigates ABE and consistently achieves new state-of-the-art performance. Our code is available at https://anonymous.4open.science/r/abe-cl-vlm.

In the animal kingdom, mirror self-recognition is a canonical probe of higher-order cognition, emerging only in some species. We ask whether an analogous functional capability emerges in embodied vision-language model (VLM) agents: can they recognize themselves in a mirror? We introduce a controlled 3D benchmark where a first-person VLM agent must infer a hidden body attribute from its reflection and select the matching target, while avoiding self–other misattribution. To separate mirror-grounded self-identification from shortcuts, we test mirror removal, misleading cues, and occluded reflections. We also evaluate the decision process through mirror seeking, temporal ordering, self-attribution, and reasoning–action consistency. Our experiments show that mirror-based self-identification emerges mainly in stronger VLMs. These models can use reflected evidence for action, whereas weaker models often inspect the mirror but fail to extract self-relevant information or misattribute their reflection. Language–vision conflict further shows that self-referential language alone is not evidence of grounded self-identification. Overall, mirror-based evaluation provides a diagnostic for whether embodied self-grounding is causally rooted in perception and action rather than priors, prompt compliance, or confabulation.