NeurIPS 2026 Papers

Large language models (LLMs) have demonstrated promising capabilities in tool calling. However, existing benchmarks predominantly evaluate models in an idealized setting where ground-truth tool definitions are provided in context. This assumption diverges significantly from real-world deployment scenarios, where the model must first identify user intent and retrieve relevant tools from a large, heterogeneous tool repository before invoking them. In this work, we introduce a Retrieval-augmented Tool-calling Benchmark (RTCBench). In RTCBench, we collect 1,840 tasks from existing benchmarks and augment them with two tool environments: 1) None: no available tools provided in the prompt and 2) Distractor: strong distractor tools, while the target tools are excluded. By removing oracle tool access, RTCBench compels LLMs to actively retrieve relevant tools from a large-scale repository via a unified search tool before making tool calls, thereby faithfully simulating the end-to-end complexity of real-world tool-use pipelines. Through comprehensive evaluation of diverse LLMs ranging from 4B-parameter models to proprietary frontier models, we identify two core deficiencies: unreliable tool discovery, where models fail to initiate effective retrieval actions, and post-retrieval grounding failures, where models cannot reliably select and invoke the correct tools from semantically similar retrieved candidates. Furthermore, to mitigate these issues, we release 13,278 retrieval-augmented tool-calling expert trajectories with high-quality chain-of-thought reasoning for LLM post-training. Experiments show that a 4B-parameter model post-trained with GRPO improves its tool-calling accuracy on RTCBench from approximately 47.9\% to over 85.8\%, approaching the performance of the state-of-the-art model Gemini-3.1-Pro.

V2X collaborative perception improves single-vehicle perception by aggregating multi-view features from collaborative agents. However, existing methods typically rely on simple spatial warping to align the collaborative features, which can cause semantic misalignment during fusion. Meanwhile, these deterministic warping-based methods are highly sensitive to the positional noises. In this paper, we propose a plug-in module called Robust Semantic and Spatial Aligner (RSSA), which performs semantic and probabilistic spatial transformations for robust feature alignment. Specifically, a Semantic Transformation (SemT) block modulates feature channels based on the observed relative pose to correct semantic inconsistencies. Subsequently, a Probabilistic Spatial Transformation (P-SpaT) block performs probabilistic warping by sampling from the Gaussian-modeled relative pose space. A Relative Transformation Augmentation (RT-Aug) strategy is further introduced to augment the relative poses and facilitate the training of the RSSA. Extensive experiments on both simulated and real-world datasets demonstrate that RSSA improves the state-of-the-art methods by a large margin under the configuration of both clean and noisy positions.

Traditional change detection identifies where changes occur, but does not explain what changed in natural language. Existing remote sensing change captioning datasets typically describe overall image-level differences, leaving fine-grained localized semantic reasoning largely unexplored. To close this gap, we present RSRCC, a new benchmark for remote sensing change question-answering containing 126k questions, split into 87𝑘 training, 17.1𝑘 validation, and 22𝑘 test instances. Unlike prior datasets, RSRCC is built around localized, change-specific questions that require reasoning about a particular semantic change. To the best of our knowledge, this is the first remote sensing change question-answering benchmark designed explicitly for such fine-grained reasoning-based supervision. To construct RSRCC, we introduce a hierarchical semi-supervised curation pipeline that uses Best- of-𝑁 ranking as a critical final ambiguity-resolution stage. First, candidate change regions are extracted from semantic segmentation masks, then initially screened using an image-text embedding model, and finally validated through retrieval-augmented vision-language curation with Best-of-𝑁 ranking. This process enables scalable filtering of noisy and ambiguous candidates while preserving semantically meaningful changes. The dataset is available at https://huggingface.co/datasets/google/RSRCC.

Point cloud registration remains challenging in scenarios with low overlap, repetitive structures, and local geometric ambiguities, where candidate correspondences established based on local geometric similarity often lack global structural consistency and are prone to unreliable matches. To address this issue, this paper proposes RSPReg, a reliability-aware structural prototype learning framework for robust point cloud registration. RSPReg learns structural prototypes from rotation-invariant coarse-level superpoint features to characterize the structural attributes of local regions within the entire point cloud. These prototypes are further introduced as structural priors into cross-cloud correspondence reasoning, thereby improving the structural consistency of candidate correspondences. Furthermore, RSPReg estimates the uncertainty of structural prototypes and adaptively modulates the role of structural priors in the matching and pose estimation processes, reducing the negative impact of ambiguous and non-overlapping regions on registration results. Experiments on 3DMatch, 3DLoMatch, KITTI, and ETH demonstrate that RSPReg achieves the highest registration recall and exhibits clear advantages in low-overlap and cross-domain scenarios.

Vision-Language Models (VLMs) have achieved remarkable advances in general autonomous driving. Nevertheless, effectively exploiting the reasoning capability of Chain-of-Thought (CoT) to enhance the robustness and accuracy of decision and planning in long-tail scenarios still remains a challenging open problem. To this end, we propose a \textbf{R}easoning-\textbf{S}upervised \textbf{P}olicy \textbf{O}ptimization (RSPO) algorithm to boost reasoning and decision-making performance under long-tail driving scenarios. Concretely, we first conduct an empirical analysis of the core challenges faced by current VLMs in autonomous driving decision and planning tasks, and formally define reasoning and decision-making tasks tailored to long-tail scenarios. On this basis, we design a curriculum-guided supervised fine-tuning strategy to enable the model to rapidly fit the mean distribution of model outputs. Furthermore, we propose a reasoning-supervised optimization algorithm, which enhances the robustness and accuracy of reasoning and decision in long-tail scenarios by guiding the optimality consistency between reasoning processes and final decisions. Finally, we construct multiple verifiable autonomous driving reasoning and decision datasets based on open-source benchmarks and conduct extensive multi-dimensional comparative experiments. Experimental results show that the proposed algorithm achieves consistent improvements in prediction accuracy across multiple datasets compared with existing methods. In particular, it achieves a 6.7\% relative improvement over the strong baseline on the CODA-LM task and also yields lower trajectory-prediction errors on downstream planning tasks.

Vision Transformers (ViTs) achieve strong performance on image recognition and mobile vision applications, but their high-dimensional linear projections and attention computations still impose substantial storage and inference costs. Extremely low-bit quantization is a promising solution, yet ViTs often suffer severe accuracy degradation because conventional real-valued scalar codebooks are poorly matched to the directional geometry of Transformer projections. We present RPFQ-ViT, a Rotated Phase-Frame Quantization method that quantizes paired channels in two-dimensional phase planes, enabling low-bit codes to better preserve projection directions while recovering magnitude with lightweight scaling. RPFQ-ViT serves as a drop-in QAT replacement for \texttt{nn.Linear} and does not modify the standard real-valued attention, normalization, or activation computation graph. On ImageNet-1K, RPFQ-ViT-B/16 reaches 79.33\% Top-1 / 94.48\% Top-5 under W2/A4, Swin-T reaches 79.30\% Top-1 / 94.79\% Top-5 under W2/A8, and DeiT-S reaches 77.41\% Top-1 / 93.11\% Top-5 under W2/A8. Ablations, phase-geometry analysis, and direction-preservation metrics show that channel pairing, learnable rotation, phase-anchor learning, and residual phase refinement each improve quantization quality. We further deploy RPFQ-ViT image-classification models on native iOS and Android runtime stacks; with 2-bit packed weights, model size shrinks by roughly $5.4$--$7.1\times$ relative to FP32 and end-to-end on-device latency drops by $1.4$--$1.6\times$. All ImageNet results trained in our codebase use a matched 300-epoch recipe and are reported as mean accuracies over three independent runs. These results show that RPFQ-ViT provides a favorable trade-off among accuracy, compression, and practical mobile deployment for extremely low-bit ViTs.

Reinforcement learning is crucial for improving large language models’ reasoning and generalization. It relies on massive rollouts whose lengths become increasingly long-tailed as context windows grow. In on-policy training, these long-tail rollouts can result in GPU bubbles, reducing system utilization and limiting RL scalability. Asynchronous or partial-rollout methods improve throughput by relaxing synchronization, but inevitably introduce stale off-policy samples (trajectories) that may hurt final accuracy. Existing approaches mainly mitigate this off-policy issue by reweighting off-policy samples during training, yet they can still leave a performance gap compared to fully on-policy training. In this work, rather than passively reweighting samples during training, we propose RollVerify, a lightweight RL framework built on partial rollout that actively verifies and repairs samples before they enter training. Specifically, it introduces an off-policy shift metric OPS, to quantify the off-policy deviation of partially generated trajectories. Guided by the OPS constraint, RollVerify performs both sequence-level and token-level verification to identify and truncate invalid suffixes of trajectories. This yields high-quality samples that protect the models’ accuracy while preserving the efficiency gains of partial rollout. Experiments across model scales, architectures, and task domains show that RollVerify matches the performance of on-policy training while significantly improving training efficiency.

Vision-Language-Action (VLA) models enable instruction-following robotic manipulation, but they are typically pretrained on 2D data and lack 3D spatial understanding. An effective approach is representation alignment, where a strong vision foundation model is used to guide a 2D VLA model. However, existing methods usually apply supervision at only a single layer, failing to fully exploit the rich information distributed across depth; meanwhile, naïve multi-layer alignment can cause gradient interference. We introduce ROCKET, a residual-oriented multi-layer representation alignment framework that formulates multi-layer alignment as aligning one residual stream to another. Concretely, ROCKET employs a shared projector to align multiple layers of the VLA backbone with multiple layers of a powerful 3D vision foundation model via a layer-invariant mapping, which reduces gradient conflicts. We provide both theoretical justification and empirical analyses showing that a shared projector is sufficient and outperforms prior designs, and further propose a Matryoshka-style sparse activation scheme for the shared projector to balance multiple alignment losses. Our experiments show that, combined with a training-free layer selection strategy, ROCKET requires \textit{only about 4% of the compute budget while achieving 98.5% state-of-the-art success rate on LIBERO}. We further demonstrate the superior performance of ROCKET across LIBERO-Plus, RoboTwin, multiple VLA models, and a real-world industrial warehouse picking task. The code and model weights will be released upon acceptance of the paper.

We present $\textbf{ROCKET}$, a training-free model compression method that achieves state-of-the-art performance in comparison with factorization, structured-sparsification and dynamic compression baselines. Operating under a global compression budget, ROCKET comprises two key innovations: First, it formulates layer-wise compression allocation as a multi-choice knapsack problem, selecting the optimal compression level for each layer to minimize total reconstruction error while adhering to a target model size. Second, it introduces a single-step sparse matrix factorization inspired by dictionary learning: using only a small calibration set, it sparsifies weight coefficients based on activation-weights sensitivity and then updates the dictionary in closed form via least squares bypassing iterative optimization, sparse coding, or backpropagation entirely. $\textbf{ROCKET}$ consistently outperforms existing compression approaches across different model architectures at 20–50\% compression rates. Notably, it retains over 90\% of the original model’s performance at 30\% compression without any fine-tuning. We will release the code implementing ROCKET upon paper acceptance.

Driver monitoring concerns determining not only what a driver is doing, but whether the behavior violates a driver-behavior safety rule under the current driving context. Existing distracted-driving benchmarks largely reduce this problem to closed-set action recognition, often missing cases where the same behavior changes meaning with road scene, vehicle state, duration, and rule-specific exceptions. We present **ROAD**, a rule-grounded context-aware open-world framework for driver anomaly detection. ROAD represents driver-monitoring constraints as structured rule cards and jointly reasons over in-cabin driver evidence, road-context video, optional vehicle-state metadata, and open-world semantic cues to produce structured anomaly judgments: driver action, violated rule, anomaly score, supporting evidence, exception status, and explanation. Unlike generic MLLM prompting, ROAD separates perception, rule grounding, and semantic generalization. Road context modulates driver-centric evidence, rule-card encodings select rule-relevant evidence, and a LoRA-adapted MLLM extracts open-vocabulary safety cues from compressed video evidence. This design targets two forms of open-world generalization: recognizing unseen driver behaviors beyond a fixed action taxonomy and adapting to new or revised rule cards. We further introduce ROAD-Bench, a rule-grounded benchmark with context, exception, evidence, anomaly-score, and explanation annotations. With a three-stage curriculum from action primitives to context-rule alignment and anomaly adaptation, ROAD outperforms evaluated baselines across driver-distraction recognition, open-world behavior recognition, and rule-grounded anomaly detection, using an 8B multimodal backbone. It achieves 92.01\% average accuracy over seven driver-recognition benchmarks, 77.9\% AUROC and 46.9\% unseen-class F1 in behavior recognition, and 73.6\% AUROC, 68.3\% rule accuracy, and 54.5\% Expl.-Rule F1 on ROAD-Bench.

Online reinforcement learning (RL) has achieved significant success in decision-making tasks, but modern online RL remains highly sensitive to the quality of training experience. Understanding the role of training data in online RL is challenging because the training distribution evolves together with the policy, causing the influence of the training data to propagate across future optimization and future data collection. Existing data-attribution methods for online RL primarily focus on local influence within a single training round, overlooking the non-local data influence across multiple training rounds. In this work, we formalize data attribution in online RL through a trajectory-level leave-one-out quantity, RL-LOO, which measures how one training sample influences a downstream target after subsequent online updates have taken place. We then derive RL-Inf, a first-order estimator that propagates data influence through both optimization effects and policy-induced sampling effects, and provide theoretical guarantees on its approximation accuracy under smoothness assumptions. Our analysis further disentangles these two effects and shows both theoretically and empirically that the full RL-Inf estimator can often be well approximated using the optimization effect alone. Building on this observation, we develop RINSE, a practical data filtering method for online RL. Experiments on standard control benchmarks and an RLHF-style toxicity-mitigation setting show that RINSE improves training efficiency and final performance over standard PPO and local attribution baselines. These results suggest that RL-Inf provides a practical and principled framework for understanding and improving online RL.

Time-aware retrieval is crucial for long-horizon LLM agents to effectively leverage ever-growing long-term memory. Existing methods focus on lexical or semantic similarity, leading to target memories being crowded out by temporally mismatched yet semantically similar candidates. Even time-aware approaches inject temporal cues only during memory organization or answer generation, overlooking temporal constraints at the retrieval stage, where candidate selection occurs—the root cause of the displacement. Inspired by the temporal contiguity effect in human episodic memory, we propose TIDER (Temporal Interval-driven Dual-channel Evidence Retrieval), a time-aware agentic retrieval framework. Central to TIDER is a reinforcement learning (RL)-trained temporal interval locator that infers the temporal windows in which the evidence lies. RL optimizes window localization as a discrete, non-differentiable retrieval decision, using composite rewards to convert sparse answer supervision into fine-grained retrieval feedback. These temporal windows first define the temporal retrieval channel; the retrieved candidates are then merged with results from the semantic channel and reranked by proximity to the windows. This temporal grounding enables TIDER to retrieve temporally aligned evidence that would otherwise be drowned out by the volume of semantically similar yet temporally mismatched memories. With a lightweight 1.7B locator, TIDER achieves state-of-the-art accuracy on both long-horizon memory benchmarks—61.52% on LifeBench (+8.82%) and 78.96% on LoCoMo (+3.66%), which indicates that temporal cues are crucial for pinpointing target evidence amid a vast pool of semantically similar historical events in long-term daily-life memory.

Verifying that two quantum circuits are equivalent is a critical challenge in compiler validation. A common approach is to transform the circuits into a tensor network and then iteratively contract tensors in the network. The order in which contractions are applied has a crucial influence on the complexity, as tensor size may grow exponentially in the number of qubits. This motivated research into heuristic policies to find good contraction orders, including recent work based on reinforcement learning (RL). Orthogonally, to mitigate the complexity of an explicit tensor representation, tensor decision diagrams (TDDs) have been proposed as a compact symbolic representation. However, also the TDD performance is sensitive to the contraction order, and policies designed for tensor networks do not perform well in case of a TDD representation. In this paper, we propose an RL framework to obtain a graph-neural-network policy that navigates the trade-off between arithmetic complexity and TDD size. Experimental results demonstrate that a policy learned with our framework is effective at taming the symbolic representation size.

The standard LLM training pipeline applies reinforcement learning (RL) only after pre-training and supervised fine-tuning (SFT). We question this status quo by training a LLM from scratch and applying RL, SFT, and SFT followed by RL directly to intermediate pre-training checkpoints. We find that RL is effective very early, and often matches the full SFT$\to$RL pipeline early as well. Through experiments on harder problems, we find that targeted pre-training data composition is a strong lever for RL effectiveness, even more so than model scale. Beyond reasoning accuracy, applying RL directly to base checkpoints expands the model's distribution; the sharpening effect reported in recent work arises only when RL follows SFT. The general capabilities of the model remain essentially unchanged by RL, while they degrade following SFT. Finally, we merge RL and SFT objectives by \textit{parallel averaging}, which outperforms across all other training methods discussed, across metrics, while preserving general capabilities. Together, these results suggest that LLM training might benefit from an expanded use of RL.

Large language models are increasingly deployed as long-lived agents that must adapt across users, tasks, domains, modalities, and feedback regimes without access to model weights. Existing black-box adaptation methods typically optimize a single prompt, maintain an undifferentiated memory, or rely on repeated rollout-heavy search. However, these designs struggle when streams of input are nonstationary, feedback is sparse, and failures from one task family can contaminate behavior on another. We introduce RIZZ (**R**outing **I**nteractions to Near **Z**ero-interference **Z**ones), a continual adaptation framework for compound language-model systems that learns entirely through verifier-gated memory, routing, and prompt compilation. RIZZ organizes input streams into dynamically spawned memory branches. At inference time, either while online or offline, a context-aware router selects or creates a branch that retrieves branch-local, global, graph-structured, and working-memory context, which is compiled into a bounded prompt together with retrieved task evidence. After the model acts, task verifiers score the output, and only verified interactions can update memory, promote reusable rules, demote harmful rules, or create anti-patterns. This yields a black-box agent that improves through persistent natural-language feedback while explicitly controlling interference. RIZZ targets the regime where adaptation must occur online under context budgets. Finally, we demonstrate the effectiveness of our framework against state-of-the-art baselines on competitive benchmarks.

Interactive regex search over large text columns needs exact matches under low latency, but indexes help only when a pattern exposes selective fragments. For LIMIT-k queries, the immediate bottleneck is choosing a few indexable keys that reach match-rich candidate buckets before regex verification. RIVET casts this step as constrained regex-to-key generation: a column-specific translator emits short keys, an existing index retrieves candidates, and the native regex engine verifies every output record. Training from repeated regex-key hits moves probability mass toward effective keys, so finite online sampling spends fewer regex checks before collecting the requested matches. Across four large datasets, RIVET reaches 35.8 ms median latency, improves over the strongest baseline REI by 3.4–36×, and reaches up to 1773× speedup over sequential scan. The same mechanism improves TPC-H query plans, preview retrieval under latency budgets, and 80 human-authored OOD regex queries, showing that regex-to-key generation translates into end-to-end system gains.

On modern production text-to-image systems, successful policy violations are rare, and previously effective human-written seeds are often patched out. Current automated red-teamers are poorly matched to this regime in two ways: unreliable success measurement and poor exploration. First, we find that judges widely used in prior T2I red-teaming work are unreliable under vague unsafe-content targets: they either miss true violations or reward benign borderline images on hardened APIs. We therefore define strict category-specific success criteria and calibrate strong VLM judges against human labels. Second, we show that broadly used prompt-modification pipelines do not solve the exploration problem: on harder guardrail settings they remain tied to seed prompts, fail to transfer, or cannot bootstrap positive examples. We introduce RISE, which evolves reusable strategies used to generate prompts rather than rewriting them one by one. The best discovered strategies are then reused to generate attacks across new scenarios. On DALL·E 3, Nano Banana 2 (Google) and GPT-Image-2, RISE reaches up to 13\% human-verified ASR; under the same calibrated evaluation, prior methods with reported ASR as high as roughly 30\% fall to near zero.

Millimeter-wave radar is promising for privacy-preserving human sensing, yet current radar-to-language pipelines typically map sparse point clouds into either global clip features or discrete token codes that are only weakly matched to the structure expected by large language models (LLMs). We present RILA, a radar-native structured interface layer for large-language-model adaptation, which converts sparse mmWave radar point clouds into LLM-readable event-aware tokens. RILA combines kinematic phase-space tokenization, flow-aware dual-order serialization, dual-scan selective state-space encoding, and an event-aware token abstraction that exposes both global clip context and temporally localized motion units to the language model. The same interface supports three language tasks through a shared decoder: clip summary generation, ordered event description, and temporal question answering. To stabilize the interface before instruction tuning, we introduce a structured interface-language alignment objective over clip and event tokens. We evaluate RILA using physics-aware synthetic pretraining and real-world mmWave adaptation under controlled radar-native baselines and matched decoder settings. This formulation reframes radar-to-LLM modeling as an interface design problem and studies how radar-native structure, rather than only generic cross-modal projection, determines whether sparse mmWave point clouds can be effectively understood by LLMs.

LLM-based multi-agent systems (MAS) improve complex reasoning through collaborative decomposition, verification, and synthesis, but their performance can degrade sharply when compromised agents inject misleading intermediate messages. Existing topology methods rely on fixed communication patterns or query-adaptive graphs, but rarely infer or intervene runtime reliability risk, leaving them fragile when compromised agents vary across queries. We study stochastic adversarial topology adaptation, where compromised agents shift across graph positions and rare cascades dominate the high-loss tail. To address this setting, we propose RIGOR, a topology-adaptation framework that couples runtime risk inference with tail-aware training. RIGOR combines Risk-Gated Topology Construction (R-GTC), which probes agent behavior, estimates per-agent risk, and edits the graph through soft quarantine and edge masking, with CVaR-Guided Robust Optimization (C-GRO), which trains the editor on high-loss episodes rather than average gain alone. Across six benchmarks and two backbones, RIGOR consistently outperforms existing MAS baselines under query-varying misleading-message attacks, achieving an average 21.09\% accuracy improvement over the strongest baseline on Qwen3.5-35B-A3B. Tail-subset and position-wise analyses further show that these gains stem from improved robustness against worst-case cascades and dynamically changing compromised-agent positions. The code is available for anonymous access at https://anonymous.4open.science/r/RIGOR-0D7D.