NeurIPS 2026 Papers

Generative models such as Diffusion Models and Flow Matching have improved multimodal trajectory planning in autonomous driving, but still suffer from inference latency and the lack of tractable probability densities, relying on heuristic trajectory selection. We propose BiDrive, a real-time end-to-end planning framework based on Normalizing Flows (NFs). By leveraging invertibility and exact likelihood estimation, BiDrive directly evaluates trajectory probabilities in a single forward pass, enabling principled maximum-likelihood-based decision making. To address the autoregressive bottleneck of expressive flows, we further develop a bidirectional distillation framework that compresses an autoregressive flow into a one-pass generator, achieving real-time inference (50 FPS) while preserving modeling capacity. The autoregressive model is used only during training as a probabilistic teacher. In addition, we introduce a latent-space optimization mechanism that incorporates differentiable safety and dynamic constraints for efficient test-time trajectory refinement. Experiments on NAVSIM closed-loop benchmarks demonstrate state-of-the-art performance in both safety and efficiency, highlighting the potential of Normalizing Flows for real-time trajectory planning. Code is available at: https://anonymous.4open.science/r/BiDrive.

Autoregressive video generation has emerged as a powerful paradigm for World Action Models (WAMs). However, existing approaches suffer from slow training convergence particularly at high frame rates, limited converged accuracy, and slow inference due to iterative video denoising, as the training supervision is confined to the current chunk without explicit signals about future dynamics. In this paper, we present Next Forcing, a multi-chunk prediction (MCP) framework for causal world modeling that enables faster training, higher accuracy, and accelerated inference. Inspired by multi-token prediction in large language models, Next Forcing introduces an MCP training objective that augments the main model with lightweight auxiliary MCP modules to simultaneously denoise video chunks at multiple future temporal horizons (next$^1$, next$^2$, next$^3$ chunks). These MCP modules form a causal chain across prediction depths, where intermediate features fused from multiple layers of the main model are leveraged to predict future dynamics, allowing near-future predictions to inform farther-future ones and providing dense multi-scale temporal supervision back to the main model. During training, the MCP modules significantly accelerate convergence and improve converged accuracy, especially at high frame rates. At 50 fps, our method achieves a 93.1\% relative improvement over the baseline LingBot-VA at 5k training steps and achieves 2.3$\times$ faster convergence. At inference, the MCP modules can be retained to predict the next video chunk in parallel with the current one, accelerating generation. Next Forcing establishes new state-of-the-art results on the RoboTwin benchmark (94.1\%/93.5\% on Clean/Random) and demonstrates significant improvements on PhyWorld, a benchmark evaluating adherence to physical laws in video generation.

Task-oriented dexterous grasp generation aims to produce dexterous grasp poses that are both physically plausible and functionally suitable for specified manipulation tasks. Existing diffusion-based methods often address these two requirements in a decoupled manner: they first train a grasp diffusion model for task alignment and then rely on post-generation refinement to improve physical plausibility. However, this after-the-fact correction strategy applies physical plausibility guidance only once the grasp has already been generated, leaving the generation trajectory itself unguided by physical constraints and potentially leading to suboptimal grasps. To address this problem, we propose a novel framework that directly injects physical plausibility guidance into the denoising process of a task-aligned grasp diffusion model in a practical and effective manner, even when physical plausibility constraints are non-differentiable. This allows physical plausibility to shape grasp generation throughout denoising while preserving task alignment. Extensive experiments demonstrate the efficacy of our framework.

Online decision-making faces a tension between efficient inference and long-horizon reasoning. Value-based methods rely on short-horizon bootstrapping that limits the propagation of long-term information, while trajectory optimization methods plan explicitly but must solve a high-dimensional optimization problem at every decision step. We propose Generative Trajectory Planning (GTP), a model-based framework that addresses this tension by learning a generative model as a proposal distribution over action sequences. Instead of optimizing sequences from scratch, GTP samples candidate sequences from a diffusion prior and refines them using learned dynamics, reward, and value models, enabling efficient trajectory-level reasoning. Across continuous control and long-horizon manipulation tasks, GTP matches or exceeds state-of-the-art model-free, diffusion-policy, and model-based baselines, and remains stable in regimes where strong baselines degrade. Our results suggest that generative models are best understood not as policies, but as structured proposal distributions for planning, offering a new perspective on how to integrate generative modeling with model-based decision-making.

Autoregressive generation is natural for language, where predicted tokens can be directly reused as the next prediction state, but trajectory forecasting lacks such a clean token: motion is continuous, multimodal, and expressed in local coordinate frames that evolve with the predicted trajectory. We present $\textbf{MoCAR}$ ($\textbf{Mo}$tion-code $\textbf{C}$oordinate-aware $\textbf{A}$uto$\textbf{R}$egression), a decoder-only framework that casts trajectory forecasting as next-code prediction in a coordinate-aware continuous latent space. MoCAR learns a continuous motion-code space for endpoint-normalized trajectory segments, where each code jointly captures local trajectory geometry and the reference-frame transition induced by that segment. Historical motion codes are used as a teacher-forced prefix, future codes are generated autoregressively under temporal, map, agent, and mode interactions, and predicted codes persist in latent memory while decoded endpoints update the local scene context. This enables rollout without trajectory-space re-tokenization, trajectory queries, goal candidates, or proposal-and-refinement pipelines. On Argoverse (AV) benchmarks, MoCAR achieves top-tier performance with a simple single-stage architecture, transfers strongly from AV2 to AV1 in zero-shot evaluation, and improves on turn-heavy scenarios. Ablations confirm that the learned continuous motion-code space, latent alignment, weak KL regularization, and joint tokenizer-predictor optimization are essential for stable latent autoregression.

Neural trajectory predictors are increasingly used in physical-system pipelines for reconstruction, simulation, forecasting, and downstream analysis. In these settings, low prediction error alone is insufficient. A trajectory may remain statistically plausible while violating dynamics, actuator limits, or state constraints, making it unreliable for physical interpretation or control-aware reasoning. This problem is especially difficult when controls are unobserved and the available dynamics model is only partially specified. We introduce the Markovian Dynamics Enforcer (MaDE), a framework for post-hoc feasibility enforcement under learned controlled dynamics. MaDE acts as a time-invariant correction operator that maps state-transition proposals onto a learned feasible dynamics manifold. For each transition, it infers latent controls through inverse dynamics, recomputes state evolution using a known-physics model augmented with a learned residual, and corrects controls through gradient-based inequality reduction while re-integrating the dynamics after each correction step. MaDE is trained from feasible state observations without ground-truth controls and is designed to operate as a frozen downstream layer for arbitrary trajectory predictors. We evaluate MaDE on simulated controlled systems spanning fully specified and underspecified dynamics, with deterministic bound-violation and Gaussian observation-noise stress tests. MaDE is the only method that drives known-, learned-, and true-dynamics residuals to essentially zero across the fully specified systems. In the underspecified dynamic-bicycle setting, it reduces true-dynamics residual from 1.40--2.90 for the baselines to 0.33, while reducing trajectory fidelity error by more than 60% relative to the next-best method. These results show that MaDE enforces model-relative physical feasibility while preserving proximity to the original trajectory.

Embodied agents are often evaluated in the same environment in which they were explored or trained, making it difficult to assess whether a learned world model supports planning after the world changes. Existing evaluations can conflate memory of the explored environment, belief update after change, and planning under the updated belief. We introduce MapShift, an executable benchmark for controlled post-intervention evaluation (CPE): an agent explores a base environment without a task reward; the environment is modified by a controlled intervention in the metric, topology, dynamics, or semantics; and the agent is evaluated on post-intervention planning, inference, and adaptation tasks. The contribution is measurement infrastructure: matched base/intervened pairs, family-wise estimands, severity ladders, invariant validators, benchmark-health gates, protocol-comparison tooling, and reproducible artifact generation. In the expanded 24-motif release, health gates pass with zero fatal leakage, zero task rejections, perfect reference solvability, no intervention-validator failures, and no severity-magnitude failures. A deterministic mechanism diagnostic shows that same-environment evaluation underestimates the belief-update advantage by 3x on topology shifts, from $\Delta=0.304$ under CPE versus $0.102$ same-environment, and by 7x on semantic shifts, from $\Delta=0.724$ versus $0.102$; planning slices also exhibit protocol-induced rank reversals.

Large language models (LLMs) have shown strong reasoning capabilities in embodied planning tasks. However, they often fail to achieve good results on long-horizon and multi-robot tasks under strict physical constraints, and generate smooth but infeasible trajectories. In real-world robotic hardware deployment, systems also face two fundamental challenges: First, the fine-tuning method for open-source models, Direct Preference Optimization (DPO), typically only makes a binary true or false judgment on the planning results, without distinguishing the severity of the planning error. As a result, they fail to impose effective penalties for catastrophic violations of physical constraints. Second, methods that search over multiple candidate plans during inference do not filter out invalid plans in advance, often leading to combinatorial explosion and high inference latency, which makes them difficult to apply to real-time robotic operations. To address these limitations, we propose a complete training-to-inference architecture consisting of two modules. First, we introduce Margin-Aware Direct Preference Optimization (mDPO) during training, which distinguishes the severity of errors in negative samples, enabling the fine-tuned model to avoid generating severe errors that would render a task irreparable. Second, we deploy Verifier-Guided Search (VGS) during inference, using a symbolic verifier to hard-prune invalid search paths, thereby repairing infeasible task plans and reducing inference latency. In addition, latency analysis shows that our approach reduces the average end-to-end runtime from 12.814s to 10.969s compared with retry-based repair, while maintaining higher final success, making it more suitable for time-sensitive robotic deployment.

Robot policy pretraining based on human videos is crucial for improving the policy’s generalization ability. One main challenge for this task is the lack of explicit action-relevant representations in such unlabeled data. Recent works tend to estimate 3D hand motion trajectories from videos using 3D foundation models (3D FMs). However, they keep solely the sparse trajectories as supervision, ignoring the rich, fine-grained information produced during the trajectory extraction process. To address this issue, we propose $\textbf{LSA}$, a hierarchical robot policy pretraining framework that includes three phases: $\textbf{L}$ift, $\textbf{S}$ee, and $\textbf{A}$ct. Specifically, the Lift phase elevates 2D video observations into dense 3D representations by adaptively aligning features in the policy's shallow layers with the intermediate representations extracted from multiple 3D FMs. Building upon these lifted representations, the See phase equips the policy with explicit geometric and interactive awareness through dual guidance. It introduces depth-map reconstruction to help the model comprehend 3D spatial layouts and utilizes hand region cues to explicitly supervise the encoder’s attention toward task-relevant interaction hotspots. Empowered by the dense 3D representation learning and precise spatial guidance, our policy achieves robust and accurate robotic manipulation in the Act phase. Extensive experiments on diverse simulation and real-world tasks demonstrate that LSA significantly outperforms current state-of-the-art approaches. The code of this work will be released soon.

Flow-based vision-language-action (VLA) models provide expressive action generation for embodied control, but supervised fine-tuning (SFT) often confines them to narrow expert behaviors. Online reinforcement learning (RL) can improve beyond demonstrations through environment interaction, yet flow-based VLAs struggle with sparse long-horizon rewards and intractable action likelihoods. We propose \textbf{\emph{LiFT}}, a likelihood-free tree policy optimization framework for flow-based VLAs. LiFT first addresses sparse-reward credit assignment by creating sibling continuations from shared rollout histories and using their final outcomes to compute branch level relative advantages. LiFT further uses local action-flow instability as an expansion criterion, focusing rollout budget on ambiguous action-generation regions that are most likely to reveal meaningful outcome differences. Finally, LiFT applies branch-level advantages through a decision-level surrogate ratio on executed action chunks, matching policy updates to the granularity of tree-based credit assignment. This yields a likelihood-free policy update without attributing chunk-level credit to individual denoising steps. Evaluations on in-distribution and out-of-distribution benchmarks show that LiFT improves task success and robustness to distribution shifts.

Vision-language-action models (VLA) promise human-steerable autonomous driving, but their training is bottlenecked by the scarcity of frames paired with natural-language instructions: while camera streams and expert trajectories are logged at scale, language annotations (e.g., "turn left at the intersection") remain scarce and expensive to acquire. To address this challenge, we introduce Latent Action Driving Annotations (LADA), a three-stage pipeline that transforms abundant unlabelled observation–trajectory pairs into a substrate for language-conditioned control. First, we train a latent action model with a vector-quantised bottleneck, producing a compact codebook of high-level vehicle intents. Second, a small language-annotated subset is used to train a vision-language translator to map observations and language instructions into this codebook. Third, we train a driving VLA on observation–latent-action pairs over the full unlabelled corpus. Using fewer than 5% of language annotations and without leveraging any auxiliary chain-of-thought reasoning or visual question answering streams, LADA achieves a Driving Score of 87.98 and a Success Rate of 70.46% on the closed-loop Bench2Drive benchmark, matching or surpassing fully supervised baselines.

Human videos contain rich manipulation priors, but using them for robot learning remains difficult because raw observations entangle scene understanding, human motion, and embodiment-specific action. We introduce MoT-HRA, a hierarchical vision-language-action framework that learns human-intention priors from large-scale human demonstrations. We first curate HA-2.2M, a 2.2M-episode action-language dataset reconstructed from heterogeneous human videos through hand-centric filtering, spatial reconstruction, temporal segmentation, and language alignment. On top of this dataset, MoT-HRA factorizes manipulation into three coupled experts: a vision-language expert predicts an embodiment-agnostic 3D trajectory, an intention expert models MANO-style hand motion as a latent human-motion prior, and a fine expert maps the intention-aware representation to robot action chunks. A shared-attention trunk and read-only key-value transfer allow downstream control to use human priors while limiting interference with upstream representations. Experiments on hand motion generation, simulated manipulation, and real-world robot tasks show that MoT-HRA improves motion plausibility and robust control under distribution shift.

Reinforcement learning has achieved remarkable success in robot learning. However, under challenging exploration and contact-rich dynamics, early-stage training is frequently dominated by premature terminations such as collisions and falls. As a result, learning is overwhelmed by short-horizon, low-return trajectories, which hinder convergence and limit long-horizon exploration. To alleviate this issue, we propose a technique called Failure Episodic Memory Alert (FEMA). FEMA explicitly stores short-horizon failure experiences through an episodic memory module. During interactions, it retrieves similar failure experiences and prevents the robot from recurrently relapsing into unstable states, guiding the policy toward long-horizon trajectories with greater long-term value. FEMA can be combined easily with model-free reinforcement learning algorithms, and yields a substantial sample-efficiency improvement of 33.11\% on MuJoCo tasks across several classical RL algorithms. Furthermore, integrating FEMA into a parallelized PPO training pipeline demonstrates its effectiveness on a real-world bipedal robot task.

Mobile sensing for real-time anomaly detection couples routing, sampling, and stopping: dispatch determines which locations are observed, while observations update the evidence used for future dispatch. We study this problem for a fleet of unmanned aerial vehicles (UAVs), where each UAV observes only its current location and anomalies may occur at unknown locations and times. The goal is to detect anomalies quickly while controlling false alarms under decision-dependent partial observations, local mobility, and collision-avoidance constraints. Existing quickest-detection methods typically ignore mobility and collision constraints, whereas learning-based routing methods scale to large fleets but are not designed for statistically calibrated detection. We formulate route-wise monitoring as a partially observable Markov decision process (POMDP) and propose DispatchPPO, a deep reinforcement learning method for long-horizon detection-aware dispatch. Its policy architecture, DispatchNet, uses autoregressive decoding with dynamic feasibility masking to generate collision-free joint UAV moves without enumerating the exponential joint action space. We establish theoretical properties for persistent coverage, evidence-based focusing, and detection delay through a resource-allocation information-rate bound. Experiments show that DispatchPPO reduces detection delay relative to heuristic and optimization-based baselines, while transferring effectively to real-robot and wildfire monitoring case studies.

Behavior policies are often formulated as continuous generative models, whose iterative denoising processes are expressive but difficult to interpret and prone to producing implausible actions. We propose the Large Discrete Policy (LDiP), a fully discrete behavior modeling framework that selects actions from a large vocabulary of physically plausible candidates. Rather than perturbing actions, LDiP improves expressivity through stochastic iterative scoring: it progressively re-scores and prunes candidates with score-space stochasticity, enabling fine-grained ranking and exploration among plausible actions while preserving an explicit decision process. Across end-to-end planning, closed-loop driving, robotic manipulation, and vision-language-action settings, LDiP consistently outperforms strong discrete and continuous baselines in autonomous driving, and exceeds or matches continuous generative policies in robotic manipulation. These results show that discrete policies, when equipped with effective scoring mechanisms, offer an expressive, plausible, and interpretable alternative for behavior modeling.

We investigate the automated generation of executable robot reinforcement learning (RL) policies from natural-language task descriptions. Rather than using large language models (LLMs) as execution-time decision makers, we use them as designers that construct RL pipelines before deployment, including Markov decision process (MDP) formulation and simulator implementation. We identify a phase-wise structure in this problem: MDP formulation is dependency-structured and sequential, whereas simulator implementation becomes modular once the formulation is fixed. Based on this structure, we propose ARPG, a dependency-aware agentic framework that performs sequential MDP modeling, localized verification and correction, and simulator-structured implementation while externalizing intermediate artifacts. ARPG targets a key failure mode of automated policy generation: executable code may still yield non-learnable policies when state, action, reward, and termination artifacts are not semantically aligned. Experiments in Isaac Sim show that ARPG improves modeling correctness, environment executability, and policy-generation success compared with monolithic LLM baselines and off-the-shelf agentic systems. Additional demonstrations on custom robotics, standard RL, and engineering-domain MDPs further suggest that the same modeling pipeline can be reused beyond the main benchmark suite.

World models enable prediction, planning, and decision-making by learning internal simulators of environment dynamics. Although substantial progress has been made in world models from pixel-based observations, many physical systems are instead observed through vector-based measurements that are noisy and only partially informative of the underlying state. We introduce K-PWM, a Koopman-structured probabilistic world model for prediction and control under partial observation. K-PWM combines a parameterized Koopman-structured state-space model with a nonlinear observation decoder, and performs probabilistic latent-state inference directly through the learned model parameters rather than through a separately trained inference network. We train K-PWM using a generalized expectation conditional maximization (GECM) procedure with principled initialization. Across Gymnasium MuJoCo tasks, K-PWM improves sample efficiency, yields reliable long-horizon predictions, and supports downstream control through model predictive path integral (MPPI) control. These results suggest that structured probabilistic latent dynamics provide a useful route toward data-efficient world modeling and control in partially observable vector-valued settings.

Articulated objects used in simulation and embodied AI are typically specified by geometry and kinematic structure, but lack the fine-grained dynamical effects that govern realistic mechanical behavior, such as frictional holding, detents, soft closing, and snap latching. Existing approaches either ignore the detailed structure of dynamics entirely, or use simple models with limited expressiveness. We introduce JODA, a framework for generating joint-level dynamics as a structured three-channel field over the joint degree of freedom, capturing conservative forces, dry friction, and damping. Instantiated using shape-constrained piecewise cubic interpolation (PCHIP), this formulation defines a compact and expressive function space that is both interpretable and compatible with differentiable simulation. Building on this representation, we develop methods for inferring and refining joint dynamics from multimodal inputs. Given visual observations and joint context, a vision-language model proposes structured dynamical primitives, which are composed into a unified dynamics field. The resulting representation supports both direct manipulation and gradient-based refinement. We demonstrate that JODA enables plausible and controllable modeling of diverse joint behaviors, providing a unified interface for inference, editing, and optimization.

Recent advances in robotics foundation models have shown promising multi-task generalization, yet their reliance on specialized embodiment-specific data and model fundamentally prevents them from exploiting the far richer physical interaction knowledge present in abundant human manipulation videos. Existing approaches that leverage human videos suffer from misaligned representations that preclude explicit unified interaction modeling over human video and robot demonstration, failing to capture unified physical interaction dynamics essential for cross-domain transfer. We address this by constructing a unified representation that aligns human videos and robot tasks through scene point clouds with hand-to-gripper mapping via spatial tracking and hand pose estimation. Leveraging this representation, we propose a structured graph transformer that explicitly models spatial, semantic, and intentional interaction entities through graph attention mechanisms. The modeled multi-level interaction features then enable interaction-aligned cross-domain learning, transferring manipulation knowledge from human videos to robot tasks via discrete interaction abstraction and hierarchical distribution alignment. Experiments on RLBench, ManiSkill2, and real-world robot demonstrate state-of-the-art performance on diverse benchmarks with fewer demonstrations and effectiveness of our designed modules. Remarkably, our method achieves zero-shot transfer on robot tasks when trained exclusively on human video, providing strong evidence for effective human-to-robot transfer through interaction alignment.