NeurIPS 2026 Papers

Multimodal large language models (MLLMs) have rapidly advanced machine intelligence by integrating visual perception with powerful linguistic reasoning. However existing visual backbones predominantly encode videos in a rigid per-frame manner and delegate temporal modeling entirely to the language model. This paradigm fundamentally constrains long-range temporal understanding and imposes severe computational inefficiencies. To resolve these bottlenecks we propose FuxViT, a unified Vision Transformer natively supporting a 64K-token context window to seamlessly process diverse inputs ranging from static images to hour-scale videos. By employing a joint spatiotemporal pretraining paradigm, FuxViT generates highly coherent and information-rich visual representations. This strategic early fusion frees the downstream language model from reconstructing low-level temporal structures to focus entirely on high-level reasoning. Furthermore we introduce Frame Selective Attention (FSA) as a native sparse routing mechanism. FSA dynamically attends to the most relevant historical frames to achieve exceptional computational efficiency without sacrificing fine-grained spatial or temporal details. Extensive experiments across 16 image and video benchmarks demonstrate that FuxViT consistently outperforms existing baselines and establishes itself as a vastly superior alternative to standard static ViTs for multimodal understanding. We will fully open-source all model checkpoints.

We study language generation in the limit under a global preference ordering on strings, as introduced by Kleinberg and Wei. We aim for breadth, but impose an additional requirement of timeliness: higher-ranked strings should be generated earlier. A string is then only credited if it is generated before a deadline, where its deadline is defined by a function that maps a string’s rank in the target language to the time by which it must be produced. This is in keeping with a central consideration in machine learning, where inductive bias favors "simpler" or "more plausible" outputs, all else being equal. We show that timely generation is impossible in a strong sense for eventually consistent generators—the protagonists of most prior related work. Under what is perhaps the mildest natural relaxation of consistency, a hallucination rate that vanishes over time, we show that we can circumvent our impossibility result. In particular, we can achieve optimal density with respect to any superlinear deadline function. We also show this is tight by ruling out timely generation with linear deadlines and vanishing hallucination rate.

Code assistants are increasingly utilized in test-driven software development, yet the theoretical mechanisms behind their environment-interaction strategies remain underexplored. We provide a probabilistic framework for two dominant paradigms: code selection after generation using the execution environment, and code generation conditioned on environment feedback. First, we formalize several well-established selection heuristics as environment-aware estimators of code correctness. We theoretically prove that estimators based on fuzzy functional similarity add an inductive bias and strictly dominate estimators based on functional equivalence in terms of signal-to-noise ratio. Second, we frame backprompting as an in-context approximation of Thompson sampling. We derive a novel regret bound for reward functions with unobservable components, theoretically explaining why the effectiveness of backprompting is limited by the ambiguity of the informal task description (an irreducible regret). Using five state-of-the-art open weight models, we corroborate these findings across BigCodeBenchHard, LeetCodeDataset, and QiskitHumanEvalSim. Our formalization also suggests how to improve task descriptions effectively, leading to a new benchmark, QiskitHumanEvalSimX.

Current world models operate at a single level of abstraction, with most prioritizing perceptual fidelity while lacking the spatial reasoning and semantic understanding required for real-world downstream tasks. We present a hierarchical driving world model that factorizes future prediction across two levels operating at distinct temporal and abstraction scales: a high-level predictor that forecasts coarse scene structure over extended temporal horizons, and a low-level generator that produces detailed predictions conditioned on the high-level output. This decomposition yields high perceptual fidelity while also capturing strong spatial and semantic representations. We further show that pretraining with a diffusion forcing objective yields substantially richer internal representations than the standard teacher forcing objective, while teacher forcing — predicting only the next frame from clean context — produces more stable autoregressive rollouts. We therefore introduce a generic two-stage training paradigm that pretrains the model with diffusion forcing and fine-tunes with teacher forcing, combining the representational benefits of the former with the rollout stability of the latter. Our approach achieves state-of-the-art results across the standard suite of driving world model evaluations on established benchmarks, including long-horizon generation fidelity, steering responsiveness evaluated on counterfactual scenarios, and internal representation quality.

Few-step diffusion sampling sharpens the tension between ODE and SDE dynamics. ODE samplers are stable but can lose diversity, while the standard SDE endpoint can become unreliable at low numbers of function evaluations (NFE). The challenge is therefore not simply whether stochasticity should be added, but which driver properties allow stochastic correction to contribute diffusion without losing finite-step control. We show that this tradeoff is governed by two driver-level quantities, the asymptotic covariance $Q$, which sets the effective diffusion level on the ODE-to-SDE spectrum, and the Poisson solution norm $C_\chi$, which controls the finite-step remainder. Gaussian drivers have $C_\chi = \infty$, so this finite-step remainder is not uniformly controlled by the same bound. Balancing the diffusion level set by $Q$ with the finite-step control set by $C_\chi$ formalizes a favorable regime for bounded structured drivers with intermediate diffusion strength. This regime directly gives a structured correction method, instantiated as Fast-Driver Sampler (FDS) with the shuffled-Lorenz driver. Across DiT-XL/2 and EDM2, FDS establishes a stronger low-NFE frontier among evaluated training-free samplers, improving ODE baselines while avoiding the degradation of the SDE endpoint.

Dynamic Scene Graph Generation (DSGG) aims to capture objects and their evolving relations in videos. Despite recent progress, the practicality and quality of generated scene graphs remain limited compared to the rapid advances in Multimodal Large Language Models (MLLMs). In this work, we revisit DSGG from two fundamental perspectives: task setup and model design. From the task setup perspective, we identify two key limitations of the current recall-oriented evaluation protocol: (i) a severe precision–recall trade-off, and (ii) uninformative and redundant relation generation. To better assess practical usefulness, we introduce four additional metrics that provide a more comprehensive evaluation of the quality of generated dynamic scene graphs. From the model design perspective, we explore directly using MLLMs for scene graph generation and establish a strong MLLM-based DSGG baseline through three design changes. First, we replace the conventional bottom-up pipeline with a top-down reason-then-locate strategy. Second, we reformulate frame-wise dynamic graphs as Temporal Relation Set (TRS) prediction, improving both efficiency and performance. Third, we introduce Importance-Aware Finetuning (IAF) to encourage more relevant and diverse relation generation. Extensive experiments on Action Genome, VidVRD, and PVSG show that our approach consistently achieves state-of-the-art performance across multiple metrics.

We present 3D consistency tokens, a compact set of 3D anchors that ground cross-view correspondence in explicit geometry. While feedforward Gaussian models amortize per-scene optimization into a single forward pass, their rendering quality still falls short of optimization-based pipelines. A primary cause lies in the relatively inaccurate correspondences they recover by comparing each image token to others across views based on appearance and semantic cues, which can be ambiguous in repetitive or semantically uniform regions. The proposed consistency tokens alleviate such ambiguity by routing image tokens that look alike but originate from different parts of the scene to distinct anchors, so that cross-view matches are established through shared 3D locations rather than appearance alone. We construct the consistency tokens from point clouds produced by either Structure-from-Motion or modern geometry foundation models, both of which can be noisy or incomplete. To address this, we adopt a bidirectional cross-attention architecture in which the two token sets co-refine one another, with image tokens gaining geometric grounding from the consistency tokens and the consistency tokens being corrected by the appearance cues of the images. On DL3DV-10K and in cross-dataset evaluations on Mip-NeRF 360 and Tanks \& Temples, our approach surpasses prior feedforward models and attains quality competitive with optimization-based pipelines, while preserving single-pass efficiency.

Training-free camera control for pretrained flow-matching video generators is a partial-observation inverse problem: a depth-warped guidance video supplies noisy evidence on a subset of latent sites, which the sampler must reconcile with the pretrained prior. Existing methods struggle to balance the trade-off between trajectory adherence and visual quality and the heuristic guidance-strength tuning lacks robustness. We propose $h$-control, which resolves this dilemma through a structural change to the sampler: each outer hard-replacement guidance step is augmented with an inner-loop block-conditional pseudo-Gibbs refinement on the unobserved complement at the same noise level, with provable convergence to the partial-observation conditional data law. To accelerate convergence on high-dimensional video latents, we exploit their conditional locality, partitioning the unobserved complement into 3D patches, each tracked by a custom mixing indicator that adaptively freezes converged patches. On RealEstate10K and DAVIS, $h$-control attains the best FVD against all seven training-free and training-based competitors, outperforming every training-free baseline on every reported metric.

While text-to-image (T2I) diffusion models exhibit strong semantic disentanglement at the object level, they struggle to localize fine-grained object parts. Although the latent visual features in these models are sufficiently fine-grained, the text–image interaction (cross-attention) fails to effectively exploit this information. This results in coarse and ambiguous localization, particularly for spatially distinct components such as `left` and `right` limbs. To address this limitation, we introduce **PartConcepts, a mechanism that encodes textual part descriptions into compact, learnable representations**. These PartConcept tokens are trained to selectively attend to their corresponding part regions in the image. To evaluate the part-level understanding of PartConcept tokens, we first probe its effectiveness for *part segmentation*. We then evaluate whether the improved localization enables *fine-grained instruction following* in T2I generation. On the standard **Open-Vocabulary Part Segmentation (OVPS) benchmark, our method outperforms dedicated segmentation baselines**, establishing a new state-of-the-art. We further evaluate our method on the more complex Pascal-Part benchmark for *part instance segmentation*, which contains fine-grained spatial annotations (e.g. `left lower arm`). Here, we outperform all baselines by a very significant margin. Finally, our qualitative and quantitative results show that this strong localization of PartConcept tokens directly enables *fine-grained instruction following* in T2I generation: the textual attributes (e.g., color) *inherently* bind well to corresponding PartConcept tokens, without explicit attribute control mechanisms. This indicates that PartConcept tokens compose seamlessly with other text tokens, highlighting their applicability for **fine-grained text-based generative control**.

Many machine learning tasks are invariant under the action of a group $G$ of transformations: signal classification can be invariant under translations, image classification under 2D rotations, and spherical-image classification under 3D rotations. The $G$-bispectrum is a principled complete invariant of a signal )retaining all signal's information up to the group action) with proven benefits in machine learning and as a pooling layer in deep networks. However, its deployment has been hampered by high computational cost and a patchwork of group-specific implementations. We present bispectrum, an open-source, fully unit-tested PyTorch library that implements selective $G$-bispectra for seven different group actions, as differentiable modules that can be directly incorporated into machine learning pipelines and deep learning architectures. For finite groups $G$, selectivity reduces the computational cost from $O(|G|^2)$ to $O(|G|)$. For planar rotations, we leverage the disk bispectrum. For spherical 3D rotations, we introduce an augmented selective bispectrum at band-limit $L$ which reduces the cost from $O(L^3)$ to $\Theta(L^2)$ coefficients. We profile the entire library (for

Generating property-optimized mRNA sequences is central to applications such as vaccine design and protein replacement therapy, but remains challenging due to limited data, complex sequence-function relationships, and the narrow space of biologically viable sequences. Generative methods that drift away from the data manifold can yield sequences that fail to fold, translate poorly, or are otherwise nonfunctional. We present RNAGenScape, a property-guided manifold Langevin dynamics framework for mRNA sequence generation that operates directly on a learned manifold of real data. By performing iterative local optimization constrained to this manifold, RNAGenScape preserves biological viability, accesses reliable guidance, and avoids excursions into nonfunctional regions of the ambient sequence space. The framework integrates three components: (1) an autoencoder jointly trained with a property predictor to learn a property-organized latent manifold, (2) a denoising autoencoder that projects updates back onto the manifold, and (3) a property-guided Langevin dynamics procedure that performs optimization along the manifold. Across three real-world mRNA datasets spanning two orders of magnitude in size, RNAGenScape increases median property gain by up to 148% and success rate by up to 30% while ensuring biological viability of generated sequences, and achieves competitive inference efficiency relative to existing generative approaches.

Vehicle trajectory generation is critical for autonomous driving simulation. However, current generative models face two coupled limitations: (i) imbalanced trajectory distributions bias them toward frequent motions, such as straight driving or stationary states, while weakening their ability to synthesize rare but safety critical motions, such as turning. (ii) Euclidean intermediate spaces provide limited geometric inductive bias for modeling vehicle pose evolution. Together, these limitations lead to mode biased generation and physically less coherent trajectories. To address this issue, we propose SEA, an \underline{\textbf{SE}}(2)-\underline{\textbf{A}}ware vehicle trajectory generation method that realizes the conditional transport path through planar rigid body poses. At the \textbf{data distribution} level, SEA learns vehicle trajectory distributions from traffic data to capture individual motion characteristics. At the \textbf{physical-space} level, motivated by the geometry dependence of {\bf optimal transport} interpolation, SEA realizes intermediate transport states through an $SE(2)$-consistent construction, thereby preserving geometric interpretability and physical coherence during generation. Extensive experiments on large scale real world datasets show that our method outperforms state of the art approaches by generating more realistic, diverse and dynamically consistent trajectories.