NeurIPS 2026 Papers

Synthetic data can scale visual pretraining, but current selection criteria only partially explain which datasets train strong representations. We propose using the energy profile of a dataset---the distribution of log-probability levels it occupies under a normalised real-image density model---as an audit and curation principle for synthetic data. To operationalise this view, we introduce Energy-Level Distribution Distance (Eld), which compares real and synthetic energy distributions. Real images span sparse high-likelihood scenes and dense low-likelihood textures, so useful synthetic data should cover this energy range rather than collapse to likely but low-variation samples. Across diverse synthetic datasets, Eld exposes mismatches missed by FID and predicts pretraining performance beyond FID, LPIPS variation, and feature-space recall, including across CNN and ViT self-supervised settings. We then use Eld to resample and mix synthetic sources, matching the real-image energy histogram while preserving diversity and improving downstream transfer.

Although audio-driven human animation has achieved impressive realism, it still lacks effective Multi-Scene Temporal Control: orchestrating background transitions at precise timestamps while preserving a coherent foreground subject. We show that this challenge stems from the diffusion transformer's self-attention mechanism, and addressing it requires two capabilities: (1) Semantic-Aware Temporal Decoupling, which isolates background attention across scenes while retaining cross-scene foreground attention for subject consistency; and (2) Foreground Motion Localization, which accurately tracks the foreground subject across latent frames, especially under large motion. To address these challenges, we introduce SceneShifter, a training-free framework for precise multi-scene control in audio-driven human animation. SceneShifter guides spatio-temporal self-attention by suppressing cross-scene attention among background tokens while preserving cross-scene attention among foreground tokens. To support this guidance under large motion, SceneShifter extracts dynamic foreground masks from the self-attention layers and heads that best capture subject trajectories. We further introduce SceneShifterBench, a benchmark designed to evaluate scene-timing accuracy, foreground preservation, and visual quality under large motion, multiple subjects, and occlusions. Experiments show that SceneShifter achieves frame-level scene timing, strong subject preservation, and high visual quality, outperforming existing baselines. Code and examples can be found at https://anonymous.4open.science/r/SceneShifter_neurips_26.

We study the problem of human gaze modeling, which aims to generate the gaze patterns a viewer produces while observing a visual stimulus. Gaze is primarily captured through two modalities: continuous eye-tracking trajectories, which describe fine-grained motion dynamics, and discrete scanpaths, which describe high-level fixation structure. Because gaze varies substantially across viewers and trials, we treat this variability as a defining property rather than noise and model gaze as a stochastic generative process. Existing generative gaze models supervise on only one of these two representations in isolation. We hypothesize that trajectories and scanpaths describe gaze at complementary scales and are jointly informative during training, and test this hypothesis through ST-DiffEye, a joint trajectory-scanpath diffusion framework that couples both modalities by concatenating them as an additional raw input channel, requiring no architectural overhead beyond an input and output channel expansion. We further introduce a principled evaluation framework based on the Continuous Ranked Probability Score (CRPS), which generalizes any existing sequence similarity metric into a proper scoring rule that jointly assesses the accuracy and diversity of generated gaze. Experiments on task-driven visual search, covering both target-present and target-absent scenarios, and on free-viewing benchmarks demonstrate state-of-the-art performance. These results, along with detailed ablations, confirm the benefit of joint modeling and the value of distribution-aware evaluation in capturing the intrinsic variability of human gaze.

Multimodal large language models (MLLMs) often answer visual reasoning questions by relying on linguistic priors rather than task-relevant visual evidence. Textual chain-of-thought reasoning can partially mitigate this issue by encouraging models to decompose visual questions into intermediate evidence-seeking steps, but generating these steps autoregressively increases inference cost. Latent reasoning avoids explicit rationale generation, but existing approaches provide limited control over what intermediate states encode, making it difficult to impose separate supervision for planning, grounding, and evidence selection. We propose Structured Latent Visual Reasoning (SLVR), a training framework that bridges explicit chain-of-thought and latent reasoning by organizing multimodal reasoning into typed latent stages for planning, grounding, evidence selection, and reasoning integration. SLVR first trains the model to rely on the image by masking answer-revealing text and contrasting the correct answer with visually plausible distractors. It then organizes reasoning into latent stages for planning, grounding, evidence selection, and integration, supervising each stage with the corresponding signal: plans, boxes, visual evidence, and final rationales. This gives latent reasoning an explicit functional structure while avoiding generated textual chains at inference time. Built on Qwen2.5-VL-7B, SLVR improves consistently across multimodal reasoning benchmarks, with absolute gains of +9.4 on MMVP and +14.2 on BLINK Relation, as well as improvements on V*, MathVista, and ChartQA. These results suggest that structured latent supervision can improve fine-grained visual reasoning without the decoding overhead of textual CoT. Code is available \href{https://anonymous.4open.science/r/SLVR-681F/README.md}{here}.

We show that Fréchet Distance (FD), long considered impractical as a training objective, can in fact be effectively optimized in the representation space. Our idea is simple: decouple the population size for FD estimation (e.g., 50k) from the batch size for gradient computation (e.g., 1024). We term this approach FD-Loss. Optimizing FD-Loss reveals several surprising findings. First, post-training a base generator with FD-Loss in different representation spaces consistently improves visual quality. Under the Inception feature space, a one-step generator achieves 0.72 FID on ImageNet $256\times256$. Second, the same FD-Loss repurposes multi-step generators into strong one-step generators without teacher distillation, adversarial training or per-sample targets. Third, FID can misrank visual quality: modern representations can yield better samples despite worse Inception FID. This motivates FDr$^{k}$, a multi-representation metric. We hope this work will encourage further exploration of distributional distances in diverse representation spaces as both training objectives and evaluation metrics for generative models. Code and checkpoints will be open-sourced.

Existing MLLM-based UI-to-Code generation methods can generate complete HTML/CSS codes through end-to-end or multi-stage pipelines, yet the generated codes often contain structural errors such as incorrect repeated layouts, wrong component boundaries, or missing interaction slots. Subsequent generations may continue along the erroneous structure, causing the error to propagate and amplify. This paper studies post-generation repair: how can we locate and repair unreliable local DOM structures? We observe that this repair process is ambiguous in two ways: a mismatch between the target screenshot and the output may correspond to multiple plausible DOM subtrees, and a single DOM subtree may admit multiple repairs with different rendered behaviors. We propose Render Structure Uncertainty (RSU)for partial DOM repair, a browser grounded uncertainty layer that can be attached to any base HTML generator. RSU parses the target screenshot into a visual structure graph and converts the browser execution trace of the generated code into a DOM-anchored render structure graph. By matching these graphs, RSU yields structural mismatches and a small set of candidate DOM holes, preserving error attribution uncertainty. For each hole, RSU generates multiple local candidates, renders them in the full page context, and clusters them using render-structure signatures that encode layout relations, repeated patterns, content slots, semantic roles, and invalid states, thereby modeling repair uncertainty. The retained representative candidates form a compact Partial DOM candidate set, over which we perform global reranking to produce the final HTML code. In this way, RSU reframes HTML repair into uncertainty-guided search in render-structure space, avoiding premature commitment to either a single error location or a single local repair. Experiments show that RSU improves the generation quality of base models, producing higher-quality HTML with better layout consistency and fewer local structural errors.

Recent advances in 3D reconstruction have progressed from per-scene optimization to feed-forward inference, and semantic scene understanding has followed suit—yet existing methods remain confined to object-centric perception, neglecting spatial relations between objects. We introduce \emph{3D spatial relation segmentation}, a task that requires models to identify a target object satisfying a given spatial relation with respect to a subject, consistently across multiple views. To this end, we propose RelationVGGT, a novel feed-forward framework that integrates semantic features from a visual foundation model with geometry-aware representations from a 3D geometry foundation model and leverages a relation transformer for subject-conditioned, cross-view relation prediction—requiring neither per-scene optimization nor known camera poses. We additionally provide a fully automated annotation pipeline built on ScanNet++ with VLMs and LLMs, enabling scalable training data generation for this new task.

Visual Geometry Grounded Transformer (VGGT) recovers dense 3D scene structure from multi-view images in one forward pass, but quadratic cross-frame attention limits its scalability. Existing training-free accelerators reduce computation uniformly along one axis, missing layer heterogeneity. Our spectral, probing, and causal analyses reveal three regimes: shallow layers lack cross-view structure, middle layers drive cross-view alignment, and deep layers are redundant for dense geometry yet their cross-frame attention remains essential for pose. RegimeVGGT applies layer-wise U-shaped compression along two axes: Saliency-Guided Banded Merging protects geometry- and edge-salient tokens, while Selectively Protected K/V Downsampling preserves cross-frame spatial coverage and the pose-critical path through a phase-shifted spatial grid, a reference-frame anchor, and uncompressed camera/register tokens. Training-free, RegimeVGGT achieves a $6.7\times$ speedup over VGGT* at matched reconstruction quality. Code: \url{https://anonymous.4open.science/status/RegimeVGGT-9477}

Image tokenization aims to compress visual signals into latent codes that remain effective for reconstruction and generation. Driven by the multi-scale nature of images, organizing latent along a coarse-to-fine refinement process emerges as a promising direction for visual tokenization. Existing scale-wise tokenizers often introduce this structure through token ordering or latent prefixes, but each target scale is typically decoded directly from its prefix, making different prefixes behave like separate reconstruction codes. This can lead to redundant encoding of global structure across scales and limits the effectiveness of compact token budgets. We propose RefineTok, a scale-wise continuous tokenizer that makes decoding itself progressive. RefineTok carries a visual state across scales and uses each latent prefix to refine the previous state into the next scale, turning scale-wise latents into incremental refinement signals rather than independent reconstruction codes. This design yields a compact and structured latent interface for both reconstruction and downstream generation. Together with prefix-level training, a coarse-centric scale trajectory, and a semantic anchor, RefineTok achieves strong reconstruction fidelity among scale-wise tokenizers and competitive class-conditional generation on ImageNet-1K under compact latent budgets. Meanwhile, it preserves a progressive scale-wise interface that single-scale tokenizers do not provide.

Camouflaged Object Segmentation (COS) aims to identify objects visually hidden in their surrounding environments. Existing COS benchmarks mainly focus on image-level or video-level visual perception, while real-world camouflaged scenes are often accompanied by audio signals and user intentions, where sound semantics and textual expressions provide critical information for localizing hidden targets. To extend the boundary of COS, we introduce a new task, termed Referring and Reasoning Audio-Visual Camouflaged Object Segmentation (R2-AVCOS), which aims to segment camouflaged targets in audio-visual scenes according to textual expressions with referring or reasoning intentions. This task emphasizes understanding audio content and incorporates complex reasoning and world knowledge into expressions. To support this task, we construct R2-AVCOSBench, the first audio-visual benchmark with pixel-level annotations for camouflaged objects specified by referring expressions or inferred through reasoning. It contains 2,654 audio-visual camouflaged videos, 21,232 annotated frames, and 32,329 expressions, including 17,543 referring and 14,786 reasoning expressions. Furthermore, we propose Camouflaged Instructed Segmentation Assistant (CISA), a baseline model built upon a Multimodal Large Language Model (MLLM). CISA understands complex textual and audio-visual cues and performs referring- and reasoning-based camouflaged object segmentation. Extensive experiments show that CISA achieves strong referring and reasoning segmentation ability in audio-visual camouflaged scenes and obtains competitive results on related tasks.

Image-to-video generation powers a vast array of downstream applications. However, while the de facto standard, ie, latent diffusion models, typically employ heavily conditioned denoising networks, their decoders often remain unconditional. We observe that this architectural asymmetry leads to significant loss of detail and inconsistency relative to the input image. To address this, we argue that the decoder requires equal conditioning to preserve structural integrity. We focus specifically on image-based generation tasks, where maintaining the structural and textural integrity of the source image is paramount. We introduce RefDecoder, a reference-conditioned video VAE decoder by injecting the reference image directly into the decoding process via reference attention. Specifically, a lightweight image encoder maps the reference frame into the detail-rich high-dimensional tokens, which are co-processed with the denoised video latent tokens at each decoder up-sampling stage. We demonstrate consistent improvements across several distinct decoder backbones (eg, Wan 2.1 and VideoVAE+) achieving consistent improvement on three reconstruction benchmarks. Notably, RefDecoder can be directly swapped into existing video generation systems without additional fine-tuning, and we report across-the-board improvements in subject consistency, background consistency, and overall quality scores on the VBench I2V benchmark. Furthermore, RefDecoder generalizes well to other visual generation tasks such as style transfer.

Modern Voice Cloning (VCL) can synthesize speech that closely matches a target speaker from only seconds of reference audio, enabling applications such as personalized speech interfaces and dubbing. In practical deployments, modern audio generation models inevitably encounter noisy reference audios, imperfect text prompts, multilingual and long-form generation settings, downstream post-processing, and adversarial perturbations, all of which can significantly hurt robustness. Despite rapid progress in VCL driven by autoregressive codec-token language models and diffusion-based models, robustness under realistic deployment shifts remains underexplored. This paper introduces RVCBench, a comprehensive dataset and benchmark that evaluates Robustness in Voice Clone across the full generation pipeline. RVCBench contributes a large-scale, task-aligned robustness dataset that instantiates realistic deployment shifts through controlled text-audio pairing, multilingual and long-form scenarios, expressive prompts, post-processing conditions, and passive or proactive audio perturbations. Covering 18 robustness evaluations, 225 speakers, and 14,370 utterances, RVCBench enables unified evaluation of input sensitivity, generation stability, output resilience, and perturbation robustness. We evaluate 18 representative modern VCL models and reveal systematic vulnerabilities in content consistency, speaker similarity, long-form stability, post-processing resilience, detectability, and adversarial robustness.

We explore whether Omni-Language Models (OLMs) can be directly applied to zero-shot Semantic Audio-Visual Navigation (SAVN). Recent work demonstrates that even state-of-the-art specialized models still struggle to achieve generalist multimodal navigation, despite extensive task-specific training. In this paper, we introduce **RAO-Nav**, short for Reasoning All-in-One OLM, a deployment pipeline for zero-shot SAVN. By leveraging the rich implicit audio-visual knowledge encoded in OLMs, the embodied agent is enabled to hear, see, reason, and act in the environment. To further elicit the built-in thinking ability of OLMs, we propose a test-time Latent Navigation Reasoning (LNR) module that can be seamlessly integrated into the decoding space. LNR encourages the model to retrieve more target-relevant observations and make effective navigation decisions. Through comprehensive experiments, we show that our framework surpasses existing state-of-the-art baselines on public SAVN benchmarks without using any training data. Moreover, we introduce a new Global Navigation Instruction setting to further evaluate the ability of OLMs to serve as embodied navigation agents. Results under this setting posit that applying OLMs to real-world natural-language-guided audio-visual navigation remains challenging.

Large vision-language models (LVLMs) rely on increasingly long visual contexts, making the KV cache a major inference-time bottleneck. Existing KV cache compression methods typically use pre-decode signals such as prefill attention, visual redundancy, objectness, or layer-wise allocation, but these criteria do not directly indicate which visual tokens in the KV cache will be reused by future answer tokens. We introduce Q-ViK, a question-conditioned visual utility framework for LVLM KV cache eviction. During offline training, we run full-cache decoding and aggregate answer-to-visual attention over cached visual tokens, yielding a privileged *future-utility signal* that reflects the visual KV entries actually used during generation. We use this signal to train a lightweight question-conditioned scorer that predicts visual cache utility from prefill representations alone. At inference time, Q-ViK requires neither full-cache lookahead nor generated traces: it preserves textual KV entries and evicts low-utility visual KV entries with a single post-prefill scoring step. Experiments on seen and held-out multimodal benchmarks show that Q-ViK preserves task-relevant visual evidence in the KV cache more effectively than pre-decode saliency, especially under aggressive KV cache compression. Code will be available at [https://anonymous.4open.science/r/Q-ViK-1A75/README.md](https://anonymous.4open.science/r/Q-ViK-1A75/README.md).

Vector polygon generation converts visual inputs, e.g., remote sensing (RS) images, into vectorized polygonal geometries, supporting applications such as autonomous driving, vector map construction, and remote sensing. Early pipelines predict raster masks and post-process them into polygons, which prevents end-to-end optimization and may miss small objects or introduce inaccurate vertices. Recent methods directly generate vector polygons, but most focus on simple exterior contours, while they either cannot represent complex polygons with holes or fail to preserve their topology. In this paper, we propose PolyTopoBench, a unified evaluation framework for vector polygon generation from RS images with explicit emphasis on complex polygons. PolyTopoBench evaluates both exterior and interior rings, and benchmarks representative methods, including segmentation-based polygonization pipelines, vision foundation model baselines, and specialized vector polygon generators, on two RS-image datasets covering buildings, roads, vegetation, and unvegetated regions. Experiments show that existing methods often recover simple exterior boundaries but degrade substantially on polygons with holes or multiple rings. These results reveal complex polygon generation as an unresolved challenge and motivate topology-aware benchmarks and model designs.

Audio-language models (ALMs) are increasingly used in real-world applications that require understanding music, from music tutoring and transcription to captioning, recommendation systems, and music production. More broadly, they are becoming an important component of multimodal AI systems that must reason from sensory input rather than text alone. This makes reliable musical perception a critical prerequisite: if a model cannot accurately hear the structure of sound, it cannot be trusted to reason about, teach, transcribe, or act on audio in the real world. Yet existing benchmarks rarely assess one of the most fundamental musical abilities underlying such perception: pitch hearing. Current evaluations tend to probe pitch hearing only indirectly, through higher-level tasks and often in multiple-choice formats, leaving open how reliably ALMs identify fine-grained pitch across instruments, acoustic conditions, and response formats. We introduce PitchBench, an evaluation suite that systematically measures pitch hearing in ALMs. PitchBench comprises 28 experiments spanning absolute and relative pitch perception within sequences and chords, while varying loudness, note duration, sound source, time stretching, background noise, and other acoustic conditions. Tasks range from identifying individual pitches in isolation to tracking a melodic line within a four-part musical texture. Evaluating frontier ALMs, we find that pitch hearing remains highly unreliable: models perform consistently poorly across settings, with accuracy varying sharply by sound source, note duration, and notation format. Current ALMs do not yet possess stable pitch perception, even for controlled synthetic and instrumental stimuli. Alongside the benchmark, we release PitchBench as a Python package containing the evaluation data and data generation tools to support future work on pitch-aware audio-language modeling.

Realistic physical interaction is a cornerstone of embodied intelligence, yet obtaining high-fidelity tactile data remains significantly more expensive than visual data. This scarcity necessitates Visual-to-Tactile synthesis to bridge the Sim-to-Real gap. However, existing data-driven approaches often treat this as a naive image-to-image translation task, neglecting fundamental contact mechanics. Moreover, the inherent spatial misalignment in real-world visual-tactile datasets frequently forces generative models to average out spatial uncertainties, resulting in blurry and textureless outputs. To address these limitations, we introduce PhysTacGen, a physics-aware generation framework that enforces consistency between visual appearance and tactile mechanics. Our system makes three technical contributions. Firstly, we propose Group Tactile Policy Optimization (GTPO), a reinforcement learning alignment strategy that refines Large Multimodal Models to infer latent physical properties by rewarding physics-consistent reasoning chains. Secondly, we introduce a Semantic-Driven Data Curation and Visual Prior pipeline, leveraging DINOv2 to filter spatially misaligned data and extracting pure depth maps to decouple macro-geometry from surface texture. Finally, we present a physics-driven generative architecture based on Stable Diffusion XL (SDXL). It synthesizes tactile images via a ControlNet conditioned on GTPO-generated physical descriptions and depth-augmented visual priors. Extensive experiments demonstrate that PhysTacGen achieves state-of-the-art perceptual quality and significantly enhances grasp force prediction for zero-shot Sim-to-Real transfer.

Multimodal large language models (MLLMs) have achieved remarkable progress on vision–language tasks, yet their reasoning processes remain sometimes unreliable. We introduce PRISM-Bench, short for Puzzle Reasoning with In-Sequence Mistakes, a benchmark of puzzle-based visual challenges designed to evaluate not only whether models can solve problems, but how their reasoning unfolds. Unlike prior evaluations that measure only final-answer accuracy, PRISM-Bench introduces a diagnostic task: given a visual puzzle and a step-by-step chain-of-thought (CoT) containing exactly one error, models must identify the first incorrect step. This setting enables fine-grained assessment of logical consistency, error detection, and visual reasoning. The puzzles in PRISM-Bench require multi-step symbolic, geometric, and analogical reasoning, resisting shortcuts based on superficial pattern matching. Evaluations across state-of-the-art MLLMs reveal a persistent gap between fluent generation and faithful reasoning: models that produce plausible CoTs often fail to locate simple logical faults. By disentangling answer generation from reasoning verification, PRISM-Bench offers a sharper lens on multimodal reasoning competence and underscores the need for diagnostic evaluation protocols in the development of trustworthy MLLMs.

Joint audio-video generation aims to synthesize temporally synchronized and semantically coherent visual and acoustic content. Existing open-source methods mainly follow either dual-tower designs, which generate audio and video in separate streams and rely on posterior alignment, or fully unified tri-modal designs, which mix textual context, audio, and video in a single shared space. These paradigms may either weaken fine-grained audio-video co-evolution or couple semantic conditioning with low-level synchronization. We propose NAVA, a Native Audio-Visual Alignment framework that formulates generation as context-conditioned native audio-visual alignment. NAVA first establishes audio-video correspondence in a dedicated alignment space and then applies context as external conditioning to guide the aligned representation. We instantiate this formulation with an Align-then-Fuse MMDiT architecture, which progressively bridges modality-aware alignment and unified audio-video denoising. To support controllable speech generation, we further introduce Timbre-in-Context Conditioning, which binds reference timbre cues to corresponding speech spans through the context pathway. Experiments on Verse-Bench and the Seed-TTS benchmark demonstrate that NAVA achieves superior audio-visual synchronization and video quality, competitive audio quality, and substantially improved reference-timbre controllability with only 6.2B parameters.