NeurIPS 2026 Papers

Current Audio-Visual LLMs (AVLLMs) struggle with reasoning over videos featuring multi-speaker dialogues. In such videos, resolving "who says what" is crucial, which necessitates trimodal (text-audio-visual) binding. Motivated by these challenges, we systematically investigate how this trimodal binding is achieved in AVLLMs. Specifically, we identify emergent symbolic trimodal binding mechanisms in AVLLMs that utilize modality-specific symbolic variables. By encoding auditory and visual components into symbolic variables-capturing temporal utterance sequences and spatial entity coordinates, respectively-the model establishes cross-modal linking within this abstract space. Crucially, we reveal that when trimodal binding fails, the breakdown predominantly stems from misaligned audio-visual connections. To overcome this bottleneck, we introduce an audio-visual prompting method utilizing an off-the-shelf Active Speaker Detection (ASD) model. By simply overlaying visual bounding boxes on active speakers, this training-free approach yields immediate performance gains across three conversation-centric benchmarks. Moreover, lightweight fine-tuning of fewer than 300 steps on these ASD-prompted-videos extends these gains to three general AV benchmarks, suggesting the generalizability of our method.

We propose a label-free approach to adapt powerful but generic vision foundation models to specialized scientific domains. Standard supervised fine-tuning is often ill-suited to these settings: labels are scarce, and task-specific training can collapse the model's generality and hurt robustness. We instead leverage metadata to adapt representations to new domains in a self-supervised manner. Our method, META-DINO, combines a standard self-supervised objective with flexible metadata guidance that handles both highly granular discrete metadata and continuous metadata. It encourages the representation to preserve informative factors while suppressing spurious ones. Across subcellular fluorescence microscopy, Earth observation, wildlife monitoring, and medical imaging, META-DINO consistently outperforms standard unsupervised domain adaptation and fully supervised adaptation. It also exceeds highly-specialized domain-specific state of the art, while using no task labels for backbone adaptation and only lightweight probes for supervision.

Deciding which member of a group produced a vocalization and where that vocalization was produced are central to studying social communication but are rarely evaluated with physically verified ground truth. Existing resources either localize sounds without identifying the caller, or recognize individuals from isolated vocalizations without reliance on the 3d candidate geometry. We close this gap with V33DA, a multimodal benchmark for spatial vocal attribution in social zebra finches: it comprises 33,625 verified events coupling 5-channel audio with three camera views, calibrated 3D pose for every visible candidate, and per-bird radio telemetry across 10 individuals and three experiments. Caller identity is verified from an on-body accelerometer-derived vibration signal; this channel is withheld from benchmark models and used only by an oracle ceiling. We also provide V33DA++ as an auxiliary extension with verified overlapping-call events and $\pm 2$s context windows, intended to support future source-separation and vocal-activity-detection studies. V33DA separates familiar-individual recognition from transferable spatial attribution. We evaluate candidate-conditioned caller attribution, with 3D source localization as a secondary diagnostic, under three regimes: session-disjoint testing, held-out-experiment transfer, and leave-one-individual-out evaluation. A broad set of reference methods exposes a consistent gap between in-domain accuracy and transfer under identity shift: methods that can exploit familiar vocal identity perform well on known callers but degrade sharply on unseen ones, whereas candidate-aware methods preserving explicit spatial reasoning transfer substantially better. The accelerometer oracle stays near-perfect across regimes, confirming the internal consistency of the accelerometer-verified labels and quantifying the ceiling available when the withheld verification channel is observed. We release V33DA with fixed attribution/localization protocols and V33DA++ with overlap and long-context tasks, together with code for the evaluated methods.

Video Large Language Models (Video-LLMs) have made rapid progress on temporal video understanding, yet many fail at a basic perceptual primitive: signed image-plane motion direction. On simple videos of a single object moving left, right, up, or down, most Video-LLMs perform near chance, with above-chance cases largely attributable to prediction biases rather than genuine direction understanding. We call this failure directional motion blindness. We localize the failure by tracing motion direction information through the Video-LLM pipeline. Motion direction remains linearly accessible from the vision encoder, projector, and LLM hidden states, but the readout fails to bind this signal to the correct verbal answer option, revealing a direction binding gap. Although synthetic motion direction instruction tuning reduces this gap on the source domain, motion direction concept vector analysis shows that visual complexity weakens the signal magnitude and limits out-of-domain generalization. We introduce MODIRECT, a dataset family for motion direction instruction tuning and evaluation, and DeltaDirect, a diagnosis-driven, projector-level objective that predicts normalized 2-D motion vectors from adjacent-frame feature deltas. On MODIRECT-SYNBENCH, instruction tuning with DeltaDirect improves motion direction accuracy from 25.9% to 85.4%. On MODIRECT-REALBENCH, DeltaDirect improves real-world motion direction accuracy by 21.9 points over the vanilla baseline without real-world tuning data, while preserving standard video-understanding performance.

Token merging has emerged as an effective training-free strategy for accelerating diffusion models by reducing redundant token computation while minimizing generation degradation. A key challenge is determining which tokens can be safely merged and which should be preserved for subsequent computation. Existing method addresses this by preserving prompt-aligned foreground regions, thereby protecting semantically relevant structures. However, this text-relevance criterion overlooks background objects, local details that are not explicitly mentioned in the prompt, leading to degraded structural coherence. In this work, we propose Dynamics-aware Token Merging (DaTMe), a training-free framework that grounds token merging in the underlying diffusion dynamics. Our key insight is that token importance should reflect whether a token remains unresolved during denoising and whether it carries structural information. To this end, DaTMe constructs a dynamics-aware importance map from the Tweedie denoised estimate. The map combines a temporal signal, measuring inter-step changes in the predicted clean image, with a structural signal, capturing local high-frequency content such as boundaries and fine details. Using this map, DaTMe adaptively assigns tokens into merging roles under a given compression ratio: important tokens are preserved as anchors, while tokens that are both temporally stabilized and structurally homogeneous are selected as safe merge candidates. Experiments on PixArt-$\alpha$ and FLUX demonstrate that DaTMe maintains visual fidelity across the entire image, highlighting diffusion dynamics as an effective token-level merging decision in efficient generation.

Direct Preference Optimization (DPO) has become a standard method for preference-based fine-tuning in RLHF pipelines. A practical but largely heuristic step in DPO is dataset curation: given a fixed budget of preference labels, which candidate pairs should be compared to produce the most informative training set and the best downstream policy? This paper studies DPO dataset curation as a sampling-design problem. We model the curated dataset by a sampling design over candidate pairs and analyze how this design propagates through DPO training to the KL-regularized RLHF objective. Our main results provide matching upper and lower bounds on the RLHF optimality gap of the policy learned from a curated DPO dataset. The bounds reveal a simple structural message: the effect of pair selection enters only through a single design-dependent matrix that summarizes how informative the collected comparisons are for learning the policy parameters. This leads to a trace-form criterion that characterizes both an achievable performance guarantee for the DPO estimator and an information-theoretic lower bound for any induced policy estimator, thereby identifying a canonical objective for comparison curation under budget constraints.

Self-supervised learning on 3D medical images commonly relies on random sub-volume sampling during pretraining. However, random cropping implicitly treats all spatial regions as equally worth observing, despite anatomical priors and spatial redundancy that make regions inherently unequal in their value for representation learning. To this end, we first show through controlled crop-quality manipulations that better sub-volumes lead to better downstream transfer, revealing the observation policy as an overlooked bottleneck in 3D medical SSL. Motivated by this finding, we revisit sub-volume sampling as a where-to-look problem. We propose VolumeProbe, a plug-and-play framework that replaces random cropping with an offline coarse-to-fine probe bank guided by anatomy-awareness, informativeness, and diversity. We instantiate VolumeProbe as a lightweight Lite variant using geometric and image-statistical cues, and a diagnostic Feat variant using frozen visual features, to examine whether lightweight volume-intrinsic priors suffice or richer semantic cues reliably provide additional value. Extensive experiments across multiple 3D medical SSL backbones and downstream tasks demonstrate that VolumeProbe-Lite consistently improves transfer performance over random sampling. Mechanistic analyses further attribute these gains to more anatomically plausible and informative observations, reduced spatial redundancy, more efficient learning dynamics, and more structured representations.

Text-to-image diffusion models fail on semantically rare attribute--object compositions, but it is unclear whether this brittleness arises during denoising or is already present in conditioning representations. We study this question at three levels of granularity. First, using a controlled stress protocol across SD1.5, SDXL, and SD3, we show that a Compositional Stress Index derived from text-side non-factorization residuals separates anchor from stress prompts with perfect discrimination (AUC${=}1.0$) cross-model, establishing a robust pre-denoising risk axis. Second, embedding-level low-rank repair reduces representation residuals without any positive semantic improvement, while a downstream cross-attention intervention succeeds on the same subset---a site-sensitive boundary between diagnosis and control. Third, we decompose this boundary at block-group resolution under three intervention modalities. Diagnostic accessibility peaks at the deep encoder (D2) across all conditions, but token-level intervention of either sign produces no measurable improvement at any block. Only broad cross-attention ablation moves semantics at D2, while decoder blocks respond to selective boost and collapse under ablation. Together, these results establish that compositional defects are diagnosable before denoising, but \textbf{where to look is not how to fix}: the diagnostic site and the control site require different intervention modalities, and even at the correct site, the wrong modality is futile.

A central yet underexplored question in Multimodal Large Language Models (MLLMs) is where to connect — how visual information from a hierarchical vision encoder should be wired into the layer-wise semantics of a large language model. Mainstream MLLMs inject a fixed single-layer visual representation into the LLM and treat all LLM layers as a single consumer, ignoring that different layers demand different visual granularities. Recent multi-layer fusion methods either compress hierarchical features on the encoder side or rely on predefined sparse or hierarchical layer-to-layer connections, and therefore stop short of modeling adaptive cross-layer interactions between the ViT and LLM hierarchies. We address this gap with DGP (Dynamic Gated Pathways), a routing mechanism that establishes all-to-all connections between every ViT layer and every LLM layer, with each LLM layer dynamically gating its preference over visual layers conditioned on its own semantic state. Built on top of LLaVA-1.5-7B and 13B, DGP delivers consistent gains over the LLaVA-1.5-7B baseline (e.g., +4.2 on SQA$^{I}$, +3.3 on MM-Vet, +3.1 on VizWiz and LLaVA$^{W}$) and surpasses prior multi-layer visual fusion methods on most benchmarks. Layer-wise routing analyses further show that different LLM layers do select different visual granularities on demand, supporting both the effectiveness and interpretability of the proposed dynamic gated pathways. We will open-source code/weight/demo soon.

Probabilistic neurosymbolic methods rely on weighted model counting (WMC) to combine neural predictors with symbolic constraints. Exact computation of the WMC, a \#P-hard problem, typically scales poorly, so many methods resort to approximations. By unifying existing approaches under a three-step bottom-up *approximate compilation* pipeline, we study *where* approximation budget is best allocated. Inspired by tensor networks, we instantiate two steps with tensor train decompositions: we derive new pipelines that exactly multiply approximated factors and recompress their products via SVD- or interpolation-based schemes. On Sudokus of increasing size, these pipelines produce WMC estimates that are many orders-of-magnitude more accurate than prior methods. Yet, when it comes to neurosymbolic learning, even crude approximations reach competitive accuracy, suggesting that approximation quality matters far more for inference fidelity than for downstream learning. Our pipelines open significant design space for effective approximate compilation of challenging probabilistic reasoning tasks.

Fine-tuning large transformers routes gradients through every token position, paying the cost in activation memory that scales linearly with sequence length and dominates the GPU budget. Existing efficient methods reduce the parameter side---low-rank adapters, prefix tuning, weight-decomposed updates---but leave the activation footprint untouched. We present the Token Hypothesis, a memory-efficient fine-tuning procedure motivated by a Neural Tangent Kernel analysis of how fine-tuning signal is distributed across token positions. We prove that every position is a low-dimensional capacity bottleneck whose width does not grow with the model, that positions with aligned hidden-state geometry are interchangeable in bidirectional models, and that causal positions form a monotone capacity chain in which later positions strictly dominate earlier ones. The analysis pins down how many positions are needed and which ones, both computable from forward passes alone: any subset suffices in bidirectional models, while the last few positions suffice in causal ones. We translate this into a procedure that backpropagates through only a small subset of positions while running the full forward pass over the rest. Empirically, the procedure matches or exceeds full-sequence fine-tuning across six task families---commonsense reasoning, mathematical reasoning, visual question answering, video question answering, GLUE, and image classification---and five backbones spanning bidirectional and causal architectures. It reduces activation memory substantially, translating to up to ∼ 20% lower total GPU memory (∼ 33GB) on long-context tasks, and composes cleanly with parameter-efficient methods such as LoRA and DoRA.

Tabular foundation models are used as off-the-shelf predictors for heterogeneous tabular tasks, but it remains unclear how they will perform on real genotype-to-phenotype tabular datasets, which carry unique challenges. One such challenge is ancestry-dependent non-stationarity in allele effect sizes (the effects of genetic variation on disease risk): the predictive contribution of a given genetic variant may change significantly across the ancestry spectrum. This is especially consequential for patients from ancestry groups that are underrepresented in existing datasets, because models that are unaware of this non-stationarity are more likely to perform poorly for them. The battery of synthetic tasks used to pretrain tabular foundation models does not capture this structure, and we show that it leads to systematic performance degradation. Using controlled hierarchical Gaussian-process stress tests, we demonstrate that both off-the-shelf TabICL and TabPFN are robust when ancestry dispersion in the data is low, but as dispersion grows, the latent ancestry-dependent non-stationarity of effect sizes becomes a first-order problem, and the predictive performance of both models deteriorates. We confirm the same pattern on All of Us (AoU) (a large biobank containing whole-genome sequencing and electronic health record (EHR) data spanning the ancestry spectrum) across an extensive panel of cancer phenotypes evaluated with the two leading tabular foundation models: TabICL and TabPFN. Holding the in-context training-set size, phenotype, and feature set fixed, ancestry-specific in-context tables, which are less dispersed in ancestry space, consistently outperform size-matched meta-ancestry tables, isolating ancestry dispersion as the driver of degradation. To address the failure without changing the base architecture, we construct two synthetic task families that explicitly encode ancestry-dependent effect drift and instruction-tune an off-the-shelf TabICL model on tasks sampled across both families. On held-out AoU evaluations covering cancer and respiratory disease phenotypes, the tuned model yields more stable performance across ancestry-distance bins and is especially strong in the bins farthest from the center of the in-context exemplars in ancestry space, that is, on subjects whose ancestry is most underrepresented among the provided exemplars. The paper contributes an evaluation protocol, a failure analysis, and two synthetic task families targeted at ancestry-dependent non-stationarity in precision-medicine tabular modeling.

Society constantly has to determine the boundaries of what it deems acceptable, from legislative decisions to the guardrails governing autonomous systems. We initiate the axiomatic study of _collective consent_: given individuals' attitudes toward options, which options should receive societal consent? We organize our analysis around three principles: sufficient support, minority protection, and dominance by decisively better options. Each captures a distinct reason for withholding societal consent from an option. For each principle, we develop a corresponding solution concept that transparently implements the principle and is canonical in a mathematically precise sense. For example, for minority protection, the resulting concept is a consent-adapted version of Moulin's Proportional Veto Core. Balancing multiple principles simultaneously is more challenging. To address this, we develop a game-theoretic characterization of our veto core that naturally gives rise to a family of related concepts. From this family, we identify the _Approval-Weighted Veto Core_ as particularly desirable. By making minorities' blocking power depend on the approval support of the options being challenged, it smoothly interpolates between proportional minority protection and majority support. Experiments on five datasets spanning _high-stakes_ decision-making (such as political elections, ethical AI evaluations, and moral decision-making) show that solution concepts violating a principle in theory also violate it empirically.

Identifying root cause of an anomaly among hundreds of sensors is critical for preventing safety incidents and costly downtime in complex monitored systems. Existing studies evaluate root cause analysis (RCA) methods using top-k accuracy. We show that this metric has a fundamental blind spot: it conflates two failure modes: *retrieval failure*, where the true cause is never considered, and *reranking failure*, where it is considered but ranked too low. In this work, we introduce a retrieval-reranking decomposition and audit four well-known benchmarks to expose this blind spot. Our experiments show that, on benchmarks with complex faults, statistical baselines mis-rank the true cause 78-100\% of the time, and graph-based methods perform poorly because of unreliable causal graphs constructed using short fault windows. Meanwhile, on simple benchmarks where faults manifest significantly at their origin, retrieval is trivially solved at 100\%. Guided by the decomposition, we build a two-stage pipeline combining a multi-signal retriever with an LLM reranker that lifts top-1 accuracy by $+$2 to $+$26 points over the best baseline on complex-fault benchmarks and $+$9 to $+$13 points on simple benchmarks, with no causal graph or labeled data required.

Mechanistic studies of grokking often begin after an endpoint circuit has been identified. We study the preceding retrospective triage problem: given a solved final checkpoint and a long saved trajectory, which path and window should be inspected first? We propose solution-frame path triage. The final checkpoint fixes a solution frame; earlier checkpoints are then scored path by path. The Geometric Coherence Score (GCS) measures coherence in this procedure: it asks whether neighboring inputs in the final frame undergo similar local Jacobian transformations through the attention, MLP, or full block measurement maps. Centered Kernel Alignment (CKA) and $L_2$ distance to the final path state provide the complementary proximity scores. Across 104 modular arithmetic Transformer runs, GCS reveals a repeatable path schedule that simple norm and dimensionality summaries miss: attention coherence rises near grokking and often turns over, while MLP maps diversify and later reconverge. After selection, held-out checks show that the resulting windows are enriched for accuracy and algebraic changes, align with Fourier progress on modular addition, and identify attention-to-MLP interface states with high replacement cost. The method returns a short list of path/window hypotheses for endpoint analysis. It is retrospective by design: the final checkpoint is part of the procedure.

Parameter-efficient fine-tuning (PEFT) and probing enable adaptation of foundation models using only a small number of trainable parameters, making it attractive for video understanding where annotation and computation are expensive. However, video PEFT has focused on adapting image-pretrained models, while standard PEFT methods can also be applied to video representations. These settings are rarely compared and both confine temporal reasoning to a single component of the model, leaving open how temporal context should be distributed across backbone, PEFT and probe. In this work we provide a systematic study of model adaptation strategies for video understanding. We evaluate methods across appearance-focused, motion-focused and spatially dense settings, with a particular focus on scenarios with limited data where parameter-efficiency is most beneficial. Our results provide new insights into PEFT and probing across settings and demonstrate the importance of temporal context allocation for effective video adaptation.

Detailed image captioning is usually framed as an amount problem: a model should mention more visual facts while avoiding hallucination. We argue that this framing hides a basic structure of long-form captioning: a caption is an ordered sequence, and different failures appear at different positions. We introduce a position-aware rubric representation that decomposes dense human references into atomic visual claims, verifies generated captions against those claims, and induces a quartile-wise composition of supported, hallucinated, and generic content. This lens reveals two qualitatively different failures in long captions: models tend to produce unsupported specificity in the middle, while the final portion often escapes into vague, non-evidential language rather than simply accumulating more hallucination. Dense human references do not exhibit the same position-dependent pattern under the identical pipeline, suggesting that the effect is a model behavior rather than an artifact of long descriptive text. Motivated by this diagnosis, we propose *PAGER* (**P**osition-**A**ware **G**rounded **E**vidence **R**einforcement), a GRPO training objective with separate signals for supported fine-grained middle evidence, evidence-bearing tail continuation, claim coverage, and content-aware length control. The result is a closed loop: the same ordered claim representation identifies where long captions fail and supplies the position-specific reward needed to repair those failures. Experiments on detailed-caption benchmarks show that PAGER improves position-wise grounding and pairwise caption preference while preserving competitive global caption quality.

Multimodal Large Language Models (MLLMs) have achieved remarkable progress, yet visual hallucination remains a critical barrier to reliable deployment. This paper aims to answer a fundamental question: when MLLMs hallucinate, where are they actually looking? By tracing the inter-layer consistency of visual attention, we reveal that hallucinated reasoning trajectories bifurcate into two pathological states: Attention Locking, where the gaze becomes overly focal and rigidly anchored to limited visual evidence, and Attention Collapse, where attention becomes excessively dispersed and fails to ground reasoning in meaningful cues. In contrast, faithful reasoning maintains a dynamic equilibrium between established visual evidence and broader peripheral exploration. Building on this insight, we propose ReGaze, a training-free framework that redirects the model's gaze toward this healthy equilibrium to mitigate hallucinations. Specifically, ReGaze monitors layer-wise gaze consistency and triggers timely interventions whenever an unhealthy gaze state emerges. For the locked state, ReGaze redistributes attention from over-dominant critical tokens to peripheral regions, encouraging the model to explore richer visual details. For the collapsed state, it gathers scattered attention from non-critical tokens and injects it into critical anchors, amplifying key visual signals. Experimental results demonstrate that ReGaze enables MLLMs to process visual information more effectively and significantly mitigates hallucinations.

Tabular prior-fitted networks (PFNs) work well across tabular classification problems but can fail silently at deployment when distributions shift. We propose **CALD** (Calibrated Anchor-Loss Disagreement), a lightweight unsupervised monitor that triggers an alarm to warn when a PFN might fail. By finetuning the decoder to maximize disagreement with the test-set prediction while maintaining accuracy on an ID anchor set, **CALD** identifies harmful OOD batches through the resulting anchor loss, requiring no labels on test batches. We prove that the post-finetune anchor loss has a strictly lower expectation under harmful shift than under ID eval, yielding a one-sided test with a formal power guarantee. Across 14 datasets covering tabular and structured distribution shift, on both TabPFN and TabICL, **CALD** matches or outperforms baselines on harmful shifts and is markedly more resilient to benign-shift false alarms.