NeurIPS 2026 Papers

State-of-the-art Extreme Multi-Label Text Classification (XMC) models rely on multi-label attention to focus on key tokens in input text, but learning high-quality attention weights is challenging. We introduce PLANT (Pretrained and Leveraged Attention), a plug-and-play strategy for initializing attention. PLANT works by \emph{planting} label-specific attention using a pretrained Learning-to-Rank model guided by mutual information gain. This architecture-agnostic approach integrates seamlessly with large language model backbones such as Mistral, LLaMA, DeepSeek, and Phi-3. PLANT outperforms state-of-the-art methods across tasks such as ICD coding, legal topic classification, and content recommendation. Gains are especially pronounced in few-shot settings, with substantial improvements on rare labels. Ablation studies confirm that attention initialization is a key driver of these gains. Code and trained models are available at \url{https://github.com/research-anon-487/xcube/tree/plant}.

Real-world machine learning systems often operate on imperfect measurements: sensors are noisy, annotations are produced by non-experts, and scientific observations are often indirect. In these settings, classic distribution comparison tools such as maximum mean discrepancy (MMD) can be systematically biased, matching artifacts of the measurement process rather than the underlying mechanisms of interest. We study a practical regime in which noisy observations are abundant, but only a small audited subset contains ground-truth measurements. We propose measurement-error-corrected MMD (MEC-MMD), an audit-based estimator that debiases kernel evaluations without requiring a noise model. MEC-MMD is exactly unbiased for the oracle MMD under simple random auditing, admits an explicit variance decomposition, and is consistent for the population MMD, achieving the standard $O_p(N^{-1/2})$ rate when the audit size scales with the dataset. We demonstrate the utility of distributional matching with MEC-MMD loss for both parameter estimation and unsupervised applications, showing that MEC-MMD consistently outperforms MMD and alternative methods. These results provide a principled foundation for robust distribution comparison, simulation-based inference, and learning in noisy real-world machine learning pipelines.

Large language models (LLMs) are often ensembled together to improve overall reliability and robustness, but in practice models are strongly correlated. This raises a fundamental question: which models should be selected when forming an LLM ensemble? We formulate budgeted ensemble selection as maximizing the mutual information between the true label and predictions of the selected models. Furthermore, to explain why performance can saturate even with many models, we model the correlated errors of the models using a Gaussian-copula and show an information-theoretic error floor for the performance of the ensemble. Motivated by these, we propose a simple greedy mutual-information selection algorithm that estimates the required information terms directly from data and iteratively builds an ensemble under a query budget. We test our approach on four datasets spanning binary and multi-class classification and sentiment analysis: MEDMCQA, MMLU, AG News, and IMDB. Across all datasets, we observe that our method consistently outperforms strong baselines under the same query budget.

Zeroth-order (ZO) optimization enables memory-efficient fine-tuning of large language models (LLMs) using only forward passes, but it remains unclear how useful adaptation is distributed across layers. In this work, we reveal a surprising phenomenon: ZO fine-tuning is sharply dominated by a single decoding layer. Across multiple LLM families and downstream tasks, fine-tuning this dominant layer alone consistently matches or even exceeds full-model ZO fine-tuning. We further show that the dominant layer is task-agnostic but model-specific, and can be identified before training through a simple inference-only analysis of activation outliers. Specifically, the dominant layer consistently aligns with the first activation-outlier layer in the pre-trained model. To explain this phenomenon, we analyze how perturbation effects propagate under ZO optimization. We find that the dominant layer combines two key properties: high perturbation sensitivity and early placement in the residual stream, allowing perturbation-induced effects to propagate and accumulate through remaining subsequent decoding layers. As a result, this layer produces disproportionately strong and stable optimization signals under forward-only updates. Extensive experiments on LLaMA2-7B and Qwen3-8B across nine benchmarks show that dominant-layer ZO fine-tuning improves average performance over full-model MeZO and LoRA-based ZO fine-tuning while achieving up to 4.52$\times$ training speedup.

Modern visual recognition systems increasingly rely on unlabeled images collected from diverse visual domains. A common goal in dataset construction is to avoid obvious marginal imbalance, such as skewed class totals or uneven domain sizes. However, marginal balance can be misleading in multi-domain data: even when per-class totals and per-domain sample sizes are balanced, sample allocations can be highly skewed across domain-class pairs. We call this hidden conditional skew Domain-Conditioned Class Imbalance (DCI), where class imbalance is revealed only after conditioning on domain context. For controlled evaluation, we fix class and domain marginals and vary only the joint domain--class allocation, making the effect of domain-conditioned imbalance directly measurable. To address this challenge, we propose HyperIDAC, a hypernetwork-based classifier for multi-domain imbalance. HyperIDAC infers domain priors from visual features and lightweight zero-shot predictions, then dynamically generates instance-specific classifier parameters and domain-conditioned class embeddings. This design yields domain-sensitive decision boundaries while leveraging CLIP semantics to support underrepresented domain--class pairs. Furthermore, a multi-level pseudo-labeling module combines teacher–student training with historical label tracking and a fallback for low-confidence samples, mitigating bias reinforcement in unlabeled settings. Extensive experiments on four multi-domain benchmarks under three regimes show that HyperIDAC substantially outperforms state-of-the-art methods, highlighting the value of domain-aware hypernetwork-based adaptation under DCI.

Difficulty perception is essential for adaptive reasoning in large language models (LLMs). Previous studies rely on training auxiliary models or using extra reasoning rollouts to estimate difficulty, which incurs high computational costs. In this paper, we identify an intrinsic property of LLMs: their internal representations, even before explicit reasoning, encode an informative and useful signal that correlates with problem difficulty. Inspired by this property, we propose Referential Latent Difficulty Perception (RLDP), a training-free and rollout-free method that estimates difficulty directly from hidden activations in a single forward pass, and requires only minimal reference problems. Additionally, we introduce RLDP-AdaSwitch, a lightweight controller that dynamically allocates reasoning effort based on the difficulty signals provided by RLDP, enabling efficient trade-offs between accuracy and compute. Our experimental results across multiple LLMs and diverse datasets, including math reasoning, code generation, and QA, demonstrate that RLDP provides stable and effective difficulty discrimination. This further powers RLDP-AdaSwitch to achieve 1.34×–2.00× efficiency compared to rollout-based methods, while matching the performance of training-based methods.

In modern deep learning, weight decay is often credited with "stabilizing" training dynamics, diverging from its classical role as a static regularization penalty. We investigate a fundamental question: does weight decay stabilize training dynamics, and if so, through which mechanism? Indeed, training stability is understood through different but related notions in the literature. We consider how weight decay affects the parameter-space dynamics and loss sharpness by analyzing its effects at the Edge of Stability (EoS). We show that weight decay robustly slows progressive sharpening. Furthermore, we uncover a striking architecture-dependent phase transition. In CNNs, weight decay dampens the oscillations at the EoS, while in MLPs, increasing weight decay causes a phase transition in which the sharpness stabilizes at a threshold significantly below the theoretical $\frac{2}{\eta}$ boundary. We develop a mathematical framework that accurately models these phenomena and identify the global alignment of the parameter vector and the sharpness gradient as the mechanistic driver of the phase transition. Importantly, we show that these phenomena translate into stability in terms of search in function-space (NTK).

Policy gradient computes a backward pass for every sample, even though the backward pass is expensive and most samples carry little learning value. The Delightful Policy Gradient (DG) provides a forward-pass signal of learning value: \emph{delight}, the product of advantage and surprisal (negative log-probability). We introduce the \emph{Kondo gate}, which compares delight against a compute price and pays for a backward pass only when the sample is worth it, thereby tracing a quality--cost Pareto frontier. In bandits, zero-price gating preserves useful gradient signal while removing perpendicular noise, and delight is a more reliable screening signal than additive combinations of value and surprise. On MNIST and transformer token reversal, the Kondo gate skips most backward passes while retaining nearly all of DG's learning quality, with gains that grow as problems get harder and backward passes become more expensive. Because the gate tolerates approximate delight, a cheap forward pass can screen samples before expensive backpropagation, suggesting a speculative-decoding-for-training paradigm.

Gradient descent on overparameterized neural networks typically operates at the Edge of Stability (EoS), where the largest Hessian eigenvalue hovers around a step-size-dependent threshold. We study how sparse connectivity changes generalization below this threshold in two layer ReLU networks. Prior results have shown that for fully-connected networks (FCNs), generalization guarantees in this regime degrade and become vacuous on high-dimensional spherical inputs. Our analysis reveals that sparse connectivity fundamentally alters this picture. Under sparse connectivity, the network processes a collection of low-dimensional patches rather than the full input vector, so the effective constraint imposed by the stability condition is governed by the geometry of the training patch collection. We prove that when the receptive fields are small relative to the ambient dimension, the effective constraint yields non-vacuous generalization bounds in precisely the spherical regime where FCNs provably fail. The same framework also reveals a contrasting failure mode: if the patch collection lacks geometric structure, the constraint becomes unable to prevent overfitting. We corroborate this theory by analyzing the patch geometry of natural images, showing that standard convolutional designs produce patch multiset with low-dimensional structure that facilitates generalization. This provides a principled explanation for the generalization advantage of convolutional networks. Thus, our analysis yields a unified framework that identifies how architecture, data geometry, and gradient descent jointly govern generalization performance.

*Minority collapse*, where minority classes become indistinguishable, is a significant challenge in imbalanced learning. This challenge is addressed by methods such as Mixup with class-balanced sampling. Although minority collapse has been mathematically analyzed using the layer-peeled model alongside Neural Collapse, no prior work has analyzed minority collapse under Mixup, particularly from the perspective of mixed labels. We investigate this overlooked factor and raise the question: *Is mixed label balance important for alleviating minority collapse?* Our analysis reveals that (i) mixed labels should be balanced, and (ii) in this setting, interpreting mixed labels as singletons is beneficial. Motivated by this analysis, we propose a *Balanced Mixed Label Sampler* and a *Mixed-Singleton classifier*, which balance mixed labels and treat them as singleton labels. Through theoretical analysis and experimental results, we highlight the importance of balancing mixed labels in imbalanced learning.

Flow cytometry is the primary assay for single-cell immune profiling, yet cross-study modelling remains under-explored. The challenge is two-fold: Studies measure different marker panels, inducing feature-space inconsistency, while batch effects shift signals even on shared markers, making transfer harder than in vision or language where input spaces are stable. We conduct a large-scale empirical study, pretraining an encoder on 89,324 samples from 112 public studies spanning 525 distinct marker panels. We evaluate it frozen --- isolating representation quality from task-specific adaptation, on five clinical tasks under a taxonomy of panel-overlap conditions, including three panel-zero-shot tasks whose marker panels were unseen during pretraining, and one negative control with no expected discriminative signal. We further characterise transfer across scaling axes and probe robustness to marker removal at test time. Despite heterogeneity that disrupts direct transfer, the frozen encoder matches or exceeds in-distribution specialist encoders on 4 of 5 tasks, including all three novel-panel tasks, and behaves as expected on the negative control. Scaling analyses reveal saturation in both encoder capacity and random corpus subsampling, and representations degrade gracefully under marker reduction. These results suggest a pretrained encoder can serve as a general-purpose feature extractor for clinical cytometry data, without task-specific retraining.

A central question in modern learning theory is what notion of compression, if any, explains why over-parameterized neural networks generalize. Existing answers measure complexity at different levels, ranging from weights to representations, and estimating the associated quantities either requires held-out data in principle or relies on mutual-information estimation in ultra-high dimensions. We approach the question through the minimum description length (MDL) principle and propose the prequential regret as a complementary signal that is computable from the training trajectory alone. We establish that, in the (nearly) interpolating regime, the expected prequential regret upper-bounds the description length of the trained parameter under the algorithm-induced distribution, and that this description length translates into a high-probability generalization bound in the PAC-MDL setting. Along the way, we show that the encoder-complexity correction recently added to the algorithm-level information bottleneck is upper-bounded by an MDL parameter description length, bringing the two views of compression onto a shared footing. Empirically, the prequential regret tracks the generalization error on CIFAR-10 and CIFAR-100, with Pearson correlations exceeding 0.9.

Vision-Language-Action (VLA) models are increasingly used as general-purpose policies for robotic manipulation, but completing the instructed task requires more than reaching the target state. A policy must also avoid nearby objects throughout the rollout, including approach, interaction, transport, and handover. A common approach is to invoke conventional collision avoidance mechanisms, such as motion planners or safety filters, which provide effective geometric safeguards. However, the resulting system may struggle to produce actions that are simultaneously safe and task progressing. When collision avoidance and manipulation progress require misaligned actions, the system may avoid the obstacle while slowing, disrupting, or even failing the task. To address this, we introduce DODGE-IT, a collision-aware learning framework that leverages collision signals as policy-improvement feedback to strengthen the VLA policy's collision-aware manipulation capability. DODGE-IT first designs Task-Relevant Assessment of Collision Events (TRACE) that detects and identifies task-relevant obstacles from RGB-D observations and turns executions into collision outcomes. These outcomes are then incorporated together with task success signals into representative reinforcement learning frameworks for VLA. We validate DODGE-IT on SafeLIBERO and real robot manipulation tasks, showing improved obstacle avoidance while preserving task completion across simulated and physical settings.

Evaluating whether Multimodal Large Language Models can produce trustworthy, verifiable reasoning over long, visually rich documents requires evaluation beyond end-to-end answer accuracy. We introduce DocScope, a benchmark that formulates long-document QA as a structured reasoning trajectory prediction problem: given a complete PDF document and a question, the model outputs evidence pages, supporting evidence regions, relevant factual statements, and a final answer. We design a four-stage evaluation protocol—Page Localization, Region Grounding, Fact Extraction, and Answer Verification—that audits each level of the trajectory independently through inter-stage decoupling, with all judges selected and calibrated via human alignment studies. DocScope comprises 1,124 questions derived from 273 documents, with all hierarchical evidence annotations completed by human annotators. We benchmark 6 proprietary models, 12 open-weight models, and several domain-specific systems. Our experiments reveal that answer accuracy cannot substitute for trajectory-level evaluation: even among correct answers, the highest observed rate of complete evidence chains is only 29\%. Across all models, region grounding remains the weakest trajectory stage. Furthermore, the primary difficulty stems from aggregating evidence dispersed across long distances and multiple document clusters, while an oracle study identifies faithful perception and fact extraction as the dominant capability bottleneck. Cross-architecture comparisons further suggest that activated parameter count matters more than total scale. The benchmark and code will be publicly released.

The advent of Multimodal Large Language Models (MLLMs) has unlocked the potential for end-to-end document parsing and translation. However, prevailing benchmarks such as OmniDocBench and DITrans are dominated by pristine scanned or digital-born documents. They do not adequately represent the intricate challenges of real-world capture conditions, such as geometric distortions and photometric variations. To fill this gap, we introduce DocPTBench, a comprehensive benchmark specifically designed for Photographed Document Parsing and Translation. DocPTBench comprises over 1,300 high-resolution photographed documents from multiple domains, includes eight translation scenarios, and provides meticulously human-verified annotations for both parsing and translation. Our experiments demonstrate that transitioning from digital-born to photographed documents results in a substantial performance decline: popular MLLMs exhibit an average accuracy drop of 18\% in end-to-end parsing and 12\% in translation, while specialized document parsing models show a more prominent average decrease of 22\%. This substantial performance gap highlights the unique challenges posed by documents captured in real-world conditions and reveals the limited robustness of existing models. The benchmark and related code will be made publicly available for further research.

Long-document understanding requires models to find and combine evidence across many pages, layouts, tables, figures, and charts. Existing retrieval-augmented systems usually select evidence from a static index before generation, while recent agentic systems add multi-turn tool use but often rely on frozen proprietary backbones whose behavior is set by prompts. We present DocAtlas, a system that treats long-document understanding as a mutable-state information-seeking process. We instantiate DocAtlas as a mutable document harness: an external environment that determines what document information is searched, read, stored, reviewed, and shown to the model at each step. Given a document and question, the harness exposes search, reading, note-taking, and review tools, maintains a hierarchical tree and note store, and updates both as the agent records evidence. DocAtlas combines self-improving retrieval, selective evidence access, and active working memory under a fixed context budget. The same harness supports inference-time use with large VLMs and end-to-end reinforcement learning for compact VLM agents. With GPT-5.4, DocAtlas reaches 71.4\% on MMLongBench-Doc, exceeding the human-expert reference of 65.8\%. A Qwen3.5-4B VLM trained with end-to-end RL in the DocAtlas environment reaches 63.7\%, compared with a 54.4\% direct-input baseline, showing that mutable document-harness design can improve compact document agents by a large margin.

Real-time polarization imaging with off-the-shelf color division-of-focal-plane (DoFP) cameras requires demosaicking that is both accurate and deployment-friendly. Interpolation methods are lightweight but limited in polarization fidelity, whereas learning-based methods improve quality at much higher online cost. We propose DoFP-LUT, a sensor-structured LUT framework and, to our knowledge, the first LUT-based framework for real-time color DoFP polarization demosaicking. It leverages the deterministic and memory-light nature of LUT inference while aligning lookup corrections with DoFP-specific residual structures. Unlike generic LUTs designed for RGB restoration, DoFP-LUT targets post-initialization residuals that are coupled across analyzer orientations, Stokes relations, and local spatial structures. It factorizes teacher-guided correction into shared analyzer correction, polarization redistribution, and high-frequency compensation, compiling each component into compact low-dimensional LUTs offline. Online inference then requires only nearest-neighbor lookup and fixed write-back. Experiments show that DoFP-LUT achieves a favorable quality--efficiency trade-off, improving polarization reconstruction while preserving deterministic, memory-light inference for real-time polarization imaging.

Medical foundation models generate narrative explanations but cannot quantify intervention effects, detect evidence conflicts, or validate literature claims, limiting clinical auditability. We propose causal compilation, a paradigm that transforms medical evidence from narrative text into executable code. The paradigm standardizes heterogeneous research evidence into structured estimand objects, each explicitly specifying intervention contrast, effect scale, time horizon, and target population, supporting six executable causal queries: do-calculus, counterfactual reasoning, temporal trajectories, heterogeneous effects, mechanistic decomposition, and joint interventions. We instantiate this paradigm in DoAtlas-1, compiling 1,445 effect kernels from 754 studies through effect standardization, conflict-aware graph construction, and real-world validation (Human Phenotype Project, 10,000 participants). The system achieves 98.5% canonicalization accuracy, 100% conflict detection recall, and 80.5% query executability. This paradigm shifts medical AI from text generation to executable, auditable, and verifiable causal reasoning.

Yes. We find that large multimodal models develop mental imagery when solving spatial puzzles, and they do imagine sheep when solving sheep puzzles. We fine- tune a Qwen3.5 VLM to solve twelve diverse visual reasoning tasks—including tangram, jigsaw, sokoban, 3D mental rotation, and rush hour—that require under- standing geometry, spatial constraints, and the consequences of actions. By simply predicting the open-loop sequence of actions to solve a puzzle from an initial state, we show that the model learns successfully, and that its hidden representations after each action encode meaningful visual information about the intermediate puzzle state. This finding suggests that an imperfect visual world model begins to form as a byproduct of learning to select correct actions in an open-loop fashion, in the absence of any explicit visual supervison. Building on this observation, we propose two ways to sharpen the mental images learned by the model through explicit visual supervision or explicit visual tokens. Across puzzles, we find that integrating as few as sixteen visual tokens into the chain of thought per board state improves the average solve rate from 83% to 89%, with gains exceeding 20 percentage points on reasoning-heavy games such as jigsaw and 3D mental rotation. Our results show that visual reasoning is a bottleneck, and that mental imagery may be a key mechanism for robust spatial cognition.