NeurIPS 2026 Papers

Dynamic runtime latency and memory constraints necessitate flexible large language model (LLM) deployment, where an LLM can be inferred with various quantization precisions based on available computational resources. Recent work on such any-precision quantization either relies on hardware-inefficient vector quantization or induces additional scaling factors when switching between bit-widths. Meanwhile, existing post-training quantization (PTQ) methods calibrated for a fixed low precision show poor generalizability under runtime precision change. In this work, we attribute the source of poor generalization across bit-widths to a precision-dependent \textit{outlier migration} phenomenon where the distribution of PTQ-sensitive tokens changes across precisions. Motivated by this observation, we propose \texttt{MoBiQuant}, a novel any-precision Mixture-of-Bits quantization framework that adjusts weight precision for flexible LLM inference based on token sensitivity. Specifically, we propose a many-in-one recursive residual quantization that can iteratively reconstruct higher-precision weights at runtime and mitigates \textit{outlier migration} with a token-aware router to dynamically select the optimal inference precision of each token. Extensive experiments show that \texttt{MoBiQuant} matches or surpasses frontier single-precision PTQ while exhibiting strong elasticity, achieving significant memory savings and throughput gains of up to $1.34\times$ over state-of-the-art any-precision methods. Source code is provided in the supplementary material.

The vanilla self-attention mechanism in Transformers can be viewed as a two-layer fast-weight MLP, whose weights are dynamically induced by inputs and whose hidden dimension is equal to the sequence length $N$. As the context extends, the expressive capacity of such an $N$-width MLP increases, but it becomes unscalable for extremely long sequences. Recently, this fast-weight perspective has motivated the Mixture-of-Experts (MoE) attention, which partitions the sequence into rigid blocks, treats them as fast-weight experts, and sparsely routes the tokens to them. In this paper, we elevate this perspective to a unifying framework for efficient attention mechanisms, interpreting them as making fast weights scalable through either routing or compression, and organizing them into a five-dimensional taxonomy. Then, we propose \textbf{Mi}xture-of-\textbf{T}op-$k$ \textbf{A}ttention (\textbf{MiTA}), which employs a small set of landmark queries to gather top-$k$ attended key-value pairs as query-aware and deformable routed experts, while compressing the $N$-width MLP into a narrower shared expert. Consequently, MiTA improves the flexibility of prior MoE attention, from rigid to deformable fast-weight experts, as well as the scalability of prior top-$k$ attention, from query-specific set to reusable top-$k$ set. Our experiments on vision tasks demonstrate the superior effectiveness and efficiency of MiTA, while also uncovering intriguing properties such as an emergent token-pruning effect and easy generalization from standard attention.

Existing Mamba-based diffusion backbones struggle to achieve competitive performance in vision tasks, as purely sequential propagation lacks an explicit spatial inductive bias and relies on a fixed computation pattern across timesteps. We address this limitation by proposing \textbf{Mixture-of-Hierarchical Experts (MoH)}, a Mamba-based diffusion backbone that performs timestep-conditioned routing over architectural components. MoH dynamically selects representation levels by its hierarchical expert space, adapts feature transformations, and adjusts Mamba scan depth according to the denoising state, enabling flexible modeling of spatial structure and multi-scale dependencies. This design leads to substantial performance gains. On unconditional CelebA-HQ $256 \times 256$, MoH reduces FID from 14.27 to 7.99, and on MS-COCO, it improves FID from 41.80 to 20.00, significantly outperforming prior Mamba-based models while improving training efficiency. These results suggest that routing over architectural components can provide an effective mechanism for improving Mamba-based diffusion backbones.

Many partially observed data streams are globally high-rank yet locally low-rank: user–item interactions in recommender systems span multiple preference groups, distinct operating regimes in sensor and network monitoring induce different low-dimensional structures, and the active latent subspace in adaptive systems shifts over time. We formalize this as online matrix completion on a drifting union of subspaces: each sample lies in one of K low-dimensional subspaces, only a random subset of its coordinates is observed, and the subspaces drift across epochs. We propose MoSAIC (MoE Subspace Adaptation via Incremental Completion), a method based on a routed mixture of low-rank experts trained in two phases. A base model is first pre-trained on samples from a fixed source distribution. It is then adapted on the non-stationary stream by routing each incoming sample to a single expert and updating only that expert. Our analysis identifies sufficient conditions under which the router remains correct with high probability throughout learning, while within each epoch the routed experts contract toward the current subspaces at a sublinear rate. The technical core is a uniform routing concentration argument that converts the random time steps on which each expert is updated into a deterministic time scale, reducing the per-expert analysis to a tractable stochastic recursion. Experiments on streams with repeated non-stationary changes corroborate the theory and show clear improvements over competitive baselines.

Multimodal Large Language Models (MLLMs) are typically designed under the assumption that all modalities available during training will also be accessible at inference. However, many real-world settings violate this assumption, requiring models to operate under a privileged modality setting, where auxiliary modalities are available only during training. While these modalities contain valuable information, existing MLLMs largely fail to leverage them effectively, as they treat modalities as interchangeable inputs rather than sources of complementary supervision. We propose Mixture of Probes (MoP), a novel framework that disentangles modality-specific and modality-general signals within the MLLM, allowing the model to preserve modality-dependent structure while learning transferable representations across modalities. At its core, MoP achieves this through a structured probing mechanism that extracts and organizes information from intermediate representations of a shared modality encoder, rather than relying only on final-layer alignment as done in existing MLLMs. To support this disentanglement, we further introduce MoP Cross-modal Probe Training (MoP-X), a training strategy for MoP centered around a probe disentanglement loss that prevents probe collapse and encourages cross-modal learning. We evaluate MoP across two domains spanning eight tasks and four modalities under a comprehensive evaluation protocol tailored to the privileged modality setting, where each modality is independently treated as the sole input at inference time. MoP consistently outperforms strong MLLM baselines, achieving up to 65% relative improvement, demonstrating that auxiliary modalities, even when unavailable at inference, can provide substantial gains when effectively leveraged during training.

Pre-trained vision encoders contain layer-wise visual representations that differ in spatial granularity, semantic abstraction, and sensitivity to local details. However, most Multimodal Large Language Models (MLLMs) rely only on the final or penultimate vision encoder representations or fixed aggregation rules, making visual abstraction largely query-agnostic and limiting access to fine-grained cues such as small objects, spatial details, text, and subtle visual attributes. In this work, we propose **Mixture of Layers (MoL)**, an instruction-conditioned layer routing approach that dynamically aggregates query-relevant latent representations from intermediate vision encoder layers. Given a text query, **MoL** predicts routing probabilities over vision encoder layers and aggregates selected hidden states at either the image level, patch level, or through a hybrid routing mechanism. Our proposed **MoL** is a vision encoder-agnostic framework for layer-wise routing that selectively samples visual features useful for fine-grained visual reasoning tasks. Experiments across seven fine-grained visual reasoning benchmarks demonstrate substantial performance improvements, including +18.9% on V* overall accuracy, +4.5% on HRBench4K, and +16.3% on CharXiv compared to baseline MLLMs, *without* requiring multi-resolution inputs, simple interleaving of multiple vision encoders, or additional patch tokens. We further analyze receptive field scales and routing behaviors across vision encoder layers to explain why adaptive layer selection improves perception and reasoning. Our results suggest that conditional intermediate-layer representations are a key step toward stronger visual perception and reasoning in MLLMs. Code will be open-sourced.

Composed Image Retrieval (CIR) enables users to express specific search intent by combining a reference image with manipulation text. Despite its promise for e-commerce, existing methods ignore readily available item textual descriptions, suffer from multi-attribute confounding where samples exhibit uncontrolled variations in unrelated attributes, and lack explicit fine-grained attribute modeling. We propose $\textbf{MoA}^{3}\textbf{CIR}$, featuring an asymmetric five-tower architecture that incorporates item descriptions on both query and target sides, a self-reflective data augmentation pipeline generating controlled attribute modifications through LVLM-driven refinement, and a Mixture of Attribute-Aware Attention mechanism with grouped multi-head attention and dynamic routing coupled with a two-phase training strategy. Besides, we contribute ECCIR, the first dual-modal e-commerce CIR benchmark with 110K samples featuring authentic item descriptions. Extensive experiments demonstrate that \method~achieves SoTA performance on ECCIR, FashionIQ, and Shoes datasets, with substantial improvements over existing methods. Code and datasets will be made publicly available.

Existing latent graph diffusion models often learn graph representations and generative priors within a homogeneous latent geometry. This assumption is restrictive for graph generation with structural heterogeneity, where hierarchical, cyclic, locally regular, and densely connected patterns may coexist within the target distribution. Under such conditions, forcing all structural patterns into one latent geometry can introduce representation distortion, which may then propagate to generation. We propose CurvDiff, a latent graph generation framework based on mixed-curvature latent diffusion. The main idea is to model structurally different graph patterns in geometry-compatible latent factors rather than representing them in a single-curvature space. Specifically, CurvDiff learns graph embeddings in a product latent geometry consisting of hyperbolic, spherical, and Euclidean components, and maps the resulting representation to a shared tangent-space interface on which latent diffusion is defined. This provides a consistent latent pathway from representation learning to generation. To further preserve topology-relevant structure, CurvDiff uses Ricci-derived structural signals as topology-aware priors. These signals regularize latent representation learning and condition the diffusion prior, helping maintain structural consistency across representation learning and generation. Experiments on synthetic and real-world graph benchmarks show that CurvDiff achieves competitive or superior graph distribution matching compared with strong baselines, with improved fidelity on degree, clustering, and spectral statistics. These results support mixed-curvature latent diffusion as an effective approach to graph generation with structural heterogeneity.

Predicting the olfactory perception of scent mixtures remains a fundamental challenge in computational neuroscience. Existing methods derive mixture-level features by encoding individual components with a single-molecule feature encoder pretrained on annotated olfactory data and aggregating these features via mean pooling, concatenation, or self-attention. This paradigm faces two critical limitations: the scarcity of annotated olfactory data and the inability to capture complex interactions among mixture components. We present MixScentNet, a multiscale graph neural network framework, to address these challenges with two key innovations. First, we introduce a self-supervised pretraining strategy that uses molecular graph structures with RDKit features to predict corresponding Mordred descriptors, enabling the model to learn rich physicochemical knowledge without relying on scarce annotated olfactory data. Second, we propose a novel mixture-as-graph paradigm to model the constituent molecules as nodes in the mixture-level graph, aligning with the fact that humans holistically perceive scent mixtures. We further process the graph via a graph attention network V2 (GATv2) to capture the high-order molecular interactions. MixScentNet differs from existing methods in terms of its pretraining strategy and ability to capture mixture-level features, achieving state-of-the-art performance on both mixture-level olfactory label prediction and perceptual distance estimation tasks. We also find that MixScentNet can reproduce olfactory white phenomena, indicating that the model enjoys clear interpretability grounded in psychophysics. The demo code is available in the Anonymous link \url{https://anonymous.4open.science/r/Odor-D2DB}

Neural Combinatorial Optimization (NCO) is a promising paradigm for solving routing problems, yet its generalization across diverse data distributions remains a critical bottleneck. Existing methods typically tackle cross-distribution generalization through complex multi-distribution training or intricate meta-learning schemes, implicitly presuming that single-distribution training yields a weak zero-shot baseline. In this paper, we demonstrate that the potential of single-distribution training has been substantially underestimated and can be unlocked by a simple architectural inductive bias: Mix Normalization. To approach this, we first construct a comprehensive benchmark spanning three representative routing problems and 179 fine-grained datasets, capturing diverse shifts in node coordinates, customer demands, and time windows. Building on this benchmark, we propose MixRoute, a neural framework that adaptively combines normalization statistics at different granularities to mitigate varying distribution shifts. Trained exclusively on uniform distributions, MixRoute generalizes effectively to a wide range of unseen distributions without any distribution-specific adaptation. Extensive experiments demonstrate that MixRoute consistently achieves state-of-the-art zero-shot generalization. Our work revisits normalization as a parsimonious alternative for zero-shot generalization against complex training-adaptation schemes.

Detecting AI-generated videos (AIGV) in a generator-agnostic manner is increasingly important as generative models close the gap with real footage, and a video, with many frames, intuitively offers richer forensic evidence than a single image. Yet existing detectors fall short of this expectation: under the encode-then-fuse paradigm, increasing the frame budget from 1 to 8 improves a voting-based image detector by only 5.28%, and even temporal-modeling backbones such as TimeSformer barely improve on this. Per-frame predictions are also noisy: per-frame logits fluctuate substantially within a video and correlate weakly with any frame-level proxy. We trace the cause not to the aggregation step but to per-frame encoding severing multi-frame cues before they can interact, leaving any post-hoc selection, weighting, or aggregation strategy without a reliable basis. We therefore propose **MixForensics**, a blend-then-encode framework that combines multiple frames in the pixel domain into a single composite image (Stochastic Frame Blending, SFB) and regularizes the encoder for consistency across blended views (Blend Invariance Regularization, BIR). To stress-test this approach, we further introduce **MixForensics-Bench**, a test-only benchmark covering ten recent generators (including five closed-source commercial platforms) and two partial-forgery scenarios (temporal splicing and conditional continuation), with matched controls that isolate generative artifacts from editing artifacts. On AIGVDBench and MixForensics-Bench, MixForensics outperforms the strongest baseline by 1.27% and 8.11% on the fully-fake splits, leads on both partial-forgery scenarios by 11.49% on temporal splicing and 9.15% on conditional continuation, and uses fewer encoder forward passes than per-frame methods.

Memory-efficient personalization is essential for adapting text-to-image diffusion models while preserving user privacy and operating within the limited computational resources of edge devices. To this end, we propose Mix-Opt, a novel mixed optimization framework that adaptively chooses between backpropagation on low-resolution images (BP-low) and zeroth-order optimization on high-resolution images (ZO-high), guided by the characteristics of the diffusion process. As observed in our experiments, BP-low efficiently adapts the model to target-specific features, but suffers from structural distortions due to resolution mismatch. Conversely, ZO-high refines high-resolution details with minimal memory overhead but faces slow convergence when applied without prior adaptation. By complementing both methods, our framework leverages BP-low for effective personalization while using ZO-high for structural consistency, achieving memory-efficient and high-quality fine-tuning. To maximize the efficacy of both BP-low and ZO-high, we introduce a timestep-aware probabilistic function that dynamically selects the appropriate optimization strategy during training. This function mitigates the overfitting from BP-low at high timesteps, where structural information is critical, while ensuring ZO-high is applied more effectively as training progresses. Experimental results demonstrate that our method achieves competitive performance while significantly reducing memory consumption, enabling scalable, high-quality personalization without increasing inference latency.

Language models fine-tuned with reinforcement learning frequently learn to exploit their reward source rather than solve the underlying task. Existing fixes are typically reactive: detect a specific exploit, then patch the reward. We instead ask whether reward hacking can be prevented by constraining how the model represents the task. We introduce prompt KL regularization, a single auxiliary loss that adds a KL penalty between the actor's and reference model's per-token distributions on the prompt only, leaving the response distribution free. Across three reward hacking tasks of different sizes, prompt KL regularization maintains low hack rates (under 3\%) when standard training yields up to 100\% hack rates. Additionally, it matches or improves ground-truth accuracy, with no changes to the reward or environment. We aim to understand why prompt KL regularization is effective. Two independent mechanistic studies suggest that changes in prompt representations are key to models learning reward hacking behavior. First, we show that prompt-space activation steering Pareto-dominates response-space steering on hack rate and accuracy. Then, we demonstrate that swapping prompt activations between models can transfer and reduce reward hacking. Together, these results suggest that the way models internally represent the task is central to reward hacking, and that this representation can be directly constrained to prevent it.

Reinforcement learning agents trained to maximize their own reward in repeated interactions can converge to supra-competitive outcomes resembling explicit collusion, without communication or shared design. Existing mitigation approaches are largely tied to specific economic settings, like two-sided platforms and auctions, leaving open how to design interventions for general repeated games. We address this gap by formalizing the connection between empirical observations from prior work on Q-learning collusion and classical theory of Simple Penal Codes (SPCs). We show any non-trivial SPC induces a quantifiable conditional dependence in agents' policies, detectable via the total variation distance between an agent's action distributions across cooperation and defection histories. Building on this connection, we propose CURB (Collusion Unwinding via Reward shaping and Belief injection), a reward-shaping framework that penalizes this Total Variation (TV) distance signal during Q-learning and is guaranteed to convert any SPC fixed point of the dynamics into a trivial one, thus precluding collusive equilibria sustained by punishment threats. Empirically, CURB substantially reduces collusion by Q-learning agents in both Bertrand and Cournot Competition Repeated Games. We further demonstrate that CURB extends to deep Q-network agents in Bertrand competition, suggesting the mechanism generalizes beyond tabular Q-learning.

Large Reasoning Language Models (LRLMs) leverage long Chain-of-Thought (CoT) reasoning to solve complex tasks, yet often engage in overthinking. Specifically, they continue to generate reasoning steps that provide meaningless contribution to or even degrade the final answer, even after sufficient reasoning has been produced. Early-exit strategies are proposed to dynamically terminate meaningless or even harmful reasoning, aiming to improve both efficiency and accuracy. However, existing methods either suffer from over-truncation that degrades accuracy below vanilla CoT, or yield only spurious efficiency gains, which reduce token counts but increase wall-clock inference time. We observe that when an LRLM's reasoning path deviates into meaningless or even harmful wandering, this deviation is accompanied by an anomalous surge of high-entropy transition tokens. Building on this insight, we propose RPDI-EE, a training-free early-exit method that monitors the Reasoning Path Deviation Index (RPDI), defined as the ratio of local to global average entropy, which serves as a proxy for reasoning path deviation. By tracking this index, RPDI-EE detects and terminates likely meaningless or even harmful reasoning while preserving productive inference. Experiments across multiple benchmarks and four LRLMs of varying types and scales show that RPDI-EE achieves the largest average accuracy improvement over vanilla CoT among all tested early-exit methods, while mitigating the spurious efficiency gains of existing alternatives.

Random Network Distillation (RND) is a scalable novelty signal for reinforcement learning, but it can overgeneralize: the predictor extrapolates the random target off the data manifold, causing novelty scores to collapse on unfamiliar inputs. This paper analyzes this failure through a kernel-theoretic lens. Under a GP target model and an NTK-regime predictor, we show that the expected RND energy is a cross-kernel residual governed jointly by the target covariance kernel and the predictor interpolation kernel. This identifies overgeneralization as spectral under-excitation of modes where predictor residuals would otherwise survive. We then explicitly formulate spectral target design as a principle for shaping RND residual geometry; one concrete instance is replacing to replace the implicit random-network target with a bandwidth-controlled Random Fourier Feature (RFF) target. Numerical experiments across offline D4RL and online Atari benchmarks show this target-design approach improves novelty discrimination and downstream RL performance.

Graph Neural Networks (GNNs) often struggle to capture long-range dependencies due to over-squashing — a phenomenon in which the repeated compression of node embeddings into finite-size messages causes representations to collapse. Oversquashing is most often diagnosed as a property of the graph topology, with effective resistance serving as a principled measure of the bottleneck. We provide a complementary view on the matter: building on cellular sheaves, we introduce sheaf effective resistance, a generalization of effective resistance that depends on the sheaf attached to the graph, and we prove that for flat vector bundles it is inversely proportional to over-squashing sensitivity in the Jacobian sense. The bottleneck thus need not lie in the graph itself: it can be relocated, and reduced, by adjusting the sheaf. We instantiate this idea in FlatNSD, a simple message-passing variant of Neural Sheaf Diffusion, and show that it implicitly learns to modulate total sheaf effective resistance, performing well on benchmarks designed to stress over-squashing without altering the original graph topology.

Large language models (LLMs) can memorize and reproduce training sequences verbatim, which undermines both generalization and privacy. Existing mitigation methods apply interventions uniformly, often degrading performance on the majority of tokens that generalize normally. We empirically find that memorization is sparse and heavy-tailed at the token level. This structure is mismatched to static, sequence-level interventions: effective mitigation must localize to where memorization actually happens. From the ideal token-level memorization reduction objective, we derive a probe-steer framework, which decomposes intervention into a probe that detects memorization-relevant activations and a steer that applies targeted correction only when the probe exceeds a threshold. We propose Gated Subspace Steering (GSS) as its practical instantiation: the optimal probe-steer pair admits a closed-form solution as the leading singular vectors of a gradient-weighted activation matrix. Across four benchmarks, GSS matches or exceeds state-of-the-art memorization reduction while requiring 10--100$\times$ less compute than optimization-based alternatives.

Large reasoning models (LRMs) improve language model capabilities by generating explicit thinking traces before final answers. In factuality-oriented question answering (QA), such thinking often improves overall performance by helping the model recover relevant knowledge and refine its answers. However, we find that this benefit is not uniform at the instance level: explicit thinking can also overturn correct non-thinking answers and lead to factual drift. We refer to this failure mode as \emph{thinking-induced hallucination}. To explain this phenomenon, we formulate explicit thinking in factuality QA as a thinking residual over the model's direct-answer tendency, which can either recover missing knowledge or introduce unsupported associations. Based on this formulation, we propose MARGO, \underline{\textit{M}}ixed-Mode \underline{\textit{A}}dvantage \underline{\textit{R}}egularization for \underline{\textit{G}}rounded \underline{\textit{O}}ptimization, a reinforcement learning framework that uses non-thinking rollouts as same-model references in advantage estimation. By constructing mixed-mode rollout groups with both thinking and non-thinking trajectories, MARGO evaluates whether explicit thinking adds factual value beyond direct answering, thereby suppressing hallucination-prone thinking while preserving beneficial thinking behaviors. Experiments across multiple factuality-oriented QA benchmarks demonstrate that MARGO improves factual reliability over strong baselines, while evaluations on mathematical benchmarks show that it preserves general reasoning ability.