NeurIPS 2026 Papers

We study the problem of learning Gaussian mixture models under overparameterization. Prior work has shown that, while overparameterization is essential for avoiding spurious local optima and enables global recovery of the ground-truth model using the gradient-EM algorithm, it can dramatically slow the local rate of convergence. Under certain assumptions on the mixture weights, we show that a standard divergence measure minimized by statistical learning procedures possesses a manifold of slow growth on which the well-known Polyak stepsize reduces the loss geometrically. We then design a gradient-based method that converges to minimizers at a locally linear rate. Additionally, we show that our method converges to nearly optimal solutions—up to a natural misspecification threshold—for mixtures with arbitrary weights. At a high level, the method alternates between several “short” gradient descent steps that approach the manifold and “long” Polyak steps that contract the distance to minimizers. Our results suggest that slow convergence is not an intrinsic challenge of overparameterization, but can be overcome by exploiting the favorable structure of the loss landscape.

Deployment-time safety methods for large language models (LLMs) are predominantly supervised and assume access to unsafe training data. Nevertheless, new attacks and harm categories regularly arise, not captured by models trained in such a supervised fashion. An alternative approach is to view this problem through the lens of anomaly detection, namely, to rely solely on modeling safe data and flagging out-of-distribution inputs. However, LLM activations lie in a high-dimensional space, raising concerns about whether anomaly detection is statistically feasible. We show that, under the linear representation hypothesis (LRH), there may indeed be hope. In the LRH concept space, which is typically recovered via a sparse autoencoder (SAE), nearby points share a small common active support. Based on this local sparsity insight, we propose a framework for locally masked SAE-based anomaly detection, and establish a sample complexity bound that scales polynomially in the local sparsity and only \emph{logarithmically} in the SAE's ambient dimension. Finally, we empirically validate our framework on various architectures and datasets, including both capability-testing datasets and safety-specific datasets.

Multi-objective reinforcement learning (MORL) aims at optimising several, often conflicting goals to improve the flexibility and reliability of RL in practical tasks. This is typically achieved by finding a set of diverse, non-dominated policies that form a Pareto front in the performance space. However, constructing such a policy set remains computationally demanding, as it requires finding not a single optimal policy, but a set of policies to cover a wide range of trade-offs among the objectives. We introduce LLE-MORL, an approach that traces policy manifolds by utilising the local relationship between the high-dimensional policy parameter space and the performance space. This structured representation enables an efficient search within contiguous solution domains via locally linear extrapolation, allowing for the rapid generation of high-quality solutions without extensive retraining. We also provide a theoretical analysis of the policy manifold structure in order to specify the applicability of the method. Experiments across diverse continuous control domains demonstrate that LLE-MORL consistently achieves higher Pareto front quality and efficiency than state-of-the-art approaches.

Diffusion models are prone to generating structural hallucinations - samples that match the statistical properties of the training data yet defy underlying structural rules, resulting in anomalies like hands with more than five fingers. Recent research studied this failure mode from several viewpoints, offering partial explanations to their occurrence, such as mode interpolation. In this work, we propose a complementary perspective that treats hallucinations as instabilities on the model-induced manifold. We begin by showing that a hallucination filter based on such instabilities matches or exceeds the performance of the recently proposed temporal one. By tracing the source of these instabilities, we identify local intrinsic dimension (LID) as their primary driver and propose Intrinsic Quenching (IQ), a direct corrective mechanism that deflates it to alleviate hallucinations. IQ consistently outperforms standard hallucination reduction baselines across a wide array of benchmarks and offers a highly promising solution for enforcing anatomical consistency in downstream medical imaging tasks.

Gaussian processes are a cornerstone of modern probabilistic machine learning, providing principled uncertainty quantification and flexible nonparametric modeling. While the Gaussian kernel is widely used in Euclidean spaces, many real-world problems involve data residing on non-Euclidean domains, particularly compact Lie groups that naturally encode \emph{symmetries}. In this work, we construct a local Gaussian kernel on arbitrary compact Lie groups by decomposing the group into a \emph{maximal torus} and its orthogonal complement. For the prominent Lie groups $\mathbf{SO}(3)$ and $\mathbf{SU}(2)$, we derive closed-form kernel expressions using the Rodrigues formula. Extensive experiments verify the positive definiteness of the proposed kernel and demonstrate its effectiveness in Gaussian process regression, achieving accurate predictions across the entire manifold.

Membership inference attacks (MIAs) are commonly formulated as single-sample hypothesis tests with false positive rate control. However, realistic adversaries often test many candidate samples and aggregate the declared members, making membership inference a multiple testing problem. In this regime, per-sample calibration can produce unreliable sets of inferred members. We develop a false discovery rate (FDR)-controlled framework for membership inference based on the classical local FDR formulation. We instantiate the framework for high dimensional ridge regression. Under a conditional sufficiency condition, we establish FDR guarantee of the proposed method, and derive an asymptotic characterization of its detection power in high-dimensional regimes under an isotropic Gaussian design. Our analysis shows that, under FDR control, stronger ridge regularization reduces membership inference risk. Experiments on synthetic and a real-world dataset validate the theoretical findings and demonstrate that the proposed methods provide more reliable membership discoveries than existing attacks.

Sparse Mixture of Experts (MoE) models offer a scalable and efficient architecture for training large neural networks by activating only a subset of parameters (“experts”) for each input. A learned router computes a distribution over these experts, and assigns input tokens to a small subset. However, without auxiliary balancing mechanisms, routers often converge to using only a few experts, severely limiting model capacity and degrading performance. Most current load balancing mechanisms encourage a uniform routing probability across experts. Early in pretraining, this can result in inconsistent routing behavior, resulting in the model spending its capacity learning redundant knowledge. We address this by introducing a novel load balancing loss that helps preserve relational structure, encouraging consistent expert choices for similar inputs during training. Our experimental results show that applying our loss to the router results in 36% faster convergence and lower redundancy compared to a popular load balancing loss.

Low-Rank Adaptation (LoRA) has become a widely adopted technique in text-to-image diffusion models, enabling the personalisation of visual concepts such as characters, styles, and objects. However, existing approaches struggle to effectively compose multiple LoRA adapters, particularly in open-ended settings where the number and nature of required skills are not known in advance. In this work, we present LoRAtorio, a novel train-free framework for multi-LoRA composition that gates each adapter using only signals already inside the denoiser, specifically, patch-level cosine (dis)similarity between each LoRA's predicted noise and the base model's. We refer to this as ``intrinsic'' guidance because no external parameters are introduced. Our method is motivated by two key observations: (1) LoRA adapters trained on narrow domains produce unconditioned denoised outputs that diverge from the base model, and (2) when conditioned out of distribution, LoRA outputs show behaviour closer to the base model than when conditioned in distribution. These patch-level similarities are used to construct a spatially-aware weight matrix, which guides a weighted aggregation of LoRA outputs. To address domain drift, we further propose a modification to classifier-free guidance that incorporates the base model's unconditional score into the composition. We extend this formulation to a dynamic module selection setting, enabling inference-time selection of relevant LoRA adapters from a large pool. On the ComposLoRA benchmark, LoRAtorio achieves state-of-the-art performance, with a CLIPScore that does not deteriorate as the number of composed adapters grows, surpassing the strongest prior method by up to 1.3% in CLIPScore, and reaching a 76.92% GPT-4V win rate against the closest baseline. We further show that the framework is backbone-agnostic: the intrinsic-similarity principle transfers to rectified-flow architectures (FLUX.1-dev) without retraining. Code will be made available.

Low-rank adaptation (LoRA) attaches small low-rank matrices to a frozen backbone, and public repositories like Hugging Face host many such LoRAs per backbone, each for a different concept, character, style, or task. Users routinely want to combine several LoRAs in a single generation, but two existing strategies involve a trade-off with each other. First, combining the LoRA weights into a single update before inference (e.g., static merge) needs only one forward pass, but quality drops, because LoRAs trained for different concepts interfere when their weights are summed into a single update, distorting each LoRA's contribution. Second, running each LoRA in a separate forward pass and summing the outputs (e.g., composite) preserves output quality, but each LoRA adds a full backbone pass, so inference time scales linearly with the active LoRA count. To recover composite-level quality at merge cost, we study a property of LoRA pools overlooked by prior multi-LoRA work. When LoRAs in a pool are trained on related concepts, their low-rank factors overlap substantially. Using singular value decomposition, we find the minimal subspace that preserves 99% of the factor energy. On three diffusion pools, only 46% to 57% of the sum of individual LoRA ranks is enough to represent the pool, and the subspace grows sub-linearly with pool size. A 48-LoRA sub-pool of LoRA-Hub on FLAN-T5 NLP tasks shows comparable overlap (about 62%), while the diversity-curated LoRA-Retriever benchmark sits near the no-overlap baseline at 96%, indicating that across these five pools pool composition rather than modality predicts where overlap is large. We use this overlap to construct LoRASpace, a training-free pool-wide method where the subspace matrices are computed once per pool and reused across two inference modes: a static mode at merge inference cost, and a dynamic mode for input-dependent routing at substantially lower cost than composite. On the diffusion pools, static LoRASpace matches composite quality at merge cost, and dynamic LoRASpace achieves more than 6× speedup over composite at six active LoRAs on a 17-LoRA pool. In our ablation, we show that the quality gain comes from truncating to this low-rank shared subspace, which acts as a form of regularisation. On the near-orthogonal LoRA-Retriever benchmark, the framework predicts limited compression benefit, with gains coming from external retrieval coupling.

Vision-Language Models (VLMs) have achieved substantial progress across a wide range of understanding and reasoning tasks, driven by large-scale image-text training aimed at multimodal fusion. Ideally, replacing a textual question with its rendered-image counterpart should leave model performance essentially unaffected. In practice, however, such modality substitution induces dramatic performance degradation. We attribute this carrier sensitivity issue to an inherent bias in current training corpora. Across prevalent datasets such as image captioning, VQA, OCR, and web-sourced interleaved data, text and images are typically organized into distinct and asymmetric roles, with text serving as linguistic queries and images as visual references. Such data bias leads VLMs to exhibit distinct preferences for information acquisition across different modalities. Consequently, VLMs fail to align representations of semantically equivalent content across textual and visual carriers, making model reasoning fragile under modality substitution. To address this, we propose Local Modality Substitution (LoMo), a lightweight, architecture-agnostic data curation paradigm designed to provide supervision for cross-modal representational invariance between semantically equivalent text and image carriers. LoMo achieves this by reformulating single-modality prompts into seamlessly interleaved multimodal sequences. It dynamically selects target text spans and recasts them as rendered images, thereby preserving the same semantics across "text, visual, text" carriers. Extensive experiments across 13 diverse multimodal benchmarks demonstrate that LoMo significantly improves overall multimodal reasoning and yields deeper cross-modal fusion. Specifically, it delivers consistent gains across foundational models, improving over standard SFT by 2.67 points on LLaVA-OneVision-1.5-8B and 2.82 points on Qwen3.5-9B.

Large language models (LLMs) and multi-agent systems (MAS) have shown promise in financial decision-making, yet existing evaluations focus on equity trading and primarily assess directional prediction, overlooking the structural complexity of derivative markets. Option trading introduces fundamentally different challenges, including nonlinear payoffs and multi-leg strategy construction, requiring structured decisions rather than simple directional bets. We introduce LiveOption, an evaluation framework for LLM-based and reinforcement learning agents in option trading. LiveOption formulates the problem as structured sequential decision-making under realistic execution and capital constraints, and provides a reproducible environment with standardized interaction protocols. The framework includes several task suites covering portfolio overlays, open-ended strategy construction, event-driven earnings trading, and 0DTE intraday trading. We further propose a hierarchical metric suite that evaluates action validity, decision quality, risk characteristics, and outcome-level performance. Experiments show that current agents often fail to construct valid strategies and handle nonlinear payoff dynamics, or achieve competitive returns in most scenarios. LiveOption offers a principled testbed for evaluating structured decision-making beyond outcome-based metrics.

Interactive streaming music generation promises the use of generative models for live performance and co-creation that is impossible with offline models. However, SOTA models exist in the discrete-AR regime, requiring industrial levels of compute for both training and inference. In this work, we investigate whether audio diffusion models, with their wide support in the open-source community but non-streaming bidirectional nature, can be repurposed efficiently into interactive models accessible on consumer hardware. By taking a critical look at the modern pipeline for block-wise outpainting diffusion, we identify critical inefficiencies during inference that result in strictly worse computational efficiency than their discrete-AR counterparts. We propose Live Music Diffusion Models (LMDMs), a simple modification of the generative diffusion process that recovers, and then outperforms, the inference complexity of the discrete Live Music Models (LMMs) through block-wise KV Caching. Unlike LMMs, LMDMs further enable stable post-training alignment through our novel ARC-Forcing paradigm, reducing error accumulation without any explicit RL or reward models. We demonstrate the application of LMDMs in a number of creative domains, including text-conditioned generation, sketch-based music synthesis, and jamming. We finally show how LMDMs can be used as a generative instrument in a real artist-AI collaboration, utilizing LMDMs as a “generative delay” to transform musicians’ improvisation live for variable timbral effects while running locally on a consumer gaming laptop.

Remote sensing lithology interpretation is fundamental to geological surveys, mineral exploration, and regional geological mapping. Unlike general land-cover recognition, lithology interpretation is a knowledge-intensive task that requires experts to infer rock types from various features, e.g., subtle visual, making reliable automated interpretation highly challenging. Geological knowledge-guided large multimodal models offer new opportunities, yet their evaluation remains constrained by the lack of benchmarks that capture lithological annotations, multi-level geological semantics, and expert-informed assessment. Here, we propose LithoBench, a multi-level benchmark for evaluating geological semantic understanding in remote sensing lithology interpretation. LithoBench contains 10,000 expert-annotated interpretation instances across 12 representative lithological categories, including 4,000 multiple-choice and 6,000 open-ended tasks organized into five cognitive levels: Identification and Description, Comparative Analysis, Mechanism Explanation, Practical Application, and Comprehensive Reasoning. We further develop an expert-in-the-loop, knowledge-grounded semi-automated construction pipeline, coupling multi sub-processes, e.g., structured geological image descriptions, to enhance geological validity and evaluation reliability. Experiments with multiple large vision-language models reveal substantial limitations in geological semantic understanding, particularly on higher-order explanation, application, and reasoning tasks. Taken together, LithoBench establishes a dedicated testbed for advancing geological knowledge-guided multimodal models, supporting systematic model evaluation, domain-specific training and adaptation, and knowledge-enhanced reasoning toward expert-level lithology interpretation from remote sensing imagery.

Decision trees are prized for their interpretability and strong performance on tabular data, but popular greedy top-down induction algorithms can yield suboptimal and unnecessarily complex structures. Optimal decision tree methods address this through global optimization, yet remain restricted to axis-aligned threshold splits, which limit the expressivity of each node and often force deep, complex trees to capture non-linear feature effects. Shape Generalized Trees (SGTs) generalize threshold splits to learnable univariate shape functions, improving expressivity and enabling more compact trees. However, existing SGT induction algorithms are greedy and offer no optimality guarantees. In this work, we introduce Literati, the first algorithm for optimal SGT induction. We propose a novel AND/OR graph formulation of the problem that jointly optimizes tree structure and shape function complexity. To solve this AND/OR graph, we develop an AO*-based algorithm with two enhancements that improve anytime performance while preserving optimality: a secondary heuristic for OR-node selection and a round-robin policy for AND-node exploration. Across 24 real-world datasets, Literati achieves higher training and test accuracy than state-of-the-art tree approaches.

Outdoor robot navigation in dynamic, unknown environments remains a formidable challenge, necessitating robust traversability assessment and collision-free planning. While traditional map-based approaches struggle with adaptability to novel scenes, existing map-free methods frequently incur high computational costs and depend heavily on LiDAR data. In response, we propose LiteNav, a vision-only for exteroceptive sensing and strictly map-free outdoor navigation system. Relying solely on a single off-the-shelf RGB camera and GPS for goal specification, LiteNav estimates traversability, encodes navigation goals, and generates candidate trajectories via a diffusion-based joint encoding model. These trajectories are then refined by a goal-oriented planner to ensure correctness, efficiency, and safety. Remarkably, LiteNav achieves performance comparable to LiDAR-equipped methods without ever constructing or referencing explicit maps. Deployed on a low-power NVIDIA Jetson Xavier NX, it operates in real time, consuming only one-fifth of the computational resources required by prior approaches. Experiments demonstrate that LiteNav achieves state-of-the-art results on the GND dataset and outperforms existing methods in real-world scenarios. In summary, LiteNav is a map-free outdoor navigation system that operates using monocular RGB perception and GPS goal cues on an embedded platform, demonstrating strong potential for scalable deployment.

Large language models (LLMs) routinely fail to output the correct option in multiple-choice question answering (MCQA) while encoding the answer internally. We expose this latent knowledge via the Query–Key (QK) score, defined for an attention head as the inner product between the last-token query and the key at the end-of-line token following option $i$, evaluated before rotary positional embedding is applied. Its argmax identifies a universal class of select-and-copy heads in middle layers that perform option selection through semantic query–key alignment, mechanistically distinct from induction and copy-suppression heads (Olsson et al., 2022): they are invariant to label symbols, and solve a synthetic task with zero surface overlap—properties no positional-copy account explains and that critically require stripping RoPE. Across $24$ models from $1.5\mathrm{B}$ to $72\mathrm{B}$ parameters (LLaMA-2/3/3.1/3.3, Qwen-2.5, Gemma, Phi-3.5, DeepSeek-R1-Distill), a single head's QK-score exceeds the model's own zero-shot accuracy by up to $+27.4$ pp on HellaSwag and $+49.8$ pp on HaluDialogue; causal zero-ablation collapses MCQA accuracy to near-random. To remove any dependence on labeled validation data, we introduce an unsupervised HeadScore that ranks heads from unlabeled inputs and recovers the supervised top-$k$ heads on every tested model. Against four positional-debiasing baselines (e.g., PriDe, Wiegreffe, Wang), QK-score is complementary by construction: debiasing re-weights output logits, whereas QK-score reads the model's selection from a middle-layer head before decoding. We release a one-line drop-in $\mathtt{HeadScore}$ script and per-model head indices, making every result one-command reproducible across all $24$ models and four benchmarks.

Interactive person retrieval extends text-to-image person retrieval by allowing follow-up questions when the initial description is incomplete. The core challenge is to ask questions that uncover the most useful missing information for identifying the target person from visually similar candidates. Existing methods guide question selection with retriever-external signals, which may not reflect the information most useful to the current retriever. To address this, we propose SCOUT, an interactive person retrieval framework that derives questions from the retriever's own response. Specifically, we introduce a Perturbation-Sensitive Question Selection strategy that selects the next question based on Retrieval Perturbation Sensitivity (RPS), a test-time measure of perturbation-induced ranking change. Additionally, SCOUT requires no additional training and builds on off-the-shelf text-to-image person retrievers, avoiding the dialogue-data curation and retraining overhead of existing interactive methods. Extensive experiments on three standard benchmarks demonstrate that SCOUT yields consistent improvements over baseline methods, validating this RPS-based strategy for interactive person retrieval.

We study for the first time, stochastic dueling bandits over continuous action spaces with Lipschitz structure, where feedback is purely comparative. While dueling bandits and Lipschitz bandits have been studied separately, their combination has remained unexplored. We propose the first algorithm for Lipschitz dueling bandits, using round-based exploration and recursive region elimination guided by an adaptive reference arm. We develop new analytical tools for relative feedback and prove a regret bound of $\tilde O\!\left(T^{\frac{d_z+1}{d_z+2}}\right)$, where $d_z$ is the zooming dimension of the near-optimal region. Further, our algorithm takes only logarithmic space in terms of the total time horizon, best achievable by any bandit algorithm over continuous action space.

As AI agents are given more autonomy in software engineering workflows, the risks grow if they pursue goals different from the ones their users intended. The field of AI control develops control protocols that prevent such harm without restricting the agent's ability to do useful work. We introduce \textbf{LinuxArena}, a control setting for testing such protocols, where agents operate directly on live, multi-service production environments. LinuxArena contains 10 public environments (with an additional 10 held out privately), providing 906 main tasks representing legitimate software engineering work and 92 side tasks representing safety failures such as data exfiltration and access-control bypassing, making it the largest and most diverse control setting for agentic software engineering to date. We demonstrate LinuxArena's utility by running sabotage and monitor evaluations: against a GPT-5 Nano trusted monitor at a 1% step-wise false positive rate, Claude Opus 4.6 achieves an undetected sabotage rate of 42%. We additionally release \textbf{LaStraj}, a dataset of human-crafted attack trajectories that achieves an undetected sabotage rate of 97% against the same monitor. Together, these results suggest meaningful headroom for both attackers and defenders, making LinuxArena a strong testbed for developing and evaluating future control protocols.