NeurIPS 2026 论文列表

Sparse mixture-of-experts (MoE) models raise a natural inference-time question: can experts not selected by the router improve predictions or estimate their errors without retraining? We study this question in vision MoEs across five RGB and multispectral classification datasets and multiple expert counts. We evaluate additional expert inference for three outcomes: top-1 accuracy, raw ranking of default router errors, and calibrated prediction of whether the router-selected classification is wrong. An upper-bound analysis shows that non-selected experts often contain useful class signal when the routed path fails. However, this signal does not reliably improve deployable top-1 accuracy, and raw multi-forward uncertainty scores yield only limited gains for ranking router errors. The strongest benefit appears in calibrated router-error prediction: post-hoc models that combine router signals with features from non-selected experts reduce Router-error Expected Calibration Error (ECE) by roughly 10-13% on average, with several settings exceeding 20% relative improvement over a calibrated router-only baseline. Overall, additional inference-time expert usage in vanilla vision MoEs is most useful for estimating the reliability of the routed decision, rather than directly improving classification accuracy.

Compositional generalization is an important form of out-of-distribution generalization, defined as the ability to train on some combinations of modules (e.g., functions such as `sort`, `filter`) and then generalize to unseen combinations. Despite substantial empirical work evaluating compositional generalization in transformers, findings have been inconsistent because evaluations were often conducted on a limited set of train-test splits, defined in an ad hoc manner. Through a systematic evaluation, we identify two key aspects of the train-test distribution that strongly affect performance: **composition equivalence** and **module coverage**. We show that the apparent performance of direct models (trained only on final outputs) results primarily from exploiting composition equivalences—different sequences of modules that reduce to identical end-to-end functions. When benchmarks eliminate these equivalences, performance drops significantly, showing their inability to generalize to compositions of known modules that produce novel end-to-end functions. We further demonstrate that performance degrades monotonically as module coverage shifts increase—when modules appear in different contexts between train and test—affecting both direct and step-by-step models (trained on intermediate and final outputs). These findings have concrete implications for benchmark design: benchmarks that include compositional equivalences can lead to significant overestimation of the generalization abilities of direct models, and benchmarks that include both module coverage violations and compositional equivalences can exacerbate shortcut learning. We further demonstrate the effect of compositional equivalence on the generalization performance of the pre-trained model Llama3-8B on the code-output prediction task in the coding benchmark [LiveCodeBench](https://livecodebench.github.io/) and discuss its real-world implications.

Frontier large language models can often recognize when they are being evaluated, a capability known as evaluation awareness. If models behave differently in evaluations than in deployment, this undermines the validity of evaluation results, which are a crucial component of current AI safety frameworks. We introduce EvalAwareBench, an open pipeline and benchmark for measuring evaluation awareness that works with any Inspect-compatible evaluation, allowing practitioners to test against current and future benchmarks. EvalAwareBench ships with a newly curated transcript suite covering current frontier system-card evaluations and diverse deployment sources. The benchmark serves two purposes: measuring how reliably frontier LLMs recognize that they are being evaluated, and assessing how detectable individual benchmarks are as evaluations. We identify two methodological choices in the existing literature that introduce systematic bias: the identity of the model that generated the deployment transcripts accounts for 11.25% of measurement variance and can reorder model rankings; and elicitation prompts selected for high performance on one model can perform near chance on others. EvalAwareBench corrects for both via per-model probe calibration and a stratified generator-harmonisation procedure. We release EvalAwareBench as an open pipeline and dataset.

The goal of asymmetric pansharpening is to synthesize a high-resolution multispectral (HR-MS) image by integrating panchromatic (PAN) and low-resolution multispectral (LR-MS) images. While State Space Models (SSMs), particularly Mamba, offer a promising paradigm for global dependency modeling, their standard selective scan mechanisms operate in discrete Cartesian grids, which are ill-suited for distinguishing between structural and spectral features in complex-valued frequency representations. To resolve this, we propose Euler-Mamba, a physics-inspired framework that reformulates the selective scan as a continuous state evolution on polar manifolds. We develop an Eulerian Selective Scan (ESS) mechanism that treats the fusion task as a trajectory of frequency growth. By performing radial scans within the polar domain, ESS governs complex-valued state transitions through the decoupled architecture: the Explicit Phase Rotation (EPR) block facilitates explicit phase rotations for geometric alignment, while the Implicit Magnitude Mapping (IMM) block performs magnitude scaling for spectral calibration. This polar-decoupled evolution allows the model to capture the continuous transition from global spectral backgrounds to fine structural details with linear complexity. Consequently, by modeling the fusion as a continuous functional evolution flow, Euler-Mamba achieves resolution-invariant performance and zero-shot generalization across varying spatial scales, bypassing the computational complexity inherent in traditional discrete-integration frameworks. Extensive experiments demonstrate that Euler-Mamba establishes a new Pareto frontier for asymmetric pansharpening, achieving a superior balance between physical interpretability, computational efficiency, and reconstruction quality.

Modern machine learning depends heavily on massive datasets, but obtaining high-quality annotations at scale is often expensive. As a result, learning from noisily-labeled data has become common, making accurate estimation of the label-noise transition matrix crucial. However, existing transition matrix estimators rely on the fragile estimation of class-posteriors and do not provide finite-sample performance guarantees. In this work, we propose a novel methodology to estimate the transition matrix based on one-sided selective classification. This approach bypasses class-posterior estimation, provides finite-sample performance guarantees, and leverages flexible learning methods for binary classification. Moreover, we introduce effective algorithms to implement the proposed methodology and provide their refined finite-sample performance bounds.

By far the most common way to estimate an expected loss in machine learning is to draw samples, compute the loss on each one, and take the empirical average. However, sampling is not necessarily optimal. Given an MLP at initialization, we show how to estimate its expected output over Gaussian inputs without running samples through the network at all. Instead, we produce approximate representations of the distributions of activations at each layer, leveraging tools such as cumulants and Hermite expansions. We show both theoretically and empirically that for sufficiently wide networks, our estimator achieves a target mean squared error using substantially fewer FLOPs than Monte Carlo sampling. We find moreover that our methods perform particularly well at estimating the probabilities of rare events, and additionally demonstrate how they can be used for model training. Together, these findings suggest a path to producing models with a greatly reduced probability of catastrophic tail risks.

Continual fine-tuning is essential for large language models (LLMs) to dynamically adapt to real-world environments, yet it inevitably suffers from catastrophic forgetting, particularly the performance degradation of previous tasks and LLMs' general-purpose knowledge. Although existing methods, such as orthogonal gradient projection, mitigate the forgetting across various fine-tuning tasks, they fundamentally fail to preserve pre-training LLMs' inherent general-purpose knowledge because the original data and gradients of off-the-shelf pre-training LLMs required by these methods are strictly unknown and highly diverse. To bridge this critical gap, we propose EoupCT, a novel framework designed to Estimate and Orthogonalize Unknown Pre-training gradients for Continual LLM fine-Tuning. Specifically, EoupCT estimates pre-training gradients by dynamically generating pseudo data that is most susceptible to forgetting for new tasks through a learnable soft prompt equipped with Gumbel-Softmax relaxation. Furthermore, we formulate a multi-objective optimization problem and introduce a first-order efficient Pareto optimizer that jointly optimizes LLM parameters and the soft prompt, rigorously enforcing orthogonality between new task updates and the estimated pre-training gradients. Extensive experiments across multiple LLMs demonstrate that EoupCT effectively preserves both task-specific proficiency and inherent general-purpose knowledge, successfully mitigating the catastrophic forgetting.

Membership inference attacks (MIAs) have emerged as the standard tool for evaluating the privacy risks of AI models. However, state-of-the-art attacks require training numerous, often computationally expensive, reference models, limiting their practicality. We present a novel approach for estimating model-level vulnerability to the Likelihood Ratio Attack (LiRA), the strongest available attack, directly from the train and test loss distributions of the target model and without training any reference models. We show that LiRA's per-sample signal decomposes into a variance-ratio term and a residual mean-shift term, with the relative contribution of each determined by how much training collapses model uncertainty at the trained sample. This places models on a continuum, with different regimes calling for different reference-free loss-based statistics as proxies for LiRA TPR. The shapes of the loss distributions themselves indicate which proxy applies. We instantiate the framework with two natural proxies. At the heavy-tailed end, the LOSS attack TNR predicts LiRA TPR@FPR=$10^{-3}$ with RMSE 0.03 across 9 image classification architectures and 4 datasets, outperforming low-cost reference-model attacks such as RMIA. At the symmetric end, the LOSS attack AUC predicts LiRA TPR with RMSE 0.01 across five GPT-2 sizes from 10M to 1B parameters. We also show these proxies to outperform both low-cost (few reference models) attacks such as RMIA and other measures of distribution difference.

Deep learning systems are known to exhibit *implicit regularization* (alt. *implicit bias*), favoring simple solutions instead of merely minimizing the loss function. In some cases, we can analytically derive the implicit regularization -- connecting it to an equivalent penalty that augments the learning objective. However, modern deep learning systems are complex, carrying modifications to the training procedure and architecture (e.g. early stopping, minibatching, dropout) whose effects are not always directly interpretable. Although estimating the resulting implicit regularization could aid theorists in algorithm design and practitioners in interpreting their hyperparameter choices, this problem has received little direct attention. It is also tractable: regularization makes weight updates deviate from loss gradients, promising a signal for identifying implicit bias. Here we provide gradient matching methods that can be used to empirically estimate the implicit regularization. Our method works on networks with known regularization, recovering popular explicit penalties like $\ell_1$ and $\ell_2$. It also replicates known implicit effects, like the quadratic weight penalty induced by early stopping in gradient descent, demonstrating that it can be used to test theories of implicit regularization. Crucially, because our method is empirical, it can handle implicit regularization in arbitrary networks. We demonstrate this use by characterizing the effects of dropout in deep networks, showing implicit $\ell_2$ effects in this popular method. Our work shows that practitioners can use gradient matching to understand regularization in networks with implicit biases that are too complicated to derive analytically.

Flow and diffusion-based generative models achieve state-of-the-art quality, but require multiple forward passes at inference time, making them computationally costly. Consistency models overcome this by learning a flow map directly, enabling few‑step generation and dramatically reducing the computation needed at test time. However, formal statistical guarantees for such one‑step generators are scarce. Existing results either suffer from the curse of dimensionality or rely on impractical loss functions. We investigate whether one‑step flow-based models can escape the curse of dimensionality under the assumption that the data distribution is generated by a low-dimensional surface perturbed with Gaussian noise. In this setup, we derive approximation and generalization error bounds for the flow map estimate that circumvent the curse of dimensionality. Our findings provide the first principled explanation for practical success of one-step flow-based generative models.

In the past year, custom and unreleased math reasoning models reached gold medal performance on the International Mathematical Olympiad (IMO). Similar performance was then reported using large-scale inference on publicly available models but at prohibitive costs (e.g., 3000 per problem). In this work, we present an inference pipeline that attains best-in-class performance on IMO-style math problems at an average inference cost orders of magnitude below competing methods while using only general-purpose off-the-shelf models. Our method relies on insights about grader failure in solver-grader pipelines, which we call the Cognitive Well (iterative refinement converging to a wrong solution that the solver as well as the pipeline's internal grader consider to be basically correct). Our pipeline addresses these failure modes through conjecture extraction, wherein candidate lemmas are isolated from generated solutions and independently verified alongside their negations in a fresh environment (context detachment). On IMO-ProofBench Advanced (PB-Adv), our pipeline achieves 67.1\% performance using Gemini 3.0 Pro with an average cost per question of $\sim$ \$31. This surpasses the performance of the DeepThink IMO gold-winning model, and more than doubles the success rate of the next best publicly accessible pipeline, all at a fraction of the cost.

Inference-time search (e.g., Tree of Thoughts, ToT) with large language models (LLMs) often concentrates on a small set of structurally or semantically similar trajectories, leaving alternatives underexplored—a failure mode we call \textit{reasoning basin collapse}. We introduce \textsc{BASIN}, a training-free selection method that penalizes repeated visits to the same reasoning basin, defined symbolically for arithmetic tasks or semantically via hypothesis clustering, thereby reallocating search across diverse reasoning strategies. Under matched inference budgets, \textsc{BASIN} improves over ToT by up to $+22$pp on Game of 24. To explain why this quality-agnostic penalty is effective, we introduce the \emph{redundancy gap} $\Delta$, which measures how differently search concentrates for correct versus incorrect predictions: standard ToT often operates near $\Delta \approx 0$, failing to distinguish correct from incorrect searches by their visit patterns, while \textsc{BASIN} consistently induces $\Delta > 0$, concentrating search on correct basins while dispersing incorrect ones. More broadly, \textsc{BASIN} suggests structure-aware selection as a simple and general approach to improving inference-time reasoning.

Offline goal-conditioned reinforcement learning remains challenging in stochastic long-horizon settings, where compounding value estimation errors hinder reliable goal reaching. While prior methods have sought to address this challenge through various approaches, a key challenge remains path mirage, where agents overcommit to spuriously successful trajectories in offline datasets induced by stochasticity, often leading to failure on difficult tasks. To address this issue, we propose Branch-aware Graph Planning (BGP), which captures stochastic branching structures by identifying high-variability points as graph nodes and designing goal-conditioned edges with segmented learning to avoid unreliable branches and reduce unnecessary stochasticity along paths. As a result, BGP enables reliable goal reaching even in highly stochastic long-horizon environments, and experiments on diverse OGBench tasks show that it substantially outperforms prior state-of-the-art offline goal-conditioned RL methods.

We derive a tight generalization bound for deterministic neural networks based on their last hidden layer representations. By evaluating representation quality with a parameter-free $k$-nearest-neighbor ($k$-NN) classifier, our certificate avoids traditional parameter-space complexity measures, KL divergences to a posterior, weight norms, and compression. The bound is tight: trained from scratch, it reaches an 8.0 percentage point gap to test error on CIFAR-10 (ResNet-18) and yields the first non-vacuous certificates we are aware of on CIFAR-100; with frozen DINOv2 representations, it certifies at 1.4\% on CIFAR-10 and 23.1\% on 1,000-class ImageNet. It decomposes additively into three empirical, independently measurable quantities: local class geometry ($k$-NN decoder error), decoder-network agreement, and training stability. This allows the bound to act as a diagnostic tool: we demonstrate that actively destroying local class geometry eliminates generalization while leaving dataset memorization intact, identifying class geometry as a causally relevant mechanism for generalization. Finally, we show that learned representations are significantly less sensitive to weight noise than linear readouts, suggesting why parameter-space certificates often struggle to remain tight.

Diffusion Transformers enable powerful in-context image editing from a reference image and a text prompt, but also make it easier to maliciously edit public images while preserving the private identity of the reference. Existing DiT-oriented protections, such as DeContext, mainly suppresses context-to-target attention to weaken reference utilization. However, this strategy can be insufficient for complex edits where prompt semantics already dominate generation. Instead, we study proactive protection for DiT-based editors from a conditional-flow perspective, and use a local geometric interpretation of in-context editing, where the denoising trajectory is empirically characterized as being steered by prompt-consistent and reference-consistent directions. Thus, effective protection should not simply corrupt the output, but should drive it away from the original context manifold. To this end, we propose Conditional Flow Hijacking, which adds imperceptible perturbations to the reference image. Instead of suppressing context attention, our method preserves the context pathway, but redirects the context-induced steering signals in the intermediate transformer layers, causing the final trajectory to drift away from the reference consistent solution region. Experiments on FLUX.1-Kontext with different datasets and prompts including attribute and complex scene editing show that our method significantly suppresses reference identity leakage while maintaining image quality and prompt alignment, outperforming prior protection baselines, especially under complex scene editing prompts. We further verify the robustness of the proposed protection method as well as its transferability across other models, highlighting the need for proactive defenses for in-context generative models.

We study semi-offline value estimation, where a small on-policy sample supplements a large off-policy dataset. This regime is common in recommender systems and pre-deployment A/B testing, and the challenge is to use the limited on-policy data to correct systematic bias in a given value predictor. Recent work on Bellman calibration (BC) gives a post-hoc, model-agnostic correction along the predicted value. We propose Error-Guided Bellman Calibration (GBC), which trains an error model on the on-policy sample and calibrates the predictor along this learned error axis. We prove a completeness-free refinement guarantee, a theorem showing that the correction adapts to the intrinsic dimension of the prediction bias, and a finite-sample bound on the value-estimation error that isolates our advantage in the refinement term. Across synthetic, CRM, and D4RL benchmarks, GBC improves over existing calibration baselines.

Designing continuous trajectories whose time-averaged occupancy provably matches a prescribed spatial density (the *ergodic coverage* problem) is central to UAV-assisted data collection and sensing, robotic exploration, and mobile monitoring. For flying agents in particular, this challenge is acute: trajectories must balance coverage fidelity against tight energy budgets, no-fly zones, and acceleration limits. Existing methods either re-optimize each trajectory online (with cost growing in the horizon and re-running for every target, agent, and realization) or rely on bespoke analytical constructions that must be re-derived for each new constraint. We propose a *pushforward* framework that decouples ergodicity from density matching: an analytic latent trajectory provides exact uniform ergodicity on a simple annular domain, and a single map, learned offline by optimal-transport conditional flow matching, transports this latent occupancy onto the prescribed target density. The composed trajectory is then asymptotically ergodic with respect to the learned pushforward distribution, with deviation from the target controlled by the flow-matching training loss. Once trained for a given target density and constraint set, the map serves an unbounded number of trajectories and a multi-agent fleet without per-agent retraining, and many differentiable operational constraints (no-fly zones, acceleration ceilings, or fairness penalties) enter as additive soft penalties in the training loss without re-deriving the design. We prove three results (an acceleration-energy bound, an $O(1/\sqrt{K})$ ergodic convergence rate in the number of trajectory cycles $K$, and an approximation-error bound) that combine into an end-to-end coverage bound estimable from CFM training diagnostics (certified given an architectural Lipschitz bound on $v_\theta$). Experiments on synthetic targets empirically support each theoretical envelope; on a real UAV-coverage dataset, the proposed method gives the strongest observed coverage--energy and coverage--constraint tradeoffs among the evaluated time-warping, optimization, and concurrent flow-matching baselines.

We consider the problem of synthesizing Clifford quantum circuits for devices with all-to-all qubit connectivity. We approach this task as a reinforcement learning problem in which an agent learns to discover a sequence of elementary Clifford gates that reduces a given symplectic matrix representation of a Clifford circuit to the identity. This formulation permits a simple learning curriculum based on random walks from the identity. We introduce a novel neural network architecture that is equivariant to qubit relabelings of the symplectic matrix representation, and which is size-agnostic, allowing a single learned policy to be applied across different qubit counts without circuit splicing or network reparameterization. On six-qubit Clifford circuits, the largest regime for which optimal references are available, our agent finds circuits within one two-qubit gate of optimality in milliseconds per instance, and finds optimal circuits in 99.2\% of instances within seconds per instance. After continued training on ten-qubit instances, the agent scales to unseen Clifford tableaus with up to thirty qubits, including targets generated from circuits with over a thousand Clifford gates, where it achieves lower average two-qubit gate counts than Qiskit's Aaronson-Gottesman and greedy Clifford synthesizers.

In many interactive settings, from social and institutional environments to multi-agent systems, agents must infer latent behavioral structure from limited signals while their own actions shape the data they observe. This endogeneity violates the stationary, exogenous assumptions underlying standard likelihood-based methods, leading to failures in adaptation despite accurate clustering. We propose \emph{Epistemic Social Learning} (ESL), a two-timescale framework that couples belief-based inference with adaptive representation learning under endogenous interaction. At a fast timescale, agents maintain Bayesian beliefs over shared behavioral prototypes. At a slower timescale, prototypes are updated from belief-weighted interaction data via the recursion $\Theta_{m+1} = \Theta_m + \gamma_m \widehat{H}_m$, aligning learned representations with the interaction distributions induced by agent behavior. We formalize ESL as a stochastic approximation process with controlled Markov noise and show that its dynamics track the differential inclusion $\dot{\Theta} \in \mathcal{G}(\Theta)$, whose limit sets correspond to self-consistent interaction regimes. Across repeated games with behavioral shifts, ESL reduces post-switch regret by $26\%$ over strong baselines while improving decision-relevant prediction. These results reveal a fundamental gap between latent identification and adaptive performance under endogenous interaction.