NeurIPS 2026 Papers

Reliable evaluation is essential for progress in image generation, yet widely used metrics rely on assumptions that are misaligned with the geometry of real data. We revisit two dominant families of metrics: Fréchet-based metrics such as FID, and precision/recall-style metrics based on KNN balls. We show that FID reduces complex feature distributions to a Gaussian moment summary, making it blind to differences beyond mean and covariance. We further show that KNN-ball metrics construct inflated isotropic support estimates in high-dimensional feature spaces, causing binary membership queries to accept off-manifold samples and making recall unstable under standard finite-sample evaluations. We propose Chamfer distance as a simple alternative: instead of constructing inflated support estimates, it directly measures cross-set nearest-neighbor distances between real and generated samples. The resulting one-sided distances provide interpretable signals for fidelity and coverage, while reducing sensitivity to the neighborhood hyperparameter and remaining computationally efficient.

Probabilistic Circuits (PCs) are deep generative models that support exact and efficient probabilistic inference. Yet in autoregressive language modeling, PCs still lag behind Transformer-based large language models (LLMs), suggesting an important expressivity gap. In this work, we compare PCs and LLMs under a unified autoregressive formulation. First, an output bottleneck: PCs parameterize predictions as convex combinations in probability space, which struggles to represent the sharp distributions typical of language; adopting a logit-space parameterization substantially narrows this gap. Second, a context-encoding bottleneck: we prove that structured-decomposable PCs can match Transformer separation rank on vtree-aligned partitions, but show, both theoretically and empirically, that this capacity is limited to partitions aligned with the fixed routing structure, leading to severe degradation when the data exhibits heterogeneous dependency topologies. We further prove that decomposable PCs are strictly more expressive than structured-decomposable ones, though effectively optimizing them remains an open challenge.

We envision a new era of AI, termed agentic organization, where agents solve complex problems by working collaboratively and concurrently, enabling outcomes beyond individual intelligence. To realize this vision, we introduce asynchronous thinking (AsyncThink) as a new paradigm of reasoning with large language models, which organizes the internal thinking process into concurrently executable structures. Specifically, we propose a thinking protocol where an organizer dynamically assigns sub-queries to workers, merges intermediate knowledge, and produces coherent solutions. More importantly, the thinking structure in this protocol can be further optimized through reinforcement learning. Experiments demonstrate that AsyncThink achieves 28% lower inference latency compared to parallel thinking while improving accuracy on mathematical reasoning. Moreover, AsyncThink generalizes its learned asynchronous thinking capabilities, effectively tackling unseen tasks without additional training.

Spiking neural networks (SNNs) offer a compelling pathway toward energy-efficient artificial intelligence, fundamentally anchored in presynaptic input-driven sparsity. However, a substantial yet chronically overlooked source of computational overhead persists: the continuous subthreshold dynamics of postsynaptic neurons. Quantitative analysis reveals that up to 44\% of neurons in an ImageNet-trained ResNet-18 model reside in a deeply inhibited state, imposing a persistent computational burden despite possessing a negligible probability of firing. To eradicate this inefficiency, the Dormant Spiking Neuron (DSN) model is introduced. By incorporating a state-driven dynamic gate, the DSN allows a neuron to enter a dormant state when its membrane potential falls below a predefined negative threshold, dynamically pruning redundant computations at the instance level. Crucially, this dormancy mechanism is directly integrated into the neuronal dynamics and trained end-to-end via backpropagation through time using surrogate gradients, enabling the network to actively learn a robust dormancy policy that adaptively preserves critical information flow. Extensive experiments across static and neuromorphic datasets demonstrate that the DSN framework consistently reduces computational energy by 40\% to over 70\%. Notably, on the large-scale ImageNet dataset, the DSN outperforms state-of-the-art static pruning techniques by achieving equivalent energy savings while simultaneously improving upon the baseline accuracy by 1.34\%. By establishing a trainable, state-driven sparsity paradigm orthogonal to conventional input-driven methods, the DSN unlocks synergistic efficiency gains, presenting a highly potent approach for advancing neuromorphic computing. Source code will be made publicly available.

Noisy gradient estimates are ubiquitous in machine learning, arising from stochastic sampling, distributed computations, or privacy- preserving mechanisms like Differential Privacy (DP). This paper introduces a general, optimizer-agnostic framework for denoising these estimates. Our approach operates as a modular wrapper that intercepts noisy gradient observations and provides denoised estimates to the optimizer, requiring no internal modifications to algorithms like SGD or Adam. We start by studying the problem of optimal denoising gradients in quadratic and cubic optimization problems. We develop maximum likelihood estimation of gradient and Hessian of the objective. We also develop other computationally-efficient order optimal algorithms for denoising gradients in such a setting. Finally, we utilize the developments for the cubic and quadratic setting to develop a more general denoising mechanisms for general smooth nonconvex optimization problems. Theoretically, by leveraging higher-order smoothness, we establish an improved convergence rate of $O(T^{-12/19})$ for smooth non-convex optimization. While our recursive algorithm requires two gradient queries per iteration, we show that the improved convergence rate yields a lower total oracle complexity than the standard $O(T^{-1/2})$ rate of SGD. This implies the gain in convergence speed asymptotically outweighs the additional computational cost. We apply our algorithm to various denoising problems, particularly nonconvex DP-training. Our experiments, including training private classifiers on CIFAR-10, demonstrate significant improvements over baselines.

Submission timing in human evaluation systems is universally treated as an irrelevant detail. We identify a systematic temporal leniency bias: across 129,023 reviews spanning three consecutive years of a large-scale ML venue, evaluators submitting closer to the deadline assign higher within-paper scores and produce less thorough assessments. The pattern holds in all 15 robustness specifications across all three years, and early evaluators achieve measurably lower Brier scores ($\Delta \approx 0.005$--$0.006$, stable across years). We propose two complementary corrections. Optimal Timing-Calibrated Aggregation (OTCA) derives a linear weight function $w(t) = a^{\star} t + 1$ by minimising paper-level decision loss, significantly outperforming heuristic alternatives (McNemar $\chi^2 = 46.3$, $p < 0.001$). Adversarial Temporal Debiasing (ATD) learns timing-invariant representations via Gradient Reversal, reducing timing discriminability by $37.4\\%$ while preserving predictive utility. Combined in a joint scheme, the two methods improve borderline decision accuracy by up to $11.25\\%$ over uniform averaging (McNemar $p < 0.001$), consistently across all three years. Both methods require only submission timestamps, already logged by every major evaluation platform, making deployment cost-free.

Zeroth-order optimization (ZOO) has emerged as a promising approach for fine-tuning Large Language Models (LLMs) when gradient access is unavailable or memory-prohibitive. However, ZOO relies on estimating gradients via finite differences between two function evaluations at perturbation scale $\mu$, making it inherently noisier than its first-order counterpart. This raises a fundamental question: \textit{under what conditions does ZOO yield reliable gradient estimates?} In this work, we provide a rigorous variance-theoretic answer through a generic noise-amplification framework and demonstrate that the reliability of ZOO gradient estimations is structurally determined by whether the two finite-difference evaluations are performed on \textit{matched data}. Mismatched evaluations, as induced by on-policy Reinforcement Learning (RL), produce a $1/\mu^2$ oracle-noise amplification that sets an irreducible noise floor, whereas matched evaluations, as realized by offline optimization methods such as Boltzmann Targeted SFT (BOLT), eliminate this amplification entirely via noise cancellation. We further derive convergence guarantees with explicit constants that cleanly separate the two regimes. Controlled experiments on math reasoning tasks validate these findings, using ZooBOLT and ZooGRPO --- zeroth-order adaptations of BOLT and GRPO --- as representatives of the matched- and mismatched-data regimes respectively. On GSM8K with Qwen2.5-1.5B, ZooBOLT recovers $96.1\%$ of first-order BOLT's performance gains, while ZooGRPO exhibits near-zero learning despite its first-order counterpart achieving strong results. This qualitative separation persists across harder benchmarks (MATH), larger model scales (Qwen2.5-7B), and different model variants (DeepSeek-R1-Distill-Qwen-1.5B). Additionally, ZOO methods reduce peak memory to near-inference levels, validating its practicality as a memory-efficient fine-tuning paradigm.

Self-attention without positional information is permutation equivariant, therefore positional encodings are required whenever a transformer must distinguish input order. However, injecting absolute position can also weaken length generalization. We formalize this tension for additive positional encodings by introducing positional spread, the total variance of the positional embedding matrix. For additive PE in softmax attention with mean pooling, we prove an upper bound demonstrating that the equivariance breaking component of risk is controlled by $\|\widetilde E\|_F^2$, along with a matching lower bound for self-separable additive PEs. We finally prove a spread-expressivity tradeoff: any additive PE that distinguishes absolute positions with minimum separation $\delta$ must have spread $\Omega(\delta^2 n)$. The theory predicts two regimes: when the target is near permutation invariant, lower spread improves length extrapolation; when the task requires absolute position, additional spread reduction destroys expressivity. We test this prediction using controlled synthetic tasks, spread-regularized additive PE, and long context retrieval and reasoning tasks. Across low positional sensitivity tasks, spread strongly predicts OOD accuracy; on absolute position tasks, performance demonstrates the predicted non-monotone tradeoff. For score-based methods such as RoPE and ALiBi, we include a displacement based comparison rather than matching bounds, clarifying both the reach and limits of the spread theory.

Frontier AI systems function as \emph{constitutional institutions}: each deployed model encodes an implicit ranking among safety, helpfulness, honesty, autonomy, and equity. We ask whether the supply of frontier constitutional types covers human demand. Combining a paraphrase-controlled audit of $23$ frontier LLM archetypes with a pairwise-tradeoff study of $1{,}649$ humans on the same instrument, we report three facts. Demand is broad: it spans all five values, with the largest constituency under one-third. Supply is narrow and drifting: the $23$-archetype hull occupies $0.10\%$ of the demand hull, and across six model families autonomy decreases in $5/6$, equity increases in $5/6$, and safety increases in $4/6$. The importance of this drift is not that the frontier moves but that it moves away from a value that is already undercovered, mechanically worsening the welfare floor for the users least well served by the static menu. The fix is sparse: a $2$-vertex menu $\{e_{\mathrm{HON}}, e_{\mathrm{AUT}}\}$ beats the full $23$-archetype frontier by $47$ percent on mean regret ($95$ percent); three vertex additions cut mean regret by up to $81$ percent and worst-group regret by up to $64$ percent. We formalize these findings as a budgeted-pluralism trilemma and show the binding regime is empirically realized.

Memory is becoming a core component of long-horizon AI agents, allowing agents to reuse past experience when operating web browsers, software tools, and other interactive environments. Existing work mostly treats memory as a supply problem, asking what experience to write, how to store it, and which entry to retrieve for the next task. Yet we still lack a clear account of how models consume retrieved memory across a multi-step action trajectory. This consumption process matters because it determines not only what memories should be retrieved, but also what models and control policies are needed to use them safely. To diagnose this process, we propose Entry--Propagation--Recovery (E-P-R), a trajectory-level framework that asks where memory first changes an action, whether that change carries forward, and whether the agent can recover after leaving a correct path. We instantiate E-P-R on WebArena and on MemTrapBench, a controlled benchmark we build to isolate these phases. We find that the main failure often begins at entry: agents adopt conflicting memory at the first exposed decision point even when the recommendation is task-wrong. Repeated exposure then amplifies this early error, while recovery after divergence is weak. Together, these effects create a compliance trap: across models, conflicting memory induces similar compliance rates, but once agents comply, their success rates collapse to a low floor. Stronger agents therefore suffer larger absolute damage because each compliance event erases more baseline capability. These results suggest that memory-augmented agents should be evaluated not only by retrieval quality or final success rate, but by how they consume memory throughout the trajectory.

When language models reason in chain-of-thought or collaborate in pipelines, they serialize structured information into natural language. How much compositional structure survives this bottleneck? We propose a round-trip protocol that answers this question empirically for tree-structured symbolic expressions. A generator converts a procedurally generated arithmetic expression into a word problem, a separate extractor recovers the expression from the word problem alone, and symbolic equivalence provides an exact oracle. Evaluating all pairwise combinations of fifteen models yields a communication matrix whose marginals separate generation quality from extraction quality. Three main findings emerge. First, the channel is lossy and asymmetric: swapping which model generates and which extracts shifts accuracy by up to 53.2 points, and the best pair reaches 82.0\% by combining different models on each end rather than the same model on both. Second, at least 84\% of round-trip failures originate at generation, and difficulty is driven by tree structure (operator count, depth, right-branching) rather than model family. Third, the channel is trainable: $\sim3600$ fine-tuning examples that share the evaluation's operators and tree shapes lift every open-weight model above untrained GPT-5, an upper bound under matched semantics. A disjoint-domain regime with new operators and vocabulary also raises every open-weight model, confirming the gain is not an artifact of matched semantics, though a gap to the frontier remains. Together these results identify compositional serialization as a primary limiting factor when models communicate hierarchical structure through natural language.

Group Relative Policy Optimization (GRPO) is widely used to post-train reasoning language models, and its rollout group size $G$ is typically treated as a compute hyperparameter. We argue that $G$ also sets a difficulty threshold below which prompts produce no learning signal. Under binary verifier rewards, a rollout group only produces a learning signal when it contains both a success and a failure, and the rate at which a prompt yields such mixed groups turns on sharply near $p_c\simeq\ln 2/G$. Below that threshold the prompt is invisible to the relative-advantage estimator regardless of optimizer state. We verify this gate directly in pass@256 rollout traces from Qwen3-1.7B, Qwen3-8B, and Llama-3.1-8B, where empirical mixed-group rates match the binomial prediction within $1.4$–$1.9$ percentage points on average per model. Sweeping fixed-$G$ GRPO with $G\in\{8,16,32,64,128\}$ then shows that difficulty-binned per-prompt improvement after training tracks the frontier shape with binned $R^2$ between $0.962$ and $0.984$ across all three model families. Treating $p_c$ as a moving target, we instantiate a frontier-tracking group-size schedule that improves the best fixed-$G$ baseline by $4.2$–$5.3$ points on MATH-500 and $2.4$–$3.4$ points on GSM8K under matched rollout budget. Ablations attribute the gain primarily to frontier-aware switch timing.

Memory-augmented reasoning agents typically store a single realized reasoning trajectory as the unit of experience. This abstraction is brittle under stochastic decoding: the same task can induce multiple plausible reasoning paths with different decompositions, assumptions, and failure modes. We introduce Branch-Structured Distributional Memory (BranchDIME), a framework for constructing memory from branch-structured trajectory sets rather than isolated traces. BranchDIME samples short reasoning prefixes, clusters them into prefix-induced branches, expands representative prefixes into full trajectories, and distills the resulting set into reusable memories containing positive strategies, anti-patterns, and contrastive rules. Across GPQA-DIAMOND and MATH500, BranchDIME improves over single-trajectory memory and alternative trajectory-selection baselines at comparable inference latency, yielding relative gains of 8.7% and 2.2% over single-trajectory memory, respectively. Coverage analysis shows that BranchDIME spans all discovered branches, compared with 20.0% coverage for consensus selection and 64.9% for random selection, while also achieving the highest downstream accuracy. BranchDIME further transfers across tasks and models, improving over single-trajectory memory by up to 15.4% on AIME25 and OlymMATH. These results support a shift from trajectory as experience to trajectory distribution as experience: memory should capture the local reasoning landscape induced by stochastic generation, not merely one sampled path.

Certifying authorities for safety-critical software audit recovered trace matrices module by module, not on average. Every existing distribution-free calibration delivers a marginal certificate, and we show the gap to the auditor's quantity is structural and unbounded. Under within-block exchangeability with cross-block heterogeneity in the conditional null score distribution, every procedure controlling marginal $\mathrm{FDR}$ at level $\alpha$ on a bipartite trace matrix with $K$ source modules admits worst-module false discovery proportion at least $1{-}\alpha{-}O(1/K)$ with probability at least $1{-}2/K$. We name this the block catastrophe and close it with $\mathrm{CERTRA}$, which wraps any black-box scorer, computes a within-block rank-indicator conformal $e$-value, and applies $e$-$\mathrm{BH}$ separately within each source module. The construction yields finite-sample per-module $\mathrm{FDR}$ control under arbitrary intra-module dependence, with matching power up to a rank-quantization factor. We release IndTrace-2026, a 12{,}406-link composite across aerospace, automotive, and medical-device specifications with professional ASIL/DAL/SIL annotation (Cohen's $\kappa{=}0.86$). $\mathrm{CERTRA}$ holds worst-block $\mathrm{FDP}$ at 0.118 where split-conformal drifts to 0.413, retains 0.741 of true links versus 0.572 for the strongest fair-operating-point baseline, and reduces audit deficiency 3.3-fold. Per-module distribution-free calibration is what an industrial audit can sign.

We study how temporal correlations in the data can make certain sparse learning problems efficiently learnable by gradient-based methods. Our focus is on Boolean k-juntas, a canonical sparse learning problem known to pose barriers for gradient-based methods under independent uniform samples. We show that this picture changes when the samples are generated by a lazy random walk on the hypercube. In this setting, the temporal dependencies can be exploited by a two-layer ReLU network trained using stylized-SGD with a temporal-difference loss, which compares target and predicted increments across consecutive samples. For every fixed k, the resulting sample complexity is essentially linear in the ambient dimension d. By contrast, we show that for large-batch gradient methods using standard convex pointwise losses, temporal correlations do not provide the same advantage.

Humans systematically misrepresent probability in a stereotyped inverse-S pattern. It has been documented for decades, but its origin remains unexplained. We propose a Bayesian encoding–decoding account in which probabilities are represented by noisy internal signals and decoded by Bayes-risk minimization. For bounded probability stimuli, we show that distortion decomposes into boundary regression, likelihood repulsion, and prior attraction, yielding a key prediction: the classic inverse-S-shaped weighting pattern implies a U-shaped allocation of encoding precision with greater sensitivity near 0 and 1. Across judgment of relative frequency, lottery pricing, and risky choice, this U-shape is recovered from data without imposing any functional form on the encoding, and our framework outperforms deterministic weighting functions, bounded log-odds models, uniform-encoding Bayesian accounts, and matched efficient-coding models on held-out data. In a new dot probability estimation experiment with bimodal stimulus statistics, the recovered prior tracks the new distribution while the recovered encoding remains U-shaped. Together, these results identify the inverse-S-shaped probability weighting function as the joint product of a stable U-shaped encoding and a flexible prior, integrated by optimal Bayesian decoding.

The Bayesian learning rule (BLR) exploits the relation between Bayesian updating and the Kullback-Leibler divergence (KLD) to derive efficient additive updates for exponential-family variational posteriors. The KL geometry is powerful but restrictive: combined with posterior families with parameter-dependent support it can lead to undefined objectives. We ask which non-KL geometries preserve the additive BLR computation. Previous works that use $f$-divergences suggest that this is only possible with KLD. Here, we show that this is not true. We show that additive information-geometric optimization algorithms can be derived for the class of mixed-space Bregman or Fenchel-Young discrepancies. These discrepancies induce generalized maximum entropy families, dual coordinates, and generalized BLR canonical gradient updates, recovering the classical BLR. The generalized BLR leads to new algorithms that \textit{cannot} be derived from the KL-based BLR. For the Tsallis negentropy with quadratic statistics, the generalized BLR yields bounded-support and heavy-tailed $q$-Gaussian posteriors and an easy Newton-like implementation. Experiments up to CIFAR-100 illustrate controllable support and tail behavior and competitive performance.

Approximate machine unlearning aims to remove the influence of specific training data from a trained model without retraining from scratch. We identify a previously undocumented confound in how unlearning is evaluated on BatchNorm-based architectures: a single forward pass over retain data, an operation that modifies no weight, can deterministically rewrite the model's normalization state and reverse the apparent surface-metric forgetting. We formalize this operation as a weight-preserving fixed-point operator and prove that any pre-versus-post gap it induces is provably attributable to BN running statistics rather than to any modification the unlearning method made to the weights. This attribution claim cleanly separates measurement failure (BN artifact) from encoder failure (residual weight-encoded information, recently documented in concurrent work), and the same operator framework yields a unique decomposition of linear-probe elevation into BN-measurement-bias and encoder-geometry components. Empirically, the artifact reverses headline forget accuracy by up to $78$pp across nine published methods on standard benchmarks; an attacker with as few as $10$ unlabeled images recovers most of the masked accuracy; and a strict GroupNorm control reduces the artifact to zero across all methods.

This paper introduces the BV4 Benchmark, a collection of high-dimensional spectral datasets for unsupervised anomaly detection in vacuum environment data streams. Collected via Optical Emission Spectroscopy (OES), the benchmark comprises nine experiments representing practical scenarios structured according to established anomaly detection literature and validated by domain experts to address challenges such as spatial anomalies, temporal anomalies, and concept drift. BV4 datasets are composed of 2048 distinct wavelengths recorded over time, accompanied by timestamps and ground truth labels, enabling rigorous evaluation of machine learning models under dynamic environmental changes within vacuum chambers. We demonstrate the utility of BV4 through a simple comparative evaluation against previous datasets for anomaly detection in spectral data streams and discuss the current limitations and future improvements of the benchmark to support meaningful evaluative claims in the area.