NeurIPS 2026 Papers

Language models are increasingly trained to "reason" before answering users' queries, outputting hundreds of intermediate tokens before a final answer. While reasoning is designed with direct model capabilities in mind, we argue that reasoning language models (RLMs) should also be assessed by interaction effects between their reasoning traces and weaker models. In contexts like safety monitoring and distillation, the performance of weak models increasingly depends on legibility---how thoroughly and precisely a reasoning trace externalizes an RLM's actions. Existing efficiency-based metrics for legibility fail to capture "thoroughness", instead focusing on conciseness. Thus, we introduce *transfer utility*, a method for measuring trace legibility derived from interactions where weak models finish tasks given incomplete traces. Evaluating 85k traces from 12 RLMs across three tasks, we find that reasoning traces generated by the highest-performing and the most efficient models rank the *lowest* for transfer utility. We also find evidence that transfer utility predicts the ability of weak monitors to verify procedurally dense traces for math or logical problem-solving, though the effect moderates when verifying fact-driven traces. Finally, we show that open-source reward models decouple from transfer utility signals on correctness. Together, these findings surface status quo trade-offs in weak-to-strong legibility, motivating an interaction-first framework for future applications.

Biological visual systems compute reliably under tight metabolic and wiring constraints despite noisy inputs and stochastic circuit components. Artificial vision systems tend to invert both properties: They consume vastly more energy and can be sensitive to adversarial perturbations. Whether artificial networks with structural properties of real brain wiring are more robust at lower computational cost is unclear. We address this with FlyNet (Lappalainen et al. 2024), a task-optimized, continuous-time neural network model of the fly visual system constrained by its synaptic connectome. We benchmark it against CNN, RAFT (Teed and Deng, 2020), and a sparse continuous-time recurrent neural network without connectome constraints (CTRNN) on a looming-based collision-detection task within a differentiable 3D rendering pipeline. We find that FlyNet is more robust than CNN controls to flicker, unstructured noise, and motion jitter. Compared with RAFT controls, FlyNet is more robust to flicker and motion jitter but less robust to all gradient-based attacks. Yet, FlyNet uses up to five orders of magnitude fewer FLOPs than RAFT, achieving higher robustness on common video corruptions at far lower cost. FlyNet is less robust than CTRNN across all tested perturbations, suggesting that FlyNet's connectome constraints contribute to cost, not robustness. To probe FlyNet's failure modes, we use counterfactual restoration: textures equally far from training as adversarial textures that recover correct classification also recover position and velocity representations, ruling out unfamiliar textures as the cause of failure. Classical reverse-phi illusions degrade the same downstream position and velocity readouts, suggesting that both perturbations may exploit similar computational vulnerabilities. Together, these results show that FlyNet is robust for its computational cost on an ecologically important task, and indicate that different architectures specialized on the same task have specific robustness profiles across perturbation types. Our differentiable pipeline enables computational tests of this hypothesis and generates targeted adversarial stimuli for in vivo experiments.

Modern large language models keep the feed-forward network (FFN) intermediate width identical across every layer, a convention inherited from the original Transformer. Yet mechanistic work suggests layers at different depths do qualitatively different work, raising a natural question: are their capacity demands also non-uniform, and if so, can a fixed parameter budget be allocated more wisely than uniformly? To probe this, we apply principal component analysis (PCA) to the input of each FFN down-projection and measure, for every layer, the smallest number of principal components needed to reconstruct the activation up to a fixed cosine similarity --- the layer's intrinsic dimensionality. Scanning 58 open-weight LLMs spanning 14 families and 70M to 72B parameters, we find a recurring middle-heavy profile in which middle layers carry higher intrinsic dimensionality than edge layers; the profile is robust to the calibration corpus, even across English and Chinese, and is largely preserved through post-training fine-tuning. As a direct test that this signal is structurally meaningful, we use it to guide width pruning in a knowledge-distilled student under a matched parameter budget: the PCA-guided allocation outperforms the uniform-width convention by $+2.7$ points on Qwen2.5-3B and $+2.3$ on Qwen3-8B, while a same-scale teacher (LLaMA-3-8B) shows only $+0.9$, and a budget-matched inverted control consistently does worst. Across these three teachers, the size of the gap scales with how unevenly the teacher's intrinsic dimensionality is distributed across layers --- a single-number summary we call $\sigma_{99}$ --- and the advantage persists through downstream supervised fine-tuning.

Multi-agent large language model (LLM) debate is often evaluated by whether final answers improve, but movement is not necessarily improvement: the same discussion can rescue an initially wrong majority or destroy an initially correct one. Standard final-accuracy evaluations conflate these opposing mechanisms. We introduce an auditable protocol for homogeneous debate on multiple-choice questions (MCQs) that records each run as a transition ledger over collapse, correction, onset, and signed intervention utility. On $6{,}925$ MMLU-Pro debates, the protocol identifies $253$ collapses and a parallel correction ledger that changes how interventions should be judged. Replay experiments reveal the central tradeoff: a leave-one-model-out probe-gated freeze prevents $29$ collapses but loses $108$ corrections under equal weights, so collapse prevention alone can recommend the wrong policy. A compact pre-debate $8$-probe screen is a triage signal: its unadjusted family-level association with conditional-collapse risk is high ($G{=}7$, Spearman $\rho=0.893$, exact two-sided $p=0.0123$), but initial-majority accuracy is a close comparator ($\rho=0.821$; family partial $\rho=0.767$, $p=0.0877$), so we do not treat it as calibrated or capability-adjusted prediction. Round-level traces localize many collapses to the first debate round, where early disagreement can precede both harmful cascades and useful recovery. We release replayable schemas, coders, audits, cost cards, and zero-API rebuild scripts so future model--scaffold rows can be compared under the same denominators and signed utility ledger.

Many decision making procedures that make use of probabilistic predictors assume that training on new data acts as Bayesian conditioning. When this assumption breaks, the downstream procedure can perform poorly. E.g. we show that in active learning, incoherent updates cause the procedure to prefer to acquire suboptimal points. To quantify how far a model's predictive updates depart from conditioning, we introduce the _incoherence ratio_. We empirically find that amortised predictors such as TabPFN can be more coherent than standard parametric approximations to Bayesian inference. In synthetic active learning experiments, more coherent models acquire better data, an effect not attributable to better predictive performance. In many tasks, decisions depend on a predictor only through its induced action. We therefore introduce action coherence, a decision-theoretic relaxation that measures only the incoherence affecting that action. These diagnostics make coherence testable and quantifiable, enabling incoherence to be diagnosed and addressed.

We evaluate the autonomous offensive cyber capabilities of frontier AI agents on two purpose-built cyber ranges---a 32-step corporate network attack and a 7-step industrial control system attack targeting a simulated cooling tower---complemented by a private 82-task CTF suite spanning four difficulty tiers to measure narrow cyber skills. Estimated human-expert solve times are roughly 20 hours for the corporate range, 22 hours for the cooling tower, and from 1 minute to 40 hours across the CTF suite. Across twelve models released over a twenty-month period---from GPT-4o in August 2024 to early predeployment checkpoints of Mythos and GPT-5.5---we observe two trends. First, performance increases approximately log-linearly with inference-time compute, with no plateau observed for the strongest models up to 100M tokens. Second, newer frontier models generally complete more steps at fixed token budgets. GPT-5.5 and the Mythos Preview checkpoint each completed all 32 steps in at least one run, the first end-to-end completions of the corporate range; GPT-5.5 averaged 22.3 of 32 steps across ten 100M-token runs. On the control system range, the latest models reached 4 of 7 steps, primarily by directly probing the operational-technology protocol rather than following the intended chain of IT compromise and authenticated control. These results indicate rapid improvements in autonomous cyber capability and suggest that complex but undefended multi-step ranges are no longer reliably distinguishing the strongest frontier models. Future evaluations should emphasise hardened environments, active defences, defensive-control effectiveness, and operational security---priorities that grow more pressing as the threat of AI-driven cyber attacks becomes increasingly realistic.

Recent progress in multimodal, high-dimensional learning has enabled foundation models to process heterogeneous, large-scale data. However, at test time, acquiring all features or modalities can be prohibitively costly and often redundant. Sequentially selecting informative modalities is therefore critical, yet challenging when the downstream task or prediction target is unknown. To this end, we introduce ECHO-$k$, a task-agnostic and self-supervised learning principle for modality acquisition: we use a deep model’s internal pretrained representations (e.g., from a foundation model) as proxy targets that summarize cross-modal information. We provide theoretical guarantees in a linear model setting that motivate a reinforcement learning (RL) policy for sequential modality selection. Across benchmarks, the resulting acquisition strategies transfer reliably and achieve state-of-the-art performance on downstream tasks that are entirely unseen during selection. Our method provides a principled route to cost-aware test-time deployment, with implications for any multimodal system where measurements are expensive or time-constrained, and downstream tasks unknown a priori.

Discrete diffusion language models enable parallel token generation, offering a pathway to low-latency decoding. However, selecting tokens independently by marginal confidence limits effective parallelism: tokens that appear reliable in isolation can form incompatible configurations when several positions are updated at once. We introduce a training-free decoding framework that coordinates these parallel updates. At each forward pass, the method assigns a commit score to each masked position and refines these scores using pairwise interactions derived from the model's predictive distributions. A variational relaxation yields a simple fixed-point update that suppresses conflicting simultaneous commitments within a single forward pass.This mechanism allows the decoder to commit more tokens in parallel while maintaining competitive generation quality. The method is lightweight, requires no auxiliary model or retraining, and drops into existing diffusion decoding pipelines without modification. Experiments on reasoning and code-generation benchmarks show consistent improvements in the quality-latency trade-off.

Mean-field control (MFC) offers a scalable solution to the curse of dimensionality in multi-agent systems but traditionally hinges on the restrictive assumption of exchangeability via dense, all-to-all interactions. In this work, we bridge the gap to real-world network structures by proposing a rigorous framework for MFC on large sparse graphs. We redefine the system state as a probability measure over decorated rooted neighborhoods, effectively capturing local heterogeneity. Our central contribution is a theoretical foundation for scalable reinforcement learning in this setting. We prove horizon-dependent locality: for finite-horizon problems, an agent's optimal policy at time t depends strictly on its (T-t)-hop neighborhood. This result renders the infinite-dimensional control problem tractable and underpins a novel Dynamic Programming Principle (DPP) on the lifted space of neighborhood distributions. Furthermore, we formally and experimentally justify the use of Graph Neural Networks (GNNs) for actor-critic algorithms in this context. Our framework naturally recovers classical MFC as a degenerate case while enabling efficient, theoretically grounded control on complex sparse topologies.

We address the brittleness of Bayesian experimental design under model misspecification by formulating the problem as a max--min game between the experimenter and an adversarial nature subject to information-theoretic constraints. We demonstrate that this approach yields a robust objective governed by Sibson's $\alpha$-mutual information (MI), which identifies the $\alpha$-tilted posterior as the robust belief update and establishes the R\'enyi divergence as the appropriate measure of conditional information gain. To mitigate the bias and variance of nested Monte Carlo estimators needed to estimate Sibson's $\alpha$-MI, we adopt a PAC-Bayes framework to search over stochastic design policies, yielding rigorous high-probability lower bounds on the robust expected information gain that explicitly control finite-sample error.

Estimating the number of distinct elements in a data stream is well understood when repeated elements are identical. In modern settings, however, observations are high-dimensional and noisy, so repeated instances of the same object are only approximately similar -- for example, different images of the same individual may vary significantly at the pixel level. Classical sketches such as HyperLogLog rely on consistent hash values for identical elements and break down in this regime. Recent work on robust distinct counting in general metric spaces achieves $\tilde\Theta(\sqrt{n})$ memory, which is tight in the worst case. We show that substantially improved memory guarantees are possible under geometric structure common in learned representations. We introduce MaxSketch, a simple max-linear sketch built from random Gaussian projections, and prove that it succeeds in estimating the number of distinct latent objects. Concretely, we show that under this assumption $m = \tilde{O}(\log n / \varepsilon^4)$ random projections (and hence $\tilde{O}(\log n/\varepsilon^4)$ memory) suffice to recover the true distinct count within a $(1+\varepsilon)$ factor. Experiments on image streams confirm that MaxSketch accurately estimates distinct counts and generalizes beyond the training regime. Our results bridge classical streaming algorithms and modern representation learning, showing how geometric structure can fundamentally reduce the complexity of distinct counting.

AI systems must often maintain bounded memory of unbounded interaction histories. However, optimizing the memory representation, and its updates, for downstream task utility is difficult for standard reinforcement learning (RL) due to severe credit assignment bottlenecks induced by sparse, delayed rewards. We introduce MaxIM (Maximally Informative Incremental Memory), a framework that formulates incremental summarization as a capacity-constrained, information-theoretic reward-shaping problem. Grounded in value suboptimality bounds, we prove that minimizing sequential information loss requires maximizing the predictive log-likelihood of future, task-relevant targets (predictive sufficiency). We introduce a recursive consistency condition to ensure summaries are grounded by maximizing mutual information with the observed history. We mitigate credit assignment issues with a parameterized critic that computes step-wise variational lower bounds for dense reward shaping. Empirical evaluation on long-horizon benchmarks shows that MaxIM outperforms state-of-the-art baselines on comprehensive autorater metrics and downstream task utility, without expensive human preference labels.

Sparse autoencoders decompose activations into sparse, interpretable features and have become a central tool for mechanistic interpretability. Transcoders extend this approach from representation reconstruction to computation prediction by mapping MLP inputs to MLP outputs, making them attractive for circuit-level analysis. We ask whether a prominent sparse-autoencoder reliability fix, Matryoshka nested-prefix training, transfers to this two-hook setting. In sparse autoencoders, Matryoshka training improves absorption behavior by encouraging broad features to appear in early dictionary prefixes. In transcoders, we find that this protection does not transfer automatically. Across Gemma-2-2B MLP transcoders trained under a matched recipe, Matryoshka models form genuine prefix hierarchies, with reconstruction improving as later groups are added, but they do not inherit the absorption advantage observed for sparse autoencoders. BatchTopK achieves lower full absorption, lower FVU, and higher first-letter F1@1 in the canonical 16k comparison. Bridge controls recover the expected Matryoshka advantage when the cross-space map is removed, showing that the failure is specific to predicting MLP outputs from MLP inputs. We identify the mechanism as hierarchy misalignment. Prefix losses reward covariance with the reconstruction target, so early transcoder groups prioritize target-predictive output geometry rather than broad source-space semantic parents. These results show that hierarchical sparse-dictionary objectives do not preserve interpretability properties by default. For dictionaries that map between activation spaces, the key question is not whether a hierarchy forms, but what geometry that hierarchy is trained to organize.

Large-graph spectral learning promises global, trainable propagation beyond local neighborhoods, but explicit spectral layers remain difficult to use in stochastic large-graph training: their bases are commonly obtained by a full-graph spectralization step and their graph-wide contexts are naturally full-batch. Matrix-free polynomial filters avoid eigensolvers and train stochastically, but tie the response to finite propagation bases; explicit spectral-subspace models offer more flexible global responses, but usually pay for full-graph basis construction and context evaluation. We introduce Randomized Adaptive Spectral Estimation (RASE), a matrix-free stochastic training framework for explicit low-rank spectral-context layers. RASE constructs a refreshable randomized block-Krylov/Ritz basis from probe vectors and sparse matrix--vector products, reuses probe-derived summaries as control variates for mini-batch estimates of the spectral context, and keeps the response module replaceable across polynomial, Chebyshev, and transformer-style instantiations. We analyze Krylov/Ritz polynomial-action exactness for the probe-generated subspace and a control-variate variance identity that isolates when the mini-batch context estimator reduces variance. Across 14 node-classification benchmarks, RASE matches its eigenbasis-trained reference within seed noise where the eigensolver is feasible (${\sim}67{\times}$ faster basis on PubMed), and trains the same backbone on the largest graphs where eigensolver preprocessing exceeds a 30-minute budget. The contribution is therefore not a new spectral filter family, but a training route that makes explicit low-rank spectral-context layers matrix-free and mini-batch trainable.

Natural gradient descent is a theoretical pillar of second-order optimization for neural networks. Existing implementations either rely on structural approximations of the curvature matrix or incur substantial computational and memory overhead. Direct application remains difficult at deep-network scale. We build an optimizer based on an unbiased stochastic estimator of the Fisher information matrix. It employs a truncated exponential moving average metric with damping, which we call the Stochastic Truncated Metric (STM), and is otherwise closely aligned with the original natural gradient. This gives a matrix-free parameter update with strictly $O(Kd)$ time and memory per step, where $K$ is a small constant. It is applicable across network architectures without layer-wise assumptions. We bound the error induced by the finite-memory truncation. We show that STM is stable and efficient on vision and language benchmarks.

Unit-modulus measurement vectors, due to their phase-only variations, exhibit favorable hardware compatibility and storage efficiency. In this paper, we consider matrix recovery using symmetric unit-modulus rank-one measurements. We identify a fundamental limitation of symmetric unit-modulus measurements---their inability to capture individual diagonal entries, and show that matrix recovery can be achieved using such measurements when the diagonal entries are known or directly measurable. We construct a stacked operator and establish that it satisfies a mixed-norm restricted isometry property ($\mathrm{mRIP}$) for unit-modulus measurements. Leveraging the $\mathrm{mRIP}$ condition, we derive exact and stable recovery results for matrix recovery with unit-modulus measurements in both noiseless and noisy settings. Our work is the first to provide the theoretical foundation for matrix recovery under symmetric rank-one unit-modulus measurements. Numerical experiments further corroborate these theoretical findings.

Transition state (TS) search is the rate-limiting step in high-throughput kinetics screening. Generative models have been used to accelerate TS prediction for molecules, but extending them to periodic materials has been blocked by the absence of a large-scale materials TS dataset. We release MaterialsSaddles, the first such dataset: 34.14 M (reactant, saddle, product) triplets generated with the state-of-the-art UMA-S-1.2 universal interatomic potential across four subsets, comprising 31.35 M triplets over LeMat-Bulk's 5.34 M-structure deduplication of Materials Project, OQMD, and Alexandria; 2.59 M triplets over OC20 surfaces; 167 k triplets over OC22 oxide electrocatalysts; and 35 k climbing-image nudged elastic band (CI-NEB) paths over Materials Project battery cells. The release totals 102.4 M structures (687 GB) in ASE-LMDB shards with stratified 90/5/5 splits, generated on a 500,000 GPU-hour HPC pipeline. Every triplet is connected by construction via double-minimization from the converged saddle, and every frame carries per-frame metadata including eigenmodes, curvatures, bond-change diffs, and source identifiers. As a working demonstration of the dataset's intended use, we release SaddleFlow, a flow-matching reactant-and-product-conditional saddle-point generator built on a UMA-S-1.2 backbone with FiLM-based time conditioning and an SO(3)-equivariant velocity head, trained on the mp20bat subset of MaterialsSaddles. In a pilot DFT study, we show that dimer searches initialized from a sample of mp20bat test triplets converge substantially faster and more reliably than those initialized from the standard reactant-product midpoint guess, indicating that MaterialsSaddles and SaddleFlow together offer a practical path to scaling DFT-level transition-state searches. MaterialsSaddles (CC-BY 4.0), SaddleFlow (MIT), and the data-generation engine SaddleMill (MIT) are publicly released.

Recently, Noise2Noise provides a powerful principle for learning image denoising without clean data. However, zero-shot methods for 2D images often rely on local similarity or non-local self-similarity to construct image pair for training, which is unreliable in complex scenes with highly textures. Different from 2D image, 4D Light Field (LF) captures multiple views for one specific scene. The multiple views with independent noises provides make it possible to find image pair across views. In this paper, we exploit the inter-view correspondence in LF and propose a novel way to implement the Noise2Noise in 4D LF. Specifically, we design a Matching2Matching (M2M) framework to construct matching views for each view in LF. This framework generates geometrically consistent training samples by searching for corresponding patches along the epipolar line, crucially enforcing both visible and disparity consistency constraints. The constructed views enable the network to train the denoised network in a zero-shot way without any other data. Experimental results show that our method effectively suppresses noise while outstandingly preserving the consistency and fine details. Our M2M framework significantly outperforms existing zero-shot methods in both quantitative and qualitative evaluations on synthetic and real-world noise, demonstrating superior generalization ability without relying on external training datasets.

Single-step retrosynthesis prediction serves as the core operational unit for automated synthesis planning and remains a fundamental computational challenge in pharmaceutical design. While a critical prior in organic chemistry is that chemical reactions typically preserve the majority of the molecular scaffold, current template-free deep learning models struggle to effectively exploit this characteristic. Existing methods attempt to exploit this by minimizing sequence-level edit distance through carefully chosen graph traversal orders. However, this structural correspondence is only indirectly induced and remains highly sensitive to imperfect traversal heuristics, which limits its effectiveness for complex transformations. To address this limitation, we introduce the Matching-while-Decoding paradigm, replacing implicit sequence alignment with direct structural supervision. We operationalize this paradigm within a graph-to-sequence architecture that explicitly maps generated reactant tokens back to the input product graph during decoding. Furthermore, our hierarchical autoregressive formulation utilizes a custom causal attention mask to enable highly efficient, single-pass parallel training. Extensive experiments across the USPTO-50K, USPTO-MIT, and USPTO-FULL benchmarks reveal a critical accuracy and robustness trade-off in existing methodologies, where models either over-rely on augmentation or fail to leverage it. Our approach uniquely resolves this bottleneck by delivering highly competitive canonical accuracy while scaling exceptionally well with test-time augmentation, establishing new state-of-the-art records on USPTO-MIT and augmented USPTO-FULL. By providing this balanced stability across diverse inference protocols, our approach emerges as a highly promising foundation for dependable single-step retrosynthesis prediction.