NeurIPS 2026 Papers

Training transformer-based architectures with data augmentation has become an increasingly popular approach for equivariant machine learning. Despite its empirical success, the interplay between the transformer architecture, invariance to different symmetries, and augmentation budgets remains underexplored. In this paper, we investigate the ability of a vanilla transformer to learn a wide range of symmetries through limited data augmentation. We identify that it performs strongly for common angle-preserving symmetries, while it struggles with non-angle-preserving ones. Within the angle-preserving family, we further find that base subgroups such as translation, rotation, and scale are learned more easily and accurately than compositional groups that combine them. Motivated by this observation, we perform a structural analysis of the trained models and identify interpretable mechanisms that induce invariance to each base angle-preserving symmetry. These findings support what we coin winning the *Symmetry Lottery*: the transformer architecture is aligned with learning mechanisms invariant to symmetries that happen to be prevalent in scientific domains.

Quantum machine learning (QML) aims to accelerate machine learning tasks by exploiting quantum computation. Previous work studied a QML algorithm for selecting sparse subnetworks from large shallow neural networks. Instead of directly solving an optimization problem over a large-scale network, this algorithm constructs a sparse subnetwork by sampling hidden nodes from an optimized probability distribution defined using the ridgelet transform. The quantum algorithm performs this sampling in time $O(D)$ in the data dimension $D$, whereas a naive classical implementation relies on handling exponentially many candidate nodes and hence takes $\exp[O(D)]$ time. In this work, we construct and analyze a quantum-inspired fully classical algorithm for the same sampling task. We show that our algorithm runs in time $O(\operatorname{poly}(D))$, thereby removing the exponential dependence on $D$ from the previous classical approach. Numerical simulations show that the proposed sampler achieves empirical risk comparable to exact sampling from the optimized distribution and substantially lower than sampling from the non-optimized uniform distribution, while also exhibiting exponentially improved runtime scaling compared with the conventional classical implementation. These successful dequantization results show that sparse subnetwork selection via optimized sampling can be achieved classically with polynomial data-dimension scaling on conventional computers without quantum hardware, providing an alternative to the existing quantum algorithm.

We introduce WildBox, a dataset and benchmark for monocular 3D detection of wildlife from drone video, comprising 237,505 3D bounding box annotations across seven African savanna species grouped into six benchmark classes. Annotations follow a KITTI/Omni3D-compatible format in a per-segment scale-normalised camera frame, with instance identities maintained across each segment. We evaluate two open-vocabulary monocular 3D architectures, OVMono3D-LIFT and DetAny3D, under zero-shot, ground-truth 2D box prompt, and supervised fine-tuning protocols. Open-vocabulary 2D foundation models provide usable zero-shot wildlife localisation (50.55 AP@50), but zero-shot 3D detection collapses to 0.00 AP across both architectures and every 2D-input condition tested, including ground-truth 2D box prompts—isolating the failure to the 3D stage. Fine-tuning on WildBox recovers performance to 8.68$\pm$0.47 AP-BEV@0.50 and 13.17$\pm$0.69 AP3D macro. Depth contributes 84\% of normalised Hausdorff distance after fine-tuning and over 99\% in zero-shot, identifying monocular aerial depth as the dominant open problem in this regime. A coarse-to-fine curriculum—pretraining on a merged zebra class before fine-tuning on the Grévy's/plains split—improves macro 3D performance with less total compute, with the largest gains on the two zebra subclasses. WildBox is released with video-level splits, evaluation code, and baseline checkpoints to enable progress in 3D wildlife perception from drone video.

As frontier AI technologies dramatically improve their abilities to perform cybersecurity and reverse-engineering tasks, it is critical to rigorously and realistically track their capabilities in these domains. Increased agent capabilities in reverse-engineering will scale black-box vulnerability discovery, which will have dramatic implications on software security at large. In this work, we present WiREBench, a unique benchmark derived from popular real-world mobile applications, to measure agents' end-to-end vulnerability discovery and reverse-engineering capabilities. This benchmark, containing 22 protocols and 62 subtasks, evaluates agents abilities to identify and exploit historical vulnerabilities in binaries' proprietary network cryptography, via black-box reverse-engineering. To resolve these tasks, agents must chain together static binary analysis, understanding of applied cryptography, and complex tool use via dynamic instrumentation of mobile applications. Beyond benchmarking, we ran WiREBench against real-world applications, identifying and confirming 65 protocol vulnerabilities in applications used by hundreds of millions of users. These results demonstrate WiREBench's utility not just as a benchmark to assess current agent capabilities, but also as a pipeline for auditing cryptographic protocols and reverse-engineering in the real world.

Transformer models can adapt to new tasks from input--output examples at inference time, a capability known as in-context learning (ICL). A common interpretation is that transformers implement implicit learning algorithms in their forward pass, using the context to infer the task and predict the query. We test this interpretation beyond the training distribution in controlled regression settings where classical estimators can generalize exactly. Transformers achieve near-perfect in-distribution ICL, but fail sharply under scale and support shifts, exhibiting a cliff-shaped collapse that is absent from least-squares and kernel baselines. This failure is robust across polynomial degrees and a range of training interventions. We identify two architectural mechanisms that help explain this behavior. First, final normalization imposes a readout-side scale constraint, causing predictions to saturate at fixed boundary values. Removing this constraint eliminates hard saturation, but does not restore reliable ICL under coefficient or label scaling. Second, attention can become a context bottleneck: under distribution shift, softmax attention may concentrate on a small part of the prompt, limiting the context-dependent adaptation required for ICL. These results show that strong in-distribution ICL does not imply distribution-independent ICL, and suggest caution when using ICL instead of explicit retraining or adaptation under distribution shift.

We argue that cognitive control, the capacity to hold multiple divergent candidate responses and reconcile them through categorical inhibition rather than soft blending, should be a first-class computational object in transformer-based language models. We develop this claim at two levels. At the \emph{signal level}, interference paradigms reveal a characteristic failure signature: rather than adjudicating between competing responses, current transformer-based language models blend them, and accuracy collapses as interference scales. At the \emph{goal level}, sustained agency requires maintaining concurrent goals, enforcing priority during signal level adjudication, and revising the goal set as evidence accumulates about whether those goals are being successfully pursued; current systems approximate this only through external scaffolding. The architectural gap revealed at the signal level is the same gap that will block transformers from scaling to the kind of goal modulated and self regulating cognition that long-horizon autonomous agents require. We define concrete criteria for what counts as architecturally explicit cognitive control, show that neither the dominant training pipeline nor reasoning-token training implements it, and commit to predictions that distinguish our position from scale-based optimism.

Large language models achieve high reasoning performance via explicit chain-of-thought and reinforcement learning, but require long output sequences and extended inference time. Latent reasoning reduces this cost by shifting computation into a latent space; however, continuous latent methods are hard to train, suffering from unstable and uninterpretable reasoning trajectories. We argue these issues stem from a misalignment between continuous-space reasoning and discrete symbolic supervision, as continuous states lack explicit anchors for step-by-step alignment. To resolve this, we propose **Discrete Latent Reasoning (DLR)**, the first method that converts continuous latent states into explicit discrete tokens. Inspired by render-based compression, we render textual chains of thought into images, extract visual features, and construct a discrete latent vocabulary via clustering-based fine-tuning. Expanding the vocabulary and output head enables standard autoregressive modeling over both natural language and latent tokens, supporting pretraining alignment, SFT, and RL. Experiments on five reasoning benchmarks and two model series (Qwen3-VL and LLaMA-3) confirm that **DLR** outperforms prior latent reasoning baselines with up to **20$\times$ compression**. Furthermore, the learned latent trajectories retain an interpretable semantic structure. Overall, discrete latent tokens provide a controllable and interpretable basis for efficient latent reasoning.

Mixture-of-Experts (MoE) architectures are central to large-scale deep learning, relying on sparse execution and expert specialisation. Despite their success, their large-scale training dynamics remain poorly understood, and training is often unstable. While scaling limits such as infinite-width theory have clarified the behavior of dense models, the scaling behavior of MoEs remains largely unexplored. To this end, we derive the infinite-width limits of MoE architectures under fixed number of experts with both soft and Top-$K$ routing, trained with SGD and Adam, using Tensor Programs. Our results show that under the Standard Parameterisation (SP), router updates freeze after one step of training in both soft and Top-$K$ routing MoEs. This provides, at least in part, a principled explanation for the widespread use of auxiliary losses such as load-balancing or $z$-losses in MoE training. In dense networks, Maximal Update heuristics allow for stable and non-vanishing feature updates. However, in MoEs the Maximal Update heuristic with soft-routing and softmax gating causes the experts to collapse to the same distribution and router gradients to vanish. While sigmoid allows non-trivial router evolution, they still fail to prevent expert collapse. Thus fixed expert soft-routing MoEs do not admit a parameterisation that simultaneously ensures stability, feature learning, and expert specialisation in the infinite-width limit. We then show that, beyond computational efficiency, Top-\(K\) also acts as a symmetry breaking mechanism allowing for both feature learning and specialisation. We empirically validate these predictions, observing close agreement between finite-width training dynamics and the predicted scaling behaviour.

Pass@$k$ is a widely used performance metric for verifiable large language model tasks, including mathematical reasoning, code generation, and short-answer reasoning. It defines success if any of $k$ independently sampled solutions passes a verifier. This multi-sample inference metric has motivated inference-aware fine-tuning methods that directly optimize pass@$k$. However, prior work reports a recurring trade-off: pass@$k$ improves while pass@1 degrades under such methods. This trade-off is practically important because pass@1 often remains a hard operational constraint due to latency and cost budgets, imperfect verifier coverage, and the need for a reliable single-shot fallback. We study the origin of this trade-off and provide a theoretical characterization of when pass@$k$ policy optimization can reduce pass@1 through gradient conflict induced by prompt interference. We show that pass@$k$ policy gradients can conflict with pass@1 gradients because pass@$k$ optimization implicitly reweights prompts toward low-success prompts; when these prompts are what we term \emph{negatively interfering}, their upweighting can rotate the pass@k update direction away from the pass@1 direction. We illustrate our theoretical findings with large language model experiments on verifiable mathematical reasoning and coding tasks.

We present MaskGen, a theoretically grounded and deliberately simple approach for domain generalization in 3D biomedical image segmentation. Modern segmentation models degrade sharply under shifts in modality, disease severity, clinical sites, and more, limiting their reliable adoption. Existing generalization methods address this using extreme augmentations, hand-engineered domain statistics mixing, or architectural redesigns that add significant implementation overhead while yielding inconsistent performance across biomedical settings. MaskGen instead presents a principled learning strategy with marginal overhead that utilizes both source-domain image intensities and domain-stable foundation model representations to train robust segmentation models. As a result, MaskGen achieves strong gains in both fully supervised and few-shot segmentation across broad clinical shifts in biomedical studies. Unlike prior approaches, MaskGen is architecture- and loss-agnostic, compatible with standard augmentation pipelines, easy to implement, and tackles arbitrary anatomical regions.

A solid understanding of the predictive success of deep neural networks (DNNs) remains elusive. For example, while many data-dependent metrics exist that seek to predict model quality of DNNs, these metrics may be expensive to compute for large datasets and/or may be impossible to compute when training datasets are not released. A practical and effective predictor of model quality involves analyzing DNN weight matrices, as it is known that a heavy-tailed spectral distribution often correlates with strong model quality. As such metrics lack a rigorous statistical derivation, in this work we aim to understand why they perform well, by employing a probabilistic framework to derive the marginal likelihood for the trained weights of a DNN. Across large-scale convolutional DNNs and LLMs, we find that this marginal likelihood predicts model quality. To understand the role heavy-tailed spectra play in predictive performance, we derive the limiting log-marginal likelihood for spectra that follow the heavy-tailed High-Temperature Marchenko-Pastur (HTMP) distribution. We show that this limiting quantity is convex in a *heaviness* parameter, and we derive an optimal *heaviness* that empirically predicts over-fitting for large-scale DNNs during training.

The effectiveness of modern deep learning models in forecasting *groups of time series* is often attributed to their ability to capture (long-range) dependencies across long observation windows. In this paper, we show that this forecasting task involves two objectives: (i) *generative process identification* (GPI), i.e., inferring the specific process generating the input sequence, and (ii) *conditional forecasting* (CF), i.e., predicting future values given input observations. From this perspective, optimal predictions can be interpreted as an average over plausible data-generating processes, weighted by their likelihood given the input window. This suggests a different explanation for the benefits of long context windows: they reduce the uncertainty about which specific process is generating the input time series during operation. We prove that even for processes with memory length $P$, an input window size strictly larger than $P$ is *necessary* to achieve the minimum attainable error. Finally, we show how decoupling GPI and CF can improve computational scalability without compromising accuracy. Experiments on synthetic and real-world data validate our insights and their relevance for designing forecasting architectures.

Recent advances in diffusion transformers (DiTs) have enabled promising single-turn image editing capabilities. However, multi-turn editing often leads to progressive semantic drift and quality degradation. In this work, we study this problem from a latent-space frequency perspective by decomposing the editing process into two functional components: VAE and DiT. Through systematic analysis in the VAE latent space, we uncover that the DiT introduces dominant low-frequency drift that accumulates as semantic misalignment across editing rounds, while the VAE contributes comparatively stable reconstruction bias. Based on this insight, we propose **VAE-LFA** (Low Frequency Alignment), a training-free, plug-and-play method that performs alignment in VAE latent space. VAE-LFA decomposes latent discrepancies across editing rounds via low-pass filtering, and aligns low-frequency statistics to an exponential moving average of previous rounds, effectively suppressing accumulated semantic drift while preserving high-frequency details. Our method requires *no retraining, ground-truth priors, or access to diffusion parameters*, making it applicable to both white-box and black-box DiT editors. For white-box models, VAE-LFA is seamlessly integrated into the editing pipeline by eliminating redundant VAE round trips; for black-box models, it operates via an off-the-shelf VAE to perform inter-round latent alignment. Extensive experiments demonstrate that VAE-LFA improves semantic consistency and visual fidelity across diverse multi-turn editing scenarios, including both controlled and in-the-wild images.

Cross-skeleton motion generation trains generative models to carry action structure and motion intention from one body to another. Yet an action-consistent target motion admits two observationally compatible explanations: source-preserving transfer and target-action recovery. We show that this ambiguity is structural rather than incidental: under standard generative objectives, the source-conditioned retargeting map is non-identifiable in sparse heterogeneous motion domains. Unpaired marginal matching yields \emph{gauge non-identifiability}: a relative gauge between skeleton-specific latent spaces allows different source-conditioned maps to induce the same training evidence. Sparse paired supervision admits the complementary failure mode, \emph{conditional-mean degeneration}: ambiguous action-level pairings drive squared-error objectives toward a target-action prototype that is independent of the source clip's motion intention. To make the missing evidence observable, we introduce \emph{Source-Instance Fidelity} (SIF), a diagnostic that tests whether the source-side geometry remains visible in the generated target motions after the target skeleton and action are fixed. Under this diagnostic, standard action-level success across animal motion domains recurrently coincides with behavior at the \emph{source-blind floor}, while positive cases localize source-instance information to auxiliary motion-space constraints that narrow the observational equivalence class. The implication is that retargeting requires objectives and evaluations capable of identifying the source-conditioned map that a retargeting claim asserts. [Project page](https://keen-semolina-09811d.netlify.app/).

When is it better to rely on information from others rather than experiment to discover the answer yourself? Social learning refers to the ability of humans to learn by observing, imitating, or interacting with others. In this work, we use multi-agent reinforcement learning (MARL) to investigate the conditions under which social learning achieves higher returns and improved learning efficiency relative to individual learning. We begin with a theoretical analysis and discover that, under coordinated group exploration and low skill transmission costs, social learning achieves lower sample complexity than individual learning. Motivated by this insight, we propose Selective Social Learning (SSL), a novel social learning algorithm that reduces the cost of skill transmission, and we design experiments across three MARL environments to test these predictions. Empirical results show that SSL outperforms baselines under conditions consistent with our theory. Surprisingly, we also find that in non-stationary environments with sparse rewards, which mirror the real world, social learning can emerge in standard MARL without an explicitly designed social learning mechanism. This suggests that the apparent absence of social learning in standard RL stems from limitations in training environment design rather than in RL itself. We therefore call for future MARL training environments to inherently incorporate sparsity and non-stationarity, enabling agents to naturally develop social-learning behaviors that transfer to complex real-world settings.

Cell-free DNA (cfDNA) liquid biopsy has been clinically routine for cancer detection and monitoring for years but has yet to produce applications in chronic diseases of parenchymal organs such as fibrotic liver, kidney, and lung disease, despite diseased organs shedding orders of magnitude more cfDNA into the bloodstream than tumors do. This gap is structural: methods developed for cancer have been adapted for chronic disease without a theoretical framework establishing when this should work or where it should break down. We formalize cfDNA classification as recovering a disease-induced shift on a low-dimensional submanifold of the simplex over molecular configurations, observed through sparse genomic windows. The formalization reveals that cancer and chronic disease occupy structurally distinct regimes parameterized by one dimensionless quantity: the inverse participation ratio $c(v)$ of the disease perturbation across windows. Cancer is small-$c(v)$: signal on a few genomic hotspots. Chronic disease is large-$c(v)$: signal spread across many co-regulated regulatory programs. We prove an information-theoretic lower bound: any classifier observing $k$ windows recovers at most a $\sqrt{k/c(v)}$ fraction of the available information in the local-asymptotic-normality regime, regardless of per-cfDNA model complexity. The bound applies architecture-agnostically to classifiers that focus deep computation on a small number of pre-selected genomic regions, i.e. the architectural shape that has succeeded in cancer detection. Such classifiers degrade sharply rather than gradually as $c(v)$ grows, with the transition provable from the geometry of the problem. Chronic-disease classification requires an architecture matched to its regime: compute allocated across a very large set of windows rather than into deeper per-cfDNA models, with $k \gtrsim c(v)$. We propose a concrete architecture and verify on synthetic experiments that the regime transition predicted by the theorem is observed, and the architecture outperforms cancer-style baselines on chronic-like signal. We then validate on real patient data, where the architecture exceeds both cancer-tuned methods and standard-of-care clinical tests.

Modern large language models (LLMs) often exhibit overconfident errors, assigning high confidence to incorrect predictions and thereby undermining reliability in real-world use. Existing calibration methods typically rely on post-hoc adjustment or retraining, operating at the output level without addressing the underlying causes of miscalibration. We propose CoCHI (Class-oriented Causal Head Intervention), a framework that improves calibration by identifying and intervening on internal components responsible for distinct confidence behaviours. By analysing predictions through correctness–confidence patterns, CoCHI enables targeted intervention on model internals, reducing overconfident errors while preserving correct predictions. Our approach provides a mechanistic perspective on calibration, linking reliability to identifiable structures within the model rather than treating it as an output-level issue. Experiments across multiple LLMs show consistent improvements in calibration metrics, including Expected Calibration Error, Negative Log-Likelihood, and Brier Score, while maintaining accuracy on five multiple-choice question answering benchmarks. These results suggest that effective calibration requires not only adjusting outputs, but also addressing the underlying causal mechanisms within the model. Implementation details and code are provided in the supplementary material.

Automated failure attribution uses LLMs to identify where and why agentic systems fail. As agents become more capable, their failures become subtler, making automated attribution increasingly important. We introduce Who&When Pro, a large-scale benchmark for automated failure attribution in agentic systems. Using a strictly controlled pipeline that injects a failure only after exactly replaying a successful prefix, we construct 12,326 failed trajectories with golden labels across 3 modalities and 26 benchmarks covering various scenarios. Beyond benchmarking, we conduct extensive experiments and analyses, revealing systematic patterns in how models attribute failures across modalities, protocols, and model families, and providing empirical guidance for future automated failure attribution systems.

We study identifiability and estimation of the direct causes, that is, the parents, of a designated target $Y$ in a structural causal model using only observational data. Our main assumption is an additive-noise model for $Y$: the value of $Y$ equals some function of its parents plus noise that is independent of those parents. We allow for general functional relationships and hidden confounding among all other variables. We prove that under a `no-backward' additive-noise condition the true parent set, $\mathrm{PA}(Y)$, is identified from the observational distribution by a simple population principle: among all candidate variable sets that make the regression residual of $Y$ independent of the regressors, choose the one with the smallest residual variance, and—if several tie—the smallest set. To justify the ``no-backward'' condition, we propose a novel identifiability scheme based on identifiability witnesses: we prove that in finite-dimensional analytic model classes, for which we have an identifiability witness (that is, a single identifiable model), residual independence for sets containing descendants of \(Y\) holds only on Lebesgue-null exceptional parameter sets. This scheme is strong enough to recover known identifiability results. For finite data, we propose Independent Risk Minimization (IndRM), which offers a simple, local, and non-interventional route to isolating the direct causes of $Y$ in multivariate systems, avoiding global model assumptions and full-graph search.