NeurIPS 2026 Papers

Transformer models can adapt to new tasks from input--output examples at inference time, a capability known as in-context learning (ICL). A common interpretation is that transformers implement implicit learning algorithms in their forward pass, using the context to infer the task and predict the query. We test this interpretation beyond the training distribution in controlled regression settings where classical estimators can generalize exactly. Transformers achieve near-perfect in-distribution ICL, but fail sharply under scale and support shifts, exhibiting a cliff-shaped collapse that is absent from least-squares and kernel baselines. This failure is robust across polynomial degrees and a range of training interventions. We identify two architectural mechanisms that help explain this behavior. First, final normalization imposes a readout-side scale constraint, causing predictions to saturate at fixed boundary values. Removing this constraint eliminates hard saturation, but does not restore reliable ICL under coefficient or label scaling. Second, attention can become a context bottleneck: under distribution shift, softmax attention may concentrate on a small part of the prompt, limiting the context-dependent adaptation required for ICL. These results show that strong in-distribution ICL does not imply distribution-independent ICL, and suggest caution when using ICL instead of explicit retraining or adaptation under distribution shift.

We argue that cognitive control, the capacity to hold multiple divergent candidate responses and reconcile them through categorical inhibition rather than soft blending, should be a first-class computational object in transformer-based language models. We develop this claim at two levels. At the \emph{signal level}, interference paradigms reveal a characteristic failure signature: rather than adjudicating between competing responses, current transformer-based language models blend them, and accuracy collapses as interference scales. At the \emph{goal level}, sustained agency requires maintaining concurrent goals, enforcing priority during signal level adjudication, and revising the goal set as evidence accumulates about whether those goals are being successfully pursued; current systems approximate this only through external scaffolding. The architectural gap revealed at the signal level is the same gap that will block transformers from scaling to the kind of goal modulated and self regulating cognition that long-horizon autonomous agents require. We define concrete criteria for what counts as architecturally explicit cognitive control, show that neither the dominant training pipeline nor reasoning-token training implements it, and commit to predictions that distinguish our position from scale-based optimism.

Large language models achieve high reasoning performance via explicit chain-of-thought and reinforcement learning, but require long output sequences and extended inference time. Latent reasoning reduces this cost by shifting computation into a latent space; however, continuous latent methods are hard to train, suffering from unstable and uninterpretable reasoning trajectories. We argue these issues stem from a misalignment between continuous-space reasoning and discrete symbolic supervision, as continuous states lack explicit anchors for step-by-step alignment. To resolve this, we propose **Discrete Latent Reasoning (DLR)**, the first method that converts continuous latent states into explicit discrete tokens. Inspired by render-based compression, we render textual chains of thought into images, extract visual features, and construct a discrete latent vocabulary via clustering-based fine-tuning. Expanding the vocabulary and output head enables standard autoregressive modeling over both natural language and latent tokens, supporting pretraining alignment, SFT, and RL. Experiments on five reasoning benchmarks and two model series (Qwen3-VL and LLaMA-3) confirm that **DLR** outperforms prior latent reasoning baselines with up to **20$\times$ compression**. Furthermore, the learned latent trajectories retain an interpretable semantic structure. Overall, discrete latent tokens provide a controllable and interpretable basis for efficient latent reasoning.

Despite the superior performance of Large Reasoning Models (LRMs), their reasoning behaviors are often counterintuitive, *e.g.*, excessive reasoning on simple questions or insufficient reasoning on complex ones, leading to suboptimal reasoning capabilities. This paper presents the Laws of Reasoning (LoRe), a unified framework that formalizes intrinsic reasoning patterns in ideal LRMs. We first propose *compute law* with the hypothesis that the reasoning compute should scale linearly with question complexity. Beyond compute, we extend LoRe with a supplementary *accuracy law*, positing exponential decay of accuracy with increasing complexity. Since the complexity is difficult to quantify in practice, we approximate these hypotheses via two tractable properties, *monotonicity* and *compositionality*. We therefore introduce LoRe-Bench, a benchmark that systematically measures these properties for large reasoning models. Evaluation shows that most reasoning models exhibit reasonable monotonicity but lack compositionality. In response, we develop an effective finetuning approach that enforces compute-law compositionality. Extensive empirical studies demonstrate that better compliance with compute laws yields consistently improved reasoning performance on multiple benchmarks, and uncovers synergistic gains across properties and laws.

Large language models (LLMs) have demonstrated strong capabilities on complex reasoning tasks. However, recent studies~\citep{gekhman2026thinking,song2026large,jin2025disentangling,cheng2024understanding} suggest that when handling knowledge-intensive reasoning tasks, LLMs often fail to effectively utilize the knowledge acquired during pre-training, which limits their reasoning performance. To investigate how internal knowledge (also termed memory) is used by an LLM when dealing with a reasoning query, we propose the Memory-to-Reasoning Alignment (MRA) metric to measure the memory engagement level of an LLM on the reasoning query. We empirically verify that failures in reasoning may correspond to insufficient memory engagement. Inspired by this, we propose a training-free Memory-Guided Activation Injection (MGAI) method to increase the memory engagement level of an LLM, in order to improve the reasoning capability of an LLM. Specifically, given a query, MGAI generates an intervention vector and applies it to the reasoning representation to inject the corresponding memory information into the reasoning representation of the LLM. Furthermore, we propose an adaptive intervention strategy to control which queries require intervention and the magnitude of the intervention. Experiments show that our method consistently improves performance across multiple reasoning benchmarks while largely preserving the memory and general capabilities of LLMs.

Humans can approach complex visual problems by mentally simulating intermediate visual steps, rather than reasoning through language alone. Inspired by this, several works on Vision-Language Models have recently explored chain-of-thought reasoning with continuous latent tokens as intermediate visual imagination steps. In this work, we investigate how recent models leverage such latent tokens. Surprisingly, we find that model accuracy is unaffected when latent tokens are replaced by uninformative dummy tokens. This indicates that latent tokens play a minimal causal role in the model's final prediction. To better understand this phenomenon, we analyze both the training signal provided by oracle latent representations and the quality of the latent tokens generated at inference time. Our experiments reveal two crucial issues holding back latent visual reasoning: First, in most existing datasets, oracle latent tokens provide limited additional information beyond the original image and do not substantially simplify the task, leading models to ignore them during training and effectively bypassing them at inference time. When fine-tuned on a diagnostic dataset, in which latent tokens provide sufficient support for the final prediction, we show that models can causally rely on them. Second, the latent tokens produced at inference time deviate from their corresponding oracle representations, collapsing to a narrow region and preventing benefits even when the model relies on them. Overall, our findings suggest that future progress in latent visual reasoning depends on two key pillars: high-quality datasets with informative intermediate steps and more precise latent token prediction.

Large language models are increasingly deployed with test-time strategies: sample $N$ responses, score them with a reward model or verifier, and return the best. This deployment rule exposes a mismatch in post-training: standard objectives optimize the mean reward of a single response, whereas best-of-$N$ performance is governed by the upper tail of the reward distribution. Recent test-time-aware objectives partly address this mismatch, but typically assume that training can use the same per-prompt rollout budget as deployment, which is impractical when post-training must cover many prompts while deployment can allocate much larger per-prompt test-time compute. We study this budget-mismatch regime, where only $m\ll N$ per-prompt rollouts are available during training but the target objective is best-of-$N$ deployment. Under structural assumptions on the reward tails, we show that the policy gradient of the best-of-$N$ objective can be approximated from a much smaller rollout group by extrapolating upper-tail statistics. This yields a family of Tail-Extrapolated estimators for best-of-$N$-oriented post-training: a simple direct estimator, Tail-Extrapolated Advantage (TEA), and a fixed-order debiased Prefix-TEA estimator based on moment cancellation. Experiments on instruction-following tasks show that TEA and Prefix-TEA improve best-of-$N$ performance across different language models, reward models and datasets under various training and test-time budget settings.

Evaluation of multimodal reasoning models is typically reduced to a single accuracy score, implicitly treating reasoning as a unitary capability. We introduce MathLens, a benchmark of textbook-style geometry problems that exposes this assumption by operationally decomposing performance into perception, reasoning, and multimodal-specific components. Each problem is derived from a symbolic specification and accompanied by visual diagrams, text-only variants, multimodal questions, and targeted perceptual probes, enabling controlled measurement of each component. Using this decomposition, we show that common training strategies induce systematically different capability profiles that are invisible under aggregate accuracy. Reinforcement learning primarily improves perceptual grounding and robustness to diagram variation, while textual SFT yields gains through reflective reasoning. In contrast, as perception and reasoning improve, a growing fraction of remaining errors fall outside these components and are categorized as multimodal-specific. These results suggest that apparent progress in multimodal reasoning reflects shifting balances among subskills rather than uniform advancement, motivating evaluation beyond scalar accuracy.

An important characteristic of temporal graphs is how the directed arrow of time influences their causal topology, i.e., which nodes can possibly influence each other causally via time-respecting paths. The resulting patterns are often neglected by temporal graph neural networks (TGNNs). To formally analyze the expressive power of TGNNs, we lack a generalization of graph isomorphism to temporal graphs that captures their causal topology. Addressing this gap, we introduce consistent event graph isomorphism, which utilizes a time-unfolded representation of time-respecting paths in temporal graphs. We compare this definition with existing notions of temporal graph isomorphisms. We highlight the advantages of our approach and develop a temporal generalization of the Weisfeiler-Leman algorithm to heuristically distinguish non-isomorphic temporal graphs. Building on this theoretical foundation, we derive a novel message passing scheme for temporal graph neural networks that operates on the event graph representation of temporal graphs and experimentally evaluate it in a temporal graph classification setting.

Video reasoning has a distinctive structure: in a chain of thought over video, most tokens follow from prior text, but a small minority hinge on specific moments of the visual input---and which tokens those are varies from one example to the next. Current post-training procedures like GRPO supply supervision that is coarse in exactly this dimension, applying a single scalar reward uniformly across the trajectory while requiring many rollouts and/or costly external judge models. Our key observation is that a model can audit its own visual reliance: at any position, the shift in its next-token distribution when the video is removed from context measures how much that prediction actually used what was seen. Computing this quantity for both a student and a privileged teacher yields a per-token \emph{reliance gap} that localizes positions at which the teacher's predictions depend on the video and the student's do not. We introduce Visually-Aware On-Policy Self-Distillation (VIOS), which turns this self-auditing signal into dense per-token supervision recovered from a single rollout of a single model---no external reward, judge, or larger teacher required. VIOS combines three components: an exponential moving average (EMA) teacher that co-evolves with the student to avoid the ceiling of a frozen rationalizer; a contrastive term that increases the student's sensitivity to the video input; and a reliance gap gate that concentrates supervision on the positions identified. Across 12 benchmarks spanning general, temporal, spatial, knowledge, and long-video understanding, VIOS consistently outperforms its base model and contemporary video-reasoning systems while training in over $9\times$ fewer steps.

This paper introduces Walk3D, a novel 3D Gaussian Splatting (3DGS) framework explicitly designed for 3D referring segmentation, particularly excelling at interpreting complex, long-sentence descriptions. Our primary motivation stems from observing that existing 3DGS-based open-vocabulary methods mainly focus on simple, category-level object segmentation. These methods struggle to precisely segment targets described by highly detailed, multi-sentence queries due to weak compositional language understanding and a lack of structural scene awareness. To tackle these dense and complex linguistic constraints, we propose a spatial routing network that decodes these queries into sequential chain-of-thought instructions upon a scene graph. Specifically, we construct a static supernode graph based on the geometric and semantic properties of the 3DGS space, where a spatial Graph Neural Network (GNN) dynamically pathfinds using the decomposed linguistic chain to accurately segment the referred target. Extensive experiments on complex 3D referring segmentation benchmarks demonstrate the superiority of our proposed method in interpreting lengthy, multi-conditional queries. Our code and trained models will be publicly released.

In this paper, we revisit the concept of entropy in the spatial context with the aim of deriving computable and interpretable metrics for point pattern analysis in domains such as histopathology. We discuss stochastic point process assumptions like Poisson homogeneity and Simple Sequential Inhibition (SSI) and review established results for Voronoi and Delaunay tilings of point sets for their potential to provide null distributions for rigorous empirical hypothesis testing. We present (1) a novel ``dense basin entropy'' defined in terms of the Ord distribution supported on Voronoi basins, which is shown to be sensitive to clustering, and (2) a related Markov chain designed as a simplification or approximation of Brownian motion in the underlying planar domain. We prove a lower bound on the dense basin entropy in the SSI regime (originally discovered in empirical experiments) and show that the entropy rate and spectral gap of the Markov chain lead to sharp discrimination metrics. Finally, we demonstrate the generalization to multiple-set analysis via aggregation of one set over the Markov invariant measure for another. Simulations and experiments in histopathology show that our proposed metrics offer insights complementary to other instantiations of entropy in spatial analysis.

Multimodal latent reasoning has emerged as a promising paradigm that replaces explicit Chain-of-Thought (CoT) decoding with implicit feature propagation, simultaneously enhancing representation informativeness and reducing inference latency. By analyzing token-level gradient dynamics during latent training, we reveal two critical observations: (1) visual tokens exhibit significantly smaller gradient norms than their textual counterparts due to inherent language bias, resulting in systematic visual under-optimization; and (2) semantically simple tokens converge rapidly, whereas complex tokens exhibit persistent gradient instability constrained by fixed architectural depths. To address these limitations, we propose a visual replay module and routing depth scaling to collaboratively enhance visual perception and refine complicated latents for deeper contextual reasoning. The former module leverages causal self-attention to estimate token saliency, reinforcing fine-grained grounding through spatially-coherent constraints. Complementarily, the latter mechanism adaptively allocates additional reasoning steps to complex tokens, enabling deeper contextual refinement. Guided by a curriculum strategy that progressively internalizes explicit CoT into compact latent representations, our framework achieves state-of-the-art performance across diverse benchmarks while delivering substantial inference speedups over explicit CoT baselines.

Embodied agents are expected to assist humans by actively exploring unknown environments and reasoning about spatial contexts. When deployed in real life, agents often face sequential tasks where each new task follows the completion of the previous one and may include infeasible objectives, such as searching for non-existent objects. However, most existing research focuses on isolated goals, overlooking the core challenge of sequential tasks: the ability to reuse spatial knowledge accumulated from previous explorations to guide subsequent reasoning and exploration. In this work, we investigate this underexplored yet practically significant embodied AI challenge. Specifically, we propose 3DSPMR, a 3D SPatial Memory Reasoning framework that utilizes Field-of-View (FoV) coverage as an explicit geometric prior. By integrating FoV-based constraints, 3DSPMR significantly enhances an agent’s memory, reasoning, and exploration capabilities across sequential tasks. To facilitate research in this area, we further introduce SEER-Bench, a novel Sequential Embodied Exploration and Reasoning Benchmark that spans two foundational tasks: Embodied Question Answering (EQA) and Embodied Multi-modal Navigation (EMN). SEER-Bench uniquely incorporates both feasible and infeasible tasks to provide a rigorous and comprehensive evaluation of agent performance. Extensive experiments verify that 3DSPMR achieves substantial performance gains on both sequential EQA and EMN tasks.

Reasoning models improve accuracy through extended chains of thought, but their long outputs create a memory and compute bottleneck. KV cache eviction methods reduce this cost by evicting unimportant key-value pairs from the cache, yet they suffer larger accuracy degradation than selection-based sparse attention alternatives, which keep the full KV cache. We identify key factors crucial to KV cache eviction accuracy. First, a small fraction of value states have abnormally large magnitudes, and evicting them causes catastrophic failure where models enter repetitive reasoning loops. Second, introducing stochasticity during eviction improves accuracy by increasing cache diversity. Based on these findings, we propose **V**alue-**a**ware **S**tochastic KV Cache **E**viction (VaSE), a training-free recipe that protects large-magnitude value states and promotes diverse eviction decisions. Across six reasoning tasks, Qwen3 models using VaSE with 4x KV cache compression achieve comparable or better average accuracies to the SOTA selection method under the same level of sparsity, while enabling a static memory footprint that selection methods cannot offer. Our method also generalizes as a recipe, with its key factors improving strong eviction methods by 5 points on average.

Transformers are powerful models but are inefficient with large context due to the quadratic complexity of the attention mechanism. Limiting their context window addresses the efficiency concern but at a cost: they cannot remember everything from the past, which severely limits their computational power. We show that we can simultaneously achieve efficiency and universal computation if we restrict attention heads to be 2-dimensional. First, we give the first algorithm for standard 2D attention that serves every insertion and every query in amortized polylogarithmic time per token. Prior work either restricted key/query magnitudes or paid polynomial overhead per step. Second, we show that the class of 2D-attention models is universal and efficient: a vanilla decoder transformer efficiently simulates an arbitrary RAM machine via chain-of-thought decoding so that, combined with our first result, the simulation runs in $\mathrm{polylog}(N)$ time per simulated RAM step end-to-end. In contrast, earlier universality results addressed only computability, showing Turing completeness while incurring large polynomial factors in their execution time. Third, as an application, we turn a transformer into a computer. We provide a compiler that supports the full $\texttt{i32}$ WebAssembly subset with integer computations (reachable from C via $\texttt{clang}$) and turns such programs into the weights of a transformer, which then executes them over millions of steps autoregressively. We illustrate how this can be used to solve combinatorial problems, tasks at which even large reasoning LLMs struggle.

Generalization to harder reasoning and planning problems remains a central challenge for AI systems. Although both domains can be viewed as search problems where valid solutions are easier to verify than generate, existing methods often rely on domain-specific solvers or diffusion-based energy models tailored to individual settings. We introduce Energy Minimization Search} (EMS), a unified framework for reasoning and planning that learns a single time-invariant energy landscape. Unlike diffusion-based energy-based models (EBMs), which perform annealed inference over time-dependent landscapes, EMS performs direct gradient-based optimization on one static landscape using the same training objective and inference procedure across domains. Across reasoning tasks (graph coloring, N-Queens, 3-SAT) and planning tasks (path finding and maze solving), EMS matches or outperforms diffusion-based EBMs, domain-specific solvers, and diffusion-based planners while using substantially less inference compute.

We consider two-layer neural networks trained in the feature-learning regime using gradient descent, and relate the output of the finite-width network $f_{\hat{\rho}}$ to its infinite-width counterpart $f_{\rho^{MF}}$, which evolves in the mean-field dynamics. While constant-time horizon bounds for $\|f_{\rho^{MF}} -f_{\hat{\rho}}\|$ may be obtained via standard Grönwall estimates, the long-time behavior of the fluctuation is a more delicate matter. Uniform-in-time bounds often rely on (local) strong convexity in the landscape or Logarithmic Sobolev inequalities present in noisy gradient dynamics. % In this work, we study a noiseless setting and do not make assumptions on the geometry of the landscape near the optimum. Instead, we impose a weaker condition on the convergence rate of the mean-field deterministic Wasserstein-gradient-flow dynamics. In this work, we establish non-asymptotic weak propagation-of-chaos that holds uniformly in time, obtained by exploiting instead the convergence rate of the mean-field deterministic Wasserstein-gradient-flow dynamics. Specifically, denoting by $L_t$ the mean-field loss at time $t$ and $m$ the number of neurons, under standard regularity assumptions and the condition $\int_0^\infty L_t^{1/2} dt =O_d(1)$, we obtain the uniform in time bound $\|f_{\rho^{MF}} -f_{\hat{\rho}}\|^2 \lesssim \text{poly}(d) m^{-\min(1,c/6)}$ whenever $L_t \lesssim t^{-c}$. Our result holds in a noiseless setting and does not make any assumptions on the geometry of the landscape near the optimum. A key implication of our result is that whenever the convergence rate of the mean-field, population-loss dynamics is faster than $1/t^2$, we can attain a loss of $\epsilon$ with only $\text{poly}(d/\epsilon)$ neurons, training samples, and GD steps.

Multimodal large language models are increasingly expected to perform \emph{thinking with images}, yet existing visual latent reasoning methods still rely on explicit textual chain-of-thought interleaved with visual latent tokens. This interleaved design limits efficiency and keeps reasoning fragmented across separate text and vision channels. We propose **UniVLR**, a unified visual latent reasoning framework that treats textual reasoning and auxiliary visual evidence as a shared visual workspace. Instead of preserving text CoT as an independent inference-time path, UniVLR renders reasoning traces together with auxiliary images and learns to compress this unified representation into compact visual latent tokens. At inference time, the model reasons only through visual latents and directly decodes the final answer, avoiding both external tool calls and verbose text reasoning. Experiments on real-world perception and visual reasoning tasks show that UniVLR outperforms prior visual latent reasoning methods while using substantially fewer generated reasoning tokens, suggesting a more unified and efficient paradigm for visual thinking in MLLMs. Our code is available at: https://anonymous.4open.science/r/UniVLR-1B0C/.