NeurIPS 2026 Papers

Naturalistic language encoding models based on large language model features can predict cortical responses well, but their learned representations remain difficult to interpret. Three challenges are central: semantic selectivity is fundamentally non-identifiable in high-dimensional correlated feature spaces, cortical organization remains implicit in voxelwise weight maps, and cross-subject structure is usually addressed only after model fitting. We introduce a multisubject coupled sparse autoencoder that jointly reconstructs language-model features and predicts subject-specific brain responses, learning a sparse basis of semantic-cortical atoms that makes semantic structure, cortical organization, and cross-subject correspondence explicit within a single model. In naturalistic story-listening fMRI, the model retains 88\% of matched dense-feature ridge brain prediction performance using 50 atoms and $k{=}5$ active atoms per TR. Relative to matched two-stage baselines, joint training yields the strongest overall tradeoff between prediction and interpretability. The learned atoms are semantically coherent and interpretable, recover more reproducible cross-subject brain maps than standard post hoc encoding-model analyses, align with large-scale cortical structure from Yeo~7 and Neurosynth, and transfer better than ridge to held-out subjects in the low-data regime. This framework provides a more interpretable basis for cognitive neuroscience analyses of semantic-to-brain encoding by linking semantic features to corresponding cortical response patterns across subjects.

We study learning rate (LR) scaling for hybrid architectures combining Transformer and State-Space Model (SSM) blocks, a class adopted by several recent production language models. In particular, we focus on the gap between the theoretical scaling rules derived for SSMs under zero-order-hold (ZOH) discretization at infinite width with proportional state size, and the field-standard practical implementations using simplified-ZOH Mamba at fixed state size. Surprisingly, in this practical regime hybrid architectures achieve a near-zero LR transfer gap across widths 256-2048 and depths 4-32 at sub-billion-parameter scale using only the original $\mu$P prescription, even though SSM operations fall outside its Tensor Programs representability conditions and every parameterization we test fails the standard coordinate-check diagnostic of $\mu$P correctness. We attribute this to a two-condition decomposition of LR transfer in hybrid architectures: a global update-to-weight invariance, enforced by $\mu$P's initialization and LR scaling; and a local per-component balance, provided by AdamW's $g/\sqrt{v}$ per-parameter normalization. Our observations show that the optimal LR is invariant to width up to $8\times$, as well as to depth, sequence length, batch size, and Transformer-to-SSM ratio, and transfers to Nemotron-H, a production hybrid outside our custom architecture set. We hope these findings fill the gap between theoretical scaling rules and practical hybrid implementations, and stimulate further research toward bridging it.

Low-Rank Adaptation (LoRA) is the prevailing approach for efficient large language model (LLM) fine-tuning. Building on this paradigm, recent studies have proposed alternative initialization strategies and architectural modifications, reporting substantial improvements over vanilla LoRA. However, these gains are often demonstrated under fixed or narrowly tuned hyperparameter settings, despite the known sensitivity of neural networks to training configurations. In this work, we systematically re-evaluate nine representative LoRA variants alongside vanilla LoRA through extensive hyperparameter searches. Across tasks spanning mathematical reasoning, commonsense reasoning, code generation, and instruction following at diverse model scales, we find that different LoRA methods favor distinct learning rate ranges. Crucially, once learning rates are properly tuned, all methods achieve similar peak performance (within 1-2\%), with only subtle rank-dependent behaviors. These results suggest that vanilla LoRA remains a competitive baseline and that improvements reported under a single training configuration may not reflect consistent methodological advantages. Finally, a second-order analysis attributes the differing optimal learning rate ranges to variations in the largest Hessian eigenvalue, aligning with classical learning theories.

In many safety-critical applications, control of uncertain dynamical systems relies on observers that estimate states and external disturbances. Neural network observers can improve estimation accuracy, but certifying their Lyapunov stability via Linear Matrix Inequality (LMI) constraints leads to large-scale semidefinite programs (SDPs) that are difficult to solve for large networks. To overcome this scalability bottleneck, we propose a novel two-stage training framework for provably stable neural network observers. Our approach decouples the optimization into a point-guided Lyapunov pre-training phase, which rapidly achieves high estimation accuracy and local stability over sampled states, followed by an LMI fine-tuning phase that efficiently satisfies a strict global Lyapunov stability certificate. We provide formal theoretical guarantees for local stability radii and probabilistic coverage over a prescribed compact error-state domain under specified regularity and sampling assumptions. Experiments on nonlinear control benchmarks and X-29 aircraft ablations show that our LMI-certified neural network observers train significantly faster than direct LMI-based methods and generalize robustly across diverse systems, achieving improved tracking accuracy over a range of observer baselines. The code is available at https://anonymous.4open.science/r/LearningNeuralNetworkObserver-2E1E.

In many modern applications of reinforcement learning (RL), the natural reward for a task of interest is inherently sparse: a reward of 0 is given everywhere except when the task is completed, when a reward of 1 is given. Training a policy to maximize such a sparse reward requires solving a challenging credit assignment problem, leading to slow or ineffective RL improvement. We propose a simple approach to transform a sparse outcome reward into a dense process reward. Our approach relies on training a discriminator to distinguish between previous successful and unsuccessful trajectories, and using this discriminator to incentivize the RL-learned policy to match the state-action visitations of successful trajectories, while avoiding those of unsuccessful trajectories. By incentivizing the policy to match the visitations over all states, not just those that correspond to task success, this reward provides dense feedback on whether progress is being made towards task completion, and, we show, provably achieves this without changing the optimal policy. Focusing on finetuning of robotic control policies, we demonstrate that our approach leads to significantly faster RL finetuning performance on both simulated and real-world manipulation tasks, as compared to simply maximizing the sparse outcome reward.

Infinite-dimensional orthonormal basis expansions play a central role in representing and computing with function spaces due to their favorable linear algebraic properties. However, common bases such as Fourier or wavelets are fixed and do not adapt to the structure of a given problem or dataset. In this paper, we aim to represent these bases with neural networks and optimize them. Our key idea is that any target infinite-dimensional orthonormal basis can be viewed either as a point on the Lie manifold of the orthogonal group, or equivalently, as the endpoint of a continuous path on that manifold that connects a reference basis, e.g. Fourier, to that target. Paths on the Lie manifold satisfy ordinary differential equations (ODEs) governed by skew-adjoint integral operators. Using neural networks to define finite-rank generators of such ODEs allows us to parameterize and optimize orthonormal bases in function space. While relying on finite-rank generators to model infinite operators might seem restrictive, we prove a universality result: even with a rank-2 generator, the integrated solutions of the ODE are dense in the orthogonal group under the appropriate operator topology. In other words, for any target orthonormal basis, there exists a path originating from a reference basis and driven by finite-rank generators that gets arbitrarily close to that target basis. We demonstrate the flexibility of our framework by transforming the Fourier basis into the principal components of a functional dataset, eigenfunctions of linear operators, or dynamic modes of energy-preserving physical simulations.

Learning flexible motor primitives is a hallmark of skilled motor control. Recent neuroscience theory proposes that motor primitives may be implemented as low-rank perturbations of a shared recurrent network, but leaves open how such a system is learned. We translate this principle into a novel architecture for learning motor skills end-to-end: a shared recurrent core modulated by a bank of residual adapters, each selected by a discrete latent code. Trained on closed-loop biomechanical control, the adapters develop emergent low-rank perturbations of the recurrent dynamics despite no architectural rank constraint, placing task representations in disparate subspaces of the shared core network. A simple high-level policy over the learned options, optimized while the whole network is frozen, sequences the low-rank adapters to produce novel out-of-distribution movements. We demonstrate the ability to generalize to novel motor sequences within the closed-loop control setting, improving on the generalization error of a task-input-conditioned multitask baseline by up to order of magnitude.

Budget-feasible mechanisms (BFMs) are a fundamental tool for budget-constrained procurement, but designing strong BFMs remains challenging, particularly for general valuation classes. Classical approaches rely on problem-specific analysis and worst-case approximations, often yielding conservative mechanisms in practice. This motivates data-driven methods for discovering mechanisms with stronger empirical performance. However, strict budget constraints expose a key limitation of existing neural automated mechanism design (AMD): they either fail to provide exact economic guarantees or suffer from severe scalability bottlenecks. We propose BFMNet, a scalable menu-based neural framework for learning truthful budget-feasible procurement mechanisms. BFMNet first learns *self-bid-independent* menus from market context, ensuring dominant-strategy incentive compatibility (DSIC) and individual rationality (IR) by construction, and then applies an ex-post, instance-level menu transformation to enforce budget feasibility. We prove that the resulting mechanism satisfies exact DSIC, IR, and budget feasibility. By avoiding intractable global discretization grids and restrictive Lipschitz constraints, BFMNet remains both expressive and scalable. Extensive experiments show that BFMNet improves utility over classical baselines by up to $34.1$%, remains competitive with state-of-the-art neural baselines that provide only approximate economic guarantees, and scales robustly to larger auction instances.

Discrete diffusion language models (DLMs) enable parallel generation but usually rely on fixed forward processes, such as masking, uniform noise, or precomputed hierarchies. We investigate whether the hierarchical forward process can instead be learned while preserving continuous-time tractability. We propose \textbf{Learnable Hierarchical Diffusion Language Model (LHDLM)}, which extends a learnable column-stochastic token-cluster map. This many-to-many hierarchy preserves a CTMC with block-conditional transition and closed-form CT-ELBO, while recovering HDLM in the one-hot limit and masked diffusion in the collapsed limit. We identify degenerate hierarchy maps caused by learning the same map used in both forward targets and model-induced cluster predictions, and mitigate them with an information-retention regularizer. Empirically, LHDLM is competitive with discrete diffusion approaches and reveals soft token structures --- cluster-agnostic function tokens and sharp content-token assignments --- that cannot be represented by hard partitions.

Large distribution shifts remain a primary obstacle for domain generalization and unsupervised adaptation. Although gradual domain adaptation alleviates this difficulty by introducing ordered intermediate domains, most existing approaches still operate in a discrete, step-wise manner, which can accumulate errors and often relies on unstable Min--Max optimization. We propose \textbf{FlowGDA}, a flow-matching framework that models gradual adaptation as \emph{continuous} transport on a statistical manifold in a representation space. FlowGDA learns time-dependent velocity fields that induce smooth domain trajectories, using a simulation-free regression objective rather than adversarial training. To prevent semantic drift, we couple left-to-right transport between consecutive domains with a source-anchored transport and a semantic consistency loss, and we further stabilize learning via a two-stage sequential optimization scheme. Extensive experiments demonstrate that our method outperforms state-of-the-art techniques, validating flow matching as an effective alternative for domain adaptation.

We study online identification of linear-in-parameter dynamical systems whose true parameters drift over time. Measuring non-stationarity through the path variation $V_T = \sum_{t=1}^{T-1}\|\theta_{t+1}^\star - \theta_t^\star\|$, we make two contributions. First, we give a clean blockwise dynamic-regret analysis for continuous projected online gradient descent, yielding $E[R_T] = O(T/\sqrt{\tilde W} + \tilde W V_T)$ and hence the oracle-tuned rate $O(T^{2/3}V_T^{1/3})$ . Second, we propose a residual-based adaptive step size and prove an oracle-proxy comparison theorem: when the residual proxy tracks the local prediction-error signal with lower-order overhead, the adaptive algorithm inherits the oracle scaling. Experiments on synthetic systems and real turbofan-engine degradation data (NASA C-MAPSS) support the matched-regime slope prediction on synthetic data, diagnose the failure regime when proxy overhead dominates, and show a clear unsupervised health-indicator trend in the residual proxy on FD001. A separate fast-rate guarantee under persistent excitation and a tube MPC integration are deferred to the appendix.

Prefill-Decode (PD) disaggregation improves LLM serving by isolating compute-bound prefilling and memory-bound decoding onto specialized GPU pools. However, dynamically rebalancing these pools is challenging because request arrival rates, prompt lengths, and generation lengths change over time. We formulate this autoscaling problem as latency-constrained cost minimization: the controller must minimize GPU usage while keeping P99 tail latency within TTFT and TPOT targets. Existing threshold-based heuristics require extensive parameter tuning, yet still tend to over-provision. To address this, we propose a reinforcement learning (RL) framework that learns a cost-efficient autoscaling policy. The policy uses a recurrent network to track multi-timescale workload trends and pod lifecycle states, enabling decisions that account for delayed pod readiness and delayed SLO feedback. It also restricts the action space with hard masks and anti-oscillation cooldowns, enforcing feasible scaling actions while reducing wasteful GPU transitions. We train the policy in a high-fidelity PD-disaggregated serving simulator calibrated from profiling data and validated against a physical testbed. Simulator experiments on ShareGPT and Azure Conversational traces show that the learned policy satisfies the same latency constraints as grid-searched heuristic baselines while reducing average GPU occupancy. Real-cluster deployment on ShareGPT further confirms sim-to-real transfer while satisfying the latency targets.

Traffic flow forecasting plays a pivotal role in intelligent transportation systems. However, most existing methods adopt a black-box learning paradigm, resulting in a lack of interpretability in their decision-making processes. Existing explainable methods mostly follow the feature attribution paradigm, leaving explanations at the level of static or discrete feature evidence and making it difficult to reveal the continuous evolution mechanism of spatio-temporal dependencies during dynamic traffic propagation. Meanwhile, prior methods often decouple spatial and temporal explanations, treating spatial structures and temporal dynamics independently and thus failing to capture the intrinsic coupling between spatial dependencies and temporal contexts in model decision-making. To this end, we propose STGIB, a new spatio-temporal graph information bottleneck theory for characterizing continuously evolving explanations in traffic forecasting. STGIB redefines traffic explanation as a time-indexed spatio-temporal explanation trajectory, and jointly learns dynamic key structures and their temporal representations under predictive sufficiency and information compression constraints. Furthermore, we derive tractable recursive variational bounds to make the theoretical objective optimizable, and instantiate a new explainable traffic flow forecasting model via spatio-temporal explanation trajectory generation. Experiments on real-world traffic datasets demonstrate that STGIB maintains competitive predictive performance while generating more faithful and spatio-temporally consistent explanations.

Softmax attention in Transformers suffers from quadratic complexity in sequence length, making it impractical for long-context applications. Linear attention alleviates this issue by replacing the exponential kernel with alternative functions that enable linear-time computation. Among existing linear attention approaches, recent studies have shown that polynomial kernels are particularly effective, as they exhibit sparse and spiky behavior similar to softmax attention, which emphasizes large dot products while suppressing irrelevant interactions. However, the exact computation of high-degree polynomial kernels is infeasible for high-dimensional representations. As a result, prior work relies on approximate polynomial kernels. This introduces a non-negligible approximation error. In this paper, we show that, from the perspective of polynomial kernel approximation, existing linear attention methods are still suboptimal. We propose Learnable Low-Rank Polynomial Sketch (LLoPS), a principled and flexible framework for approximating polynomial kernels with linear attention. Our method learns a low-rank polynomial sketch that provably achieves a strictly smaller approximation error than existing approaches. Experiment results show that LLoPS achieves the highest performance across extensive benchmarks, comparing with various linear attention baselines.

Training binary neural networks (BNNs) from scratch is dominated by the straight-through estimator (STE), whose forward/backward mismatch produces severe accuracy degradation as networks deepen. We study an orthogonal axis: when and where binarization is enforced during training. We introduce StoMPP (Stochastic Masked Partial Progressive Binarization), which gradually replaces clipped weights and activations with their hard binary counterparts layer by layer from input to output, using stochastic partial masks with soft refresh. StoMPP delivers two complementary benefits. As a standalone training rule, it provides a fully STE-free procedure that improves over vanilla STE with gains that grow with depth (ResNet-50 BNN: +18.0/+13.5/+3.8 on CIFAR-10/100/ImageNet), and the pattern holds across ResNet-18/34/50, MobileNetV2, and BERT fine-tuning. Composed with surrogate gradients by applying STE only to frozen entries, it reaches +27.1/+19.8/+17.7 over vanilla STE on the same setting. Underlying both regimes is a single mechanistic finding: progression order is decisive. Forward layerwise progression prevents depth collapse, reverse progression collapses to near-chance, and binary-weight networks (without binary activations) are insensitive to order. We trace this asymmetry to activation-induced gradient blockades: a committed binary activation severs gradient flow upstream, and ordering controls when these blockades form. To isolate the progression's contribution from any benefit conferred by STE, we conduct all ablations in the STE-free regime; the resulting characterization (schedule, refresh, ordering, dynamics) thus reflects the progression itself rather than its interaction with surrogate gradients.

Autoencoders (AEs) are widely used in unsupervised anomaly detection (AD) due to their ability to model complex data distributions. However, their strong reconstruction capacity, while effective for capturing normal patterns, often enables them to also reconstruct anomalous objects, including both individual and collective anomalies, which reduces detection performance. This limitation is particularly severe for collective anomalies, which are frequently misinterpreted as normal clusters. To address this issue, we propose Layer Precision Reduction (LPR), a novel technique that systematically constrains the numerical precision of weights and biases in neural network layers to strategically reduce the learning capacity of neural networks, such as the reconstruction capacity of AEs. LPR encourages the network to prioritize dominant manifold structures while suppressing the reconstruction of rare or anomalous patterns. While LPR was primarily developed for AE-based detectors (the focus of this paper), it is applicable to any neural network. Simulations are conducted with 24 AD models across 30 real-world public datasets. LPR is applied to 10 deep AD models, including 5 unsupervised AE-based methods, 2 unsupervised non-AE-based methods, 1 semi-supervised method, 1 weakly supervised method, and 1 fully supervised method. LPR consistently improves all 10 models, most notably enhancing the best-performing unsupervised AE-based detector from an average AUROC of 72.12\% to 83.06\%, surpassing the strongest unsupervised non-AE-based competitor, which achieves an average AUROC of 77.36\%, across the remaining 14 detectors. These results reveal a previously unexplored link between layer-level numerical precision and manifold pattern prioritization, demonstrating that LPR serves as a simple yet powerful mechanism for enhancing AD performance while offering broad applicability to diverse neural architectures and potential extensions to tasks beyond AD.

Reliable physics simulation requires both broad generalization across heterogeneous PDE families and stability under long autoregressive rollouts. Current neural solvers rarely provide both. Deterministic operators accumulate rollout error, while probabilistic solvers typically remain tied to a single PDE family or a short prediction horizon. We address this gap with the \textbf{Latent Generative Solver} (LGS), which couples a Physics VAE (PhyVAE) with a Pyramidal Flow-Forcing Transformer (PFlowFT). PhyVAE compresses states from twelve PDE families into a shared \emph{physical manifold}, separating dynamics-relevant structure from the ambient state space. PFlowFT then predicts the next latent state through input-noised flow matching, yielding a sufficient-condition contraction mechanism that explains improved autoregressive rollout stability. Pretrained on a 2.5\,M-trajectory, 16-system corpus at $128^2$ resolution, LGS obtains the lowest relative $L_2$ error (L2RE) on 15 of 16 systems at both 5- and 10-step rollout. At 20 steps, LGS reduces the L2RE from $56.1\%$ to $\mathbf{30.2\%}$ against matched deterministic and adapted generative baselines, while requiring $\mathbf{13}$--$\mathbf{77\times}$ less recurrent dynamics-step compute. On a held-out $256^2$ Kolmogorov flow, LGS reduces the 1-step L2RE from $0.398$ to $0.129$ within five finetuning epochs; U-AFNO achieves only $0.653$ to $0.343$ under the same protocol.

Understanding the internal “thinking” process of Large Language Models (LLMs) and the cause of hallucinations remains a key challenge. To this end, we introduce latent debate, a structured surrogate framework for interpreting model outputs on True/False prediction tasks through the lens of internal latent arguments and interactions amongst them. Unlike human debates, latent debate captures the hidden supporting and attacking signals that arise within a model during a single inference. We first present a model- and task-agnostic conceptual framework, and then instantiate it symbolically to approximate the thinking process of LLMs towards binary decisions. Empirical studies demonstrate that our latent debate is a faithful structured surrogate model that has highly consistent predictions with the original LLM, while providing a form of interpretability. We also demonstrate that our latent debate provides a strong baseline for hallucination detection. Specifically, we identify strong correlations between debate patterns and hallucinations, such as a high degree of disagreement in the middle layers of the latent debate surrogate is linked to a higher risk of hallucinations. Our findings suggest that latent debate shows potential to analyze internal signals in LLMs for binary decision settings.

In this paper, we study last-iterate convergence in stochastic constrained convex--concave minimax optimization. A key difficulty is that the last iterates of vanilla stochastic extragradient (S-EG) and stochastic optimistic gradient descent--ascent (S-OGDA) can fail to converge in the presence of gradient noise, even for simple bilinear problems. To address this issue, we regularize the original convex--concave problem into a strongly convex--strongly concave one. Applying S-EG and S-OGDA to the regularized problem gives two simple single-loop first-order methods, which we call perturbed S-EG and perturbed S-OGDA. By carefully choosing the regularization parameter and balancing the resulting regularization bias, stochastic error, and optimization error, we prove that both methods achieve a last-iterate convergence rate of $\mathcal{O}(T^{-1/4+\varepsilon})$ for any $\varepsilon>0$ in terms of the primal--dual gap. This improves upon the best-known $\tilde{\mathcal{O}}(T^{-1/7})$ rate under comparable constrained settings. Moreover, for unconstrained problems, we establish almost sure convergence under a single-timescale stepsize scheme, whereas prior almost sure convergence results typically rely on two-timescale stepsizes and are limited to S-EG.