NeurIPS 2026 Papers

Hyper-Connections (HC) extend residual connections into multiple streams, employing residual matrices for cross-stream mixing to enrich model expressivity. However, unconstrained mixing disrupts the identity mapping property intrinsic to the residual connection, causing unstable training. To address this, Manifold-Constrained Hyper-Connections (mHC) and its variants restrict these matrices to be doubly stochastic via Sinkhorn-Knopp (SK) algorithm or permutation-based parameterizations. We reveal three limitations of this doubly stochastic constraint: (1) identity degeneration, where learned matrices collapse around the identity initialization and diminish cross-stream interactions, (2) an expressivity bottleneck via spectral collapse, where the doubly stochastic constraint induces feature homogenization, and (3) parameterization inefficiencies, manifesting as unstable SK iterations or the factorial-scaling overhead of permutation-based parameterizations. To overcome these flaws, we propose Spectral-Sphere-Constrained Hyper-Connections (sHC). By confining residual matrices to a spectral norm sphere, sHC prevents spectral collapse and enables selective feature diversification. This shift eliminates unstable SK iterations and factorial parameterization, enabling expressive, non-degenerate residual matrices while preserving training stability.

Post-hoc image explanations usually reveal *where* a model looks through spatial heatmaps, or *which channels* matter through global channel scores. This separation is limiting for multi-spectral data, where channel relevance is often spatially localized and physically meaningful. We introduce **Spectral-Spatial Interpretation (SSI)**, a full-grid attribution method over position-channel players (g, c). For each spatial position, SSI fixes the observed surrounding context, defines a conditional game over the channels at that position, and applies the Shapley value to this local game. The result is a full spectral-spatial attribution matrix rather than a single heatmap or a global channel ranking. SSI has a clear local semantics: it satisfies Shapley fairness within each position, and its channel aggregation exactly recovers the position-level contextual contrast obtained by replacing that entire position with the baseline. We also distinguish SSI from context-averaged Owen-style hierarchical accounting, which respects the same position-channel grouping but averages over external spatial coalitions. On HLS Burn Scar, SSI outperforms an MC-Owen estimator across retained attribution metrics while using only about 6% of its evaluation budget. Across CUB-200, EuroSAT, and HLS Burn Scar under both per-channel mean and zero baselines, SSI achieves the best joint deletion and insertion AUC over dense, flat, and rank-1 baselines, yields the strongest spectral hit rates and channel deletion, and leads spatial localization on both annotated datasets.

Machine unlearning in large language models (LLMs) aims to remove specific learned knowledge, capabilities, or behaviors while preserving model utility. Existing methods have mostly focused on the design of unlearning objectives, while largely overlooking the intrinsic matrix structure of transformer weights. In particular, standard optimizers operate on independent scalar coordinates, thereby ignoring this matrix structure. We propose \emph{spectral unlearning}, an update framework that combines standard unlearning objectives with matrix-structured optimization. We show that the spectral update corresponds to the exact steepest-descent direction for transformers composed of linear layers with normalized inputs and outputs. Our approach is applied on top of existing unlearning methods and evaluated across complementary axes of memorization, privacy, and utility, summarized via their harmonic mean. Spectral unlearning consistently outperforms a broad range of baselines and closes most of the gap to a retain-trained reference.

Time series, spatial data, and images are natural applications of Neural Processes. However, when such data exhibit strong periodicity and quasi-periodicity, existing methods often suffer from underfitting and generalise poorly beyond the training distribution. In this work, we propose Spectral Transformer Neural Processes (STNPs), a frequency-aware extension of Transformer Neural Processes (TNPs). STNPs introduce a Spectral Aggregator that estimates an empirical context spectrum, compresses it into a spectral mixture, samples task-adaptive spectral features, and concatenates them with time-domain embeddings, thereby injecting a spectral-mixture-kernel bias into TNPs. This design reshapes the similarity geometry, allowing inputs that are distant in Euclidean space to remain close in an induced periodic manifold while enhancing time--frequency interactions. Extensive experiments on synthetic regression tasks, real-world time-series datasets, and an image dataset demonstrate that STNPs consistently improve predictive performance over existing baselines, extending Neural Processes beyond translation equivariance toward effective modelling of periodicity and quasi-periodicity.

Pre-training Graph Neural Networks (GNNs) via self-supervised learning has become a dominant paradigm, yet efficiently adapting frozen encoders remains a challenge. Graph prompting offers a parameter-efficient alternative to fine-tuning, but existing methods largely treat pre-trained models as opaque feature extractors, ignoring their internal spectral structure. In this work, we identify a systematic phenomenon in pre-trained GNNs, which we term spectral bias: optimization during pre-training disproportionately aligns representations with directions associated with large singular values, leaving low-energy directions under-explored. We show that these underutilized directions can encode complementary information that is beneficial for downstream adaptation, especially under distribution shift. To leverage this insight, we propose Spectral Reverse Prompt (SRP), a prompting framework that rebalances the spectral contributions of frozen GNN encoders. SRP applies a learnable soft-thresholding mask in the spectral domain to down-weight dominant directions while amplifying weaker ones. In addition, SRP incorporates a null-space augmentation module that captures variation in directions with minimal activation under the frozen encoder. Extensive experiments across multiple benchmarks demonstrate that SRP achieves state-of-the-art performance with minimal additional parameters, highlighting that reweighting spectral components is a principled and effective strategy for parameter-efficient graph adaptation.

Context-based offline meta-reinforcement learning (COMRL) aims to learn adaptable policies entirely from static datasets by inferring latent task representations from small contexts. Most COMRL methods rely on immediate, per-step reward signals. This can be generalized by aggregating rewards into ``bags'' and being delivered after a sequence of actions (where bag size $n=1$ is the per-step case). We formally prove that under Bagged Rewards, existing per-transition COMRL methods suffer an information-theoretic collapse ($\mathcal{O}(1/n)$), causing their task inference to degrade to behavior cloning. To solve this, we introduce SpectralMeta, an algorithm that fully decouples dynamics representation from task inference. By pre-training an energy-based spectral representation on reward-free transitions, we recast task identification as a closed-form Bayesian linear regression over bagged rewards. SpectralMeta comes with finite-sample bounds on reward recovery and policy sub-optimality. In continuous-control meta-RL benchmarks, SpectralMeta maintains robust task inference and competitive returns even when rewards are reduced to a single episode-level scalar, a regime where existing baselines fail to recover task-relevant structure.

Model merging enables the integration of multiple expert models with different capabilities into a unified model. In practical deployments, new expert models are often continuously updated and arriving. Meanwhile, due to the training and communication overhead of large language models (LLMs) and multimodal large language models (MLLMs), Low-Rank Adaptation (LoRA) is widely used to adapt large models to specific domains due to its efficiency. This scenario brings a new challenge: continual LoRA merging of deployed MLLMs to load new task capabilities. Naive merging faces catastrophic forgetting and parameter conflicts. We first observe the distinct features of the LoRA direction and layers in MLLM LoRA tuning. Based on this, we propose our Spectral Rank Calibration Merging (SRC-Merging), a data-free and train-free continual LoRA merging method tailored for MLLMs. SRC-Merging regards the deployed LoRA as a compressed spectral memory and performs calibrated rank-level merging for each incoming adapter. It aims to preserve accumulated multimodal knowledge while avoiding over-suppression of new vision-language task updates during continual merging. Extensive experiments on multiple multimodal tasks have demonstrated that our method significantly outperforms state-of-the-art merging methods.

Implicit neural representations (INRs) model signals as continuous functions from coordinates to values, but their performance depends strongly on how spectral capacity is allocated. Yet most Quantum INR (QINR) architectures treat spectral components as independent outputs of separate quantum blocks. We introduce the Spectral Quantum Memory Model (SQMM), a memory coupled quantum INR in which sequential hybrid data reuploading layers (bands) communicate through a shared coherent quantum memory register. The construction gives a controlled way to share spectral information across quantum modules while preserving the finite Fourier structure of a band. When memory read and write layers are input independent, we prove that the memory state forms an operator valued Fourier series whose support grows by Minkowski summation of local band spectra. Downstream bands then condition their Fourier coefficients on the spectral content stored by earlier bands. This theory motivates a spectral memory alignment loss that encourages memory to encode the spectrum expressed by the deployed prefix of the model. Across audio and image reconstruction benchmarks, SQMM improves over QINR and INR baselines, achieving state of the art results.

Diffusion models have been shown to implicitly generate visual content autoregressively in the frequency domain, where low-frequency components are generated earlier in the denoising process while high-frequency details emerge only in later timesteps. This structure offers a natural opportunity for efficient generation, as high-resolution computation on noise-dominated frequencies is largely redundant. We propose Spectral Progressive Diffusion, a general framework that progressively grows resolution along the denoising trajectory of pretrained diffusion models. To this end, we develop a spectral noise expansion mechanism and derive an optimal resolution schedule from the model's power spectrum. Our framework supports training-free acceleration and a novel fine-tuning recipe that further improves efficiency and quality. We demonstrate significant speedups on state-of-the-art pretrained image and video generation models while preserving visual quality.

Multi-output regression can be approached either by fitting one joint vector-valued predictor or by training separate univariate predictors for each target coordinate. This holistic-versus-separate choice is classical in the largely non-neural multi-output regression literature, but is less characterized analytically for neural networks, where joint prediction arises naturally through shared features and vector-valued output layers. Because neural networks are highly non-linear and high-dimensional, exact analysis is notoriously difficult. We study two questions using the Unconstrained Feature Model (UFM) as a tractable surrogate for exact analysis: $\textbf{(1)}$ how does joint-output regression compare with independent coordinate-wise regression under regularization? $\textbf{(2)}$ when can target whitening or normalization improve original-scale MSE? Under UFM, the minimal training MSE depends only on the regularization product and the spectrum of the target covariance. This yields two theoretical results: joint-output regression has no larger training MSE than independent coordinate-wise regression under matched regularization, and whitening or normalizing targets can help or hurt original-scale MSE depending on the average target variance. Experiments on neural networks trained with standard weight decay across robotic imitation-learning and autonomous-driving datasets support these theoretical results, with test-MSE trends following the same qualitative pattern. Our results show that simplified analytical surrogates can provide useful guidance for neural multi-output regression.

In this work, we study transformer attention through the lens of spectral geometry and operator theory. We view each attention head as a functional map between Hilbert spaces of functions on the token sequence and derive a \emph{Token Difference Operator}, whose spectral structure controls how token-space information is routed to the output. We show that standard Euclidean spectra are structurally biased by sinks, conflating mass concentration with genuine routing capacity. By recasting token space in the intrinsic probability geometry induced by attention, the token difference spectrum disentangles sink effects from routing capacity and provides a spectral description of the dimensionality of the head output. This yields a unified framework for analyzing attention maps, explaining sinks, routing collapse, and output dimensionality within a single operator-theoretic framework. In practice, by grounding attention heuristics in spectral geometry, we develop a novel uncertainty quantification method for LLMs that provides a theoretically sound alternative to traditional, probability-based estimators, with the strongest gains on long-form summarizations

Sample covariance matrices are fundamental objects in machine learning and statistics, with valuable information encoded into their eigenvalues. When treating modern problems, the scale of these matrices can become prohibitive for tractable computation. This creates a critical need for "spectral decompression": a method for inferring the eigenspectrum of a very large-scale model using only information from its smaller realizations. Existing methods fall into two main categories: those relying on spectral inversion; and those that rely on Free Decompression techniques. On one hand, spectral inversion is a fundamentally ill-posed problem that is notoriously unstable, often failing to yield reliable results in finite-size settings. On the other hand, while Free Decompression was recently proposed to address this scaling, its reliance on the Nica-Speicher framework restricts its use to unitarily invariant ensembles, a symmetry condition that subsampled covariance matrices fail to satisfy. In this work, we introduce Deformed Decompression (DD), a novel framework for the stable, forward-extrapolation of spectral densities. By leveraging properties of the companion matrix, DD recasts the spectral decompression problem as a directed evolution of the Stieltjes transform of the companion. This approach bypasses the instabilities of traditional inversion and provides a robust methodology for scaling covariance matrices and even their generalized variants (including Hessian matrices) that lack unitary invariance. We demonstrate the utility of our framework through experiments on a variety of random matrix examples, showing that DD successfully captures the complex spectral signatures of large-scale systems.

Source-free domain adaptation (SFDA) adapts a pre-trained source model to an unlabeled target domain without access to source data. However, existing SFDA studies have predominantly focused on classification tasks, leaving time series forecasting (TSF), a fundamentally different problem with continuous outputs and complex temporal dependencies, largely unexplored. In this work, we identify unique characteristics and key challenges of SFDA in TSF: 1) forecasting labels correspond to future temporal segments of the same underlying signal, enabling us to split target input and construct a self-supervised learning task, thereby training an accurate short-term forecasting model to provide high-quality pseudo-labels; 2) however, such supervision remains inherently *local* rather than *global* for long-horizon source model adaptation. To fill this gap, we propose the notion of Spectral Energy Allocation (SEA) pattern, defined over a short-long signal pair. The SEA pattern provides a structured correspondence that bridges signals across different temporal horizons, enabling reliable short-horizon pseudo-labels to guide long-horizon adaptation. Extensive experiments demonstrate that our method substantially outperforms baselines across different forecasting backbones.

Large-scale Ising optimization underlies critical optimization and learning applications, including correlation clustering, MAP inference, and energy-based models. Spectral relaxation methods run in polynomial time but yield solutions with significant optimality gaps while metaheuristics close those gaps but scale poorly to large instances. In particular, spectral methods are inherently single-shot as they solve exactly one eigenvalue problem. We introduce $\textit{Spectral Annealing}$ (SpecAnn), a new spectral optimization approach that transforms traditional spectral methods from single-shot approximations into exploration-based optimization, by sweeping a parametrized normalization path from the raw adjacency to the signed normalized Laplacian, with an $O(n)$ diagonal predictor that warm-starts successive eigensolves and GPU-batched local search refining candidates in parallel. Experiments on general Ising problems demonstrate that SpecAnn achieves competitive to near-optimal solutions depending on instance structure, with runtimes less than 7 minutes for instances of 8.4M variables. SpecAnn delivers up to 83$\times$ speedup over recent GPU-accelerated methods while maintaining superior solution quality, and scales successfully beyond 524K variables where traditional metaheuristics fail.

Forward-backward (FB) representations provide a powerful framework for learning the successor representation (SR) in continuous spaces by enforcing a low-rank factorization. However, a fundamental spectral mismatch often exists between the high-rank transition dynamics of continuous environments and the low-rank bottleneck of the FB architecture, making accurate low-rank representation learning difficult. In this work, we analyze temporal abstraction as a mechanism to mitigate this mismatch. By characterizing the spectral properties of the transition operator, we show that temporal abstraction acts as a low-pass filter that suppresses high-frequency spectral components. This suppression reduces the effective rank of the induced SR while preserving a formal bound on the resulting value function error. Empirically, we show that this alignment is a key factor for stable FB learning, particularly at high discount factors where bootstrapping becomes error-prone. Our results identify temporal abstraction as a principled mechanism for shaping the spectral structure of the underlying MDP and enabling effective long-horizon representations in continuous control.

Predicting transcriptome-wide cellular responses to genetic and biomolecular perturbations is central to functional genomics and therapeutic target discovery. Existing methods assign gene positional encodings derived from arbitrary orderings or static co-expression masks built from unperturbed cells, and cannot capture the functional rewiring induced by perturbations. We present SPEAR, a generative framework built on two complementary positional mechanisms. Spectral positional encoding grounds each gene's position in the geometry of the co-expression network via continuous coordinates derived from the graph Laplacian. Adaptive repositioning then derives each gene's position from its perturbation-conditioned hidden state at every layer, allowing the attention genes exert on one another to reorganize without explicit supervision. Evaluated on genetic, pharmacological, and cytokine perturbation benchmarks, SPEAR demonstrates excellent recovery of differential gene-expression, distributional structure, and perturbation-specific identifiability. SPEAR also captures latent positional relationships that closely map to known transcription factor-target interactions.

Routing systems for large language models, such as MasRouter, RouteLLM, FrugalGPT, and others, match queries to suppliers based on cost-quality tradeoffs. Most prior work optimizes this problem from the demand side, while the supplier-side question of how LLM suppliers should price their services within a routing mechanism has received little formal treatment. We provide the first systematic analysis of this problem. In our setup, the LLM is the priced commodity: supplier organizations set price functions for the services they offer, rather than acting as strategic agents that generate bids per query. We model supplier-side pricing as a sealed-bid first-price reverse auction over a router that allocates each query to one supplier based on cost and quality. This framework applies to any cost-quality routing system rather than to a specific implementation. Using calibrated profiles from 12 open-weight models and MasRouter as a case-study router, we characterize equilibrium behavior. Our main finding is that the Bayesian Nash equilibrium is asymmetric: capability-differentiated suppliers adopt flat, non-discounted strategies, while marginal-quality suppliers compete primarily on price. A mechanism-baseline experiment, where the case-study router is replaced by an analytical rational-decision rule, confirms that this pattern is a property of the auction mechanism rather than an artifact of router training. We further observe that price differentiation can emerge in equilibrium even under capability symmetry, because private-cost types alone produce nontrivial bid functions. The practical implication is that auction-based pricing alone does not discipline specialist rents. Achieving that goal requires additional mechanism elements, such as reserve prices or capability-blind tie-breaking.

Large language models increasingly use external tools such as web search and document retrieval to solve information-intensive tasks. However, multi-hop tool use in complex tasks introduces substantial latency, since the model must repeatedly wait for tool observations before continuing. We study how to accelerate such trajectories without changing the final trajectory the model would have taken without acceleration, assuming access to faster but less reliable speculator tools. We develop a theoretical framework for lossless speculation in multi-hop tool-use settings, characterizing the optimal achievable latency gain. We propose SpecHop, a continuous speculation framework that maintains multiple speculative threads, verifies predicted observations asynchronously as target tool outputs arrive, commits correct branches, and rolls back incorrect ones. This preserves task accuracy while reducing wall-clock latency. We show that SpecHop can approach the oracle latency gain with sufficient active threads. Empirically, we evaluate SpecHop on retrieval-augmented multi-hop tasks and find that its latency gains closely match theoretical predictions, reaching up to 40% latency reduction in some settings.

As language models become autonomous software agents, documentation shifts from explanatory prose to a natural-language behavioral specification for code generation. We study agent-oriented documentation generation, where the objective is not readability but the correctness of code synthesized from documentation alone, turning the task into a black-box search over natural-language specifications evaluated only through the generated code and its execution behavior. Its defining difficulty is output coupling: program entities are behaviorally entangled, so a revision that sharpens the specification of one entity may simultaneously destabilize its dependents, inducing the familiar whack-a-mole failure mode in iterative refinement. We introduce SpecForge, a multi-frontier tree search solution that preserves complementary search frontiers through Pareto state management to avoid premature commitment and escape certain frontier-local optima, performs dependency-constrained bandit selection for callee-before-caller refinement, and uses diversified error-conditioned expansion to remain robust to noisy test feedback. On DevEval+, SpecForge attains the best performance across five backbone models, including a 94.9% solve rate with Claude-4.5-Sonnet and a 25.7% average improvement over the leading baseline. We further demonstrate that the resulting specifications improve downstream performance on both cross-language code translation and new-feature implementation.