NeurIPS 2026 Papers

Forward-backward (FB) representations provide a powerful framework for learning the successor representation (SR) in continuous spaces by enforcing a low-rank factorization. However, a fundamental spectral mismatch often exists between the high-rank transition dynamics of continuous environments and the low-rank bottleneck of the FB architecture, making accurate low-rank representation learning difficult. In this work, we analyze temporal abstraction as a mechanism to mitigate this mismatch. By characterizing the spectral properties of the transition operator, we show that temporal abstraction acts as a low-pass filter that suppresses high-frequency spectral components. This suppression reduces the effective rank of the induced SR while preserving a formal bound on the resulting value function error. Empirically, we show that this alignment is a key factor for stable FB learning, particularly at high discount factors where bootstrapping becomes error-prone. Our results identify temporal abstraction as a principled mechanism for shaping the spectral structure of the underlying MDP and enabling effective long-horizon representations in continuous control.

Modular networks pursue specialization through learned routers, gates, and load-balancing losses. However, at matched total-parameter budgets, learned routers can underperform equal-weight No-Routing baselines. Is the bottleneck the routing algorithm, or the alignment between training-signal granularity and the target categories? We show that when each training and inference unit carries one coarse category tag, a fixed parameter-free routing scheme (SpecDrop) induces branch-category specialization and matches or exceeds the learned-routing baselines we evaluate at matched parameter budget; when this alignment breaks, the scheme matches the multi-branch No-Routing baseline. SpecDrop assigns each of $K$ branches weight $p_{\mathrm{a}}$ for its category and a small leakage $p_{\mathrm{i}}{>}0$ otherwise, merged through a category-independent fixed denominator that calibrates merged-branch magnitude to single-branch scale, with no learnable routing parameters or auxiliary losses. Category labels are required at inference, drawn from dataset metadata. On vision tasks where each image has one superclass label, SpecDrop reaches $\mathbf{79.23\%}$ on CIFAR-100 with ResNet-110, outperforming dense by $+4.75$, and $\mathbf{79.89\%}$ on ImageNet-1K with Vision Transformer ViT-S/16, outperforming the matched-supervision No-Routing baseline with a shared expert by $+6.53$. SpecDrop reaches the highest top-1 among the multi-branch routing baselines we evaluate at this parameter budget (Soft MoE and Mod-Squad on ImageNet). On NLP tasks where training units span multiple categories, SpecDrop matches the multi-branch No-Routing baseline on both SlimPajama-6B language modeling with a 30M Transformer and SuperNI instruction tuning over Llama-3.2-1B with LoRA. Furthermore, SpecDrop matches or outperforms multi-branch routing baselines including Demix and LoRAMoE, and is statistically tied with HydraLoRA. Per-branch pruning sensitivity reveals four specialization regimes ranked by category clarity: strongest on ImageNet-1K, strong on CIFAR-100, weakening on SlimPajama, and anti-aligned on SuperNI LoRA. Granularity alignment, not algorithm choice, localizes when routing helps. Code: https://anonymous.4open.science/r/C30862.

Speculative decoding accelerates LLM inference by drafting a tree of candidate continuations and verifying it in one target forward. Existing drafters fall into two camps with opposite weaknesses. Autoregressive drafters such as EAGLE-3 preserve dependence along each draft path but call the drafter once per tree depth, making drafting a non-trivial share of per-iteration latency. Parallel drafters cut drafter calls by predicting multiple future positions in one forward, but each position is predicted without seeing the others, producing paths the verifier rejects. In this paper, we propose SpecBlock, a block-iterative drafter that combines path dependence with cheap drafting. Each drafter forward produces $K$ dependent positions and we call this a block. The draft tree grows through repeated block expansions. Two mechanisms explicitly carry path dependence to keep later draft positions accurate. Within each block, a layer-wise shift carries the previous position's hidden state into every decoder layer. Across blocks, each new block can start from any position of the previous block, inheriting its hidden state to extend the path. To spend verifier budget where acceptance is likely, a co-trained rank head replaces the fixed top-$k$ tree by allocating per-position branching during drafting. To avoid training the drafter on prefixes it never produces at inference, a valid-prefix mask drops the loss at later positions once an earlier one is wrong. Beyond static drafting, a cost-aware bandit at deployment uses free verifier feedback to update the drafter selectively, only when the expected throughput gain exceeds the update cost. Experiments show that SpecBlock improves mean speedup by $8$--$13\%$ over EAGLE-3 at $44$--$52\%$ of its drafting cost, and cost-aware adaptation extends this lead to $11$--$19\%$.

Quantum state tomography of pure non-Gaussian states is fundamentally limited by scarce and noisy data due to measurement-induced collapse and experimental drift. We introduce a novel reconstruction framework that parameterizes the wavefunction directly via an overparameterized neural network, trained using rank-1 factored gradient descent. We rigorously prove that this formulation's loss landscape is devoid of spurious local minima and strict saddles, guaranteeing all critical points are global minima. By utilizing the max-sliced Wasserstein-1 distance, our loss perfectly maps to the structure of homodyne measurements, structurally preventing gradient vanishing in low-shot regimes. On noisy two-mode cat states, our method achieves continuous fidelity improvements, successfully bypassing the noise-floor plateaus that limit standard iterative maximum-likelihood estimation.

Federated intrusion detection must adapt to rare attacks from only a few labeled flows per client, while client distributions are highly non-IID and raw traffic cannot be centralized. Existing few-shot intrusion detectors typically construct class priors in centralized settings, and standard federated training does not provide an explicit mechanism for reusing cross-client class statistics during local low-shot adaptation. We propose a semantic-statistical prior bank for federated low-shot intrusion detection. Clients first participate in federated representation learning and submit protected class-wise moment statistics through secure aggregation, from which the server constructs a global prior bank without accessing raw traffic. During local episodic adaptation, label-phrase embeddings retrieve relevant priors, which are combined with scarce support statistics through a maximum a posteriori (MAP) shrinkage estimator in the feature space. The adapted class distributions are then used to train lightweight episode-specific classifiers on each client. Experiments on CICIDS2017 and CICDDoS2019 under 5-way 1-shot and 5-shot non-IID settings show consistent improvements over FedAvg and backbone-matched baselines, with the largest gains appearing in 1-shot and highly skewed client distributions. With ResNet1D, the proposed adaptation improves 1-shot F1 from 92.97 to 95.55 on CICIDS2017 and from 82.13 to 88.25 on CICDDoS2019.

Most graph neural networks propagate information through a fixed-coefficient polynomial filter applied uniformly across every node and every layer. However, the appropriate filter varies along two distinct axes that such a parameterization cannot accommodate. Within a single graph, different nodes benefit from different mixtures of low-pass smoothing, high-pass contrast, and pass-through behavior. Across graphs, signal propagation dynamics themselves differ, with some graphs benefiting from long-range propagation across many hops while others require damping to prevent over-smoothing. The appropriate filter is therefore not a single fixed object at all, but a state-dependent composition whose behavior changes according to a node's current representation and its propagation history. We propose Castor, a self-tuning graph filter that realizes this composition through state-dependent decisions made at every step of propagation. A router reads each node's current state together with its initial state, and emits per-node mixing weights over three primitives, namely low-pass smoothing, high-pass contrast, and identity. A learnable per-hop coefficient then determines how much of the most recent change in the propagation is carried forward, accelerating the iteration when positive and damping it when negative. Setting the coefficient to zero recovers standard first-order propagation. Across fourteen standard node-classification benchmarks spanning the full homophily spectrum, a single Castor instance ranks top-three on every dataset and achieves an average rank of $2.07$, more than $3.5$ ranks ahead of the next-best of fourteen baselines.

What does it mean to create a new concept, rather than retrieve a familiar one? Diffusion models sampled many times at the same prompt vary only within the narrow stylistic envelope set by a fixed prompt embedding. We propose an operational definition of creativity as the generation of concepts that are initially unfamiliar yet rapidly learnable to an adaptive observer, and formalize it as a bilevel optimization between a Creator that generates and an Appraiser that adapts: the Appraiser's improvement under a brief inner-loop adaptation provides the reward signal that the Creator maximizes. We call this framework CAMEL (Creator-Appraiser Meta-Learning), and instantiate it on MNIST with an autoencoder Appraiser and on natural images with a CLIP Appraiser based on a low-rank adapter over the text projection. CAMEL produces outputs that lie outside the basin reachable by vanilla diffusion sampling at any sample count, while remaining recognizable as the prompt's class to an off-the-shelf observer.

When deploying large language models (LLMs) to safety-critical applications, uncertainty quantification (UQ) is of utmost importance to self-assess the reliability of the LLM-based decisions. However, such decisions typically suffer from overconfidence, particularly after parameter-efficient fine-tuning (PEFT) for downstream domain-specific tasks with limited data. Existing methods to alleviate this issue either rely on a Laplace approximation based post-hoc framework, which may yield suboptimal calibration depending on the training trajectory, or variational Bayesian training that requires multiple complete forward passes through the entire LLM backbone at inference time for Monte Carlo estimation, posing scalability challenges for deployment. To address these limitations, we build on the Bayesian last layer (BLL) model, where the LLM-based *deterministic* feature extractor is followed by random last-layer parameters for uncertainty reasoning. Since existing low-rank adapters (LoRA) for PEFT have limited expressiveness due to rank collapse, we address this with Polar-decomposed Low-rank Adapter Representation (PoLAR), an orthogonalized parameterization paired with Riemannian optimization to enable more stable and expressive adaptation. Building on this PoLAR-BLL model, we leverage the variational (V) inference framework to put forth a scalable Bayesian fine-tuning approach which jointly seeks the PoLAR parameters and approximate posterior of the last-layer parameters via alternating optimization. The resulting PoLAR-VBLL is a flexible framework that nicely integrates architecture-enhanced optimization with scalable Bayesian inference to endow LLMs with well-calibrated UQ. Our empirical results verify the effectiveness of PoLAR-VBLL in terms of generalization and uncertainty estimation on both in-distribution and out-of-distribution data for various common-sense reasoning tasks.

Source-Free Domain Adaptation (SFDA) aims to adapt a source-pretrained model to an unlabeled target domain without access to the original source domain. While early single-model approaches rely on self-refinement, they are inherently susceptible to confirmation bias and struggle to correct their own systematic errors. To overcome this limitation, recent methods introduce Vision-Language (ViL) models as external knowledge sources. However, these approaches operate in a largely unidirectional paradigm, using the ViL model primarily to supervise the source-pretrained model. This overlooks a key structural property: the two models exhibit distinct failure modes -- where one produces an incorrect prediction, the other may produce a correct one, creating a natural opportunity for mutual correction within the target domain. Yet, without ground-truth labels, identifying which model is correct on any given sample is non-trivial, and naively exchanging predictions risks propagating errors across models. To address this challenge, we propose SafeCut, a novel approach that leverages the cut statistic as a label-free measure of prediction reliability to gate cross-model supervision. Our approach dynamically controls both the direction and strength of supervision based on relative reliability, selectively amplifying true corrections while suppressing miscorrections on a per-sample basis. We further provide theoretical justification showing that this reliability-gated mechanism guarantees a net-positive correction signal. Extensive experiments across diverse SFDA benchmarks demonstrate that SafeCut achieves state-of-the-art performance, highlighting the effectiveness of safeguarding mutual correction in SFDA via cut statistics. Codes are available https://anonymous.4open.science/r/SafeCut_Source_Free_Domain_Adaptation-65D0/

Low-rank adaptation (LoRA) has attracted significant attention in the parameter-efficient fine-tuning (PEFT) landscape. However, its design suffers from two fundamental limitations. First, its invariance manifold induces flat directions in the loss landscape, creating an optimization bottleneck for adaptive optimizers. Second, its effective rank scales only linearly with the parameter budget. In this paper, we propose Scalable Morpho Adaptation (SaMA), a PEFT method grounded in a principled generalization of the Kronecker product. By relaxing the shared-block constraint in the perfect-shuffle factorization, SaMA spans from low to full rank and achieves effective rank that scales \quadratically in parameters. Furthermore, we show that its invariance manifold collapses to a diagonal group, making it structurally more optimizer-friendly. Empirically, SaMA consistently outperforms strong PEFT baselines on commonsense and arithmetic reasoning benchmarks while being more parameter-efficient.

Training-memory reduction requires identifying which saved tensors drive peak usage and ensuring high-fidelity backward contractions from compressed states. We formalize saved-activation compression as a target-surface prediction problem: a pre-training framework to estimate the replaceable memory surface, computational overhead, and parameter-gradient error. We introduce STAC-R (Subspace Tracked Activation Compression with Residuals), a saved-tensor wrapper storing activations as persistent low-rank coefficients paired with a sparse quantized residual. Weight gradients are contracted directly from these factors, bypassing full decompression. To guide deployment, we propose the Saved-Surface Predictor ($\mathrm{SSP} = A_C^+ / P$), a statistic that upper-bounds peak memory reduction. We also derive a gradient bridge linking reconstruction error to parameter-gradient cosine similarity, determining if a low-precision state preserves update fidelity.Evaluations across GPT-2 (Medium/Large) and Pythia-1.4B pre-training show STAC-R reduces peak memory by 7.2-12.4\% while keeping long-horizon validation perplexity within 1.5\% of baseline (Pythia-1.4B 10k steps: +1.14\%) with mean gradient cosine $\geq$ 0.98. On BERT fine-tuning (SST-2), STAC-R achieves 12.45\% memory reduction with a 0.15pp accuracy decrease. Mechanistic benchmarks yield a mean gradient cosine of 0.9813, outperforming iso-byte SVD (0.9236) and untracked-subspace (0.6949) controls. On the empirical rate-distortion frontier, STAC-R is Pareto-dominant over controls across stored-byte budgets. Composed with GaLore, a "Turbo-fast" variant matches full STAC-R's memory footprint within 0.005 GiB while reaching 0.67𝑥 wall time. Finally, our analysis identifies memory-increasing regimes in SwiGLU architectures (Qwen-2.5/Llama-3.2), establishing STAC-R as a compressor with predictable memory and fidelity conditions checkable before training.

Aligning Large Language Models (LLMs) with human preferences typically relies on external supervision, which faces critical limitations: human annotations are scarce and subjective, reward models are vulnerable to reward hacking, and self-evaluation methods suffer from prompt sensitivity and biases. In this work, we propose **stable rank**, an intrinsic, annotation-free quality signal derived from model representations. Stable rank measures the effective dimensionality of hidden states by computing the ratio of total variance to dominant-direction variance, capturing quality through how information distributes across representation dimensions. Empirically, stable rank achieves 84.04\% accuracy on RewardBench and improves task accuracy by an average of 14.1\% relative to greedy decoding via Best-of-N sampling. Leveraging this insight, we introduce **Stable Rank Group Relative Policy Optimization (SR-GRPO)**, which uses stable rank as a reward signal for reinforcement learning. Without external supervision, SR-GRPO improves Qwen2.5-1.5B-Instruct by 10\% on STEM and 19\% on mathematical reasoning, outperforming both learned reward models and self-evaluation baselines. Our findings demonstrate that quality signals can be extracted from internal model geometry, offering a path toward scalable alignment without external supervision.

Self-speculative decoding offers a plug-and-play route to accelerating autoregressive large language model (LLM) inference, but its practical success hinges on identifying effective layer-skipping configurations online without retraining or offline profiling. Existing methods mainly optimize agreement-based proxies that capture draft-target consistency, yet overlook intrinsic runtime differences across draft configurations. Consequently, configurations with similar acceptance can still yield substantially different realized decode throughput. We present SPIRIT, a speed-driven online adaptation framework for self-speculative decoding that directly optimizes realized decode throughput, measured as generated tokens per second, using window-level runtime feedback. Our key insight is that speed is not determined by agreement alone: it is jointly shaped by draft-target consistency and the intrinsic speed advantage of the draft configuration. Empirically, the skip ratio serves as a first-order control variable, capturing the dominant consistency-speed trade-off while sharply shrinking the search space over layer-skipping patterns. SPIRIT therefore first identifies an effective skip ratio and then refines the layer-skipping pattern within it. Combined with length-aware normalization and shift-aware profile-based adaptation, this design enables rapid adaptation to task shifts in non-stationary request streams. Across diverse tasks and model scales, SPIRIT consistently outperforms strong training-free baselines and achieves up to 1.74x speedup while preserving the target model's output distribution, establishing a practical foundation for plug-and-play, high-throughput LLM inference.

Recent foundation models are moving toward native multimodal Vision-Language Models (VLMs), making VLMs a central form of next-generation foundation models. However, their large language backbones make edge deployment difficult due to high memory footprint and memory-bound autoregressive decoding. Weight-only post-training quantization is a practical solution, but pushing VLMs to extreme low bit-widths remains challenging: existing rotation-free methods suffer from outliers at 2--3 bits, while rotation-based methods improve accuracy at the cost of additional runtime overhead. We propose SPHQuant, a rotation-free spherical weight-only quantization framework for VLMs. Instead of quantizing weights directly in Cartesian coordinates, SPHQuant decomposes each 8D weight vector into coordinate signs, radius, and a positive unit direction. This representation isolates outlier magnitude into the radius while keeping directions bounded and statistically regular. Based on this insight, SPHQuant allocates extra precision to the radius to mitigate accuracy degradation induced by outliers. It further uses a compact positive-direction codebook and fine-tunes codebook entries through angular parameterization to preserve the unit-sphere constraint. We also design a hardware-friendly GEMV kernel that keeps the direction codebook small enough for shared-memory lookup and packs radial bits efficiently. Experiments show that SPHQuant matches the performance of state-of-the-art extreme low-bit quantization methods while improving decode throughput by \textbf{30.3\%}, providing a new step toward efficient edge deployment of VLMs.

Low-Rank Adaptation (LoRA) enables efficient adaptation of large pre-trained models to downstream tasks by parameterizing weight updates with low-rank matrices. In this paper, we investigate the limitations of the LoRA parameterization from a geometric perspective. Specifically, we show that when a full fine-tuning gradient is backpropagated to the low-rank matrices, it undergoes anisotropic scaling driven by their singular values. We argue that this phenomenon is undesirable because it distorts the full fine-tuning gradient by skewing it toward dominant singular directions while suppressing others. Our analyses demonstrate that anisotropic gradient scaling reduces the effective rank of the low-rank matrices' gradients and results in suboptimal alignment between the full fine-tuning gradient and its low-rank approximation in LoRA, thereby exacerbating the gap to full fine-tuning. To address these limitations, we propose a new low-rank parameterization, SDS-LoRA, which structurally decouples singular values from the backward pass. Our method ensures that the full fine-tuning gradient backpropagates only through the orthonormal bases of the low-rank matrices' subspaces, independent of their scales. Convergence analysis demonstrates that while LoRA’s convergence rate degrades with the condition number of the low-rank matrices, SDS-LoRA remains independent of it. Experimental results across natural language and vision benchmarks show that SDS-LoRA improves loss convergence and reduces the gap to full fine-tuning, significantly enhancing adaptation performance.

Few-shot adaptation of vision--language models must balance two competing goals: learning fine-grained visual distinctions from very limited supervision, and preserving the efficiency that makes frozen CLIP-like models attractive for deployment. Existing lightweight adaptation methods, including prompt tuning, LoRA, and cache-based adapters, typically supervise global image embeddings and therefore underuse token-level structure. Conversely, structure-aware methods can exploit local evidence but often introduce additional inference-time computation. We propose SD-LoRA, a structural distillation framework that uses token-level graph reasoning only during training and absorbs its effect into a lightweight LoRA-adapted CLIP model. During training, a heterogeneous graph teacher models patch-patch, global-local, and visual-text relations over high-resolution ViT tokens. Rather than keeping this graph at test time, SD-LoRA transfers its structural bias into the shared CLIP-LoRA backbone through implicit distillation and Prototype Predictive Alignment, which aligns graph-conditioned representations with support-derived class prototypes. At inference, the entire graph teacher is discarded; prediction uses only a single CLIP-LoRA forward pass and a cache-based classifier. Across 11 few-shot benchmarks, SD-LoRA improves over a strong CLIP-LoRA baseline by 1.3/1.2/2.4/1.9/2.0% in the 1/2/4/8/16-shot settings, while adding no graph computation at test time. Further results on base-to-novel generalization, cross-dataset transfer, OOD robustness, and calibration show that training-time structural supervision improves not only accuracy but also transfer reliability.

Diffusion models achieve strong performance across image and audio synthesis, but their inference remains computationally expensive, requiring tens to hundreds of sequential evaluations of a large denoising network. We observe that the effective rank of the denoiser output varies sharply across timesteps and architectures, motivating step- and block-wise adaptation of computation. We propose Step-Adaptive Rank Adjustment (SARA), a training-free framework that treats the SVD rank of pretrained linear layers as an inference-time control variable, adapted per step and block by SARA-DP and per step by SARA-Online. SARA introduces the Low-Rank Approximation Error (LRAE) — the discrepancy a rank-r truncation induces against the full-rank network — and instantiates it in two algorithms: SARA-DP measures LRAE on a few warmup prompts and solves the globally optimal rank assignment under a MACs budget by dynamic programming; SARA-Online computes a closed-form, spectrum-based LRAE from the model output at inference time, requiring no calibration and no trajectory lookahead. SARA applies uniformly to UNet (SDXL), MMDiT (SD3), DiT (Stable Audio Open), and hybrid MMDiT/DiT (TangoFlux) backbones, spanning ϵ, v, and rectified-flow parameterizations. SARA combines with existing solver- and cache-axis acceleration to yield compounded wall-clock speedups in our experiments. Together, these results introduce rank as a new axis of diffusion inference acceleration.

Convolutional architectures have emerged as powerful alternatives to Transformers for sequence modeling. The primary advantage is that they offer improved theoretical sequence length complexity by leveraging the Fast Fourier Transform (FFT). However, this theoretical improvement does not always meaningfully land in practice. One critical obstacle is that applying standard FFTs is not amenable to the large-scale training pipeline wherein data is packed from different sources into a single sequence for hardware efficiency. Indeed, standard FFT algorithms are not easily amenable to document packing. Existing workarounds suffer from severe inefficiencies, crippling the practical performance of convolutional architectures. We close this gap with RubiConv, a novel algorithm for performing hardware-efficient, boundary-respecting convolutions on packed sequences. Extensive experiments show that RubiConv achieves significant speedups over both attention and standard FFT-based baselines. This work makes the theoretical efficiency of long convolutional models a practical reality for large-scale, real-world data packing.

Quantizing Mixture-of-Experts (MoE) models is difficult for two coupled reasons: expert quantization error perturbs expert outputs, and sparse routing amplifies small perturbations into discrete expert selection changes. This coupling produces progressive output drift that is not captured by dense model quantization analyses. To address this gap, we propose a unified quantization framework that jointly handles both sources of error. First, we formulate compression-induced routing inconsistency as a form of training-inference router mismatch and use this connection to motivate limited routing replay, which replays BF16 routes during quantization-aware fine-tuning (QAT), freezes router parameters, and updates only expert-side adapter parameters. We then provide a standard convergence guarantee for the resulting fixed-route QAT objective, together with a routing-consistency bound under explicit router-margin conditions. Second, for expert restoration under a fixed rank budget, we propose load-aware low-rank compensation that assigns larger compensation rank to heavily loaded experts. Under this allocation rule, we prove monotone strict decrease of weighted compression error and characterize optimal rank assignment by marginal load-weighted spectral gains. Finally, experiments on the MXFP4 quantization of Mixtral, DeepSeek, and Qwen3 MoE models show that our method improves perplexity and downstream accuracy over prior methods. We also report system efficiency on Qwen3-30B, where uniform W4A8 serving accelerates decoding and our adapter path preserves most of that acceleration.