NeurIPS 2026 Papers

Adapting Large Language Models (LLMs) to downstream tasks is increasingly bottlenecked by the staggering memory overhead of first-order (FO) backpropagation. Zeroth-order (ZO) optimization has emerged as a compelling, memory-efficient alternative; however, it suffers from the curse of dimensionality, which introduces prohibitively high variance in gradient estimation. While LLM gradients empirically reside in low-rank subspaces, existing state-of-the-art ZO methods typically rely on randomly generated low-rank subspaces, which often fail to align with the underlying dominant gradient manifold, fundamentally limiting optimization efficiency. To bridge this gap, we propose AHZO, an efficient low-rank ZO fine-tuning algorithm that dynamically identifies dominant subspaces via the Average Historical Gradient (AHG). Motivated by the directional coherence of optimization trajectories, AHZO leverages the average historical ZO gradient over a period as a principled proxy for the true gradient. By applying singular value decomposition to AHG matrices, AHZO distills principal spectral signatures to construct low-rank bases that tightly align with the true dominant subspaces. Theoretically, we prove that AHZO significantly mitigates estimation variance, yielding superior convergence guarantees compared to random-subspace ZO methods. Extensive evaluations across diverse LLM architectures and benchmarks demonstrate that AHZO consistently outperforms existing ZO baselines, achieving performance highly competitive with memory-intensive FO fine-tuning.

Persistence diagrams provide stable, interpretable summaries of geometric and topological structure, and are useful for simulation-based inference when important information is not captured by low-order statistics. In practice, however, persistence-based pipelines require hand-chosen filtrations, vectorizations, and compressors, usually without an objective tied directly to parameter uncertainty. We introduce \textbf{TopoFisher}, a differentiable persistent-homology pipeline that learns topological summaries by maximizing local Gaussian Fisher information. From simulations near a fiducial parameter value, TopoFisher optimizes trainable filtrations, diagram vectorizations, and compressors without posterior samples or supervised regression targets, while preserving the inductive bias of stable topological descriptors. We also give sufficient regularity conditions under which the log-determinant Fisher loss is locally Lipschitz in the trainable parameters. Controlled experiments on noisy spirals and Gaussian random fields, where the total Fisher information is known, validate the pipeline: TopoFisher recovers a large fraction of the available information and improves over fixed topological vectorizations. Our main results are on weak gravitational lensing, a high-dimensional non-Gaussian field-inference problem from cosmology. There, both learned topological summaries, a fixed cubical filtration with a learned PersLay vectorization, and a learned CNN filtration with the same PersLay vectorization, reach $\log|F|\approx 21$, compared with $13.8$ for the power spectrum, $17.1$ for peak counts, and $19.3$ for wavelet scattering, approaching an unconstrained Information Maximising Neural Network baseline ($22.4$) with up to $\sim80\times$ fewer parameters. More importantly, the fixed-filtration variant generalizes better: under simulator shift from lognormal to LPT-based maps it retains $\log|F|=19.24$ while the neural baseline drops to $8.9$, and in neural posterior estimation it yields tighter constraints than the neural baseline, power spectrum, peak counts, and wavelet scattering. These results suggest Fisher-based topological optimizations as a robust, parameter-efficient front end for simulation-based inference.

Unsupervised domain adaptation has become an important paradigm for mitigating distribution shift in time series classification. However, existing time series domain adaptation methods typically align source and target data at a fixed time scale and simply fuse time-frequency features before uniform alignment. Furthermore, they do not consider the impact of noisy samples on distribution alignment. To address these limitations, we propose CSPAN (Time-Frequency Decoupled \textbf{C}ross-\textbf{S}cale \textbf{P}artial Optimal Transport \textbf{A}lignment \textbf{N}etwork) for time series domain adaptation. CSPAN first constructs segment-scale time-frequency candidates, and then uses Segment-Scale Aware Top-k Fusion to select the most suitable segment-scale feature representation. It then performs cross-domain alignment after decoupling time and frequency through Temporal Partial Optimal Transport and Frequency Partial Optimal Transport, where temporal domain alignment is guided by the transferability estimated from a probing transport plan, and frequency domain alignment is guided by the discriminability estimated by an auxiliary frequency classifier. By combining segment-scale representation selection, time-frequency decoupling, and partial distribution alignment, CSPAN enables more flexible cross-domain matching. Extensive experiments on six datasets demonstrate CSPAN’s consistent superiority, achieving an average accuracy improvement of 3.16\% in cross-domain scenarios. Code is available at the anonymous link: \url{https://anonymous.4open.science/r/CSPAN-35FB/}.

Multi-parameter persistent homology is a rapidly developing branch of topological data analysis that improves the robustness of single-parameter persistent homology to outliers, while still capturing the metric characteristics of the data. However, a notable limitation is its lack of scalability. In this paper, we introduce a novel approach for efficiently computing 2-parameter persistent homology on large point sets. Our work extends the Flood filtration, originally developed for single-parameter persistence. Our construction, called the sublevel Flood bifiltration, offers a scalable approximation of the sublevel Flood bifiltration. We show that it benefits from theoretical stability properties and describe how to compute it efficiently. We demonstrate the performance of our approach in classification tasks on low-dimensional synthetic datasets, where density awareness is critical, as well as on real-world time series datasets.

How little of a pretrained large language model has to change for it to acquire new capabilities? We find that zeroing fewer than 0.05% of its weights, with no other modification, matches full-parameter fine-tuning (FPT) at scale across the three dominant LLM fine-tuning paradigms: on-policy reinforcement learning with verifiable rewards (GRPO), supervised fine-tuning (SFT), and on-policy distillation (SDFT). We call the method Bit-Mask Tuning (BMT): a learnable binary keep/zero mask over the gate projection of transformer blocks. BMT reaches FPT accuracy at our largest backbones, with the GRPO gap closing monotonically across model sizes from 0.5B to 8B. Two practical advantages follow: the trained mask is several 1000x smaller than the full-parameter checkpoint, and BMT forgets substantially less than FPT, retaining prior-task accuracy throughout training where FPT measurably drifts. To understand why masking alone suffices, we analyze the update geometry on GRPO and find that BMT's weight delta lands closer to full-parameter updates than other adapters on three measures: effective rank, spectrum drift, and principal-weight overlap. In sum, deletion alone produces a tiny yet high-performing adapter, reduced forgetting, and an update geometry that tracks FPT's more closely than any baseline we evaluate.

Hypernetwork-based methods such as Doc-to-LoRA internalize a document into an LLM's weights in a single forward pass, but they fail systematically when the document contradicts pretraining knowledge: accuracy drops to 46.4% on the deepest conflicts. We show that this failure is primarily a magnitude problem rather than a representational one. The generated adapter reaches the relevant layers, but its adapter margin is not calibrated to the strength of the contradicted prior; as the pretrained margin grows, deep conflicts lose the override competition. This account predicts that failure should track prior strength. Sorting 194 conflicts by the base model's log-probability on the contradicted fact, baseline accuracy falls from 68% on weak-prior questions to 16% on strong-prior questions, a 52 percentage-point gap. We propose two training-free corrections. Selective Layer Boosting (SLB) scales the adapter at its highest-activity layers, and Conflict-Aware Internalization (CA) applies stronger boosting only when the base model appears confident. Together they raise deep-conflict accuracy from 46.4% to 71.0% on Gemma-2B and from 53.6% to 72.5% on Mistral-7B, while preserving novel-knowledge recall at 97.1%. They outperform vanilla retrieval-augmented generation on Gemma medium conflicts by 18 percentage points, though explicit conflict-aware prompts remain stronger when the user already knows a conflict exists. We release KID-Bench, a 489-question benchmark that separates novel recall, cross-knowledge combination, and prior-graded conflicts.

Knowledge editing aims to rewrite sensitive information in language models, yet prior work has shown that edited models remain vulnerable to extraction attacks. In this work, we identify a previously overlooked vulnerability, which we term $\textit{rank leakage}$: although editing replaces the top-1 output with a sanitized answer, the original sensitive token often remains highly ranked in the model’s next-token distribution. Exploiting this phenomenon, we propose an effective black-box Output Attack that recovers sensitive information using only a single query. Despite its simplicity, the attack achieves strong extraction performance and remains effective even against models equipped with state-of-the-art defenses. To mitigate this vulnerability, we introduce $\textbf{R}$ank-guided $\textbf{C}$oncealment $\textbf{E}$diting (RCE), a plug-in defense that enforces explicit control over the rank of sensitive tokens during editing. Although designed for the Output Attack, RCE generalizes to existing black-box and white-box extraction attacks. Extensive experiments across multiple models, datasets, and editing algorithms show that RCE consistently improves resistance to attacks while preserving edit fidelity and model utility. Our results highlight the importance of controlling the rank of sensitive tokens, rather than only the top-1 prediction, for privacy-preserving knowledge editing.

In Low-Rank Adaptation (LoRA), the scaling factor $\alpha$ is often treated as a mere complement to the learning rate, yet its role in optimization remains poorly understood. In this paper, we reveal that the scaling factor $\alpha$ and the learning rate function differently, with $\alpha$ emerging as the dominant driver of effective optimization, delivering gains that cannot be replicated by learning rate scaling alone. Through the synergy of extensive empirical analysis and a theoretical Signal-Drift framework, we uncover three findings into LoRA’s scaling mechanism: First, LoRA’s spectral suppression smooths the optimization landscape, rendering standard hyperparameters overly conservative and creating an optimization gap. Second, when leveraging this smoothness to accelerate convergence, $\alpha$ outperforms the learning rate by amplifying the task signal without increasing the drift ratio. Third, the optimal scaling factor follows a sublinear relationship with the rank, well characterized by a square-root law with an unexpectedly large coefficient, revealing the insufficient scaling of existing rank-tied heuristics. Based on these insights, we propose LoRA-$\alpha$, a minimalist framework that restores $\alpha$ to its principled regime, making LoRA compatible with standard small learning rates. Extensive evaluations across diverse tasks demonstrate that LoRA-$\alpha$ consistently improves performance while streamlining hyperparameter search, unleashing the learning potential of LoRA.

Probabilistic Circuits (PCs) are deep generative models that support exact and efficient probabilistic inference. Yet in autoregressive language modeling, PCs still lag behind Transformer-based large language models (LLMs), suggesting an important expressivity gap. In this work, we compare PCs and LLMs under a unified autoregressive formulation. First, an output bottleneck: PCs parameterize predictions as convex combinations in probability space, which struggles to represent the sharp distributions typical of language; adopting a logit-space parameterization substantially narrows this gap. Second, a context-encoding bottleneck: we prove that structured-decomposable PCs can match Transformer separation rank on vtree-aligned partitions, but show, both theoretically and empirically, that this capacity is limited to partitions aligned with the fixed routing structure, leading to severe degradation when the data exhibits heterogeneous dependency topologies. We further prove that decomposable PCs are strictly more expressive than structured-decomposable ones, though effectively optimizing them remains an open challenge.

Deployed reinforcement learning agents often face safety requirements that are specified only after training: new hazard maps, revised risk thresholds, or behavioral alignment constraints. We study zero-update deployment-time adaptation in which a fixed library of risk-neutral source policies must be reused under a newly specified reward–risk tradeoff. We propose **TRAM** (*Test-time Risk Adaptation via Mixture of Agents*), a source-scored composition rule that evaluates each source under the target reward and an occupancy-based deployment risk, then selects actions using risk-adjusted source scores. Unlike training-time risk-sensitive methods tied to a fixed surrogate such as return variance, TRAM supports spatial barrier exposure, divergence to a reference behavior, and local volatility risks specified at test time. We make the surrogate nature of the method explicit: TRAM is not claimed to solve the full occupancy-control problem of the stitched policy, but it admits a measurable source-hull mismatch term connecting source-scored risk to realized risk. Experiments in gridworlds, MuJoCo Reacher, Safety-Gymnasium, and an LLM alignment setting show that TRAM improves deployment risk while preserving reward and requires no parameter updates at test time.

Neurobiological circuits may be energy efficient in part because there is a division of labor: different subsystems compute or represent different things. In light of this, it is interesting that a major difference between subsystems is the time scale on which they typically vary. Here, we argue that temporal smoothness constraints---the idea that it may be energetically costly for systems to vary much more quickly or slowly than their typical time scale---imply a certain division of labor with respect to temporal stimulus features. In particular, slow subsystems ought to represent slowly varying stimulus features, and fast subsystems ought to represent quickly varying features. We formulate a novel efficient coding model that we use to investigate this claim, and exactly solve for the optimal division of labor. In addition to finding that slower subsystems ought to represent more slowly varying temporal features, we find surprising structure of mathematical interest: optimal codes have subsystems which represent orthogonal temporal features, and which correspond to the eigenfunctions of a Sturm-Liouville problem.

Long-Tailed Class-Incremental Learning (LT-CIL) is challenging due to severe class imbalance, where frequent head classes dominate gradient updates and suppress learning for rare tail classes. Beyond data imbalance, we identify an asymmetric likelihood signals within each session: tail classes contribute insufficient evidence to overcome prior regularization, causing their parameters to remain near the prior mean even under variational inference. This asymmetry disrupts the stability-plasticity tradeoff, leading to pronounced forgetting and poor tail-class performance. We propose a hierarchical variational framework, TailAdapt, for prompt and adapter-based continual learning that enables adaptive group-wise regularization of model parameters. TailAdapt employs a hierarchical inverse Gaussian scale-mixture prior, which induces a heavy-tailed marginal distribution over prompt and adapter parameters. This formulation encourages selective plasticity so that most parameter groups are strongly regularized to preserve previously learned knowledge, while a small, data-supported subset is allowed to adapt substantially. This selective plasticity allocates adaptation capacity where it is most needed, mitigating interference from head classes and improving learning for tail classes. Extensive experiments on LT-CIL benchmarks demonstrate consistent improvements over strong baselines, with better tail-class performance, reduced forgetting, and more efficient utilization of prompt and adapter capacity.

We introduce and analyze Target-Induced Loss Tilting (TILT) for unsupervised domain adaptation under covariate shift. It is based on a novel objective function that decomposes the source predictor as $f+b$, fits $f+b$ on labeled source data while simultaneously penalizing the auxiliary component $b$ on unlabeled target inputs. The resulting fit $f$ is deployed as the final target predictor. At the population level, we show that this target-side penalty implicitly induces relative importance weighting at the population level, but in terms of an estimand $b^*_f$ that is *self-localized* to the current error, and remains *uniformly bounded* for any source-target pair (even those with disjoint supports). We prove a general finite-sample oracle inequality on the excess risk, and use it to give an end-to-end guarantee for training with sparse ReLU networks. Experiments on controlled regression problems and shifted CIFAR-100 distillation show that \tilt improves target-domain performance over source-only training, exact importance weighting, and relative density-ratio baselines, with a stable dependence on the regularization parameter.

Model-based learning agents use learned world models to predict future states, plan actions, and adapt to new environments. However, the process of updating world models from collected experience creates a training-time attack surface: adversarially poisoned fine-tuning trajectories can manipulate the learned dynamics and thereby corrupt downstream planning. In this paper, we propose SWAAP, the first two-stage data poisoning framework for learned world models. In the first stage, SWAAP identifies a harmful target world model that induces low-return behavior under planning while remaining close to clean dynamics, using first-order bilevel optimization enabled by a transition-gradient theorem. In the second stage, SWAAP realizes this target through stealth-constrained gradient matching, modifying only a limited fraction of fine-tuning transition targets so that the induced training gradients steer the victim model toward the adversarial target, while a prediction-error regularizer encourages the poisoned targets to remain close to the world model's natural approximation error. To assess attack stealthiness, we evaluate defenses and detectability across three stages of the poisoning pipeline: pre-training detection of poisoned transitions, robust training during fine-tuning, and test-time monitoring of the resulting world model. Across diverse continuous-control tasks, SWAAP causes substantial performance degradation while keeping poisoned transitions close to clean data and avoiding detection by the evaluated defenses. These results reveal a practical vulnerability in world-model adaptation pipelines and highlight the need for robustness methods that protect both world-model training data and learned dynamics.

Determinant point processes (DPPs) have recently emerged as a kernelized alternative to vanilla independent sampling for generating efficient minibatches, coresets and other parsimonious representations of large-scale datasets. While basic theoretical foundations and promising empirical performance have been demonstrated, there are two main challenges for current proposals for DPP-based coresets or minibatches. The first is the need for families of DPPs with certain key variance reduction properties, usually constructed in a continuous setting, of which there are few known examples. The second is the need for an ad-hoc construction of a discrete DPP defined on a given dataset, that inherits such variance reduction. In this work, we contribute to the programme of establishing DPPs as a subsampling toolbox for ML by advancing on these two fronts. First, we propose new DPPs on the Euclidean space based on wavelets, with provably better accuracy guarantees than the best known rates. Second, we introduce a general method to convert such continuous DPPs, which are more amenable to proving analytical statements, into discrete kernels, which are pertinent for subsampling tasks such as minibatch and coreset constructions. This conversion mechanism simultaneously preserves the desired variance decay and reveals a low-rank decomposition of the discrete kernel, which makes sampling the corresponding DPP computationally inexpensive. En route, we enlarge the class of ML tasks amenable to improvements via DPP-based minibatches and coresets to include highly non-smooth objective functions, with arbitrarily low regularity, and rate guarantees that adapt explicitly to this regularity.

Training neural networks directly in a low-rank parameterization is an appealing route to reducing memory, compute, and storage simultaneously during both training and inference. Dynamic low-rank training (DLRT), which confines weights to a rank-$r$ manifold via the Galerkin projection of the gradient flow, is particularly attractive because it identifies efficient subnetworks on the fly without specialized initialization or post-factorization. However, DLRT fails to find trainable networks under high compression. In this paper, we derive the gradient flow of the best rank-$r$ approximation and point out that the offset of DLRT comes from a curvature-coupling term which is large and thus non-negligible under aggressive compression. Guided by this analysis, we propose a stable dynamic low-rank training method, named SDLRT, which maintains a lightweight compensation buffer that reinjects the top neglected singular directions. Additionally, we introduce a negative feedback on the truncation tolerance to stabilize each layer's rank. Experimentally, SDLRT reliably finds trainable subnetworks where DLRT collapses and as a PEFT adapter on DeBERTa-v3, it achieves the best average score on SuperGLUE at only $2.8\\%$ parameter overhead over LoRA.

Spiking Neural Networks (SNNs) promise energy-efficient inference, but adapting large pre-trained SNNs to new tasks is expensive because BPTT memory and runtime scale with the number of simulation steps $T$. We observe that weights in pre-trained SNNs are strongly concentrated near zero, and under thresholded dynamics many such synapses are functionally silent. Based on this, we propose \textbf{Spike-SFT}, a two-stage adaptation framework. Stage 1 (\textbf{Selective Parameter Enhancement}, SPE) fine-tunes only a small-magnitude subset via masked updates with low-rank regularization, progressive re-selection, and sparse-gradient storage, reducing memory while maintaining competitive accuracy. Stage 2 (\textbf{PickIt}) fuses multiple SPE-adapted models by alignment and interference-aware delta merging with spike-statistics calibration, yielding a single merged model with zero inference-time overhead. Across several SNN backbones and benchmarks, Spike-SFT offers a favorable accuracy--cost trade-off, improving fine-tuning efficiency (time/memory) while retaining strong accuracy, and enabling zero-overhead weight-space fusion via PickIt.

Pre-training Graph Neural Networks (GNNs) via self-supervised learning has become a dominant paradigm, yet efficiently adapting frozen encoders remains a challenge. Graph prompting offers a parameter-efficient alternative to fine-tuning, but existing methods largely treat pre-trained models as opaque feature extractors, ignoring their internal spectral structure. In this work, we identify a systematic phenomenon in pre-trained GNNs, which we term spectral bias: optimization during pre-training disproportionately aligns representations with directions associated with large singular values, leaving low-energy directions under-explored. We show that these underutilized directions can encode complementary information that is beneficial for downstream adaptation, especially under distribution shift. To leverage this insight, we propose Spectral Reverse Prompt (SRP), a prompting framework that rebalances the spectral contributions of frozen GNN encoders. SRP applies a learnable soft-thresholding mask in the spectral domain to down-weight dominant directions while amplifying weaker ones. In addition, SRP incorporates a null-space augmentation module that captures variation in directions with minimal activation under the frozen encoder. Extensive experiments across multiple benchmarks demonstrate that SRP achieves state-of-the-art performance with minimal additional parameters, highlighting that reweighting spectral components is a principled and effective strategy for parameter-efficient graph adaptation.

Model merging enables the integration of multiple expert models with different capabilities into a unified model. In practical deployments, new expert models are often continuously updated and arriving. Meanwhile, due to the training and communication overhead of large language models (LLMs) and multimodal large language models (MLLMs), Low-Rank Adaptation (LoRA) is widely used to adapt large models to specific domains due to its efficiency. This scenario brings a new challenge: continual LoRA merging of deployed MLLMs to load new task capabilities. Naive merging faces catastrophic forgetting and parameter conflicts. We first observe the distinct features of the LoRA direction and layers in MLLM LoRA tuning. Based on this, we propose our Spectral Rank Calibration Merging (SRC-Merging), a data-free and train-free continual LoRA merging method tailored for MLLMs. SRC-Merging regards the deployed LoRA as a compressed spectral memory and performs calibrated rank-level merging for each incoming adapter. It aims to preserve accumulated multimodal knowledge while avoiding over-suppression of new vision-language task updates during continual merging. Extensive experiments on multiple multimodal tasks have demonstrated that our method significantly outperforms state-of-the-art merging methods.