NeurIPS 2026 论文列表

Large language models have achieved remarkable capabilities by scaling model capacity and training data, yet many practical deployments still rely on smaller models with limited resources whose capabilities lag behind their larger counterparts trained with richer resources. This gap calls for efficient knowledge transfer from source models with richer resources to compact target models. While model merging provides an effective mechanism, most existing methods are based on the assumption that the source and target models are architecturally compatible, making them inapplicable to heterogeneous source and target pairs. Although a recent method based on optimal transport extends model merging to settings with different architectures, it remains limited by linear correspondence modeling, iterative transport optimization, and reliance on supervised adaptation after fusion for further performance gains. To address these limitations, we propose BDC-Merge, a framework for model merging across different architectures based on Brownian Distance Correlation (BDCorr). BDC-Merge uses a small calibration set to estimate dependencies between heterogeneous activations at the feature level and the layer level, and directly lifts these dependencies from activation space into fusion operators in weight space, enabling effective parameter fusion across different architectures without any gradient based optimization or training after fusion. Extensive experiments across four low resource language and two specialized knowledge benchmarks show that BDC-Merge consistently outperforms the state of the art baseline for model merging across different architectures and largely preserves the target model’s general capabilities.

Attribute-efficient PAC learning of sparse halfspaces has been a fundamental problem in machine learning theory. In recent years, machine learning algorithms are faced with prevalent data corruptions or even malicious attacks. It is of central interest to design computationally-efficient algorithms that are robust to malicious corruptions. In this paper, we consider that there exists a constant amount of malicious noise in the data and show that it is possible to learn an underlying $s$-sparse halfspace $w^* \in \mathbb{R}^d$ with $O(s^2\log^5 d)$ samples. Specifically, we follow a recent line of works and assume that the underlying distribution satisfies a certain concentration condition and a margin condition at the same time. As a complementary result, we provide an information-theoretic sample lower bound under such conditions. Strong evidence shows that our sample complexity is nearly optimal. To show the robustness of our algorithm, we provide a new gradient analysis that carefully handles the sparsity admitted constraints in hinge loss minimization program, which could be of independent interest.

Low-rank factorization is a standard way to scale optimization in machine learning by replacing large matrix variables with compact factors. For positive semidefinite (PSD) variables, the symmetric Burer--Monteiro factorization (sBMF) writes $Z=XX^\top$ with a single low-rank factor $X$. A recent asymmetric alternative (aBMF) writes $Z=XY^\top$ and adds a quadratic penalty $(\gamma/2)\|X-Y\|_F^2$ to encourage symmetry. This split is attractive because it yields a biconvex objective with alternating convex subproblems, but its practical value depends strongly on how the penalty parameter $\gamma$ is chosen. We study a unified regularized aBMF framework and derive an explicit lower bound on $\gamma$ that guarantees exactness: under mild assumptions, any $\gamma$ above this threshold makes aBMF and sBMF share the same critical points. This gives a principled way to use the asymmetric formulation without altering the critical-point structure of the symmetric problem. In particular, it answers the open question of whether an exact penalty exists for asymmetric relaxation.

No existing nested Archimedean copula tool handles all three of (a) arbitrary per-variable (right-)censoring in survival analysis, (b) arbitrary nesting trees, and (c) exact parameter gradients. Existing implementations handle only bivariate problems, low dimensional (i.e., $d \leq 10$) cases, two layers of nesting, or only hand-derived copula nestings. We present acopula, a JAX-native framework that, given any Archimedean generator—classical or neural—evaluates exact nested-copula likelihoods and parameter gradients under arbitrary censoring masks in polynomial time. The mechanism is polynomial powering of Taylor-mode automatic differentiation output, which replaces per-family hand-derived partial Bell polynomial tables with a single differentiable computation that any user-defined generator can drive. We conduct extensive simulations to verify the correctness of acopula. We then demonstrate (a) per-variable censoring on $85,229$ MIMIC-IV ICU admissions in high dimensions with $d=53$, fit by both classical Archimedean families and nested neural Archimedean copulas; (b) an 11-sector hierarchical model on S&P 500 daily returns at $d=98$; (c) family-agnostic censored MLE across ten families, five of them with no prior implementation, on a retinopathy study; and (d) a $\sim650\times$ per-density speedup over R's nacLL at $d=35$, scaling quadratically to $d=8,000$.

Contrastive representation learning (CRL) is a fundamental paradigm for learning transferable representations. Recent work identifies its population target as the pointwise mutual information (PMI) $ \log\frac{p(\mathbf{x},\mathbf{y})}{p_\mathcal{X}(\mathbf{x})p_\mathcal{Y}(\mathbf{y})} $ between two modalities $\mathcal{X},\mathcal{Y}$, where $p$ is the joint density function and $p_\mathcal{X},p_\mathcal{Y}$ are the marginals. However, it remains unclear why standard inner-product scores $f(\mathbf{x})^\top g(\mathbf{y})$ can effectively approximate a generally nonseparable population target PMI. In this paper, we address this question from the aspect of approximation for CRL. By linking PMI to a compact operator, we show that $D$-dimensional inner-product models perform rank-$D$ spectral approximation, with error controlled by the spectral tail. We then study neural network realizations: in general settings, we show that deep neural network scoring functions are universally consistent, and the approximation error rates depend on the smoothness of the PMI function; under structured low-rank data models we obtain improved rates attained by moderate-width encoders. Finally, we establish a fast calibration rate from contrastive excess risk to downstream retrieval error under a density-ratio Tsybakov noise condition.

With the growing number of fine-tuned variants derived from pre-trained models, model merging emerges as a key challenge: effectively integrating task-specific adaptations while preserving performance. However, naive weight averaging often fails when parameter updates are misaligned, revealing the need for a structured space in which merging becomes principled. In this work, we hypothesize that pre-trained models possess an intrinsic low-rank structure. Specifically, we posit that the transformation from input to output is mediated by a latent bottleneck of low intrinsic dimensionality; we leverage this low-dimensional space to construct a model embedding space. This space ensures that fine-tuned updates, i.e., model embeddings, of all adaptations remain confined within the intrinsic structure of the pre-trained model. As model embeddings lie in the same low-dimensional space, model merging via direct averaging becomes more effective. Moreover, we further introduce an aligned subspace of these model embeddings to improve the efficacy of model merging. Experiments across vision and multimodal tasks, including ViT and multimodal LLMs, demonstrate that our approach enables effective model merging and provides a unified geometric view of adaptation and combination in pre-trained models.

We propose a new framework for low-precision neural networks based on angular geometry and randomized low-bit estimators. Rather than approximating Euclidean linear operations under limited precision, we reinterpret neural computation in terms of directional similarity and construct quantized random-feature estimators of cosine interactions. Our approach introduces \textit{Angular Layers}, which replace standard linear transformations with low-bit projections that estimate angular similarity in the forward pass, while gradients are computed with respect to the underlying continuous cosine geometry. This decoupling enables stable optimization without straight-through estimators or heuristic surrogate gradients, even under aggressive quantization of both weights and activations. On the theory side, we introduce \emph{G-Gradient} as a gradient proxy for angular layers and show that, in a one-layer planted model, a sufficiently small \emph{G}-gradient can guarantee exact recovery of a target binary solution. We further prove uniform approximation and convergence guarantees that provide a rigorous path from tractable continuous optimization to exact recovery in the original discrete model. Empirically, we instantiate these ideas in \textit{Angular Nets}, including low-precision variants of ResNet, Vision Transformers, and BERT-style encoders, and obtain strong performance across vision and language tasks at substantially reduced precision. Overall, our results suggest that preserving angular structure provides a principled foundation for low-bit deep learning. Our full implementation is available at: \url{https://github.com/GGM2026/GGM}.

Remote photoplethysmography (rPPG) enables non-contact physiological measurement from facial videos, but performance often degrades under unseen environments due to severe domain shifts. Continual learning offers a practical paradigm for updating rPPG models over sequential domains. However, it is challenged by two coupled issues: First, adapting to new domains may overwrite previously acquired physiological representations, resulting in the degradation of physiological consistency across domains. Second, drastic environmental variations induce heterogeneous facial video distributions, making the adaptation process unstable and prone to suboptimal convergence. To address these challenges, we propose ApexPhys, a continual rPPG framework that simultaneously anchors physiological invariance and expands domain plasticity. Specifically, we introduce a Backbone Consistency Representation Preservation (BCRP) mechanism, which leverages the Fisher Information Matrix to identify and preserve parameters critical to physiological consistency. To enhance adaptation flexibility under diverse environmental shifts, we further propose a Hierarchical Expansion Strategy (HES) that autonomously perceives layer-wise distribution discrepancies and dynamically expands hierarchical adaptation branches. Additionally, we develop a Prior-Guided Pathfinding (PGP) strategy, which leverages parameter inheritance to guide the adaptation along a stable trajectory. Experiments on five public datasets demonstrate that ApexPhys achieves strong anti-forgetting performance and robust adaptation under continual learning.

Principal component analysis (PCA) is one of the most widely used unsupervised dimension reduction techniques. We study PCA for data from multiple related domains. Since principal components generally differ across domains, one way to obtain a shared low-rank embedding is to perform PCA on the pooled data. However, this approach can focus on spurious directions that exhibit high variation in only a few domains. To find a robust embedding that still explains most variance in unseen but similar domains, we propose instead to focus on shared directions of variation. To this end, we introduce Anchor PCA which trades off overall explained variance with agreement between the shared and domain-specific low-rank embeddings. Anchor PCA amounts to PCA on a modified target matrix and thus can be solved efficiently. Moreover, we show that Anchor PCA recovers a maximal invariant subspace and admits a minimax reconstruction interpretation under bounded domain-specific covariance inflations. On simulated and real-world gas sensor data with temporal drift, we demonstrate, respectively, that Anchor PCA recovers the maximally invariant subspace and yields embeddings that explain more variance on unseen domains than the pooling baseline and a worst-case alternative. Taken together, these findings establish Anchor PCA as a promising approach to robust unsupervised dimension reduction from multi-domain data.

We introduce the Thresholding Monte Carlo Tree Search problem, in which, given a tree $\mathcal{T}$ and a threshold $\theta$, a player must answer whether the root node value of $\mathcal{T}$ is at least $\theta$ or not. In the given tree, 'MAX' or 'MIN' is labeled on each internal node, and the value of a 'MAX'-labeled ('MIN'-labeled) internal node is the maximum (minimum) of its child values. The value of a leaf node is the mean reward of an unknown distribution, from which the player can sample rewards. For this problem, we develop a $\delta$-correct sequential sampling algorithm based on the Track-and-Stop strategy that has asymptotically optimal sample complexity. We show that a ratio-based modification of the D-Tracking arm-pulling strategy leads to a substantial improvement in empirical sample complexity, as well as reducing the per-round computational cost from linear to logarithmic in the number of arms.

Federated Low-Rank Adaptation (FedLoRA) enables efficient fine-tuning of large language models by exchanging low-rank adaptation parameters across clients. Several studies mainly focus on aggregation bias and design improved aggregation schemes. We argue that aggregation bias is not the fundamental bottleneck of FedLoRA. Even under ideal aggregation, heterogeneous data induce directional conflicts among client-specific LoRA updates. We show that these conflicts arise from misaligned low-rank adaptation subspaces, leading to destructive interference during collaboration. To diagnose this phenomenon, we introduce a geometry-aware metric that characterizes client updates at the subspace level. Our analysis reveals that subspace conflicts intensify with increasing heterogeneity and are concentrated in the column spaces of LoRA updates. Based on this insight, we propose ALIGN, a geometry-aware federated adaptation framework that aligns client LoRA subspaces during aggregation and constrains local updates to mitigate conflicts. Comprehensive empirical results confirm that mitigating subspace misalignment is crucial for FedLoRA, and that ALIGN consistently improves downstream performance over existing methods. The code is available for anonymous access at https://anonymous.4open.science/r/ALIGN-7C41.

Federated fine-tuning of large language models (LLMs) under heterogeneous client capacities is increasingly important, and low-rank adaptation (LoRA) with heterogeneous ranks makes it practical. Existing server-side aggregators compose local matrix updates $\Delta W_i$ from client-specific LoRA factors $(A_i,B_i)$ before aggregating them. Although these matrix updates lie in the same weight space $\mathbb{R}^{m\times n}$ and can be averaged directly, the LoRA factors that generate them may expressed client-specific local basis. In the typical compose--aggregate--factorize--dispatch loop, this basis inconsistency can make the rank-$r_i$ reference prefix dispatched to lower-rank clients unstable after server-side refactorization. In this work, we propose FedAbA, a basis-consistent federated LoRA aggregation framework for heterogeneous ranks. FedAbA first extracts each client's local basis, aligns it to the corresponding prefix of the global reference basis, and constructs a basis-consistent reconstruction of each local matrix update before aggregation. It then aggregates these reconstructions with weights that combine client data size and consistency score, followed by exact coefficient-space SVD refactorization and gauge fixing before rank-prefix dispatch. Our theoretical analysis clarifies the role of basis consistency and supports the proposed consistency-aware aggregation mechanism. Extensive experiments show that FedAbA outperforms representative federated LoRA baselines, with the largest gains under the strongest rank heterogeneity.

Diffusion Transformers (DiTs) have achieved strong performance in visual generation, but their dense token processing leads to slow training convergence and high computational cost. Recent token routing methods such as TREAD accelerate training by allowing a random subset of tokens to bypass intermediate layers. It can be expected that carefully tuning the routing ratios over time steps or layers can achieve more significant accelerations. However, the theoretical mechanisms underlying stochastic routing remain underexplored, making the tuning of routing ratios intractable. In this paper, we first theoretically show that stochastic token routing can be interpreted as an implicit route-sensitivity regularizer. Under this view, the routing ratio determines the strength of the induced regularization: larger routing ratios introduce stronger route-induced perturbations and therefore stronger regularization pressure. This further motivates a noise-conditioned routing strategy. In diffusion training, samples at larger timesteps contain heavier noise and often provide less stable gradient signals, suggesting that they may require stronger regularization. We therefore assign larger routing ratios to higher-noise samples, allowing the routing-induced regularization strength to adapt to the noise condition. Based on this insight, we propose \emph{Noise-Aware Routing}, which partitions samples within each mini-batch into low-, middle-, and high-noise regions and assigns progressively larger routing ratios to them. This simple noise-conditioned allocation synergistically bridges computational savings with enhanced model priors. Extensive experiments on class-conditional ImageNet generation show that our method accelerates training convergence and improves generation quality. Noise-Aware Routing achieves up to \textbf{15.6}$\times$ and \textbf{20}$\times$ convergence speedups over DiT baselines—and outperforms TREAD—on ImageNet-256 and ImageNet-512, respectively, measured by the training iterations required to match our 400K-step FID. In the guided setting, our method achieves an FID of \(2.67\) with a substantially shorter training iterations without architectural changes.

Evolution strategies (ES) have recently emerged as a promising alternative to gradient-based reinforcement learning for fine-tuning large language models, but classical zeroth-order optimization theory does not systematically account for the empirical observations driving this success. We formulate the continuous-time limit of Z-score-normalized ES as a unified stochastic differential equation and identify three sequential dynamical regimes characterized by the dominant gradient contribution to the population reward variance; under a basin-local low-rank Hessian assumption, this framework systematically explains (1) the information-geometric optimality of the Z-score heuristic, (2) the dimensionality paradox of sample-efficient ES with populations far smaller than the parameter dimension, (3) the same-task rise-then-decay of the training reward, and (4) catastrophic forgetting of pre-training capabilities. Synthetic experiments and direct verification on Qwen2.5-{0.5B,3B,7B}-Instruct confirm these predictions, matching the parameter drift reported by Abdi et al. (2026) within $\sim 1.5\\%$.

Many inference-time language-model pipelines combine a cheap reward signal with an expensive verifier, such as exact answer checking in mathematical reasoning or hidden-test execution in code generation. We formalize this setting using a learning-theoretic lens as generative active search: a cost-sensitive first-positive search problem in which a policy adaptively samples candidates from an unknown distribution, observes cheap scores, and pays for verifier labels until it finds a positive example. For a fixed prompt, the generator and reward model induce two unknown objects: a distribution over reward scores and a score-conditioned success function. When these quantities are known, we characterize the distribution-aware optimal policy using a dynamic programming approach. In the realistic and practical setting where both the score distribution and success function are unknown, we propose ADAP, a shellwise adaptive generate-rank-verify algorithm that progressively increases the number of sampled responses and top-ranked verifications. Under the monotonicity assumption that higher reward scores are no less likely to pass verification, we show that ADAP achieves expected cost within a constant factor of the distribution-aware optimum. We complement this result with learning-theoretic lower bounds, based on a centered star number, showing that structural assumptions on the score--label relationship are necessary. Experiments on mathematical reasoning and competitive programming validate the predicted advantage over both fixed non-adaptive policies and difficulty-adaptive baselines.

Test-Time Adaptation (TTA) for black-box models accessible only via APIs remains a largely unexplored challenge. Existing approaches such as post-hoc output refinement offer limited adaptive capacity, while Zeroth-Order Optimization (ZOO) enables input-space adaptation but faces high query costs and optimization challenges in the unsupervised TTA setting. We introduce BETA (Black-box Efficient Test-time Adaptation), a framework that addresses these limitations by employing a lightweight, local white-box steering model to create a tractable gradient pathway. Through a prediction harmonization technique combined with consistency regularization and prompt learning-oriented filtering, BETA enables stable adaptation with no additional API calls and negligible latency beyond standard inference. BETA achieves a +7.1% accuracy gain on ViT-B/16 (ImageNet-C) and +3.4% on CLIP (ImageNet-S/R), surpassing strong white-box and gray-box methods including TENT and TPT. On a commercial API, BETA achieves comparable performance to ZOO at 250x lower cost while maintaining real-time inference speed, establishing it as a practical and efficient solution for real-world black-box TTA.

Low-Rank Adaptation (LoRA) reparameterizes a weight update as a product of two low-rank factors, but the Jacobian $J\_\mathcal{G}$ of the generator mapping the factors to the weight matrix is rank-deficient, so the factor-space preconditioner $J\_\mathcal{G}^\* \mathcal{F}\_t J\_\mathcal{G}$ induced by any ${W}$-space preconditioner $\mathcal{F}\_t$ is singular, and consequently the standard chain rule cannot be uniquely inverted to map a preconditioned ${W}$-space direction back to a factor-space update. We cast existing LoRA optimizers in a unified framework parameterized by two choices: (i) which invertible surrogate for $J\_\mathcal{G}^\* \mathcal{F}\_t J\_\mathcal{G}$ to use, and (ii) which $\mathcal{F}\_t$ on ${W}$ to use. Existing methods occupy four families along these axes: factor-space adaptive updates, block-diagonal surrogates for $J\_\mathcal{G}^\* J\_\mathcal{G}$, Frobenius-residual pseudoinverse methods, and Riemannian manifold constraint. Within this design space, a gradient-statistics-aware $\mathcal{F}\_t$ paired with a closed-form factor-space solve at $\mathcal{O}((m+n)r)$ memory remains underexplored. We propose AdaPreLoRA, which fills this gap by adopting the Adafactor diagonal Kronecker preconditioner $\mathcal{H}\_t$ on ${W}$ and selecting from the resulting factor-space solution family the element minimizing an $\mathcal{H}\_t$-weighted imbalance between the two factor contributions; by construction, the resulting factor update is the closest LoRA approximation to the preconditioned ${W}$-space direction under the $\mathcal{H}\_t$-weighted norm. Across GPT-2 (E2E), Mistral-7B and Qwen2-7B (GLUE, ARC, GSM8K), and diffusion-model personalization, AdaPreLoRA is competitive with or improves over a representative set of LoRA optimizers while keeping peak GPU memory at the LoRA optimizer level.

Fine-tuning large language models with LoRA requires choosing a rank $r$ before training starts. Existing approaches either extract rank-1 components sequentially, freezing each component's error permanently into every subsequent residual, or optimize the full low-rank factorization jointly with guarantees that describe only the joint update, not individual rank-1 directions. We present AdaPaD (Adaptive Parallel Deflation), which trains all rank-1 components simultaneously: each worker refines its component against a deflation target built from the latest estimates of all predecessors, and as those estimates improve, the targets improve too. We call this property self-correction: deflation errors converge to zero over rounds rather than persisting as fixed residuals. On top of this backbone, AdaPaD adds advance learning (private pre-training before activation) and per-module dynamic rank discovery (importance-based growth until a shared budget is exhausted), making the rank distribution an output rather than an input. We prove that every component's error decays exponentially after a warm-up period, with a generalization bound that splits into a vanishing algorithmic term and an irreducible statistical floor. Empirically, AdaPaD is competitive with adaptive-rank LoRA baselines on GLUE with DeBERTaV3-base at matched parameter budgets, and competitive with fixed-rank LoRA on Qwen3-0.6B SQuAD/SQuAD v2 while deploying an adapter that is on average 30.7% smaller.

Zeroth-order (ZO) optimization enables large-language-model fine-tuning without storing backpropagation activations, while LoRA supplies compact trainable adapters. Combining them creates a rank paradox: increasing LoRA rank improves adapter capacity, but standard two-point ZO either perturbs a rank-dependent number of coordinates or, under atomwise updates, can make the finite-difference signal unobservable. This paper shows that the bottleneck is a measurement-topology problem rather than a need for an external subspace. LoRA already decomposes into matched rank-$1$ atoms, each a complete factor-coordinate block of dimension $d_\text{out}+d_\text{in}$. Querying one atom per step keeps the stored adapter rank $r$ while removing $r$ from the single-query perturbation dimension. The naive atomwise query is still miscalibrated: if it inherits canonical LoRA scaling $\alpha/r$, the active finite-difference signal shrinks as $1/r$ and the active finite-difference signal-to-noise ratio (FD-SNR) as $1/r^2$, producing directional collapse under a fixed residual evaluation-noise floor. AR1-ZO pairs alternating rank-$1$ atom queries with topology-aware scaling $\gamma=\alpha r$, restoring rank-invariant active signal without auxiliary bases, activation hooks, curvature estimates, or extra forward queries. Theory proves atom minimality, rank-independent active query dimension, directional collapse and restoration, and the remaining rank dependence as an amortized coverage cost. Experiments on OPT and Qwen3 models validate the signal mechanism and show that AR1-ZO makes high-rank LoRA effective among matched-budget ZO methods under the standard two-forward-pass query budget.