NeurIPS 2026 论文列表

Fine-tuning large-scale foundation models across resource-constrained, distributed clients presents significant computational and communication challenges. While merging Low-Rank Adaptation (LoRA) with Federated Learning (FL) enables rank-based flexibility, the interdependence of the adaptation matrices ($A$ and $B$) often complicates optimization. In this work, we theoretically demonstrate that LoRA fine-tuning with a frozen A matrix acts as a stochastic approximation of full fine-tuning, where batch gradients are subject to a random, rank-dependent perturbation. Motivated by this insight, we propose freezing and periodically resampling $A$ matrices at the client level. This strategy decouples the $A$ and $B$ gradients, diminishes LoRA low-rank constraints and reduces both computational and uplink communication overheads by 50%, allowing for the reallocation of resources toward higher adaptation ranks. Furthermore, we show that in this frozen-$A$ regime, the adaptation rank $r$ directly governs the signal-to-noise ratio (SNR) of the perturbed gradients. Specifically, we find that the SNR increases linearly with $r$, implying that clients’ learning rates must be dynamically scaled with their adaptation ranks for higher utility. Building on these foundations, we introduce Frank-LoRA, a federated rank-aware fine-tuning strategy that utilizes frozen, resampled $A$ matrices and rank-dependent learning rates for clients. We also prove the convergence of the algorithm in FL settings with heterogeneous clients resources. Our experiments across benchmark datasets demonstrate that Frank-LoRA consistently outperforms state-of-the-art baselines with half the overhead on clients.

Existing approaches to controllable generation typically rely on fine-tuning, auxiliary networks, or test-time search. We show that flow matching admits a different control interface: adaptation through examples. For deterministic interpolants, the velocity field is solely governed by a conditional endpoint mean; shifting this mean shifts the flow itself. This yields a simple principle for controllable generation: steer a pretrained model by changing the reference set it follows. We instantiate this idea in two forms. Reference-Mean Guidance is training-free: it computes a closed-form endpoint-mean correction from a reference bank and applies it to a frozen FLUX.2-klein (4B) model, enabling control of color, identity, style, and structure while keeping the prompt, seed, and weights fixed. Semi-Parametric Guidance amortizes the same idea through an explicit mean anchor and learned residual refiner, matching unconditional DiT-B/4 quality on AFHQv2 while allowing the reference set to be swapped at inference time. These results point to a broader direction: generative models that adapt through data, not parameter updates.

k-means has historically been positioned primarily as an offline processing primitive, typically used for dataset organization or embedding preprocessing rather than as a first-class component in online systems. In this work, we revisit this classical algorithm under the lens of modern AI system design and enable k-means as an online primitive. We point out that existing GPU implementations of k-means remain fundamentally bottlenecked by low-level system constraints rather than theoretical algorithmic complexity. Specifically, the assignment stage suffers from a severe IO bottleneck due to the massive explicit materialization of the N × K distance matrix in High Bandwidth Memory (HBM). Simultaneously, the centroid update stage is heavily penalized by hardware-level atomic write contention caused by irregular, scatter-style token aggregations. To bridge this performance gap, we propose Flash-KMeans, an IO-aware and contention-free k-means implementation for modern GPU workloads. Flash-KMeans introduces two core kernel-level innovations: (1) FlashAssign, which fuses distance computation with an online argmin to completely bypass intermediate memory materialization; (2) Sort-Inverse Update, which explicitly constructs an inverse mapping to transform high-contention atomic scatters into high-bandwidth, segment-level localized reductions. Furthermore, we integrate algorithm-system co-designs, including chunked stream overlap and cache-aware compile heuristic, to ensure practical deployability. Extensive evaluations on NVIDIA H200 GPUs demonstrate that Flash-KMeans achieves up to 17.9× end-to-end speedup over best baselines. At the kernel level, FlashAssign and Sort-Inverse Update deliver up to 21.2× and 6.3× speedups. By systematically restructuring execution around underlying hardware constraints, Flash-KMeans delivers mathematically exact, scalable, and highly deployable acceleration across diverse AI workloads.

Time series forecasting is an actively researched problem with diverse applications. Recent work has shown that simple linear models can compete with complex deep learning architectures in terms of forecasting accuracy. This has led to the design of several small models, yet the reasons behind their effectiveness remain insufficiently understood. We address this gap through a theoretical framework grounded in reduced-rank regression and kernel analysis. We show both analytically and empirically that the intrinsic complexity of many TSF tasks is lower than commonly assumed and that the performance of linear models is linked to their alignment with low-rank data structures. Building on these insights, we propose a new design for low-rank neural networks that incur lower computational cost than linear models. Specifically, we introduce a simple filter bank architecture that provides a principled way to control model complexity through tunable and interpretable hyperparameters that directly correspond to the rank of the forecasting model. The proposed architecture also serves as a versatile building block for constructing deeper networks. Our evaluation shows that filter bank architectures achieve state-of-the-art results on most long-term TSF benchmarks, with lower computational cost than recently proposed solutions.

Multimodal foundation models integrate heterogeneous signals across modalities, yet it remains poorly understood how their predictions depend on specific internal feature groups and whether such reliance can be deliberately controlled. Existing studies of shortcut and spurious behavior largely rely on post hoc analyses or feature removal, offering limited insight into whether reliance can be modulated without altering task semantics. We introduce FiLoRA (Focus-and-Ignore LoRA), an instruction-conditioned, parameter-efficient adaptation framework that enables explicit control over internal feature reliance while keeping the predictive objective fixed. FiLoRA decomposes adaptation into feature group-aligned LoRA modules and applies instruction-conditioned gating, allowing natural language instructions to act as computation-level control signals rather than task redefinitions. Across text-image and audio-visual benchmarks, we show that instruction-conditioned gating induces consistent and causal shifts in internal computation, selectively amplifying or suppressing core and spurious feature groups without modifying the label space or training objective. Further analyses demonstrate that FiLoRA yields improved robustness under spurious feature interventions, revealing a principled mechanism to regulate reliance beyond correlation-driven learning.

Low-rank adaptation (LoRA) enables communication-efficient federated fine-tuning of large language models, but standard FedLoRA averages the factors $A$ and $B$ separately even though the model update is determined by their product $\Delta W=sBA$. Under task heterogeneity, factor-wise aggregation can further dilute directions that are locally useful but not strongly shared across clients; we call this phenomenon \emph{directional dilution}. We analyze this effect through the singular directions of client LoRA updates and find that low-strength directions are most vulnerable to dilution. Their repeated owner-specific recovery suggests client-specific adaptation signals rather than random noise. We propose \textsc{FedReCall}, a client-private plug-in that captures observable dilution gaps, selects reliable diluted directions into a private frozen cache, and recalls them through a low-rank bypass guarded by a single-batch loss probe when the client next participates. As a plug-in, \textsc{FedReCall} can be seamlessly integrated into existing FedLoRA methods without changing their aggregation rules, communication payloads, or trainable-parameter sets, consistently improving personalized adaptation and improving global evaluation metrics on most aggregators.

Low-Rank Adaptation (LoRA) is a popular parameter-efficient fine-tuning (PEFT) method for fine-tuning of large language models (LLMs), especially in federated learning, due to its strong performance and communication efficiency. In practice, LoRA is applied to the query and value projections of transformer layers without considering the distinct roles of these components. In this work, we analyze the impact of these projections in federated fine-tuning of LLMs using LoRA and observe that the value projection converges significantly faster than the query projection. Motivated by this finding, we propose FedLoVA, a federated fine-tuning framework that trains LoRA adapters on both query and value projections locally while sharing only the value projection with the server. This approach reduces communication overhead while improving performance and convergence efficiency. Experiments on natural language understanding, multilingual sentiment analysis, and generation tasks show that FedLoVA reduces the communication cost by up to $7.8\times$ compared to existing federated LLM fine-tuning baselines while achieving comparable performance.

Federated adaptation of large vision-language models (VLMs) is appealing for privacy-sensitive applications but must contend with client data heterogeneity, strict communication budgets, and on-device inference constraints. Recent methods predominantly introduce additive adaptation interfaces, such as learnable prompts or adapter modules, which attach extra trainable components to the frozen backbone, inflating both parameter counts and inference latency. We challenge this additive paradigm and ask: can robust federated VLM adaptation be achieved through a compact intrinsic interface alone? We answer this question with \textbf{FedIBS} (Federated Intrinsic Bias Selection), a module-free framework that fine-tunes only the biases in feed-forward network projections. FedIBS operates on existing backbone parameters and introduces zero additional inference cost. To make such compact intrinsic adaptation viable under heterogeneous clients, we further propose a Fisher-guided progressive masking mechanism that disentangles shareable and personalized bias coordinates. Each client retains personalized coordinates locally and communicates only the shareable subset for aggregation, enabling sparse communication while preserving client-specific adaptation. Extensive experiments on diverse datasets and heterogeneity settings demonstrate that FedIBS consistently outperforms recent federated prompt- and adapter-based baselines in both personalization and generalization, all while achieving substantially lower communication cost.

3D semantic occupancy prediction provides dense geometric and semantic scene representations for autonomous driving, where both high accuracy and low response latency are crucial for safe downstream forecasting and planning. Existing methods usually exhibit different accuracy-latency characteristics. Performance-oriented models provide stronger geometric and semantic predictions, while deployment-friendly models provide faster responses. This motivates EvoOcc, a fast-slow evolutionary occupancy prediction framework that jointly exploits their complementary strengths to improve the accuracy-latency balance without changing the upstream architectures. EvoOcc models evolutionary occupancy prediction as a continuous time controlled dynamical system, enabling flexible adaptation to irregular prediction arrival of both fast and slow system. The framework further decomposes the evolution process into dual state evolution where ego state evolution for the deterministic coordinate shifts and scene state evolution for the genuine geometric and semantic scene changes. Extensive experiments on three different fast-slow system pairs and irregular prediction arrivals show that EvoOcc consistently improves over fast system baselines while maintaining much lower latency than the slow system, indicating a favorable accuracy-latency balance and stable behavior under practical inference situations.

Vital sign measurement using cameras presents opportunities for comfortable, ubiquitous health monitoring. Remote photoplethysmography (rPPG), a foundational technology, enables cardiac measurement through minute changes in light reflected from the skin. However, practical deployment is limited by the computational constraints of performing analysis on front-end devices and the accuracy degradation of transmitting data through compressive channels that reduce signal quality. We propose a memory efficient rPPG algorithm - FacePhys - built on temporal-spatial state space duality, which resolves the trilemma of model scalability, cross-dataset generalization, and real-time operation. Leveraging a transferable heart state, FacePhys captures subtle periodic variations across video frames while maintaining a minimal computational overhead, enabling training on extended video sequences and supporting low-latency inference. FacePhys establishes a new state-of-the-art, with a substantial 49% reduction in error. Our solution enables real-time inference with a memory footprint of 3.6 MB and per-frame latency of 9.46 ms -- surpassing existing methods by 83% to 99%. The training code, a real-time inference demo for mobile browsers, and the demonstration video can be found in the supplementary materials.

Low-Rank Adaptation (LoRA) is a popular parameter-efficient fine-tuning (PEFT) method that adapts the large model with few trainable parameters. Federated LoRA extends LoRA to federated learning (FL), enabling clients to collaboratively fine-tune a shared model without sharing their raw data by exchanging LoRA parameters. However, due to distributed data heterogeneity, local LoRA updates are highly inconsistent across clients, hindering robust global generalization. Moreover, common direct aggregation introduces aggregated bias, yielding noisy updates and further harming the global generalization. Existing bias-mitigation approaches often rely on symmetrically alternating LoRA modules, overlooking matrix distinct characteristics during training. Therefore, we propose FLoRA-Chef, an asymmetric Federated LoRA that decides when and how to aggregate LoRA components. FLFire adaptively selects the next aggregation target by tracking cross-clientupdate dispersion. FLSauce applies matrix-distinct reweighting to control diversity and prioritize stability. Experiments on various scenarios demonstrate consistent improvements, yielding an average gain of 6.94% over counterparts, validating the benefit of adaptive asymmetric aggregation for federated LoRA.

Low-Rank Adaptation (LoRA) is a popular parameter-efficient fine-tuning method for Large Language Models (LLMs), and practical systems usually need to serve many concurrent LoRA adapters to process different tasks. This is challenging due to the inherent tension between \textit{fairness} and \textit{efficiency}, i.e., achieving fairness requires frequent switching among the adapters such that they have similar service quality, while good efficiency requires to minimize adapter switching to reduce memory movement overheads. Existing schedulers can only achieve a fixed trade-off between fairness and efficiency, and their trade-offs are often suboptimal. To tackle this problem, we propose \textbf{F}airness-aware \textbf{E}fficient \textbf{L}oRA \textbf{P}riority \textbf{S}cheduling (FELPS), which explicitly decomposes fairness and efficiency considerations. In particular, for fairness, FELPS adapts the classic proportional fairness and ensures that the amount of service received by each adapter is proportional to its workload. For efficiency, FELPS prioritizes adapters that are already loaded to GPU memory. To conduct scheduling, FELPS combines to the fairness and efficiency terms with an adjustable weight to achieve arbitrary trade-offs between fairness and efficiency. Evaluations on real workloads show that FELPS achieves a superior trade-off between fairness and efficiency than all baselines.

Large language model training with sequence-parallel data-parallel (SP-DP) is bottlenecked by the interplay between intra-step activation redistribution and inter-step gradient synchronization. Existing infrequent-communication methods like FedAvg keep states local or reset them lack convergence guarantees and are unstable under long-context sequence-parallel regimes where each device holds only a partial activation shard; Local Adam synchronizes all states jointly and is convergent but triples the collective payload, and its uniform synchronization period cannot distinguish the semantically mandatory intra-step All-to-All from the temporally deferrable inter-step AllReduce. We propose FAUST (Federated Asynchronous Update with Staggered Timescales), a compiler-orchestrated training system that composes compile-time sequence-parallel graph rewriting with tiered runtime synchronization, assigning independent averaging periods to parameters ($K_x$), first moments ($K_u = 3K_x$), and second moments ($K_v = 6K_x$) according to their impulse-response half-lives, while preserving convergence under the orthogonal collective constraint that intra-step All-to-All must retire before inter-step AllReduce begins. Our analysis shows that while first-moment drift dominates the convergence rate in-distribution, high-fidelity convergence guarantees require at least periodic synchronization of the second moment. Experiments on language models up to 1.3B parameters show that FAUST incurs $303{\times}$ less wall-clock communication overhead than standard DDP and $1.78{\times}$ less than the previous state-of-the-art DES-LOC, while achieving perplexity within $4.6\%$ of the fully-synchronized baseline in $27\%$ less wall-clock time. On bandwidth-constrained links, FAUST delivers $48$--$303{\times}$ speedup over DDP.

Feed-forward 3D reconstruction from unconstrained multi-view images has recently emerged as a dominant paradigm in 3D computer vision. Various foundational reconstruction models have been proposed with different architectures and reconstruction paradigms. Despite being trained on large-scale perspective datasets with substantial computational resources, these models generally fail when deployed on images whose camera models deviate from the training distribution. Applying such models to downstream applications with arbitrary, unknown, and potentially mixed camera types remains challenging. To maximally leverage the strengths of existing reconstruction models with minimal retraining efforts, we propose a low-rank adaptation framework that efficiently extends perspective-pretrained models to fisheye and panoramic imagery. Extensive experiments demonstrate that, with only 0.1% additional parameters and one day of training, our method can adapt a foundational 3D reconstruction model to non-pinhole cameras while fully preserving its original performance on perspective inputs.

Standard In-Context Reinforcement Learning (ICRL) typically assumes stationary dynamics within an interaction history, a restriction that limits its real-world applicability. We formalize Lifelong ICRL, a more realistic paradigm in which agents must continuously adapt to diverse non-stationary dynamics, ranging from abrupt random shifts to structured temporal drifts within a single lifetime. To probe the adaptation limits of this setting, we conduct a large-scale empirical study across non-stationary environments that span discrete symbolic reasoning and continuous physics-based control. We systematically investigate three core dimensions governing adaptation: (i) Training Regimes, comparing the generalization boundaries of domain randomization, stationary training, non-stationary training, and their staged combinatorial strategies; (ii) Model Architectures, evaluating modern sequence models, including Linear Attention and hybrid designs; and (iii) Impact of Scale, analyzing the relative contributions of interaction scale and model capacity. Our results reveal how these factors jointly shape robust in-context adaptation under unknown dynamics, yielding actionable insights for the design of generalist agents.

We derive a tight generalization bound for deterministic neural networks based on their last hidden layer representations. By evaluating representation quality with a parameter-free $k$-nearest-neighbor ($k$-NN) classifier, our certificate avoids traditional parameter-space complexity measures, KL divergences to a posterior, weight norms, and compression. The bound is tight: trained from scratch, it reaches an 8.0 percentage point gap to test error on CIFAR-10 (ResNet-18) and yields the first non-vacuous certificates we are aware of on CIFAR-100; with frozen DINOv2 representations, it certifies at 1.4\% on CIFAR-10 and 23.1\% on 1,000-class ImageNet. It decomposes additively into three empirical, independently measurable quantities: local class geometry ($k$-NN decoder error), decoder-network agreement, and training stability. This allows the bound to act as a diagnostic tool: we demonstrate that actively destroying local class geometry eliminates generalization while leaving dataset memorization intact, identifying class geometry as a causally relevant mechanism for generalization. Finally, we show that learned representations are significantly less sensitive to weight noise than linear readouts, suggesting why parameter-space certificates often struggle to remain tight.

Speculative decoding accelerates LLM inference but suffers from performance degradation when target models are fine-tuned for specific domains. A naive solution is to retrain draft models for every target model, which is costly and inefficient. To address this, we introduce a practical parameter- and data-efficient framework named Efficient Draft Adaptation, abbreviated as EDA, for efficiently adapting draft models. EDA introduces three innovations: (1) a decoupled architecture that utilizes shared and private components to model the shared and target-specific output distributions separately, enabling parameter-efficient adaptation by updating only the lightweight private component; (2) a data regeneration strategy that utilizes the fine-tuned target model to regenerate training data, thereby improving the alignment between training and speculative decoding, leading to higher average acceptance length; (3) a sample selection mechanism that prioritizes high-value data for efficient adaptation. Our experiments show that EDA effectively restores speculative performance on fine-tuned models, achieving superior average acceptance lengths with significantly reduced training costs compared to full retraining. Code will be released upon acceptance.

We study the classical problem of linear regression under heteroskedastic noise, where observations arise from multiple sources with unknown and potentially unequal variances. Specifically, we consider a setting with $n$ sources, each providing a linear measurement of an unknown $d$-dimensional parameter vector $\beta^*$, where each source is associated with an unknown noise variance. We focus on the classic "subset-of-signals'' model, where one assumes that $m$ of the sources have a relatively low noise, formally, with variance of at most 1. Our goal is to accurately estimate $\beta^\*$ despite the presence of high-variance sources. We show that if $m$ is sufficiently large, it is possible to recover $\beta^*$ with sub-constant error. Specifically, we prove a recovery error bound of $\tilde{O} \left(\frac{\sqrt{nd}}{m}\right)$, significantly improving upon previously known results for this setting.

Reinforcement-learning policies for robot locomotion can fail at deployment when operating conditions shift (e.g., rough or slippery terrain, payload changes, sensing degradation), and retraining or manual recalibration is often infeasible under on-device latency and memory constraints. We study test-time adaptation (TTA) for robotic control: adapting a pretrained policy online and self-supervisedly during execution, without access to rewards. Our starting point is a lightweight baseline that updates only the actor using a frozen pretrained critic as a deployment-time performance surrogate. Building on this, we introduce TEMPO, a test-time entropy-regularized, memory-guided policy optimization method that stabilizes critic-driven updates under severe shift. TEMPO combines (i) an entropy-promoting regularizer on the critic’s predictive distribution to mitigate overconfident extrapolation, and (ii) a score-based memory, inspired by prioritized replay, that prioritizes low-value (“hard”) observations and applies importance sampling to focus limited computation on the most informative deployment conditions. Across Go1, Go2, and G1 in MuJoCo Playground under four representative shifts and multiple architectures, TEMPO yields consistent gains, reaching up to 80\% improvement in a single-device setting. We further validate on a physical Unitree Go2 with an added calf payload, where fully on-device adaptation improves performance over the non-adapted baseline across real-world episodes.