ICML 2026 Papers

Long input sequences are central to in-context learning, document understanding, and multi-step reasoning in Large Language Models (LLMs). However, the quadratic attention cost of Transformers makes inference memory-intensive and slow. While context distillation (CD) can transfer information into model parameters, per-prompt distillation is impractical due to training costs and latency. To address these limitations, we propose Doc-to-LoRA (D2L), a lightweight hypernetwork that meta-learns to perform approximate CD within a single forward pass. Given an unseen prompt, D2L generates a LoRA adapter for a target LLM, enabling subsequent queries to be answered without re-consuming the original context, reducing latency and KV-cache memory consumption during target LLM inference. On a long-context needle-in-a-haystack task, D2L successfully learns to map contexts into adapters that store the needle information, achieving near-perfect zero-shot accuracy at sequence lengths exceeding the target LLM’s native context window by more than 4x. On real-world QA datasets with limited compute, D2L outperforms standard CD while significantly reducing peak memory consumption and update latency. We envision that D2L can facilitate rapid adaptation of LLMs, opening up the possibility of frequent knowledge updates and personalized chat behavior. Code and checkpoints are available at https://github.com/SakanaAI/doc-to-lora.

As pre-trained models evolve rapidly, transferring fine-tuning knowledge to updated models without retraining has become a critical challenge. Most existing methods reuse parameter updates, yet the same dataset can induce substantially different updates across base models due to mismatched local loss landscapes, making such transfer unstable. We instead adopt a Bayesian-updating perspective: a base model defines a prior, while fine-tuning contributes a task-update factor that is prior-agnostic, thereby making it feasible to reuse the update across base models. Specifically, we formalize a reusable task-update factor by requiring *invariance across base models* and *a fixed-dimensional parameterization*. Our main theoretical result shows that such reusable factors exist when the variational family is a half-space, and it is already maximal among convex families. In particular, an ideal regime arises when the priors and their Bayesian posteriors remain within a shared exponential family, as it always admits a reusable update factor. Building on this existence, we propose ***B**ayesian Task Update **Transfer*** (*BTransfer*), which extracts a reusable task-update factor from a single fine-tuning run and applies it to a new prior. For deep networks, we implement *BTransfer* with a ``lift–transfer–return'' pipeline: 1) lift model parameters to distributions; 2) transfer the extracted task-update factor in the exponential family distributions; and 3) return the updated posterior distribution to parameter space. Extensive experiments demonstrate that our approach effectively reuses fine-tuning knowledge across models without post-training.

Robust signal enhancement at low SNR is fundamentally challenging because noise becomes strongly entangled with the signal and corrupts local time–frequency (TF) evidence. In this regime, fixed resolution short time Fourier transform (STFT) enhancement with purely data driven convolutional biases can become overconfident in unreliable TF regions, causing unstable suppression or residual artifacts. We propose FracKGMF, which couples Fractional Distance Decay Convolution (FracConv) with Knowledge Guided Multi-view Fusion (KGMF) for expressive TF modeling and reliability aware decisions under heavy corruption. FracConv introduces a lightweight fractional distance decay family that reshapes local interactions into long tailed receptive patterns, enabling aggregation of weak but globally consistent cues when per-bin observations are ambiguous. KGMF uses a wiener inspired reliability prior to calibrate multi-view fusion and reduce excessive suppression in uncertain regions. Experiments on speech and EM benchmarks show consistent improvements over state-of-the-art baselines, with particularly large gains under extremely low SNR, including a 33 dB average improvement on EM signals at -20 dB.

Time-series analysis is critical in real-world applications, yet the explosion of time-series data imposes severe burdens on storage and computational resources. Recently, dataset condensation has emerged as a promising data-centric solution by synthesizing compact yet informative datasets to replace large-scale raw data. However, existing methods are largely vision-centric, failing to capture unique temporal properties of time series, or task-specific, tightly coupling the condensed data to a particular downstream objective. As a result, these approaches suffer from feature mismatch and fail to generalize across diverse time-series tasks. To bridge this gap, we propose UniTSC, the first unified dataset condensation framework for general time-series analysis. UniTSC employs a multi-view hybrid encoder to capture task-invariant representations across temporal, spectral, and topological perspectives. Building upon this representation, we design a tri-space alignment paradigm that jointly aligns optimization trajectories, power spectral densities, and multivariate dependency structures, enabling comprehensive information preservation under extreme compression. Extensive experiments show that UniTSC retains up to 97.9\% of downstream performance using as little as 0.01\% of the original training data, within our experimental settings tied to standard sequence lengths, revealing that a single batch-equivalent budget ($\textless$ 128 samples) is sufficient to capture the essential dynamics of complex time-series data.

Sliced optimal transport (SOT), or sliced Wasserstein (SW) distance, is widely recognized for its statistical and computational scalability. In this work, we further enhance computational scalability by proposing the first method for estimating SW from sample streams, called streaming sliced Wasserstein (Stream-SW). To define Stream-SW, we first introduce a streaming estimator of the one-dimensional Wasserstein distance (1DW). Since the 1DW has a closed-form expression, given by the integral of the absolute difference between the quantile functions of the compared distributions, we leverage quantile approximation techniques for sample streams to define a streaming 1DW estimator. By applying the streaming 1DW to all projections, we obtain Stream-SW. The key advantage of Stream-SW is its low memory complexity while providing theoretical guarantees on the approximation error. We demonstrate that Stream-SW achieves a more accurate approximation of SW than random subsampling, with lower memory consumption, when comparing Gaussian distributions and mixtures of Gaussians from streaming samples. Additionally, we conduct experiments on point cloud classification, point cloud gradient flows, and streaming change point detection to further highlight the favorable performance of the proposed Stream-SW.

Unsupervised video domain adaptation (UVDA) is a practical but under-explored problem. In this paper, we propose a frustratingly easy UVDA method, called \emph{MetaTrans}. Specifically, \emph{MetaTrans} adopts a concise learning objective that contains only two fundamental loss terms. Despite the simplicity of the learning objective, \emph{MetaTrans} embodies an advanced UVDA idea, that is, handling the spatial and temporal divergence of cross-domain videos separately, through a subtle model architecture design. By implementing a temporal-static subtraction module, \emph{MetaTrans} effectively removes spatial and temporal divergence. Extensive empirical evaluations, particularly on various cross-domain action recognition tasks, show substantial absolute adaptation performance enhancement and significantly superior relative performance gain compared with state-of-the-art UVDA baselines.

Emerging LLM workloads demand extreme memory agility. However, state-of-the-art inference systems such as vLLM rely on software-defined paging, which sacrifices the contiguous tensor abstraction. This rigid interface exposes fragmentation complexity to developers, imposing a severe engineering burden that stifles algorithmic innovation. We introduce CONTINUUM, a tensor memory virtualization subsystem implemented as a PyTorch extension. By bypassing serialized OS bottlenecks through a lightweight GPU driver extension, CONTINUUM significantly reduces mapping costs by orders of magnitude, from milliseconds to microseconds. Built atop this low-latency API, CONTINUUM provides Elastic Tensor, a set of flexible tensor operations that natively support complex memory dynamics and zero-copy topological aliasing. Evaluations demonstrate that CONTINUUM achieves significantly higher throughput across diverse dynamic scenarios, effectively lowering the barrier to implementing next-generation LLM applications.

In this paper, we present a novel federated independence testing (FedIT) method that addresses both theoretical and practical challenges arising from client heterogeneity. We begin by revisiting existing federated independence testing methods and showing why they fail to provide valid guarantees or maintain statistical power under data distributional shift across clients. Building on this analysis, we develop a copula-based marginal alignment technique together with a stacking-based aggregation strategy that amplifies intra-client dependence while mitigating inter-client variation, resulting in a theoretically sound and powerful global test. For practicality, we further accelerate the aggregation step and incorporate a privacy-preserving mechanism. On the theoretical side, we prove both the correctness of our method and the validity of the test. Empirically, we conduct extensive experiments on both synthetic and real-world datasets, which demonstrate the superiority of our solution over existing methods.

Existing symbolic regression approaches primarily focus on learning explicit input-output mappings, often neglecting relational structures among data instances. This paper introduces Contrastive Symbolic Regression (CSR), a feature-construction-based symbolic regression approach that integrates evolutionary feature construction with contrastive learning to shape a representation space where geometric proximity reflects similarity in the target space. CSR employs a contrastive objective that optimizes a linear transformation of constructed features, with a closed-form solution for aligning the feature space with the target space. The constructed features are applied to K-nearest neighbor regression, where we propose an efficient leave-one-out cross-validation (LOOCV) method that addresses standard LOOCV's computational expense and adaptively selects the neighborhood size, along with a linear-rank weighted K-nearest neighbor variant for faithful assessment of representation quality during evolution. A determinantal point process-based ensemble selection mechanism further enhances robustness by jointly considering model quality and diversity. Extensive experiments on 58 real-world regression datasets demonstrate that CSR consistently surpasses both traditional symbolic regression and modern machine learning counterparts, highlighting CSR as a promising direction for interpretable and effective regression modeling.

Estimating probability density and its score from samples remains a core problem in generative modeling, Bayesian inference, and kinetic theory. Existing methods are bifurcated: classical kernel density estimators (KDE) generalize across distributions but suffer from the curse of dimensionality, while modern neural score models achieve high precision but require retraining for every target distribution. We introduce DiScoFormer (Density and Score Transformer), a ``train-once, infer-anywhere" equivariant Transformer that maps i.i.d. samples to both density values and score vectors, generalizing across distributions and sample sizes. Analytically, we prove that self-attention can recover normalized KDE, establishing it as a functional generalization of kernel methods; empirically, individual attention heads learn multi-scale, kernel-like behaviors. The model converges faster and achieves higher precision than KDE for density estimation, and provides a high-fidelity plug-in score oracle for score-debiased KDE, Fisher information computation, and Fokker-Planck-type PDEs.

Stochastic Frank-Wolfe is a classical optimization method for solving constrained optimization problems. On the other hand, recent optimizers such as Lion and Muon have gained quite significant popularity in deep learning. In this work, building on recent initiatives, we provide a unifying perspective by interpreting these seemingly disparate methods through the lens of Stochastic Frank-Wolfe. Specifically, we show that Lion and Muon with weight decay can be viewed as special instances of a Stochastic Frank-Wolfe, and we establish their convergence guarantees in terms of the Frank-Wolfe gap, a standard stationarity measure in non-convex optimization for Frank-Wolfe methods. We further find that convergence to this gap implies convergence to a KKT point of the original problem under a norm constraint for Lion and Muon. Moreover, motivated by recent empirical findings that stochastic gradients in modern machine learning tasks often exhibit heavy-tailed distributions, we extend Stochastic Frank-Wolfe to settings with heavy-tailed noise by developing two robust variants with strong theoretical guarantees that hold for general compact convex sets without the need for a large batch size, filling the gap in the literature on Stochastic Frank-Wolfe for non-convex optimization. Our contributions in the later part of this work, in turn, yield new variants of Lion and Muon, that better accommodate heavy-tailed gradient noise, thereby enhancing their practical scope.

In many applications, practical constraints prevent measuring covariates and outcomes on the same units, resulting in unpaired data. We study the problem of estimating causal effects under hidden confounding in the following unpaired data setting: we observe some covariates $X$ and an outcome $Y$ under different experimental conditions (environments) but do not observe them jointly -- we either observe $X$ or $Y$. Under appropriate regularity conditions, the problem can be cast as an instrumental variable (IV) regression with the environment acting as a (possibly high-dimensional) instrument. When there are many environments but only a few observations per environment, standard two-sample IV estimators fail to be consistent. We propose a GMM-type estimator based on cross-fold sample splitting of the instrument–covariate sample that also applies in standard IV settings. We prove that it is consistent as the number of environments grows but the sample size per environment remains constant. We further extend the method to sparse causal effects via $\ell_1$-regularized estimation and post-selection refitting.

Unsupervised multi-source domain adaptation (UMDA) leverages labeled data from multiple source domains to generalize to an unlabeled target. While federated UMDA addresses privacy by avoiding raw data sharing, existing methods scale poorly as the number of sources increases, often suffering from high computational overhead or training instability. We propose GALA, a scalable and robust federated UMDA framework designed for high-diversity settings. GALA achieves scalability by coupling a novel inter-group discrepancy minimization objective that approximates pairwise alignment with linear complexity alongside a temperature-controlled, centroid-based weighting strategy for dynamic source prioritization. These components enable stable, parallelizable training across many heterogeneous sources, addressing a critical scalability bottleneck that remains largely unaddressed in current literature. To evaluate performance in high-diversity scenarios, we introduce Digit-18, a new benchmark comprising 18 datasets with varied synthetic and real-world domain shifts. Extensive experiments demonstrate that GALA achieves state-of-the-art results on standard benchmarks and significantly outperforms prior methods in large-scale settings where others either fail to converge or become computationally infeasible.

In many modern machine learning pipelines, abundant pretrained representations serve as noisy proxy covariates, while task-specific labels remain scarce. We study semi-supervised regression in this setting, and propose a simple two stage estimator that learns kernel eigenfeatures from all proxy covariates and fits a ridge predictor on labeled data. We derive finite sample bounds showing that fast labeled sample rates are recovered when proxy perturbation is controlled and unlabeled proxy covariates are sufficiently abundant. We also show that distribution regression is a direct special case, with analogous guarantees when the finite bag size is large enough. Experiments show consistent gains over supervised and semi-supervised baselines, especially in low label regimes.

Difficulty replicating baselines, high computational costs, and required domain expertise create persistent barriers to clinical AI research. To address these challenges, we introduce PyHealth 2.0, an enhanced clinical deep learning toolkit that enables predictive modeling in as few as 7 lines of code. PyHealth 2.0 offers three key contributions: (1) a comprehensive toolkit addressing reproducibility and compatibility challenges by unifying 15+ datasets, 20+ clinical tasks, 25+ models, 5+ interpretability methods, and uncertainty quantification including conformal prediction within a single framework that supports diverse clinical data modalities—signals, imaging, and electronic health records—with translation of 5+ medical coding standards; (2) accessibility-focused design accommodating multimodal data and diverse computational resources with up to 39× faster processing and 20× lower memory usage, enabling work from 16GB laptops to production systems; and (3) an active open-source community of 400+ members lowering domain expertise barriers through extensive documentation, reproducible research contributions, and collaborations with academic health systems and industry partners, including multi-language support via RHealth. PyHealth 2.0 establishes an open-source foundation and community advancing accessible, reproducible healthcare AI.

Tabular anomaly detection (TAD) aims to identify samples that deviate from the majority in tabular data and is critical in many real-world applications. However, existing methods follow a ``one model for one dataset (OFO)'' paradigm, which relies on dataset-specific training and thus incurs high computational cost and yields limited generalization to unseen domains. To address these limitations, we propose OFA-TAD, a generalist one-for-all (OFA) TAD framework that only requires one-time training on multiple source datasets and can generalize to unseen datasets from diverse domains on-the-fly. To realize one-for-all tabular anomaly detection, OFA-TAD extracts neighbor-distance patterns as transferable cues, and introduces multi-view neighbor-distance representations from multiple transformation-induced metric spaces to mitigate the transformation sensitivity of distance profiles. To adaptively combine multi-view distance evidence, a Mixture-of-Experts (MoE) scoring network is employed for view-specific anomaly scoring and entropy-regularized gated fusion, with a multi-strategy anomaly synthesis mechanism to support training under the one-class constraint. Extensive experiments on 34 datasets from 14 domains demonstrate that OFA-TAD achieves superior anomaly detection performance and strong cross-domain generalizability under the strict OFA setting. The source code is available at https://github.com/Shiy-Li/OFA-TAD.

Optimal transport (OT) finds a least-cost transport plan between two probability distributions. Unlike standard OT, which infers unstructured pointwise mappings, low-rank optimal transport explicitly constrains the rank of the transport plan to infer latent structure. This improves statistical robustness, yields sharper parametric rates for estimating Wasserstein distances, and generalizes $K$-means to co-clustering. However, these advantages come at the cost of a non-convex and NP-hard optimization problem. We introduce Transport Clustering, an algorithm to compute a low-rank OT plan that reduces low-rank OT to a clustering problem on correspondences obtained from a full-rank transport registration step. We prove that this reduction yields polynomial-time, constant-factor approximation algorithms for low-rank OT: specifically, a $(1+\gamma)$ approximation for negative-type metrics, a $(1+\gamma+\sqrt{2\gamma}\,)$ approximation for kernel costs, and a $(1+\gamma+\rho)$ approximation for general metrics satisfying the triangle inequality. Here, $\gamma \in [0,1]$ is the cost ratio of the optimal full-rank to low-rank solutions, and $\rho\in[0,1]$ is an asymmetry coefficient on the cluster variances. Numerically, Transport Clustering outperforms existing solvers on synthetic benchmarks and large-scale datasets.

Confidence calibration for classification models is vital in safety-critical decision-making scenarios and has received extensive attention. General confidence calibration methods assume training and test data are independent and identically distributed ($i.i.d.$), limiting their effectiveness under covariate shifts. Previous calibration methods under covariate shift struggle with class-wise or canonical calibrations and often rely on unstable importance weighting when density ratios are large or unbounded. Given the above limitations, this paper rethinks confidence calibration under covariate shifts. First, we derive a necessary and sufficient condition for confidence calibration under covariate shifts, named Expectation consistency condition, which reveals covariate shifts do not necessarily lead to uncalibrated confidence and provides a weaker condition for confidence calibration than global covariate distribution alignment. Then, utilizing Expectation consistency condition, this paper proposes an unsupervised domain adaptation loss to calibrate confidence of the target domain, named Expectation consistency loss (ECL), which is compatible with canonical calibration, class-wise calibration, and top-label calibration. Third, we prove that computing ECL loss has the same sample complexity as Expected Calibration Error (ECE) and provide a theoretically grounded mini-batch trainable scheme for ECL loss. Finally, we validate the effectiveness of our method on both simulated and real-world covariate shift datasets.

Data valuation quantifies the intrinsic quality of individual samples to enable principled data curation, quality control, and robust learning. For time series in critical domains such as healthcare, finance, and industrial monitoring, effective valuation methods are essential yet fundamentally lacking. Existing approaches are either model-dependent, limiting their generalizability, or designed for i.i.d. data and thus fail to capture temporal dependencies, multi-scale patterns, and non-stationary dynamics inherent to sequential data. We introduce TimeLAVA, a learning-agnostic framework that values temporal segments by their marginal contribution to minimizing distributional discrepancy between evaluated and reference data. At its core is a novel Selective Wavelet-based Wasserstein ($\mathcal{W}_\text{SW}$) discrepancy combining multi-scale wavelet transforms for temporal localization with unbalanced optimal transport for robustness to distributional shifts. Segment values are efficiently computed via sensitivity analysis without requiring model training and aggregated into point-wise scores. We provide theoretical guarantees linking valuation to model-agnostic generalization and prove bounded sensitivity to outlier contamination. Extensive experiments across anomaly detection, data pruning, and label noise detection demonstrate that TimeLAVA produces significantly more informative value scores than existing methods on diverse real-world datasets.