ICML 2026 论文列表

By processing electronic health records (EHRs) as natural language sequences, large language models (LLMs) have shown potential in clinical prediction tasks such as mortality prediction and phenotyping. However, longitudinal or highly frequent EHRs often yield excessively long token sequences that result in high computational costs and even reduced performance. Existing solutions either add modules for compression or remove less important tokens, which introduce additional inference latency or risk losing clinical information. To achieve lossless compression of token sequences without additional cost or loss of performance, we propose Medical Token-Pair Encoding (MedTPE), a layered method that extends standard tokenisation for EHR sequences. MedTPE merges frequently co-occurring medical token pairs into composite tokens, providing lossless compression while preserving the computational complexity through a dependency-aware replacement strategy. Only the embeddings of the newly introduced tokens of merely 0.5-1.0\% of the LLM’s parameters are fine-tuned via self-supervised learning. Experiments on real-world datasets for two clinical scenarios demonstrate that MedTPE reduces input token length by up to 31\% and inference latency by 34-63\%, while maintaining or even improving both predictive performance and output format compliance across multiple LLMs and four clinical prediction tasks. Furthermore, MedTPE demonstrates robustness across different input context lengths and generalisability to scientific and financial domains and different languages. The code is available in the GitHub repository.

Accurate long-horizon vessel trajectory prediction remains challenging due to compounded uncertainty from complex navigation behaviors and environmental factors. Existing methods often struggle to maintain global directional consistency, leading to drifting or implausible trajectories when extrapolated over long time horizons. To address this issue, we propose a semantic-key-point-conditioned trajectory modeling framework, in which future trajectories are predicted by conditioning on a high-level Next Key Point (NKP) that captures navigational intent. This formulation decomposes long-horizon prediction into global semantic decision-making and local motion modeling, effectively restricting the support of future trajectories to semantically feasible subsets. To efficiently estimate the NKP prior from historical observations, we adopt a pretrain-finetune strategy. Extensive experiments on real-world AIS data demonstrate that the proposed method consistently outperforms state-of-the-art approaches, particularly for long travel durations, directional accuracy, and fine-grained trajectory prediction.

Despite their strong general capabilities, large language models (LLMs) often remain unreliable when outputs must be numerically precise. A key reason is the training objective: standard cross-entropy treats numeric tokens as unstructured categories and ignores the metric structure of their values. We address this mismatch by proposing **S**mooth **M**aximum **M**ean **D**iscrepancy (**SMMD**), which builds on the classic MMD by incorporating value-distance kernels over numeric tokens and graph-based smoothness. With this kernel defined over a numeric sub-vocabulary, SMMD aligns the predicted numeric distribution to the target via kernel matching and smooths the prediction--target residual over the induced kernel graph to encourage local consistency. We evaluate SMMD on four numeric-target tasks---mathematical reasoning, arithmetic calculation, clock-time recognition, and chart question answering---across multiple open-weight LLM and VLM backbones. SMMD consistently improves accuracy over both cross-entropy and recent numeric-target losses; analyses show complementary effects between MMD and smoothness and underscore the importance of distance-based kernel design.

Reliable uncertainty quantification at unobserved spatial locations is a key challenge in spatial statistics, particularly for complex and heterogeneous datasets. While traditional methods such as Kriging rely on strong distributional assumptions, conformal prediction (CP) offers a distribution-free alternative. However, although non-i.i.d. CP theory is well established for time-series data, a significant gap remains for spatial data, where the lack of a natural ordering and discrete index complicates theoretical guarantees. Existing CP theory for spatial data often relies on exchangeability. We propose Localized Spatial Conformal Prediction (LSCP), a model-agnostic framework that bridges this gap by coupling local quantile regression with conformal calibration. LSCP conditions on spatial neighborhoods to capture local heterogeneity. We show that LSCP retains finite-sample marginal coverage under spatial exchangeability and attains asymptotic conditional coverage under stationarity and spatial mixing. Across synthetic and real datasets, LSCP consistently achieves near-nominal coverage with tighter and more stable prediction intervals than existing methods that fail to capture these spatial dependencies.

Robust conformal prediction (RCP) extends conformal prediction (CP) to noisy inputs by producing prediction sets with guaranteed coverage, ensuring that the true label is contained in the set with a user-specified probability even under worst-case perturbations. Recent works use randomized smoothing, as it provides robustness for black-box models at larger radii. Currently, there exist two setups for smoothing-based RCP: one requires extensive Monte Carlo sampling at calibration and test time but results in smaller prediction sets; the other setup produces larger prediction sets but uses a single sample at both stages. In deployment, calibration—as a one-time pre-processing step—can accommodate substantially higher computational overhead than inference. Inspired by this observation, we introduce an RCP framework that strikes a balance between the two extremes of this trade-off: we increase the sample rate at calibration time while keeping it either one or very low during test time. This calibration-time sampling opens the possibility of reducing the size of the prediction sets. In production, where the number of test predictions typically far exceeds the size of the calibration set, our Front-Loaded RCP matches the computational complexity of the state of the art while producing considerably smaller prediction sets at larger radii.

While deep learning is facing an homogenization across modalities led by Transformers, they are still challenged by shallow linear models in the time series forecasting task. Our hypothesis is that models should learn a direct link from past to future data points, which we identify as a learning dynamics capability. We develop an original $\texttt{PRO-DYN}$ nomenclature to analyze existing models through the lens of dynamics. Two observations thus emerge: **1.** under-performing architectures learn dynamics at most partially, **2.** the location of the dynamics block at the model end is of prime importance. Our systemic and empirical studies both confirm our observations on a set of performance-varying models with diverse backbones. We propose a simple plug-and-play methodology guiding model designs and improvements.

We study the problem of monitoring model performance in dynamic environments where labeled data are limited. To this end, we propose prediction-powered risk monitoring (PPRM), a semi-supervised risk-monitoring approach based on prediction-powered inference (PPI). PPRM constructs anytime-valid lower bounds on the running risk by combining synthetic labels with a small set of true labels. Harmful shifts are detected via a threshold-based comparison with an upper bound on the nominal risk, satisfying assumption-free finite-sample guarantees on the type-I error. We demonstrate the effectiveness of PPRM through extensive experiments on image classification, large language model (LLM), and telecommunications monitoring tasks.

Connectionist models and symbolic models have long embodied two divergent paradigms: the former excel at differentiable representation learning yet struggle with transparency, while the latter deliver explicit rule-based reasoning but resist gradient-based optimization. We introduce Arboreal Neural Networks (ArbNN), a neural-symbolic framework that unifies these paradigms both computationally and conceptually. At the design level, ArbNN departs fundamentally from prior neuralized-tree models through a depth-aware routing mechanism and a topology-informed softmax aggregation, which together enable one-shot full-path gradient propagation and consequently achieving rapid and well-conditioned optimization dynamics. At the conceptual level, ArbNN reveals that decision-tree branching and self-attention routing are two realizations of the same conditional computation primitive. We prove a structural isomorphism between a decision tree and a single-query attention head, enabling a differentiable architecture that faithfully preserves symbolic decision logic. The defining property of ArbNN is Bidirectional Fidelity, ensuring that the neural module can be compiled from—and losslessly decompiled back into—a symbolic tree, yielding both ordering consistency in ranking behavior and explicit, auditable interpretability via reconstructed if-else rules. ArbNN further supports GBDT-based initialization, allowing it to inherit strong inductive biases and integrate seamlessly with existing production workflows. Empirically, ArbNN achieves state-of-the-art performance on various public tabular benchmarks and delivers consistent gains under temporal distribution shift in large-scale industrial credit-risk systems. To support realistic evaluation, we additionally construct TabCredit, a feature-rich, temporally partitioned dataset built from millions of real-world loan applications. Together, these results demonstrate that ArbNN forms a unified, reversible, and practically deployable bridge between symbolic reasoning and neural computation for high-stakes tabular domains.

Neural networks often map low-dimensional embeddings to high-dimensional output spaces. Usually, the output layer is linear, which can create a *rank bottleneck* that limits the functions a model can represent. Such bottlenecks are ubiquitous in link prediction models, such as knowledge graph embeddings (KGEs), as the output space of entities can be orders of magnitude larger than the embedding dimension. We investigate how rank bottlenecks limit model expressivity for fitting the training data. While previous work focused on sufficient bounds on the embedding dimension required for specific KGEs, we show necessary bounds for *all* KGEs with a linear output layer, which grow with graph size and connectivity. We also consider a non-linear output layer using mixtures to break the bottleneck without significant parameter overhead. Empirically, we show that models using this non-linear layer improve in ranking performance and probabilistic fit for large and dense datasets at a low parameter cost, as predicted by our theory. Our work reveals how linear output layers limit KGEs and motivates non-linear alternatives for scaling to large and dense graphs.

Large Language Models (LLMs) obey consistent scaling laws---empirical power-law fits that predict how loss decreases with compute, data, and parameters. While predictive, these laws are descriptive rather than prescriptive: they characterize typical training, not optimal training. Surprisingly few works have successfully challenged the data-efficiency bounds implied by these laws---which is our primary focus. To that end, we introduce the Geodesic Hypothesis, positing that token sequences trace geodesics on a smooth semantic manifold and are therefore locally linear. Building on this principle, we propose a novel Semantic Tube Prediction (STP) task, a JEPA-style regularizer that confines hidden-state trajectories to a tubular neighborhood of the geodesic. STP generalizes JEPA to language without requiring explicit multi-view augmentations. We show this constraint improves signal-to-noise ratio, and consequently preserves diversity by preventing trajectory collisions during inference. Empirically, STP allows LLMs to match baseline accuracy with 16$\times$ less training data, directly violating the data term of fine-tuning scaling laws and demonstrating that principled geometric priors can surpass brute-force scaling. Code: \url{https://github.com/galilai-group/llm-jepa#stp}

Tree ensembles are machine learning models with strong predictive performance and interpretability, and remain widely used for tabular data. Standard pruning methods for tree ensembles typically optimize an accuracy-compression trade-off and may change a subset of predictions, potentially compromising decision consistency. Faithful pruning methods address this issue by preserving prediction equivalence over the entire input space, but this requirement leads to lower compression ratios. We propose **PINE**, a pruning method that provides strong guarantees within an in-distribution region. PINE preserves prediction equivalence within this region and controls the region size using a single parameter $\alpha$ via conformal calibration. Experiments on 12 public tabular datasets show that PINE improves the compression ratio by up to 30% while preserving predictions at a comparable level to existing faithful pruning methods.

Conventional autoregressive models achieve causality through next-token prediction, but suffer from prohibitive latency and typically underperform non-causal alternatives such as masked prediction and diffusion. We propose self-token prediction, which enables predicting a flexible number of tokens per step, and introduce SAR, the first spatially autoregressive image generator built on this paradigm. SAR delivers markedly faster inference speeds and consistently outperforms prior autoregressive baselines, achieving performance on par with state-of-the-art non-causal models. Our findings highlight self-token prediction as a crucial step toward a high-quality autoregressive paradigm for visual generation.

Online structured prediction, including online classification as a special case, is the task of sequentially predicting labels from input features. In this setting, the *surrogate regret*—the cumulative excess of the actual target loss (e.g., the 0-1 loss) over the surrogate loss (e.g., the logistic loss) incurred by the best fixed estimator—has gained attention because it admits a finite bound independent of the time horizon $T$. However, such guarantees break down in *non-stationary* environments, where every fixed estimator may incur surrogate loss that grows linearly with $T$. To address this limitation, we obtain an upper bound of $F_T+O(1+P_T)$ on the cumulative target loss, where $F_T$ is the cumulative surrogate loss of any comparator sequence and $P_T$ is its *path length*. This bound depends on $T$ only through $F_T$ and $P_T$, thus offering stronger guarantees under non-stationarity. Our core idea is to combine the dynamic regret analysis of online gradient descent (OGD) with the *exploit-the-surrogate-gap* technique. This viewpoint sheds light on the usefulness of a Polyak-style learning rate for OGD, which systematically yields target-loss bounds and performs well empirically. We then extend our approach to broader settings beyond prior work via the *convolutional Fenchel–Young loss*. Finally, a lower bound shows that the dependence on $F_T$ and $P_T$ is tight.

Conformal prediction provides distribution-free prediction sets with finite-sample conditional guarantees. RKHS-based frameworks—while promising for complex covariate shifts—suffer from prohibitive computational costs. To guarantee conditional validity under such shifts while ensuring feasibility, we build upon the framework of Gibbs et al. (2025) by introducing a stable and efficient algorithm that computes the full solution path of the regularized RKHS conformal optimization problem, at essentially the same cost as a single kernel quantile fit. Our approach provides simultaneous hyperparameter tuning which provides smoothness control and data-adaptive calibration. To extend the method to high-dimensional settings, we further integrate our approach with low-rank latent embeddings that capture conditional validity in a data-driven latent space. Empirically, our method provides reliable conditional coverage across a variety of modern black-box predictors, improving the interval length of Gibbs et al. (2025) by 30%, while achieving a 40-fold speedup.

Battery lifetime early prediction is crucial for safety assessment and decision planning, yet early-stage degradation signals are extremely weak and difficult to distinguish from stochastic noise. Existing methods primarily rely on denoising or signal decomposition, which may lose critical degradation cues. In nature, most organisms exhibit the binaural effect, exploiting discrepancies between left and right auditory inputs to enhance perceptual reliability. Inspired by this, we propose DITING, a weak degradation listener for battery lifetime early prediction. We first employ optimal-transport-based selective matching to extract a robust health template from initial cycles, and further design a tri-coupled degradation manifestation mechanism to distinguish degradation signals from noise. By exploiting the randomness of noise, matched responses under symmetric coupling suppress stochastic fluctuations, while degradation-driven cumulative deviations propagate through the coupling process to form stable bilateral discrepancies, thereby amplifying weak early-stage cues for lifetime prediction. Experiments on various datasets demonstrate that DITING achieves state-of-the-art performance and provides more reliable early support for full-lifecycle battery management.

Conformal prediction is challenging for time series with time-varying conditional distributions. Existing sequential conformal methods can yield volatile, non-nested prediction intervals due to noisy tail conditional quantile estimation and quantile crossing issue. To overcome these challenges, we construct prediction intervals for time series via a novel method called Conditional Quantile Adjusted Conformal Prediction (CQACP), which stabilizes sequential conformal calibration by modeling the conditional quantile curve of nonconformity score. At each time step, CQACP evaluates a base conditional quantile learner on a grid of quantile levels and fits a Cornish--Fisher approximation parameterized by conditional moments of nonconformity score with monotonicity constraints. Asymptotically, we prove the conditional validity of the prediction interval under serial dependence and show improved conditional quantile estimation accuracy. Experiments on multiple real-world datasets demonstrate that CQACP maintains accurate coverage and produces smooth, narrow, and nested prediction intervals across different significance levels and prediction models.

While large language models (LLMs) perform strongly on diverse tasks, their trustworthiness is limited by erratic behavior that is unfaithful to their internal knowledge. In particular, LLMs often fail on multiple-choice questions (MCQs) even if they encode correct answers in their hidden representations, revealing a misalignment between internal knowledge and output behavior. We investigate and mitigate this *knowledge-prediction gap* on MCQs through a three-step analysis of hidden representations. First, we quantify the prevalence and magnitude of the gap across models and datasets. Second, we provide a geometric interpretation by identifying distinct *knowledge* and *prediction* subspaces in the residual stream. Third, we introduce **KAPPA**, a lightweight inference-time intervention that aligns the two subspaces within the residual stream to reduce the knowledge-prediction gap. Our results provide a geometric and interpretable explanation of the knowledge-prediction gap in LLMs. Furthermore, KAPPA effectively reduces the gap across diverse MCQ benchmarks and models, and generalizes to free-form settings.

We introduce test prediction variance (TPV)—the first-order sensitivity of a trained model's outputs to parameter perturbations—as a unifying framework for analyzing post-training robustness. TPV's trace form $\mathrm{Tr}(H_{\mathrm{eff}}C)$ separates the geometry of the trained model $H_{\mathrm{eff}}$ from the perturbation covariance $C$, placing SGD noise, label noise, quantization, and pruning under a single lens. The resulting expressions recover the wide-minima hypothesis for SGD and quantization noise, and yield a distinct Jacobian-spectral characterization for label noise connecting label-noise TPV with benign overfitting in nonlinear networks. Theoretically, we prove that training-set TPV converges to its test-set counterpart in the overparameterized limit, irrespective of generalization performance, providing the first result that prediction variance under local parameter perturbations can be inferred from training inputs alone. Empirically, this stability holds far more broadly, including at very low widths. Further, TPV correlates well with test loss, enabling practical applications: JBR, a label-free pruning criterion derived from TPV geometry matching state-of-the-art baselines; and training-set based model selection signal for in-distribution and transfer learning scenarios. \href{https://github.com/devansharpit/TPV/tree/main}{Code Available Here}

The integration of Large Language Models (LLMs) into ad auctions for dynamic creative enhancement presents a paradigm shift, yet introduces significant computational costs disrupting traditional mechanism design. This paper provides a comprehensive game-theoretic and algorithmic framework for such LLM-augmented auctions. We model the system as a dynamic Stackelberg game where the platform (leader) strategically invests in creative enhancement to maximize net revenue, while autobidding agents (followers) respond to enhanced ad qualities under budget constraints. To endogenize inference costs, we propose the Platform-Investment Mechanism (PIM). We develop the Online Dual-Descent Bidding with Regularization (ODDB-R) algorithm for agents to learn optimal bidding strategies in this non-stationary environment. For the platform, we formulate the investment problem as a continuous control task and solve it using a Two-Timescale Stackelberg Learning with Proximal Policy Optimization (TTSL-PPO) algorithm, which provably converges to a Stackelberg Stationary Point. Extensive experiments on large-scale real-world datasets and state-of-the-art LLMs demonstrate that our framework significantly outperforms heuristic baselines in revenue, social welfare, and user engagement.