AISTATS 2026 论文列表

State-of-the-art large language models rely on randomization to respond to a prompt. Consequently, a model may respond differently to the same prompt if asked multiple times. In this work, we argue that the evaluation and ranking of large language models should control for this randomization. Our starting point is the development of a causal model for coupled autoregressive generation, which allows different large language models to sample responses with the same source of randomness. Building upon our causal model, we first show that, on evaluations based on benchmark datasets, coupled autoregressive generation leads to the same conclusions as vanilla autoregressive generation but using provably fewer samples. However, we further show that, on evaluations based on pairwise comparisons, the two approaches can surprisingly lead to different rankings when comparing more than two models. To complement our theoretical results, we conduct experiments with several models from the , and families. We find that, across multiple benchmark datasets, coupled autoregressive generation requires up to 75

Causal discovery and data imputation are often treated separately, yet both face challenges when data are missing. Existing causal discovery methods discard incomplete samples, losing valuable information, while standard imputation relies on spurious correlations that obscure the causal signal. We propose LOGIC, a framework that performs causal discovery and causally consistent imputation jointly. In contrast to prior work that assumes all source variables are observed, we derive a verifiable criterion for this assumption under MCAR and MAR missingness, grounded in the Algorithmic Markov Condition. LOGIC then proceeds layer by layer: identifying sources, recovering downstream relations, and imputing missing values, while explicitly declaring unknowns when imputation is unsupported. This design preserves causal reasoning even in challenging missingness regimes. Experiments on synthetic and real-world data show that LOGIC outperforms state-of-the-art baselines in both structure recovery and imputation accuracy.

In high-stakes AI applications, even a single action can cause irreparable damage. However, nearly all of sequential decision-making theory assumes that all errors are recoverable (e.g., by bounding rewards). Standard bandit algorithms that explore aggressively may cause irreparable damage when this assumption fails. Some prior work avoids irreparable errors by asking for help from a mentor, but a mentor may not always be available. In this work, we formalize a model of learning with unbounded rewards without a mentor as a two-action contextual bandit with an abstain option: at each round the agent observes an input and chooses either to abstain (always 0 reward) or to commit (execute a preexisting task policy). Committing yields rewards that are upper-bounded but can be arbitrarily negative, and the commit reward is assumed Lipschitz in the input. We propose a caution-based algorithm that learns when not to learn: it chooses a trusted region and commits only where the available evidence does not already certify harm. Under these conditions and i.i.d. inputs, we establish sublinear regret guarantees, theoretically demonstrating the effectiveness of cautious exploration for deploying learning agents safely in high-stakes environments.

Recent Transformer- and MLP-based models have demonstrated strong performance in long-term time series forecasting, yet Transformers remain limited by their quadratic complexity and permutation-equivariant attention, while MLPs exhibit spectral bias. We propose HaKAN, a versatile model based on Kolmogorov-Arnold Networks (KANs), leveraging Hahn polynomial-based learnable activation functions and providing a lightweight and interpretable alternative for multivariate time series forecasting. Our model integrates channel independence, patching, a stack of Hahn-KAN blocks with residual connections, and a bottleneck structure comprised of two fully connected layers. The Hahn-KAN block consists of inter- and intra-patch KAN layers to effectively capture both global and local temporal patterns. Extensive experiments on various forecasting benchmarks demonstrate that our model consistently outperforms recent state-of-the-art methods, with ablation studies validating the effectiveness of its core components.

The synthetic control (SC) framework is widely used for observational causal inference with time-series panel data. Despite its success across diverse applications, existing SC methods typically treat pre-intervention time indices as exchangeable, meaning they may fail to exploit temporal structure when strong trends are present. We propose Time-Aware Synthetic Control (TASC), a method that addresses this limitation by adopting a state-space model with a constant trend component while preserving the low-rank structure of the signal. TASC uses the Kalman filter and the Rauch–Tung–Striebel smoother in two steps: it first fits a generative time-series model with expectation–maximization and then performs counterfactual inference. We evaluate TASC on simulated and real-world datasets spanning policy evaluation and sports prediction. Our results demonstrate that TASC offers advantages in settings with high observation noise and long prediction horizons.

Zeroth-order or derivative-free optimization (MeZO) is an attractive strategy for finetuning large language models (LLMs) because it eliminates the memory overhead of backpropagation. However, it converges slowly due to the inherent curse of dimensionality when searching for descent directions in the high-dimensional parameter space of billion-scale LLMs. We propose ConMeZO, a novel zeroth-order optimizer that accelerates convergence by adaptive directional sampling. Instead of drawing the direction uniformly at random, ConMeZO restricts the sampling to a cone centered around a momentum estimate. This concentrates the search in directions where the true gradient is more likely to lie and thus reduces the effect of high dimensions. We prove that ConMeZO achieves the same worst-case convergence rate as MeZO. Empirically, when finetuning LLMs on natural language tasks, ConMeZO is up to 2× faster than MeZO while retaining the low-memory footprint of zeroth-order methods.

We study low-rank matrix completion in the semi-random model, where each entry (i, j) is observed independently with an unknown probability p_i,j≥ p, in contrast to the standard model with a uniform probability p. While prior work has shown that nonconvex approach succeeds in the semi-random model for exact observations [CG18], it remains unclear whether similar guarantees extend to more general observation model, such as noisy or one-bit measurements. In this paper, we give a unified framework for semi-random matrix recovery applicable to a broad family of observation models. Our approach builds on the preprocessing step of [CG18] to restore regularity conditions that are violated under adversarial sampling, and leverages the primal-dual framework of [ZWYG18] to obtain near-optimal recovery guarantees. As concrete corollaries, we show that for both noisy and one-bit matrix completion in the semi-random model, after the preprocessing step, every local minimum of the non-convex objective yields an approximate recovery of the ground-truth matrix.

We study a data marketplace where a broker intermediates between buyers, who seek to estimate the mean μ of an unknown normal distribution N(μ, σ^2), and contributors, who can collect data from this distribution at a cost. The broker delegates data collection work to contributors, aggregates reported datasets, sells it to buyers, and redistributes revenue as payments to contributors. We aim to maximize welfare or profit under key constraints: individual rationality for buyers and contributors, incentive compatibility (contributors are incentivized to comply with data collection instructions and truthfully report the collected data), and budget balance (total contributor payments equals total revenue). We first compute welfare/profit-optimal prices under truthful reporting; however, to incentivize data collection and truthful data reporting, we adjust them based on discrepancies in contributors’ reported data. This yields a Nash equilibrium (NE) where the two lowest-cost contributors collect all data. We complement this with two hardness results: (𝒾) no nontrivial dominant-strategy incentive-compatible mechanism exists in this problem, and (𝒾𝒾) no mechanism outperforms ours in a NE.

When training large models on limited data, avoiding overfitting is paramount. Common grid search or smarter search methods rely on expensive separate runs for each candidate hyperparameter, while carving out a validation set that reduces available training data. In this paper, we study gradient-based learning of hyperparameters via the evidence lower bound (ELBO) objective from Bayesian variational methods. This avoids the need for any validation set. We focus on scenarios where the model is over-parameterized for flexibility and the approximate posterior is chosen to be Gaussian with isotropic covariance for tractability, even though it cannot match the true posterior. In such scenarios, we find the ELBO prioritizes posteriors that match the prior, leading to severe underfitting. Instead, we recommend a data-emphasized ELBO that upweights the likelihood but not the prior. In Bayesian transfer learning of image and text classifiers, our method reduces the 88+ hour grid search of past work to under 3 hours while delivering comparable accuracy. We further demonstrate how our approach enables efficient yet accurate approximations of Gaussian processes with learnable lengthscale kernels.

Frontier models can be prompted or conditioned to do many tasks, but finding good prompts is not always easy, nor is understanding some performant prompts. We view prompting as finding the best conditioning sequence on a near-optimal sequence predictor. On numerous well-controlled experiments, we show that unintuitive optimal conditioning sequences can be better understood given the pretraining distribution, which is not usually available. Even using exhaustive search, reliably identifying optimal prompts for practical neural predictors can be surprisingly difficult. Popular prompting methods, such as using demonstrations from the targeted task, can be surprisingly suboptimal. Using the same empirical framework, we analyze optimal prompts on frontier models, revealing patterns similar to the binary examples and previous findings. Taken together, this work takes an initial step towards understanding optimal prompts, from a statistical and empirical perspective that complements research on frontier models.

We formulate a point-source localization problem in d dimensions, where a source inside the ball of radius R emits a signal that is picked up by various sensors located at the surface of the ball. For d=3, this can model problems in neural sensing, where a net of electroencephalography (EEG) or magnetoencephalography (MEG) sensors try to locate the source of a distinct neural event such as a seizure. For a power law decay model with exponent α>0 for the sensors, we obtain a lower bound on the minimax risk for localizing the source that is asymptotically d^2σ^2R^2α+2/nα^2PK under mean-squared error loss, where σ^2 is the noise variance, P is the signal power, K is the number of sensors, and n is the number of independent measurements. In the case d≤ 2(α+1) with uniformly distributed sensor locations, we then give a matching upper bound, including getting the exact constant correct, for the asymptotic minimax rate in a neighborhood of the origin. We show that there is a phase transition at d=2(α+2), above which a certain Fisher information quantity is minimized at the boundary of the ball, and below which it is minimized at the origin. At the critical dimension d=2(α+2), the Fisher information is constant throughout the entire parameter space. For the special case d=3, we supplement and compare this information-theoretic analysis with a simulated forward EEG model that uses a realistic head model derived from population-averaged magnetic resonance imaging data.

Reinforcement Learning from Human Feedback (RLHF) has emerged as a powerful approach for aligning generative models, but its reliance on learned reward models makes it vulnerable to mis-specification and reward hacking. Preference-based Reinforcement Learning (PbRL) offers a more robust alternative by directly leveraging noisy binary comparisons over trajectories. We study the best policy identification problem in PbRL, motivated by post-training optimization of generative models, for example, during multi-turn interactions. Learning in this setting combines an offline preference dataset—potentially biased or out-of-distribution and collected from a rater of subpar ’competence’—with online pure exploration, making systematic online learning essential. To this end, we propose Posterior Sampling for Preference Learning (𝖯𝖲𝖯𝖫), a novel algorithm inspired by Top-Two Thompson Sampling that maintains posteriors over the reward model and dynamics. We provide the first Bayesian simple regret guarantees for PbRL and introduce an efficient approximation that outperforms existing baselines on simulation and image generation benchmarks.

A/B tests in online experiments face statistical power challenges when testing multiple candidates simultaneously, while adaptive experimental designs (AED) alone fall short in inferring experiment statistics such as the average treatment effect, especially with many metrics (e.g., revenue, safety) and heterogeneous variances. This paper proposes a fixed-budget multi-metric AED framework with a two-phase structure: an adaptive exploration phase to identify the best treatment, and a validation phase with an A/B test to verify the treatment’s quality and infer statistics. We propose SHRVar, which generalizes sequential halving (SH) with a novel relative-variance-based sampling and an elimination strategy built on reward z values. It achieves a provable error probability that decreases exponentially, where the exponent H3 generalizes the complexity measure for SH and SHVar with homogeneous and heterogeneous variances, respectively. Numerical experiments demonstrate its performance and robustness.

We propose a novel deep symbolic regression (DSR) approach to enhance the robustness and interpretability of data-driven mathematical expression discovery. Existing DSR methods are built on recurrent neural networks, solely guided by data fitness, and potentially meet tail barriers that can zero out the policy gradient, causing inefficient model updates. To address these issues, we design a decoder-only architecture that performs attention in the frequency domain and introduce a dual-indexed position encoding to conduct layer-wise generation. Second, we propose a Bayesian information criterion (BIC)-based reward function that can automatically adjust the trade-off between expression complexity and data fitness, without the need for explicit manual tuning. Third, we develop a ranking-based weighted policy update method that eliminates the tail barriers and enhances training effectiveness. Extensive benchmarks and systematic experiments demonstrate the advantages of our approach. We have released our implementation at <https://github.com/ZakBastiani/CADSR.>

Active subspace analysis uses the leading eigenspace of the gradient’s second moment to conduct supervised dimension reduction. In this article, we extend this methodology to real-valued functionals on Hilbert space. We define an operator which coincides with the active subspace matrix when applied to a Euclidean space. We show that many of the desirable properties of Active Subspace analysis extend directly to the infinite dimensional setting. We also propose a Monte Carlo procedure and discuss its convergence properties. Finally, we deploy this methodology to create visualizations as well as improve modeling and optimization on complex test problems.

The ROC curve is widely used to assess binary classifiers. Yet for some applications, such as alert systems for monitoring hospitalized patients, conventional ROC analysis cannot meet two key deployment needs: enforcing a constraint on precision to avoid false alarm fatigue and imposing an upper bound on the number of predicted positives to represent the capacity of hospital staff. The usual area under the curve metric also does not reflect asymmetric costs for false positives and false negatives. In this paper we address all three of these issues. First, we show how the subset of classifiers that meet precision and capacity constraints occupy a feasible region in ROC space. We establish the polygon-shaped geometry of this region. We then define the partial area of lesser classifiers, a performance metric that is monotonic with cost and only accounts for the feasible region. Averaging this area over a desired distribution for cost parameters results in the partial volume over the ROC surface, or partial VOROS. In experiments predicting mortality risk from vital sign history on several datasets, we show this cost-aware metric can outperform alternatives at ranking classifiers for in-hospital alerts.

We study a nonstationary bandit problem where rewards depend on both actions and latent states, the latter governed by unknown linear dynamics. Crucially, the state dynamics also depend on the actions, resulting in tension between short-term and long-term rewards. We propose an explore-then-commit algorithm for a finite horizon T. During the exploration phase, random Rademacher actions enable estimation of the Markov parameters of the linear dynamics, which characterize the action-reward relationship. In the commit phase, the algorithm uses the estimated parameters to design an optimized action sequence for long-term reward. Our proposed algorithm achieves 𝒪̃(pT^2/3) regret where p is the action dimension. Our analysis handles two key challenges: learning from temporally correlated rewards, and designing action sequences with optimal long-term reward. We address the first challenge by providing near-optimal sample complexity and error bounds for system identification using bilinear rewards. We address the second challenge by proving an equivalence with indefinite quadratic optimization over a hypercube, a known NP-hard problem. We provide a sub-optimality guarantee for this problem, enabling our regret upper bound. Lastly, we propose a semidefinite relaxation with Goemans-Williamson rounding as a practical approach.

The computation of inner products between high-order tensor data points is a fundamental task in numerous machine learning and scientific applications. However, the naive approach to these computations incurs exponential time complexity with respect to the number of modes. The work of Rakhshan and Rabusseau (AISTAT, 2020) introduced an extension of the random projection tailored for tensor datasets, which compresses large tensors into compact vectors (a.k.a sketches). Their approach provides unbiased estimates of the original pairwise inner products. However, the variance of their estimates grows exponentially with the number of modes, making their estimates less reliable for small sketch sizes. In this work, we propose improved sketching algorithms that provide unbiased estimates for pairwise inner products, with significantly lower variance - independent of the number of modes—compared to that of Rakhshan and Rabusseau (AISTAT, 2020). Furthermore, our approach offers asymptotically improved time complexity. Our sketching algorithm builds on the framework of Ahle et al. (SODA 2020), which proposed sketching techniques for high-degree polynomial kernels.

This paper studies the Exponential Weights (EW) algorithm with an isotropic Gaussian prior for online logistic regression. We show that the near-optimal worst-case regret bound O(dlog(Bn)) for EW, established by Kakade and Ng (2005) against the best linear predictor of norm at most B, can be achieved with total worst-case computational complexity Õ(B^3 n^5). This substantially improves on the O(B^18n^37) complexity of prior work achieving the same guarantee (Foster et al., 2018). Beyond efficiency, we analyze the large-B regime under linear separability: after rescaling by B, the EW posterior converges as B→∞ to a standard Gaussian truncated to the version cone. Accordingly, the predictor converges to a solid-angle vote over separating directions and, on every fixed-margin slice of this cone, the mode of the corresponding truncated Gaussian is aligned with the hard-margin SVM direction. Using this geometry, we derive non-asymptotic regret bounds showing that once B exceeds a margin-dependent threshold, the regret becomes independent of B and grows only logarithmically with the inverse margin. Overall, our results show that EW can be both computationally tractable and geometrically adaptive in online classification.