AI conference papers

AI conference paper library

Search papers from 30+ top conferences, including NeurIPS, ICLR, ICML, CVPR, ACL and AAAI. Filter by keyword, then open any paper for its abstract, authors and links.

arXiv

Gaussian-Mixture-Model Q-Functions for Policy Iteration in Reinforcement Learning

Minh Vu, Konstantinos Slavakis

Unlike their conventional use as estimators of probability density functions in reinforcement learning (RL), this paper introduces a novel function-approximation role for Gaussian mixture models (GMMs) as direct surrogates for Q-function losses. The proposed estimators, termed GMM-QFs, possess substantial representational capacity, as they are shown to be universal approximators over a broad class of functions. They are further embedded within Bellman residuals, where their learnable parameters--a fixed number of mixing weights, together with Gaussian mean vectors and covariance matrices--are inferred from data via optimization on a product Riemannian manifold. This geometric perspective on the parameter space naturally introduces Riemannian optimization into the policy-evaluation step of standard policy-iteration (PI) frameworks. Moreover, rigorous theoretical analysis establishes performance bounds on Q-function estimation error under the proposed PI scheme. Supporting numerical tests show that GMM-QFs deliver competitive performance and, in some cases, outperform state-of-the-art approaches across a range of benchmark RL tasks, all while maintaining a significantly smaller computational footprint than deep-learning methods.

arXiv

Fluid Reasoning Representations

Dmitrii Kharlapenko, Terry Jingchen Zhang, Arth Singh, Alessandro Stolfo, Arthur Conmy, Mrinmaya Sachan, Zhijing Jin

Frontier large language models increasingly solve complex tasks involving abstract concepts through extended test-time thinking. Yet we lack a mechanistic account of how extended thinking changes hidden-state representations over the course of a reasoning trace. We introduce \textit{Fluid Reasoning Representations} (FRRs), a representation-level account of how LLMs organize action and predicate concepts during self-generated reasoning, and test them on obfuscated planning, symbolic, and mathematical tasks where task-relevant words are replaced while problem structure is preserved. Across open-weight base, instruction-tuned, and extended-thinking LLMs, representations of the same action or predicate become more similar across wordings and move toward the corresponding unobfuscated concepts over the reasoning trace. Causal probes show that these representations affect behavior: cross-naming steering improves held-out accuracy beyond Gaussian and shuffled controls, symbolic patching retains more action information than shuffled patching, and subtracting refined directions degrades accuracy; together, these results suggest that extended thinking strengthens a representation dynamic already present at lower magnitude in base and instruction-tuned LLMs. Our codebase is open-sourced \href{https://github.com/AI4Collaboration/Fluid-Reasoning-Representation}{here}.

arXiv

Training and Simulation of Quadrupedal Robot in Adaptive Stair Climbing and Descending for Indoor Firefighting: An End-to-End Reinforcement Learning Approach

Baixiao Huang, Baiyu Huang, Yu Hou

Quadruped robots are used for primary searches during the early stages of indoor fires. A typical primary search involves quickly and thoroughly looking for victims under hazardous conditions and monitoring flammable materials. However, situational awareness in complex indoor environments and rapid stair climbing and descending across different staircases remain the main challenges for robot-assisted primary searches. In this project, we designed a two-stage end-to-end deep reinforcement learning (RL) approach to optimize both navigation and locomotion. In the first stage, the quadrupeds, Unitree Go2, were trained to climb and descend stairs in Isaac Lab's pyramid-stair terrain. In the second stage, the quadrupeds were trained to climb and descend various realistic indoor staircases in the Isaac Lab engine, with the learned policy transferred from the previous stage. These indoor staircases are straight, L-shaped, and spiral, to support climbing and descending tasks in complex environments. This project explores how to balance navigation and locomotion and how end-to-end RL methods can enable quadrupeds to adapt to different stair shapes. Our main contributions are: (1) A two-stage end-to-end RL framework that transfers climbing/descending skills from abstract pyramid terrain to realistic indoor stair topologies. (2) A centerline-based navigation formulation that enables unified learning of navigation and locomotion without hierarchical planning. (3) Demonstration of policy generalization across diverse staircases using only local height-map perception. (4) An empirical analysis of success, efficiency, and failure modes under increasing stair difficulty.

arXiv

SKETCH: Semantic Key-Point Conditioning for Long-Horizon Vessel Trajectory Prediction

Linyong Gan, Zimo Li, Wenxin Xu, Xingjian Li, Jianhua Z. Huang, Enmei Tu, Shuhang Chen

Accurate long-horizon vessel trajectory prediction remains challenging due to compounded uncertainty from complex navigation behaviors and environmental factors. Existing methods often struggle to maintain global directional consistency, leading to drifting or implausible trajectories when extrapolated over long time horizons. To address this issue, we propose a semantic-key-point-conditioned trajectory modeling framework, in which future trajectories are predicted by conditioning on a high-level Next Key Point (NKP) that captures navigational intent. This formulation decomposes long-horizon prediction into global semantic decision-making and local motion modeling, effectively restricting the support of future trajectories to semantically feasible subsets. To efficiently estimate the NKP prior from historical observations, we adopt a pretrain-finetune strategy. Extensive experiments on real-world AIS data demonstrate that the proposed method consistently outperforms state-of-the-art approaches, particularly for long travel durations, directional accuracy, and fine-grained trajectory prediction.

arXiv

Dr. Zero: Self-Evolving Search Agents without Training Data

Zhenrui Yue, Kartikeya Upasani, Xianjun Yang, Suyu Ge, Shaoliang Nie, Yuning Mao, Zhe Liu, Dong Wang

As high-quality data becomes increasingly difficult to obtain, self-evolution without curated training data has emerged as a promising paradigm. This approach allows large language models (LLMs) to autonomously generate and solve complex problems, thereby improving their reasoning capabilities. However, multi-turn search agents struggle in this setting due to limited question diversity and the substantial compute required for multi-step reasoning and tool use. In this work, we introduce Dr. Zero, a framework that enables search agents to effectively self-evolve without human-annotated training data, relying solely on an external search engine as their knowledge environment. In particular, we design a self-evolution feedback loop where a proposer generates structurally diverse questions to train a solver initialized from the same base model. As the solver evolves, it incentivizes the proposer to produce increasingly difficult yet solvable tasks, thus establishing an automated curriculum to refine both agents. To enhance training efficiency, we also introduce hop-grouped relative policy optimization (HRPO). This method clusters structurally similar questions to construct group-level baselines, effectively minimizing the sampling overhead in evaluating each query's individual difficulty and solvability. Consequently, HRPO significantly reduces the compute requirements for proposer training and reward estimation without compromising performance or stability. Extensive experimental results demonstrate that Dr. Zero matches or surpasses fully supervised search agents on several question answering benchmarks, showing that strong agentic search and evidence-grounded reasoning can emerge solely through self-evolution.

arXiv

DeepPAAC: A New Deep Galerkin Method for Principal-Agent Problems

Michael Ludkovski, Changgen Xie, Zimu Zhu

We consider numerical resolution of principal-agent (PA) problems in continuous time. We formulate a generic PA model with continuous and lump payments and a multi-dimensional strategy of the agent. To tackle the resulting Hamilton-Jacobi-Bellman equation with an implicit Hamiltonian we develop a novel deep learning method: the Deep Principal-Agent Actor Critic (DeepPAAC) Actor-Critic algorithm. DeepPAAC is able to handle multi-dimensional states and controls, as well as constraints. We investigate the role of the neural network architecture, training designs, loss functions, etc. on the convergence of the solver, presenting five different case studies.

arXiv

Hierarchical Physics-Embedded Learning for Partially Known Spatiotemporal Dynamics

Xizhe Wang, Xiaobin Song, Hongbo Zhao, Qingshan Jia, Qianchuan Zhao, Hao Sun, Benben Jiang

Partial physical knowledge--governing structures known, constitutive relations or their combinations not--pervades spatiotemporal systems. Existing scientific machine learning paradigms learn evolution largely from data, impose equations as soft constraints, or hard-code physical terms into network updates; none exploits knowledge of this form. Here we introduce the hierarchical physics-embedded adaptive Fourier neural operator, encoding such knowledge as computational architecture rather than penalizing or appending it: a first level learns or embeds fundamental physical expressions as intermediate representations, and a second level learns or embeds their governing combination, with adaptive Fourier layers capturing nonlocal, high-order couplings at each level. We prove a hierarchical error decomposition--embedding known components removes or shrinks their terms, and a parameter-complexity advantage: when the hierarchy aligns with the compositional structure of the dynamics, the number of learnable Fourier parameters sufficient for a prescribed accuracy grows strictly more slowly than for a single-level operator. Across canonical phase-field systems and experimental hydrofoil wake data, our method reduces long horizon extrapolation errors by up to ~70% relative to state-of-the-art physics encoded and neural operator baselines, while preserving physically meaningful morphology, energetic consistency, and spectral structure, and maintaining robust performance under sparse and noisy observations. The separated intermediate representations further enable symbolic recovery of unknown constitutive relations in partially specified PDEs. These results establish hierarchical physics embedding as a theoretically grounded route to prediction and discovery when governing laws are neither fully known nor absent, but partially known and compositionally organized.

arXiv

SHUFFLESPARSE: Learned Shuffles for Structured Sparse Networks

Abhishek Tyagi, Arjun Iyer, Liam Young, William H Renninger, Christopher Kanan, Yuhao Zhu

Structured weight sparsity accelerates training and inference on modern GPUs, but it trails unstructured dynamic sparse training (DST) in accuracy especially at extreme sparsity. We pinpoint the reason for this difference in performance to a lack of expressivity: a dense layer can implement any pattern of non-zero weights, whereas structured patterns are restricted to only a small set of weight configurations. We introduce SHUFFLESPARSE, a single permutation primitive that applies uniformly across DST-from-scratch and one-shot pruning, and across N:M, block etc. We close most of this gap by learning a single permutation matrix jointly with the structured weight matrix. When used on three different types of structures (block, N:M, and diagonal), SHUFFLESPARSE is able to reduce the structured-vs-unstructured accuracy gap on ViT-B16 (ImageNet-1K) and GPT-2 (WikiText-103) at 90-95% sparsity, while adding minimal inference overhead (< 8.7% inference overhead) and preserving any training acceleration the host structure provides. The same permutation formulation transfers to one-shot 2:4 pruning of pretrained LLMs, where it improves zero-shot accuracy by 4.6 points on LLaMA-2 7B. Together, these results establish learned permutations as a general tool for recovering unstructured-level accuracy from structured sparsity patterns.

arXiv

ImplicitRDP: An End-to-End Visual-Force Diffusion Policy with Structural Slow-Fast Learning

Wendi Chen, Han Xue, Yi Wang, Fangyuan Zhou, Jun Lv, Yang Jin, Shirun Tang, Chuan Wen, Cewu Lu

Human-level contact-rich manipulation relies on the distinct roles of two key modalities: vision provides spatially rich but temporally slow global context, while force sensing captures rapid local contact dynamics. Integrating these signals is challenging due to their fundamental frequency and informational disparities. In this work, we propose ImplicitRDP, a unified end-to-end visual-force diffusion policy that integrates visual planning and reactive force control within a single network. We introduce Structural Slow-Fast Learning, a mechanism utilizing causal attention to simultaneously process asynchronous visual and force tokens, allowing the policy to perform rapid force control at the action rate while maintaining the temporal coherence of action chunks. Furthermore, to mitigate modality collapse where end-to-end models fail to adjust the weights across different modalities, we propose Virtual-target-based Representation Regularization. This auxiliary objective maps force feedback into the same space as the action, providing a stronger, physics-grounded learning signal than raw force prediction. Extensive experiments on contact-rich tasks demonstrate that ImplicitRDP significantly outperforms both vision-only and hierarchical baselines, achieving superior reactivity and success rates with a streamlined training pipeline. Code and videos are available at https://implicit-rdp.github.io.

arXiv

Multi-Agent Inverted Transformer for Flight Trajectory Prediction

Seokbin Yoon, Keumjin Lee

Flight trajectory prediction for multiple aircraft is essential and provides critical insights into how aircraft navigate within current air traffic flows. However, predicting multi-agent flight trajectories is inherently challenging. One of the major difficulties is modeling both the individual aircraft behaviors over time and the complex interactions between flights. Generating explainable prediction outcomes is also a challenge. Therefore, we propose a Multi-Agent Inverted Transformer, MAIFormer, as a novel neural architecture that predicts multi-agent flight trajectories. The proposed framework features two key attention modules: (i) masked multivariate attention, which captures spatio-temporal patterns of individual aircraft, and (ii) agent attention, which models the social patterns among multiple agents in complex air traffic scenes. We evaluated MAIFormer using a real-world automatic dependent surveillance-broadcast flight trajectory dataset from the terminal airspace of Incheon International Airport in South Korea. The experimental results show that MAIFormer achieves the best performance across multiple metrics and outperforms other methods. In addition, MAIFormer produces prediction outcomes that are interpretable from a human perspective, which improves both the transparency of the model and its practical utility in air traffic control.

arXiv

Low-Rank Evolutionary Deep Neural Networks via Adaptive Tangent-Space Reduction

Jiahao Zhang, Shiheng Zhang, Guang Lin

Evolutionary deep neural networks (EDNNs) solve time-dependent partial differential equations by evolving the neural-network parameters sequentially in time through a local least-squares problem. Their main computational bottleneck is that each time step requires the solution of a dense linear system whose dimension equals the total number of trainable parameters. We propose a low-rank evolutionary deep neural network (LR-EDNN) method that reduces this cost through adaptive tangent-space projection. This construction replaces direct bilinear low-rank factor evolution by a linear reduced problem while preserving the sequential-in-time structure of EDNN. We construct the reduced Jacobian directly through layerwise Jacobian-vector products, without forming the full Jacobian. We further establish a finite-time comparison estimate: the deviation of the LR-EDNN trajectory from full EDNN is bounded by a discrete Grönwall accumulation of the local tangent-space projection defects, with amplification governed by the assumed Lipschitz and directional-coercivity constants. Numerical experiments on a porous-medium equation with drift, one- and two-dimensional Allen-Cahn equations, and two-dimensional viscous Burgers' equations demonstrate that LR-EDNN substantially reduces computational cost while maintaining the accuracy and qualitative fidelity of the full EDNN solver when the rank is chosen adequately.

arXiv

Finite-Agent Stochastic Differential Games on Large Graphs: II. Graph-Based Architectures

Ruimeng Hu, Jihao Long, Haosheng Zhou

We propose a novel neural network architecture, called Non-Trainable Modification (NTM), for computing Nash equilibria in stochastic differential games (SDGs) on graphs. These games model a broad class of graph-structured multi-agent systems arising in finance, robotics, energy, and social dynamics, where agents interact locally under uncertainty. The NTM architecture imposes a graph-guided sparsification on feedforward neural networks, embedding fixed, non-trainable components aligned with the underlying graph topology. This design enhances interpretability and stability, while significantly reducing the number of trainable parameters in large-scale, sparse settings. We theoretically establish a universal approximation property for NTM in static games on graphs and numerically validate its expressivity and robustness through supervised learning tasks. Building on this foundation, we incorporate NTM into two state-of-the-art game solvers, Direct Parameterization and Deep BSDE (backward stochastic differential equation), yielding their sparse variants (NTM-DP and NTM-DBSDE). Numerical experiments on three SDGs across various graph structures demonstrate that NTM-based methods achieve performance comparable to their fully trainable counterparts, while offering improved computational efficiency.

arXiv

Learning to Make Friends: Coaching LLM Agents toward Emergent Social Ties

Philipp J. Schneider, Lin Tian, Marian-Andrei Rizoiu

Can large language model (LLM) agents reproduce the complex social dynamics that characterize human online behavior -- shaped by homophily, reciprocity, and social validation -- and what memory and learning mechanisms enable such dynamics to emerge? We present a multi-agent LLM simulation framework in which agents repeatedly interact, evaluate one another, and adapt their behavior through in-context learning accelerated by a coaching signal. To model human social behavior, we design behavioral reward functions that capture core drivers of online engagement, including social interaction, information seeking, self-presentation, coordination, and emotional support. These rewards align agent objectives with empirically observed user motivations, enabling the study of how network structures and group formations emerge from individual decision-making. Our experiments show that coached LLM agents develop stable interaction patterns and form emergent social ties, yielding network structures that mirror properties of real online communities. By combining behavioral rewards with in-context adaptation, our framework establishes a principled testbed for investigating collective dynamics in LLM populations and reveals how artificial agents may approximate or diverge from human-like social behavior.

arXiv

Beyond-Diagonal RIS Under Non-Idealities: Learning-Based Architecture Discovery and Optimization

Binggui Zhou, Bruno Clerckx

Beyond-diagonal reconfigurable intelligent surface (BD-RIS) has recently been introduced to enable advanced control over electromagnetic waves to further increase the benefits of traditional RIS in enhancing signal quality and improving spectral and energy efficiency for next-generation wireless networks. A significant issue in designing and deploying BD-RIS is the tradeoff between its performance and circuit complexity. While existing studies have explored optimal architectures to minimize circuit complexity in ideal BD-RIS, architecture discovery for non-ideal BD-RIS remains uninvestigated. Consequently, how non-idealities and circuit complexity jointly affect the performance of BD-RIS remains unclear, making it difficult to achieve the performance-circuit complexity tradeoff in the presence of non-idealities. Essentially, architecture discovery for non-ideal BD-RIS faces challenges from both the computational complexity of global architecture search and the difficulty in achieving global optima. To tackle these challenges, we propose a learning-based two-tier architecture discovery framework (LTTADF) consisting of an architecture generator and a performance optimizer to jointly discover optimal architectures for non-ideal BD-RIS given specific circuit complexities, which can effectively explore over a large architecture space while avoiding getting trapped in poor local optima and thus achieving near-optimal solutions for the performance optimization. Numerical results provide valuable insights for deploying non-ideal BD-RIS considering the performance-circuit complexity tradeoff.

arXiv

HERAKLES: Hierarchical Skill Compilation for Open-ended LLM Agents

Thomas Carta, Clément Romac, Loris Gaven, Pierre-Yves Oudeyer, Olivier Sigaud, Sylvain Lamprier

We study goal-conditioned reinforcement learning in partially observable environments with sparse rewards and large, structured goal spaces. In such settings, complex goals often require composing simpler skills, but learning these compositions efficiently remains challenging. This difficulty is particularly relevant in open-ended exploration settings, where agents are exposed to increasingly complex goals over time and must continuously expand and reorganize their skill repertoire. We assume the goal space admits prerequisite relations, enabling latent decomposition of tasks into subgoals, and leverage language to represent and reason over these goals. To exploit this structure, we propose HERAKLES, a hierarchical agent that jointly learns a high-level LLM policy and a low-level controller. The high-level policy selects subgoals among those the low-level can reliably achieve, while the low-level executes them and progressively compiles successful behaviors into reusable skills. Both policies are trained concurrently: the high-level guides exploration and structures behavior, while the low-level distills trajectories into efficient goal-conditioned skills. As training progresses, more goals become directly executable, enabling scalable skill composition. This results in a hybrid system combining planning and execution, improving efficiency and adaptation in open-ended, compositional environments.

arXiv

Robust Belief-State Policy Learning for Quantum Network Routing Under Decoherence and Time-Varying Conditions

Amirhossein Taherpour, Abbas Taherpour, Tamer Khattab, Mazen Hasna

Quantum network routing requires online decisions under probabilistic entanglement generation, finite quantum memories, decoherence, imperfect operations, and classical feedback, while the controller has incomplete knowledge of the physical state. This paper develops a robust belief-state routing framework based on a quantum partially observable Markov decision process (q-POMDP) and a feasibility-masked graph neural network (GNN). The model uses atomic micro-epochs in which each selected operation completes before the next decision boundary. This enables explicit accounting of memory reservations, pair-instance inventories, purification consumption, swapping outcomes, release decisions, queue service, and completion-time delivery fidelity. The controller maintains a classical belief over hidden physical states, including latent environmental conditions, and uses this belief to evaluate feasible actions and update posterior pair states. To make planning scalable, we introduce feasibility-stratified prototypes, identifier-free signatures, and role-aware action matching, which preserve hard resource constraints while enabling value transfer across structurally similar information states. A cached q-POMDP planner is then fused with a role-aware GNN policy through an adaptive trust rule, with a safe fallback for previously unseen feasibility signatures. We provide theoretical guarantees on feasibility, value approximation, policy performance, robustness, regret, and learning variance. Simulations over finite-memory quantum-network topologies show that the proposed hybrid controller improves high-fidelity goodput, reduces below-threshold deliveries, and maintains lower online decision cost than planner-only control, while outperforming heuristic, purification-aware, and learning-based baselines.

arXiv

RoboInspector: Unveiling the Unreliability of Policy Code for LLM-enabled Robotic Manipulation

Chenduo Ying, Linkang Du, Peng Cheng, Yuanchao Shu

Large language models (LLMs) demonstrate remarkable capabilities in reasoning and code generation, enabling robotic manipulation to be initiated with just a single instruction. The LLM carries out various tasks by generating policy code required to control the robot. Despite advances in LLMs, achieving reliable policy code generation remains a significant challenge due to the diverse requirements of real-world tasks and the inherent complexity of user instructions. In practice, different users may provide distinct instructions to drive the robot for the same task, which may cause the unreliability of policy code generation. To bridge this gap, we design \textbf{RoboInspector}, a pipeline to unveil and characterize the unreliability of the policy code for LLM-enabled robotic manipulation from two perspectives: the complexity of the manipulation task and the granularity of the instruction. We perform comprehensive experiments with 216 distinct combinations of tasks, instructions, and LLMs in two prominent frameworks. The \textbf{RoboInspector} identifies four main unreliable behaviors that lead to manipulation failure. We provide a detailed characterization of these behaviors and their underlying causes, giving insight for practical development to reduce unreliability. Furthermore, we introduce a refinement approach guided by failure policy code feedback that improves the reliability of policy code generation by up to 35\% in LLM-enabled robotic manipulation, evaluated in both simulation and real-world environments.

arXiv

Assistax: A Multi-Agent Hardware-Accelerated Reinforcement Learning Benchmark for Assistive Robotics

Leonard Hinckeldey, Elliot Fosong, Rimvydas Rubavicius, Elle Miller, Trevor McInroe, Fan Zhang, Patricia Wollstadt, Stefano V. Albrecht, Subramanian Ramamoorthy

As embodied autonomous systems capable of assisting humans in daily activities remain a major goal for robotics, efficient and appropriate reinforcement learning (RL) simulation testbeds are increasingly important. Many common RL environments are too simple to provide insight into complex robotics domains, and many robotics simulations have throughput too low for RL. Very few simulators target multi-agent interactions: most treat the robot as an isolated agent, yet real-world tasks such as home assistance and caretaking are inherently multi-agent. Assistax addresses these limitations by providing a high-throughput, scalable suite of GPU-accelerated assistive robotics tasks built on JAX and MuJoCo-MJX, and includes an active humanoid agent as a simulated human partner, trainable alongside the robot using multi-agent RL (MARL). Beyond its use as a MARL benchmark, we formulate the human-robot interaction as an Ad-Hoc Teamwork (AHT) problem, where the robot's policy must generalise to unseen humans with varying disabilities and preferences. To this end, we provide an extensive AHT benchmarking pipeline: we use MARL to pre-train a diverse population of humanoid partners, and evaluate robot policies' ability to coordinate with a withheld set of humanoid policies. In contrast to other benchmarks, we also release reactive MARL-pre-trained humanoid policies via Hugging Face, enabling faster iteration in AHT research. With hardware acceleration, Assistax achieves up to 412$\times$ faster open-loop simulation than comparable CPU-based environments on a single GPU. Our AHT pipeline reveals a coordination gap for existing RL algorithms on unseen partners with novel preference combinations. This RL-native test suite for embodied multi-agent interaction provides a practical benchmark for advancing RL in assistive care. Code is available at: https://github.com/assistive-autonomy/assistax.

arXiv

A Geometry-Aware AI Emulator for the Coupled Whole Atmosphere from Earth Surface to the Ionosphere and Thermosphere

Jiahui Hu, Wenjun Dong

Whole-atmosphere models such as WACCM-X resolve coupling from the Earth surface to the Mesosphere-Lower-Thermosphere (MLT), and Ionosphere-Thermosphere (IT) systems with expensive computational costs. Here we introduce CAM-NET, a geometry-aware Spherical Fourier Neural Operator (SFNO) surrogate for emulating WACCM-X variability from Earth surface to IT region. CAM-NET is trained on 3-hourly WACCM-X simulations and predicts neutral winds, temperature, pressure-coordinate vertical velocity, electron density, and zonal ion drift. The framework combines a Spherical Fourier Neural Operator (SFNO) backbone with a newly developed lightweight module that extends the frozen atmospheric representation to plasma variables. For the held-out simulation, CAM-NET preserves the dominant IT morphology and remains stable during multi-day autoregressive rollouts. Spherical-harmonic diagnostics show that the model retains low-degree variability while damping high-wavenumber mesospheric structures, especially near 90 km where gravity wave breaks. CAM-NET is intended as a computationally efficient emulator of WACCM-X, rather than an operational forecasting system. These results demonstrate its potential for rapid ensemble experiments, uncertainty quantification, and sensitivity studies of large-scale coupled whole atmospheric variability.

arXiv

How Benchmark Prediction from Fewer Data Misses the Mark

Guanhua Zhang, Florian E. Dorner, Moritz Hardt

Large language model (LLM) evaluation is increasingly costly, prompting interest in methods that speed up evaluation by shrinking benchmark datasets. Benchmark prediction (also called efficient LLM evaluation) aims to select a small subset of evaluation points and predict overall benchmark performance from that subset. In this paper, we systematically assess the strengths and limitations of 11 benchmark prediction methods across 19 diverse benchmarks. First, we identify a highly competitive baseline: Take a random sample and fit a regression model on the sample to predict missing entries. Outperforming most existing methods, this baseline challenges the assumption that careful subset selection is necessary for benchmark prediction. Second, we discover that all existing methods crucially depend on model similarity. They work best when interpolating scores among similar models. The effectiveness of benchmark prediction sharply declines when new models have higher accuracy than previously seen models. In this setting of extrapolation, none of the previous methods consistently beat a simple average over random samples. To improve over the sample average, we introduce a new method inspired by augmented inverse propensity weighting. This method consistently outperforms the random sample average even for extrapolation. However, its performance still relies on model similarity and the gains are modest in general. This shows that benchmark prediction fails just when it is most needed: at the evaluation frontier, where the goal is to evaluate new models of unknown capabilities.

Keep browsing

More papers from top AI conferences

30+ top conferences, by conference, year, and topic.