AI conference papers

AI conference paper library

Search papers from 30+ top conferences, including NeurIPS, ICLR, ICML, CVPR, ACL and AAAI. Filter by keyword, then open any paper for its abstract, authors and links.

arXiv

Generalization from Low- to Moderate-Resolution Spectra with Neural Networks for Stellar Parameter Estimation: A Case Study with DESI

Xiaosheng Zhao, Yuan-Sen Ting, Rosemary F. G. Wyse, Alexander S. Szalay, Yang Huang, László Dobos, Tamás Budavári, Viska Wei

Cross-survey generalization is a critical challenge in stellar spectral analysis, particularly in cases such as transferring from low- to moderate-resolution surveys. We investigate this problem using pre-trained models, focusing on simple neural networks such as multilayer perceptrons (MLPs), with a case study transferring from LAMOST low-resolution spectra (LRS) to DESI medium-resolution spectra (MRS). Specifically, we pre-train MLPs on either LRS or their embeddings and fine-tune them for application to DESI stellar spectra. We compare MLPs trained directly on spectra with those trained on embeddings derived from transformer-based models (self-supervised foundation models pre-trained for multiple downstream tasks). We also evaluate different fine-tuning strategies, including residual-head fine-tuning, LoRA, and full fine-tuning. We find that MLPs pre-trained on LAMOST LRS achieve strong performance, even without fine-tuning, and that modest fine-tuning with DESI spectra further improves the results. For iron abundance, embeddings from a transformer-based model yield advantages in the metal-rich ([Fe/H] > -1.0) regime, but underperform in the metal-poor regime compared to MLPs trained directly on LRS. We also show that the optimal fine-tuning strategy depends on the specific stellar parameter under consideration. These results highlight that simple pre-trained MLPs can provide competitive cross-survey generalization, while the role of spectral foundation models for cross-survey stellar parameter estimation requires further exploration.

arXiv

Gradient Networks for Universal Magnetic Modeling of Synchronous Machines

Junyi Li, Tim Foissner, Floran Martin, Antti Piippo, Marko Hinkkanen

This paper presents a physics-constrained neural network framework for dynamic modeling of saturable synchronous machines, including spatial harmonics. The proposed architecture embeds gradient networks directly into the fundamental machine equations to model nonlinear, coupled electromagnetic behavior. By learning the gradient of magnetic field energy, the model satisfies reciprocity and energy-balance constraints by construction. The approach can universally approximate any physically feasible magnetic characteristics while offering key advantages over lookup tables and conventional black-box networks: monotonicity, smooth outputs, and improved generalization from limited data. These properties also support robust model inversion and trajectory optimization for control. The method is validated using measured and finite-element-method (FEM) data from a 5.6-kW permanent-magnet (PM) synchronous reluctance machine and is further demonstrated experimentally in real-time closed-loop operation on an embedded control platform. The results show accurate and physically consistent modeling performance, even with limited training data.

arXiv

Breaking the Curse of Repulsion: Remoteness-Aware Control of Negative Off-Policy Updates

Yusen Huo, Changping Wang, Yangru Huang, Jun Zhang, Jie Jiang

Off-policy policy optimization reuses historical behavior, including negative-advantage samples that suppress known failures. We show that repeated reuse can turn this useful signal into excessive repulsion: as the learner moves away from a historical negative action, subsequent updates make that action increasingly remote without necessarily reducing its update strength. Our aggregate theory characterizes the resulting transition from a stable displacement beyond the positive-only target to persistent drift and the loss of finite stable equilibria; controlled strength sweeps show that an intermediate displacement can improve held-out reward. The relevant learner-relative coordinate is squared standardized distance for Gaussian policies and surprisal for categorical policies. We introduce Dynamic Remoteness-Aware Policy Optimization (DRPO), which leaves the negative update unchanged in the near field and exponentially attenuates its remote tail. DRPO restores eventual inward Gaussian drift for every fixed finite negative-to-positive mass ratio, yields an explicit ultimate-bound radius, and changes categorical support suppression from exponential to polynomial probability decay. External diagnostics and controlled interventions isolate remoteness-dependent policy geometry as a source of negative-update amplification and show that selective tapering can remove its far-field effect without discarding useful local feedback.

arXiv

Linear-LLM-SCM: Benchmarking LLMs for Coefficient Elicitation in Linear-Gaussian Causal Models

Kanta Yamaoka, Sumantrak Mukherjee, Thomas Gärtner, David Antony Selby, Stefan Konigorski, Eyke Hüllermeier, Viktor Bengs, Sebastian Josef Vollmer

Large language models (LLMs) have shown potential in identifying qualitative causal relations, but their ability to perform quantitative causal reasoning---estimating effect sizes that parametrize functional relationships---remains underexplored in continuous domains. We introduce Linear-LLM-SCM, a plug-and-play framework for evaluating LLMs on Linear Gaussian structural causal model parametrization when a directed acyclic graph (DAG) is given. The framework decomposes a DAG into local parent-child sets and prompts an LLM to produce a regression-style structural equation per node, which is aggregated and compared against available ground-truth parameters. Our experiments with seven real-world DAGs effect ground truth illustrate limitations of LLMs as quantitative causal parameterizers. Across most models, we observe variability in coefficient estimates and sensitivity to structural perturbations. We open-sourced the framework to further encourage the community to work on studies toward the use of LLM for causal effect elicitation in safety-critical domain, e.g., healthcare.

arXiv

SwiftRepertoire: Few-Shot Immune-Signature Synthesis via Dynamic Kernel Codes

Rong Fu, Yang Li, Yabin Jin, Jiekai Wu, Chunlei Meng, Youjin Wang

Repertoire-level analysis of T cell receptors offers a biologically grounded signal for disease detection and immune monitoring, yet practical deployment is impeded by label sparsity, cohort heterogeneity, and the computational burden of adapting large encoders to new tasks. We introduce a framework that synthesizes compact task-specific parameterizations from a learned dictionary of prototypes conditioned on lightweight task descriptors derived from repertoire probes and pooled embedding statistics. This synthesis produces small adapter modules applied to a frozen pretrained backbone, enabling immediate adaptation to novel tasks with only a handful of support examples and without full model fine-tuning. The architecture preserves interpretability through motif-aware probes and a calibrated motif discovery pipeline that links predictive decisions to sequence-level signals. Together, these components yield a practical, sample-efficient, and interpretable pathway for translating repertoire-informed models into diverse clinical and research settings where labeled data are scarce and computational resources are constrained.

arXiv

HeatACO: A Heatmap-Guided Max--Min Ant System for Large-Scale Travelling Salesman Problems

Bo-Cheng Lin, Yi Mei, Mengjie Zhang

Non-autoregressive neural solvers predict an edge-confidence heatmap for the Travelling Salesman Problem (TSP) in one forward pass, but a decoder must still produce a feasible Hamiltonian cycle. As instance size grows, this stage must reconcile a quadratic number of edge scores with global tour constraints. Greedy edge merging is fast and deterministic but produces low-quality tours, whereas Monte Carlo Tree Search (MCTS) over k-opt moves recovers better tours at high computational cost and requires predictor-specific tuning. We propose HeatACO, a predictor-agnostic heatmap-to-tour decoder. Its key is a capped, degree-aware evidence factor that integrates a fixed heatmap into a Max--Min Ant System (MMAS). The factor rewards only edge confidence beyond a node's tour-degree capacity, and its strength is scaled automatically from the pheromone dynamic range, allowing one configuration to decode heatmaps from different predictors without retraining or per-predictor tuning. Across four heatmap sources, HeatACO produces higher-quality solutions in less decoding time than the MCTS baseline on TSP500, TSP1K and TSP10K. Against matched standard MMAS baselines with the same search budget, heatmap guidance improves construction for all four predictors at both scales and remains beneficial with local search. HeatACO also transfers competitively to several distribution shifts and the asymmetric TSP (ATSP). Our post-hoc analysis identifies measurable heatmap properties associated with the observed performance variation.

arXiv

Dual-Prototype Disentanglement: A Context-Aware Enhancement Framework for Time Series Forecasting

Haonan Yang, Yulin HE, Jianchao Tang, Zhuo Li

Real-world time series are governed by both recurring structures, such as trends and seasonality, and infrequent yet critical variations, such as abrupt shifts and rare events. However, existing methods often lack an explicit mechanism to organize and utilize these heterogeneous patterns according to their distinct forecasting roles. Consequently, common and rare patterns can become entangled, preventing models from dynamically distinguishing and selectively leveraging them according to context. To address this issue, we propose Dual-Prototype Adaptive Disentanglement (DPAD), a model-agnostic framework that organizes temporal patterns by their forecasting roles. Specifically, we construct a Dynamic Dual-Prototype bank (DDP), comprising a common pattern bank initialized with structured temporal priors to represent prevalent dynamics, and a rare bank that adaptively memorizes infrequent deviations. Then a Dual-Path Context-aware routing (DPC) mechanism enhances outputs with selectively retrieved context-specific pattern representations from DDP. A Disentanglement-Guided Loss (DGLoss) is further introduced to ensure that each prototype bank specializes in its designated role while maintaining sufficient coverage. Extensive experiments across diverse real-world benchmarks demonstrate that DPAD consistently improves the forecasting performance of a range of time-series models.

arXiv

Benchmarking Deep Learning Models for Raman Spectroscopy Across Open-Source Datasets

Adithya Sineesh, Akshita Ramya Kamsali

Deep learning classifiers for Raman spectroscopy are increasingly reported to outperform classical chemometric approaches. However, their evaluations are often conducted in isolation or compared against traditional machine learning methods or trivially adapted vision-based architectures that were not originally proposed for Raman spectroscopy. As a result, direct comparisons between existing deep learning models developed specifically for Raman spectral analysis on shared open-source datasets remain scarce. In this work, we focus on supervised Raman spectra classification where each spectrum is assigned to a predefined material, bacterial/yeast isolate, drug treatment or pharmaceutical compound. To the best of our knowledge, this study presents one of the first benchmarks comparing three or more published Raman-specific deep learning classifiers across multiple open-source Raman datasets. We evaluate five representative Deep Learning (DL) architectures along with two conventional Machine Learning (ML) methods under a unified training and hyperparameter tuning protocol across three open-source Raman datasets selected to support standard evaluation, fine-tuning, and explicit distribution-shift testing. In this comparative study, we primarily focus on classification because the selected open-source datasets provide classification annotations, while annotations for complete structure elucidation are not available. We report classification accuracies and macro-averaged F1 scores to provide a fair and reproducible comparison of the supervised ML and DL models for Raman spectra based classification.

arXiv

Dataset Poisoning Attacks on Behavioral Cloning Policies

Akansha Kalra, Soumil Datta, Ethan Gilmore, Duc La, Guanhong Tao, Daniel S Brown

Behavior Cloning (BC) is a popular framework for training sequential decision policies from expert demonstrations via supervised learning. As these policies are increasingly being deployed in the real world, their robustness and potential vulnerabilities are an important concern. In this work, we perform the first analysis of the efficacy of clean-label backdoor attacks on BC policies. Our backdoor attacks poison a dataset of demonstrations by injecting a visual trigger to create a spurious correlation that can be exploited at test time. We evaluate how policy vulnerability scales with the fraction of poisoned data, the strength of the trigger, and the trigger type. We also introduce a novel entropy-based test-time trigger attack that substantially degrades policy performance by identifying critical states where test-time triggering of the backdoor is expected to be most effective at degrading performance. We empirically demonstrate that BC policies trained on even minimally poisoned datasets exhibit deceptively high, near-baseline task performance despite being highly vulnerable to backdoor trigger attacks during deployment. Our results underscore the urgent need for more research into the robustness of BC policies, particularly as large-scale datasets are increasingly used to train policies for real-world cyber-physical systems. Videos and code are available at https://sites.google.com/view/dataset-poisoning-in-bc.

arXiv

Online learning of subgrid-scale models for quasi-geostrophic turbulence in planetary interiors

Hugo Frezat, Thomas Gastine, Alexandre Fournier

Machine learning approaches to subgrid-scale (SGS) modelling are now well established in atmospheric and oceanic applications. Among these, online end-to-end learning, where the differentiable solver participates in the training, has shown particular promise. Yet, existing studies are largely restricted to idealised periodic domains, with no mechanical boundaries, precluding them from addressing the dynamics of bounded rotating flows relevant to planetary interiors. Here we consider two-dimensional quasi-geostrophic turbulence in a rapidly rotating annular bounded domain. We examine three configurations varying the geometry of the container and the rotation rate. The system exhibit key features such as zonal jets, Rossby waves, and, in the spherical shell geometry, a slow quasi-periodic inward drift of the jets. The spectral properties of the zonal and non-zonal flow can be understood in the framework of zonostrophic turbulence theory. We develop a differentiable solver for this system, which allows us to train SGS models online, over a time span of one turnover time, using coarse-grained data from direct numerical simulations. In all cases, a SGS model trained on a single turnover time accurately reproduce global integrated diagnostics, energy spectra as well as long-term dynamical behaviours ---such as jet migration--- occurring on timescales which exceed the training window by one order of magnitude. The online-trained model further outperforms classical hyperdiffusivity and Leith closure schemes, for which reducing the radial resolution is impractical. The resulting speed-up paves the way to further investigations of long-term geophysical fluid processes beyond the reach of direct numerical simulations.

arXiv

DeepVRegulome: DNABERT-based deep-learning framework for predicting the functional impact of short genomic variants on the human regulome

Pratik Dutta, Matthew Obusan, Rekha Sathian, Max Chao, Pallavi Surana, Nimisha Papineni, Yanrong Ji, Zhihan Zhou, Han Liu, Alisa Yurovsky, Ramana V Davuluri

Whole-genome sequencing (WGS) has revealed numerous non-coding short variants whose functional impacts remain poorly understood. Despite recent advances in deep-learning genomic approaches, accurately predicting and prioritizing clinically relevant mutations in gene regulatory regions remains a major challenge. We developed DeepVRegulome, a computational framework integrating 464 fine-tuned DNABERT models (458 transcription factor, 4 histone mark, and 2 splice site models) trained on ENCODE and GENCODE datasets. The framework pairs these deep learning models with a suite of analytical tools: quantitative variant scoring via log-odds ratios to assess functional impact, attention-based motif analysis to identify disrupted sequence patterns, and survival analysis using Kaplan-Meier and Cox proportional hazards models to link high-impact variants with clinical outcomes. To ensure the framework accurately captures variant effects on baseline binding status, we benchmarked DeepVRegulome against an independent experimental assay of allele-specific transcription factor binding (SNP-SELEX) data and compared its performance to four established variant-effect predictors. The analysis identified 572 splice-disrupting and 9,837 transcription-factor binding site-altering mutations occurring in greater than 10 percentage of glioblastoma samples. Survival analysis linked 1352 mutations and 563 disrupted regulatory regions to patient outcomes, enabling stratification via non-coding mutation signatures. All the code, fine-tuned models, and an interactive data portal are publicly available.

arXiv

A Support-Set Algorithm for Optimization Problems with Nonnegative and Orthogonal Constraints

Lei Wang, Xin Liu, Xiaojun Chen

In this paper, we investigate optimization problems with nonnegative and orthogonal constraints, where any feasible matrix of size $n \times p$ exhibits a sparsity pattern such that each row accommodates at most one nonzero entry. Our analysis demonstrates that, by fixing the support set, the global solution of the minimization subproblem for the proximal linearization of the objective function can be computed in closed form with at most $n$ nonzero entries. Exploiting this structural property offers a powerful avenue for dramatically enhancing computational efficiency. Guided by this insight, we propose a support-set algorithm preserving strictly the feasibility of iterates. A central ingredient is a strategically devised update scheme for support sets that adjusts the placement of nonzero entries. We establish the convergence of the support-set algorithm to a first-order stationary point, and show that its iteration complexity required to reach an $ε$-approximate first-order stationary point is $O (ε^{-2})$. Numerical results are strongly in favor of our algorithm in real-world applications, including nonnegative PCA, clustering, and community detection.

arXiv

Extreme Event Aware ($η$-) Learning

Kai Chang, Themistoklis P. Sapsis

Quantifying and predicting rare and extreme events is challenging because such events are infrequent, severe, and expensive to simulate. Existing data-driven methods often require multiple extremes in the training data or sampling process, leading to accurate predictions in quiescent regimes but high epistemic uncertainty in extreme-event regions. To overcome this limitation, we introduce Extreme Event Aware ($η$-) Learning, which does not require extreme events in the available data. The method reduces uncertainty even in uncharted extreme regimes by enforcing during training the statistics of an observable indicative of extremeness, obtained from qualitative knowledge or unlabeled data. This statistical regularization results in models that fit observed data while remaining consistent with prescribed observable statistics, enabling the generation of unprecedented extreme events. Optimal-transport-based theoretical results offer rigorous justification and establish key optimality properties. Numerical experiments on prototype systems and real-world precipitation downscaling problems demonstrate the effectiveness of the $η$-learning framework.

arXiv

Towards real-time surrogate-free Bayesian inversion for neutron reflectometry

Max D. Champneys, Andrew J. Parnell, Philipp Gutfreund, Maximilian W. A. Skoda, Patrick A. Fairclough, Timothy J. Rogers, Stephanie L. Burg

Neutron reflectometry (NR) is a key enabling technology for many areas of scientific development. Although the forward reflectivity model is well-known, inferring the physical properties of a sample from NR data requires the solution of an inverse problem. Increasingly, beamline scientists are using NR in fast kinetic configurations and probing highly-complex structures and interfaces, introducing significant uncertainty. Existing uncertainty quantification (UQ) approaches in NR, such as Markov-Chain Monte-Carlo (MCMC), suffer from poor sample efficiency and slow convergence times. Recently, surrogate machine learning models have been proposed as an alternative. However, physical intuition is lost when replacing governing equations with fast surrogates. Instead, we propose a rapid, surrogate-free Bayesian inversion for NR. Our approach offers a step-change in inference speed and efficiency. For the first time in NR, exact gradients through the reflectivity are computed, enabling highly performant gradient-based inference schemes: Hamiltonian Monte-Carlo offers significant advances in sample efficiency compared to MCMC. Variational inference enables approximate UQ on the order of seconds rather than hours. We demonstrate state-of-the-art performance on a thick oxide quartz film, and robust co-fitting performance in the high complexity regime of organic LED multilayer devices. Additionally, we provide an open-source library of reflectometry kernels in the python language.

arXiv

Machine Intelligence on the Edge: Interpretable Cardiac Pattern Localisation Using Reinforcement Learning

Haozhe Tian, Qiyu Rao, Nina Moutonnet, Pietro Ferraro, Danilo Mandic

Matched filters are widely used to localise signal patterns due to their high efficiency and interpretability. However, their effectiveness deteriorates for low signal-to-noise ratio (SNR) signals, such as those recorded on edge devices, where prominent noise patterns can closely resemble the target within the limited length of the filter. One example is the ear-electrocardiogram (ear-ECG), where the cardiac signal is attenuated and heavily corrupted by artefacts. To address this, we propose the Sequential Matched Filter (SMF), a paradigm that replaces the conventional single matched filter with a sequence of filters designed by a Reinforcement Learning agent. By formulating filter design as a sequential decision-making process, SMF adaptively design signal-specific filter sequences that remain fully interpretable by revealing key patterns driving the decision-making. The proposed SMF framework has strong potential for reliable and interpretable clinical decision support, as demonstrated by its state-of-the-art R-peak detection and physiological state classification performance on two challenging real-world ECG datasets. The proposed formulation can also be extended to a broad range of applications that require accurate pattern localisation from noise-corrupted signals.

arXiv

Machine Learning the H-theorem

Ruben Lier

The H-theorem provides a microscopic foundation for the Second Law of Thermodynamics and therefore occupies a central place in statistical physics. At the same time, its relation to microscopic reversibility has remained conceptually subtle. To investigate how an arrow of time may be inferred directly from microscopic data, we study the relaxation of randomly initialized hard disks in a periodic box. We construct a permutation-invariant neural network based on the DeepSets architecture. The model is trained only to assign later states a larger scalar value than earlier states. We compare the learned scalar with the Boltzmann H-functional and assess to what extent the dynamics alone lead the model toward the structure implied by the H-theorem.

arXiv

Annotation-Assisted Learning of Treatment Policies From Multimodal Electronic Health Records

Henri Arno, Thomas Demeester

We study how to learn treatment policies from multimodal electronic health records (EHRs) that consist of tabular data and clinical text. These policies can help physicians make better treatment decisions and allocate healthcare resources more efficiently. Causal policy learning methods prioritize patients with the largest expected treatment benefit. Yet, existing estimators are designed for tabular covariates under causal assumptions that may be hard to justify in the multimodal setting. A pragmatic alternative is to apply causal estimators directly to multimodal representations, but this can produce biased treatment effect estimates when the representations do not preserve the relevant confounding information. As a result, predictive models of baseline risk are commonly used in practice to guide treatment decisions, although they are not designed to identify which patients benefit most from treatment. We propose AACE (Annotation-Assisted Coarsened Effects), an annotation-assisted approach to causal policy learning for multimodal EHRs. The method uses expert-provided annotations during training to support confounding adjustment, and then predicts treatment benefit from only multimodal representations at inference. We show that the proposed method achieves strong empirical performance across synthetic, semi-synthetic, and real-world EHR datasets, outperforming risk-based and representation-based causal baselines, and offering practical insights for applying causal machine learning in clinical practice.

arXiv

TaylorPODA: A Taylor Expansion-Based Method to Improve Post-Hoc Attributions for Opaque Models

Yuchi Tang, Iñaki Esnaola, George Panoutsos

Post-hoc model-agnostic local attribution (LA) methods have been widely adopted to explain opaque AI models by quantifying feature-wise contributions. However, many existing methods rely on heuristic or only partially justified attribution mechanisms, while the quality of attribution itself is often shaped by downstream objectives without universally accepted standards. In this work, we propose Taylor exPansion-Originated aDaptive Attribution (TaylorPODA), a new post-hoc model-agnostic LA method grounded in the Taylor expansion framework. We first introduce a set of postulates, which formalize principled requirements for explicitly and exhaustively attributing Taylor terms to the corresponding features. Based on these postulates, we analyze existing post-hoc model-agnostic LA methods and identify a fundamental tension between principled attribution and adaptation toward user-defined utilities. To address this challenge, TaylorPODA introduces a controllable allocation mechanism for Taylor interaction effects, enabling attribution results to adapt to downstream objectives while preserving the proposed postulates. Furthermore, although developed from a Taylor-expansion perspective, TaylorPODA also admits a Harsanyi-dividend interpretation, allowing the attribution mechanism to extend beyond model differentiability. Theoretical analysis demonstrates that TaylorPODA satisfies all the proposed postulates together with an additional adaptation property. Empirical results across multiple datasets and both differentiable and non-differentiable models further show that TaylorPODA achieves consistently improved alignment with user-defined utilities while maintaining the communicability of the resulting explanations. Overall, this work provides a starting point toward more trustworthy XAI systems for the deployment of increasingly powerful yet opaque task models.

arXiv

Diffusion Disambiguation Models for Partial Label Learning

Jinfu Fan, Xiaohui Zhong, Kangrui Ren, Jiangnan Li, Linqing Huang, Min Gan, C. L. Philip Chen

Learning from ambiguous labels is a long-standing problem in practical machine learning applications. The purpose of \emph{partial label learning} (PLL) is to identify the ground-truth label from a set of candidate labels associated with a given instance. Inspired by the remarkable performance of diffusion models in various generation tasks, this paper explores their potential to denoise ambiguous labels through the reverse denoising process. Therefore, this paper reformulates the label disambiguation problem from the perspective of generative models, where labels are generated by iteratively refining initial random guesses. This perspective enables the diffusion model to learn how label information is generated stochastically. By modeling the generation uncertainty, we can use the maximum likelihood estimate of the label for classification inference. However, such ambiguous labels lead to a mismatch between instance and label, which reduces the quality of generated data. To address this issue, this paper proposes a \emph{diffusion disambiguation model for PLL} (DDMP), which first uses the potential complementary information between instances and labels to construct pseudo-clean labels for initial diffusion training. Furthermore, a transition-aware matrix is introduced to estimate the potential ground-truth labels, which are dynamically updated during the diffusion generation. During training, the ground-truth label is progressively refined, improving the classifier. Experiments show the advantage of the DDMP and its suitability for PLL.

arXiv

On the Convergence Analysis of Muon

Wei Shen, Ruichuan Huang, Minhui Huang, Cong Shen, Jiawei Zhang

The majority of parameters in neural networks are naturally represented as matrices. However, most commonly used optimizers treat these matrix parameters as flattened vectors during optimization, potentially overlooking their inherent structural properties. Recently, an optimizer called Muon has been proposed, specifically designed to optimize matrix-structured parameters. Extensive empirical evidence shows that Muon can significantly outperform traditional optimizers when training neural networks. Nonetheless, the theoretical understanding of Muon's convergence behavior and the reasons behind its superior performance remain limited. In this work, we present a comprehensive convergence rate analysis of Muon and its comparison with Gradient Descent (GD). We characterize the conditions under which Muon can outperform GD. Our theoretical results reveal that Muon can benefit from the low-rank structure of Hessian matrices, a phenomenon widely observed in practical neural network training. Our experimental results support and corroborate the theoretical findings.

Keep browsing

More papers from top AI conferences

30+ top conferences, by conference, year, and topic.