AutoML 2023 Papers

We introduce AutoGluon–TimeSeries—an open-source AutoML library for probabilistic time series forecasting. Focused on ease of use and robustness, AutoGluon-TimeSeries enables users to generate accurate point and quantile forecasts with just 3 lines of Python code. Built on the design philosophy of AutoGluon, AutoGluon–TimeSeries leverages ensembles of diverse forecasting models to deliver high accuracy within a short training time. AutoGluon–TimeSeries combines both conventional statistical models, machine-learning based forecasting approaches, and ensembling techniques. In our evaluation on 29 benchmark datasets, AutoGluon–TimeSeries demonstrates strong empirical performance, outperforming a range of forecasting methods in terms of both point and quantile forecast accuracy, and often even improving upon the best-in-hindsight combination of prior methods.

We present AlphaD3M, an open-source Python library that supports a wide range of machine learning tasks over different data types. We discuss the challenges involved in supporting multiple tasks and how AlphaD3M addresses them by combining deep reinforcement learning and meta-learning to effectively construct pipelines over a large collection of primitives. To better integrate the use of AutoML within the data science lifecycle, we have built an ecosystem of tools around AlphaD3M that support user-in-the loop tasks, including the selection of suitable pipelines and the development of solutions for complex systems. We present use cases that demonstrate some of these features. We report the results of detailed experimental evaluations which show that AlphaD3M is effective and derives high-quality pipelines for a diverse set of problems with performance that is comparable or superior to state-of-the-art AutoML systems.

Understanding the efficacy of a method requires ablation experiments. Current Machine Learning (ML) workflows emphasize the vertical scaling of large models with paradigms such as ‘data-parallelism’ or ‘model-parallelism’. As a consequence, there is a lack of methods for horizontal scaling of multiple experimental trials. Horizontal scaling is labor intensive when different tools are used for different experiment stages, such as for hyper-parameter optimization, distributed execution, or the consolidation of artifacts. We identify that errors in earlier stages of experimentation propagate to the analysis. Based on our observations, experimental results, and the current literature, we provide recommendations on best practices to prevent errors. To reduce the effort required to perform an accurate analysis and address common errors when scaling the execution of multiple experiments, we introduce ABLATOR. Our framework uses a stateful experiment design paradigm that provides experiment persistence and is robust to errors. Our actionable analysis artifacts are automatically produced by the experiment state and reduce the time to evaluate a hypothesis. We evaluate ABLATOR with ablation studies on a Transformer model, ‘Tablator’, where we study the effect of 6 architectural components, 8 model hyperparameters, 3 training hyperparameters, and 4 dataset preprocessing methodologies on 11 tabular datasets. We performed the largest ablation experiment for tabular data on Transformer models to date, evaluating 2,337 models in total. Finally, we open source ABLATOR; \url{https://github.com/fostiropoulos/ablator}