摘要
While Adam is one of the most effective optimizer for training large-scale machine learning models, a theoretical understanding of how to optimally set its momentum factors, β_1 and β_2, remains largely incomplete. Prior works have shown that Adam can be seen as an instance of Follow-the-Regularized-Leader (FTRL), one of the most important class of algorithms in online learning. The prior analyses in these works required setting β_1 = √(β_2), which does not cover the more practical cases with β_1 ≠√(β_2). We derive novel, more general analyses that hold for both β_1 ≥√(β_2) and β_1 ≤√(β_2). In both cases, our results strictly generalize the existing bounds. Furthermore, we show that our bounds are tight in the worst case. We also prove that setting β_1 = √(β_2) is optimal for an oblivious adversary, but sub-optimal for an non-oblivious adversary.