Multiple Descent of Generalization Curve for Optimally Regularized Ridge Regression

Abstract

It is known that optimally regularized ridge regression can exhibit non-monotone dependence of the generalization risk on the sample size. We show that this behavior can be arbitrarily intricate in finite dimensions: for every $d \ge 2$, there exists a random design linear model in $\mathbb{R}^d$ whose generalization risk, viewed as a function of the sample size, exhibits at least $d - 1$ local maxima.