Sharp Representation Theorems for ReLU Networks with Precise Dependence on Depth

Abstract

We prove dimension free representation results for neural networks with D ReLU layers under square loss for a class of functions G_D defined in the paper. These results capture the precise benefits of depth in the following sense: