Advances in PAC-Bayesian certification of deep neural networks: tighter closed-form inequalities and optimization of bounds on non-differentiable losses

摘要

This paper presents three theoretical contributions that improve the usability of risk certificates for neural networks based on PAC-Bayes bounds. First, two bounds on the KL divergence between Bernoulli distributions enable the derivation of the tightest explicit bounds on the true risk of classifiers across different ranges of empirical risk. Then, a novel method to optimize bounds on non-differentiable objectives is introduced, allowing to optimize risk bounds directly on the 0-1 loss instead of resourcing to a surrogate differentiable loss, such as the bounded cross-entropy. These theoretical contributions are illustrated with an empirical evaluation on the MNIST and CIFAR-10 datasets. In fact, this paper presents the first non data-dependent generalization bounds on the 0-1 loss for neural networks fitted on CIFAR-10.