PAC-Bayes-Empirical-Bernstein InequalityPAC-Bayes-Empirical-Bernstein InequalityTolstikhin, Ilya O. and Seldin, Yevgeny2013

Paper summarynipsreviewsThis paper derives a new empirical PAC Bayesian bound by combining an existing (non-empirical) PAC Bayesian Berstein bound (i.e., involving the true variance of the loss values) with a PAC Bayesian analysis of the concentration of the empirical variance around its true value. This new bound has the advantage of being tighter when the empirical variance is small compared to the empirical loss. Experiments on real and empirical data with simple models compare the new bound with the usual empirical PAC Bayesian bound confirming the advantage.

This paper derives a new empirical PAC Bayesian bound by combining an existing (non-empirical) PAC Bayesian Berstein bound (i.e., involving the true variance of the loss values) with a PAC Bayesian analysis of the concentration of the empirical variance around its true value. This new bound has the advantage of being tighter when the empirical variance is small compared to the empirical loss. Experiments on real and empirical data with simple models compare the new bound with the usual empirical PAC Bayesian bound confirming the advantage.