Seminar: Graduate Seminar
PAC-Bayes theory and practice for novel deep network parametrizations
Date:
September,09,2024
Start Time:
13:30 - 14:30
Location:
1061, Meyer Building
Add to:
Lecturer:
Guy Sassy
Research Areas:
| Our research explores advancements in understanding neural network generalization through PAC-Bayesian analysis, which provides a probabilistic framework for deriving generalization bounds. These bounds are split into empirical bounds, derived from observed data, and oracle bounds, which assume an idealized oracle with perfect knowledge. Our first contribution is a novel oracle bound for the density estimation learning problem, which shows a reduced dimensional dependency for Gaussian data under the submanifold assumption. The second contribution is practical and is twofold: First, we extend the work of Perez et al. 2020 by applying the PAC-Bayes with backprop (PBB) algorithm to language models. We show that by incorporating PAC-Bayes learning to language models, validation and test splits can be merged back to be trained on, as the certificate can be used as model selection metric. Moreover, an ensemble model can be created by sampling several models from the posterior to provide an uncertainty metric over the generated tokens. Second, we extend the framework of the original PBB such that the parametric posterior model is a non-diagonal Gaussian. By extending the framework to a block-diagonal parameterized posterior, we provide superior generalization abilities with high probability than diagonal PBB. This is done without compromising a lot of computation time and memory footprint by incorporating compact sparse storage and parallel computation through the weights tensors. M.Sc. student under the supervision of Prof. Ron Meir and Prof. Aviv Tamar.
|