Seminar: Machine Learning Seminar
Bayesian Approaches in Deep Learning: Theory and Applications in Continual Learning and Denoising
Date:
August,14,2024
Start Time:
10:30 - 11:30
Location:
1061, Meyer Building
Add to:
Lecturer:
Chen Zeno
Research Areas:
| In recent years, Bayesian approaches have played a significant role in advancing deep learning, particularly through the use of prior probability models. One key application is the use of neural network denoisers to solve inverse problems and generate images. In image generation, neural network denoisers estimate the score function of the perturbed distribution. The connection between the score function and the denoiser is mathematically valid only for the optimal Bayes estimator (the MMSE denoiser). However, for the estimated score to be a gradient field, the denoiser must have a symmetric Jacobian matrix. In practice, the Jacobian matrix of neural network denoisers is generally non-symmetric, which leaves open questions about why the sampling process in score-based diffusion algorithms still works effectively. To better understand these complex applications, we focus on shallow ReLU neural network denoisers. We analyze their function space characteristics in the context of interpolation with minimal L2 norm weights (minimal representation cost). The study provides theoretical insights and closed-form solutions for both univariate and multivariate data scenarios, demonstrating their efficacy in function approximation and generalization. Furthermore, we investigate the score flow process of shallow ReLU denoisers, particularly in interpolation scenarios with minimal representation costs. We explore when the score flow converges to training samples versus general points on the data manifold, shedding light on their stability and convergence properties through empirical and analytical approaches. |
| Chen Zeno is a Ph.D. student under the supervision of Prof. Daniel Soudry.
|