Diffusion Probabilistic Models in Time-Dependent Domains and Their Spectral Analysis

Diffusion models have emerged as powerful tools for modeling complex data distributions and generating realistic new samples. In this seminar, we explore the application of diffusion models to time-dependent signals, such as speech and audio, and investigate their underlying phenomena and design choices from a spectral perspective. We begin by introducing a text-to-speech diffusion model operating directly on the raw speech waveform, departing from conventional approaches that rely on intermediate representations. This motivates a broader investigation of the spectral behavior of diffusion models and, in particular, the frequency distribution of the generated samples. In the second part, we introduce a spectral perspective that describes the diffusion process through a frequency-dependent transfer function of the input noise. This formulation provides a theoretical basis for design choices that are commonly made heuristically, particularly noise-schedule design, and sheds light on the spectral bias associated with such heuristics. Finally, we extend our spectral analysis to zero-shot posterior sampling for inverse problems, where diffusion models serve as learned generative priors. This framework enables a unified analysis of different posterior samplers and provides principled recommendations for their guidance heuristics. We illustrate these ideas in real-world applications such as accelerated MRI, where improved reconstruction can reduce acquisition time.

Ph.D. student Under the supervision of Prof. Yossi Keshet.