Two Lenses on Deep Learning: Data Reconstruction and Transformer Structure

Despite the remarkable success of modern deep learning, our theoretical understanding remains limited. Many fundamental questions about how these models learn, what they memorize, and what their architectures can express are still largely open. In this talk, I focus on two such questions that offer complementary perspectives on the behavior of modern networks.

First, I examine how standard training procedures implicitly encode aspects of the training data in the learned parameters, enabling reconstruction across a wide range of architectures and loss functions. Second, I turn to transformers and analyze how architectural choices, such as the number of heads, rank, and depth, shape their expressive capabilities, revealing both strengths and inherent limitations of low-rank attention.

Together, these perspectives highlight recurring principles that shape the behavior of deep models, bringing us closer to a theoretical framework that can explain and predict the phenomena observed in practice.

Bio: Gilad is a postdoctoral research associate at the Courant Institute of Mathematical Sciences at New York University, hosted by Prof. Joan Bruna. His research focuses on the theory of deep learning models, with a recent emphasis on transformers. He also works on attacks on neural networks, particularly data reconstruction attacks and adversarial attacks. Previously, he completed his Ph.D. at the Weizmann Institute of Science under the supervision of Ohad Shamir. He has held research internships at NVIDIA, where he worked on graph neural networks, and at Google Research, where he worked on large-scale optimization. He holds a B.Sc. and M.Sc. in mathematics from Tel Aviv University.