סמינר: Machine Learning Seminar
Symmetry and Inductive Bias in Transformers: A Kernel Perspective
Date:
January,08,2025
Start Time:
11:30 - 12:30
Location:
1061, Meyer Building
Zoom:
Zoom link
Add to:
Lecturer:
Itay Lavie
Research Areas:
| In this talk, I explore the mathematical foundations of transformers in the kernel limit, focusing on how symmetries shape their inductive biases. First, I examine the case of permutation-symmetric datasets, using representation theory to analyze how these symmetries influence learnability. I will then present how these results can be extended to non-symmetric datasets, introducing the cross-dataset learnability measure. This generalization allows us to derive lower bounds on sample complexity for real-world data. Together, these results demonstrate the power of symmetry-based approaches in understanding deep learning performance across diverse data distributions. |
| Itay Lavie is currently a PhD student at the Hebrew University of Jerusalem, supervised by Prof. Zohar Ringel. He graduated summa cum laude from the Hebrew University with a Bachelor’s degree in physics and philosophy. He then completed his Master’s degree at the Hebrew University’s Racah Institute of Physics, where he researched the theory of deep learning under the supervision of Prof. Ringel and did a research internship at IBM Research. His research focuses on the intersection of theoretical physics and machine learning. |