Fundamentals of Aligning General-Purpose AI

The field of artificial intelligence (AI) is undergoing a paradigm shift, moving from neural networks trained for narrowly defined tasks (e.g., image classification and machine translation) to general-purpose models such as ChatGPT. These models are trained at unprecedented scales to perform a wide range of tasks, from providing travel recommendations to solving Olympiad-level math problems. As they are increasingly adopted in society, a central challenge is to ensure the alignment of general-purpose models with human preferences. In this talk, I will present a series of works that reveal fundamental pitfalls in existing alignment methods. In particular, I will show that they can: (1) suffer from a flat objective landscape that hinders optimization, and (2) fail to reliably increase the likelihood of generating preferred outputs, sometimes even causing the model to generate outputs with an opposite meaning. Beyond characterizing these pitfalls, our theory provides quantitative measures for identifying when they occur, suggests preventative guidelines, and has led to the development of new data selection and alignment algorithms, validated at large scale in real-world settings. Our contributions address both efficiency challenges and safety risks that may arise in the alignment process. I will conclude with an outlook on future directions, toward building a practical theory in the age of general-purpose AI.

Short bio:
Noam Razin is a Postdoctoral Fellow at Princeton Language and Intelligence, Princeton University. His research focuses on the fundamentals of artificial intelligence (AI). By combining mathematical analyses with systematic experimentation, he aims to develop theories that shed light on how modern AI works, identify potential failures, and yield principled methods for improving efficiency, reliability, and performance.
Noam earned his PhD in Computer Science at Tel Aviv University, where he was advised by Nadav Cohen. Prior to that, he obtained a BSc in Computer Science (summa cum laude) at The Hebrew University of Jerusalem under the Amirim honors program. For his research, Noam received several honors and awards, including the Zuckerman Postdoctoral Scholarship, the Israeli Council for Higher Education (VATAT) Postdoctoral Scholarship, the Apple Scholars in AI/ML PhD fellowship, the Tel Aviv University Center for AI and Data Science excellence fellowship, and the Deutsch Prize for PhD candidates.