Learning one feature at a time: A tale of phase transitions and neural scaling laws
Learning one feature at a time: A tale of phase transitions and neural scaling laws
- Event time: 3:00pm until 4:00pm
- Event date: 24th November 2026
- Speaker: Professor Florent Krzakala (École polytechnique fédérale de Lausanne, Switzerland)
- Location: Microsoft Teams - see email.
Event details
What explains the remarkable success of neural networks trained by gradient descent and the scaling laws they obey? I’ll start with an old idea from random matrix theory. At the Edwards–Jones / Baik–Ben Arous–Péché transition, an eigenvalue breaks away from the bulk and its eigenvector reveals a signal hidden in the noise. After illustrating this transition with spin glasses and community detection, I’ll show how a stylized approximation to gradient descent lets us track feature learning through the spectrum of a random matrix. In two-layer networks, progressively weaker features become detectable as the training set grows. When feature strengths follow a power law, this cascade of transitions predicts the scaling exponent for prediction error. I’ll close with the kiss of depth: In simple models of a deep network, the features learned by one layer make a previously hidden signal visible to the next, revealing why depth can give networks an advantage on hierarchical tasks.
About Statistical Physics and Complexity Group meetings
This is a weekly series of webinars on theoretical aspects of Condensed Matter, Biological, and Statistical Physics. It is open to anyone interested in research in these areas..
Find out more about Statistical Physics and Complexity Group meetings.
