Statistical physics of learning in neural networks: the importance of data structure

The highly structured character of data used in training deep networks is a crucial ingredient of their performance. Yet theoretical work has largely overlooked this structure. Modelling structured data, analyzing the learning and the generalization of deep networks trained on this data, are major challenges. This talk will describe several recent developments in this direction.

We shall introduce a generative model for structured datasets, the hidden manifold model, in which high-dimensional inputs lie on a lower-dimensional folded manifold, as in real datasets. The analytic study of learning with such data ensembles is possible due to a Gaussian equivalence stating that the key metrics of interest, such as the training and test errors, can be fully captured by an appropriately chosen Gaussian model. This can also be extended to data drawn from pre-trained generative models. The Gaussian equivalence, which can be proven in some cases, allows to apply statistical physics methods that accurately describe the learning dynamics and the phase diagram.

Event resources