This episode is an interview with Sebastian Institute de Physique Theorique, discussing highlights from his paper "Dynamics of stochastic gradient descent for two-layer neural networks in the teacher-student setup," accepted as an oral presentation at NeurIPS 2019 Conference.
Sebastian is a post-doc Ecole normale supérieure in Paris, where I work on problems at the interface of theoretical physics and machine learning with Florent Krzakala and Lenka Zdeborová. The main theme of his current research is to understand why neural networks are able to generalize well from examples in practice, when classical learning theory would predict that they cannot. Sebastian's approach is to use concepts and tools from statistical physics to build models for the key drivers of generalization of neural networks. He's also interested in using machine learning as a tool to handle the vast data sets generated by large-scale experiments, in particular in neuroscience; and as a source of novel theoretical ideas, e.g. for the thermodynamics of computation.
Paper At A Glance:
Deep neural networks achieve stellar generalization even when they have enough parameters to easily fit all their training data. We study this phenomenon by analyzing the dynamics and the performance of over-parameterized two-layer neural networks in the teacher-student setup, where one network, the student, is trained on data generated by another network, called the teacher. We show how the dynamics of stochastic gradient descent (SGD) is captured by a set of differential equations and prove that this description is asymptotically exact in the limit of large inputs. Using this framework, we calculate the final generalization error of student networks that have more parameters than their teachers. We find that the final generalisation error of the student increases with network size when training only the first layer, but stays constant or even decreases with size when training both layers. We show that these different behaviors have their root in the different solutions SGD finds for different activation functions. Our results indicate that achieving good generalization in neural networks goes beyond the properties of SGD alone and depends on the interplay of at least the algorithm, the model architecture, and the data set.