This video explores AutoML-Zero, an evolutionary search for machine learning programs. These programs are initially empty with Setup, Predict, and Learn functions that can access scalar, vector, and matrix memory addresses. Through fitness evaluation and mutation, these programs evolve to use gradient descent, dropout-like operations, and ReLU activation functions!
Abstract: Machine learning research has advanced in multiple aspects, including model structures and learning methods. The effort to automate such research, known as AutoML, has also made significant progress. However, this progress has largely focused on the architecture of neural networks, where it has relied on sophisticated expertdesigned layers as building blocks—or similarly restrictive search spaces. Our goal is to show that AutoML can go further: it is possible today to automatically discover complete machine learning algorithms just using basic mathematical operations as building blocks. We demonstrate this by introducing a novel framework that significantly reduces human bias through a generic search space. Despite the vastness of this space, evolutionary search can still discover two-layer neural networks trained by backpropagation. These simple neural networks can then be surpassed by evolving directly on tasks of interest, e.g. CIFAR10 variants, where modern techniques emerge in the top algorithms, such as bilinear interactions, normalized gradients, and weight averaging. Moreover, evolution adapts algorithms to different task types: e.g., dropout-like techniques appear when little data is available. We believe these preliminary successes in discovering machine learning algorithms from scratch indicate a promising new direction for the field.
1:30 Machine Learning from Scratch
2:20 Linear Regression Program
3:20 Animation illustrating Programs
3:42 Search Space Operations
4:20 Evolutionary Search Strategy
6:23 Evolution Speedups
8:02 Experimental Questions
9:32 Linear Regression Experiments
10:25 Evolution of CIFAR-10 Programs
12:22 Algorithmic Adaptations
13:05 Connections with Neural Architecture Search
