ICML 2020 Oral Talk: Planning to Explore via Self-Supervised World Models

ICML 2020

ICML 2020 long oral talk for the paper titled "Planning to Explore via Self-Supervised World Models". Code and paper at https://ramanans1.github.io/plan2explore/ TL;DR: RL agents are specific to the tasks they are trained on. What if we remove the task itself during training? Turns out, a self-supervised model-based planning agent can both explore efficiently without rewards and then achieve SOTA when given new tasks with zero or few (10-20 episodes) new samples in DMControl from images! Project Abstract: Reinforcement learning allows solving complex tasks, however, the learning tends to be task-specific and the sample efficiency remains a challenge. We present Plan2Explore, a self-supervised reinforcement learning agent that tackles both these challenges through a new approach to self-supervised exploration and fast adaptation to new tasks, which need not be known during exploration. During exploration, unlike prior methods which retrospectively compute the novelty of observations after the agent has already reached them, our agent acts efficiently by leveraging planning to seek out expected future novelty. After exploration, the agent quickly adapts to multiple downstream tasks in a zero or a few-shot manner. We evaluate on challenging control tasks from high-dimensional image inputs. Without any training supervision or task-specific interaction, Plan2Explore outperforms prior self-supervised exploration methods, and in fact, almost matches the performances oracle which has access to rewards. Authors: Ramanan Sekar (speaker), Oleh Rybkin, Kostas Daniilidis, Pieter Abbeel, Danijar Hafner, Deepak Pathak