[NeurIPS 2021 Outstanding Paper] On the Expressivity of Markov Reward - Research Paper Explained!

[NeurIPS 2021 Outstanding Paper] On the Expressivity of Markov Reward - Research Paper Explained!

Dec 05, 2021
|
97 views
Details
On the Expressivity of Markov Reward Paper: https://arxiv.org/abs/2111.00876 Reward Misspecification Blog : https://openai.com/blog/faulty-reward-functions/ Related Paper : Reward Is Enough (https://deepmind.com/research/publications/2021/Reward-is-Enough) Contents: 0:00 Introduction 4:25 Different Perspectives of Reward 11:11 Assumptions 14:15 What is a task? 19:52 SOAP 29:21 PO&TO 32:13 When is reward not enough? 38:08 Polynomial time reward finder 42:48 Experiments 45:16 Conclusion Abstract: Reward is the driving force for reinforcement-learning agents. This paper is dedicated to understanding the expressivity of reward as a way to capture tasks that we would want an agent to perform. We frame this study around three new abstract notions of "task" that might be desirable: (1) a set of acceptable behaviors, (2) a partial ordering over behaviors, or (3) a partial ordering over trajectories. Our main results prove that while reward can express many of these tasks, there exist instances of each task type that no Markov reward function can capture. We then provide a set of polynomial-time algorithms that construct a Markov reward function that allows an agent to optimize tasks of each of these three types, and correctly determine when no such reward function exists. We conclude with an empirical study that corroborates and illustrates our theoretical findings. Authors: David Abel, Will Dabney, Anna Harutyunyan, Mark K. Ho, Michael L. Littman, Doina Precup, Satinder Singh

Comments
loading...