[KDD 2020] Off-policy Bandits with Deficient Support
CrossMind.ai logo

[KDD 2020] Off-policy Bandits with Deficient Support

Dec 16, 2020
|
58 views
|
Details
Learning effective contextual-bandit policies from past actions of,a deployed system is highly desirable in many settings (e.g. voice,assistants, recommendation, search), since it enables the reuse of,large amounts of log data. State-of-the-art methods for such offpolicy learning, however, are based on inverse propensity score,(IPS) weighting. A key theoretical requirement of IPS weighting is,that the policy that logged the data has "full support", which typically translates into requiring non-zero probability for any action,in any context. Unfortunately, many real-world systems produce,support deficient data, especially when the action space is large, and,we show how existing methods can fail catastrophically. To overcome this gap between theory and applications, we identify three,approaches that provide various guarantees for IPS-based learning,despite the inherent limitations of support-deficient data: restricting,the action space, reward extrapolation, and restricting the policy,space. We systematically analyze the statistical and computational,properties of these three approaches, and we empirically evaluate,their effectiveness. In addition to providing the first systematic,analysis of support-deficiency in contextual-bandit learning, we,conclude with recommendations that provide practical guidance.

Comments
loading...