[KDD 2020] On Sampling Top-K Recommendation Evaluation
Aug 13, 202010 views
Recently, Rendle has warned that the use of sampling-based top𝑘,metrics might not suffice. This throws a number of recent studies,on deep learning-based recommendation algorithms, and classic,non-deep-learning algorithms using such a metric, into jeopardy.,In this work, we thoroughly investigate the relationship between,the sampling and global top𝐾,Hit-Ratio (HR, or Recall), originally,proposed by Koren [,2,] and extensively used by others. By formulating the problem of aligning sampling top𝑘,(,𝑆𝐻𝑅,@,𝑘,) and global,top𝐾,(,𝐻𝑅,@,𝐾,) Hit-Ratios through a mapping function,𝑓,, so that,𝑆𝐻𝑅,@,𝑘,≈,𝐻𝑅,@,𝑓,(,𝑘,),, we demonstrate both theoretically and experimentally that the sampling top𝑘,Hit-Ratio provides an accurate,approximation of its global (exact) counterpart, and can consistently predict the correct winners (the same as indicate by their,corresponding global Hit-Ratios).