Overcoming accuracy-diversity tradeoff in recommender systems: A variance-based approach

Collaborative filtering and, more generally, recommender systems represent an increasingly popular and important set of personalization technologies that help people navigate through the vast amounts of information. The performance of recommender systems can be evaluated along several dimensions, such as the accuracy of recommendations for each user and the diversity of recommendations across different users. Intuitively, there is a tradeoff between accuracy and diversity, because high accuracy may often be obtained by safely recommending to users the most popular ("bestselling") items, which can lead to the reduction in recommendation diversity, i.e., less personalized recommendations. And conversely, higher diversity can be achieved by trying to uncover and recommend highly idiosyncratic/personalized items for each user, which are inherently more difficult to predict and, thus, may lead to a decrease in recommendation accuracy. In our research we explore different ways to overcome this accuracy-diversity tradeoff, and in this paper we discuss a variance-based approach that can improve both the accuracy and diversity of recommendations obtained from a traditional collaborative filtering technique. We provide empirical results based on several real-world movie rating datasets.