Recbench: Benchmarks for evaluating performance of recommender system architectures

Justin J. Levandoski, Michael D. Ekstrand, Michael J. Ludwig, Ahmed Eldawy, Mohamed F. Mokbel, John T. Riedl

Research output: Contribution to journalConference articlepeer-review

10 Scopus citations


Traditionally, recommender systems have been "hand-built", implemented as custom applications hard-wired to a particular recommendation task. Recently, the database community has begun exploring alternative DBMS-based recommender system architectures, whereby a database both stores the recommender system data (e.g., ratings data and the derived recommender models) and generates recommendations using SQL queries. In this paper, we present a comprehensive experimental comparison of both architectures. We define a set of benchmark tasks based on the needs of a typical recommender-powered e-commerce site. We then evaluate the performance of the "hand-built" MultiLens recommender application against two DBMS-based implementations: an unmodified DBMS and RecStore, a DBMS modified to improve efficiency in incremental recommender model updates. We employ two non-trivial data sets in our study: the 10 million rating MovieLens data, and the 100 million rating data set used in the Netflix Challenge. This study is the first of its kind, and our findings reveal an interesting trade-off: "hand-built" recommenders exhibit superior performance in model-building and pure recommendation tasks, while DBMS-based recommenders are superior at more complex recommendation tasks such as providing filtered recommendations and blending text-search with recommendation prediction scores.

Original languageEnglish (US)
Pages (from-to)911-920
Number of pages10
JournalProceedings of the VLDB Endowment
Issue number11
StatePublished - Aug 2011
Event37th International Conference on Very Large Data Bases, VLDB 2011 - Seattle, United States
Duration: Aug 29 2011Sep 3 2011


Dive into the research topics of 'Recbench: Benchmarks for evaluating performance of recommender system architectures'. Together they form a unique fingerprint.

Cite this