Rethinking the recommender research ecosystem: Reproducibility, openness, and LensKit

Michael D. Ekstrand, Michael Ludwig, Joseph A Konstan, John T. Riedl

Research output: Chapter in Book/Report/Conference proceedingConference contribution

111 Scopus citations

Abstract

Recommender systems research is being slowed by the difficulty of replicating and comparing research results. Published research uses various experimental methodologies and metrics that are difficult to compare. It also often fails to sufficiently document the details of proposed algorithms or the evaluations employed. Researchers waste time reimplementing well-known algorithms, and the new implementations may miss key details from the original algorithm or its subsequent refinements. When proposing new algorithms, researchers should compare them against finely-tuned implementations of the leading prior algorithms using state-of-the-art evaluation methodologies. With few exceptions, published algorithmic improvements in our field should be accompanied by working code in a standard framework, including test harnesses to reproduce the described results. To that end, we present the design and freely distributable source code of LensKit, a flexible platform for reproducible recommender systems research. LensKit provides carefully tuned implementations of the leading collaborative filtering algorithms, APIs for common recommender system use cases, and an evaluation framework for performing reproducible offline evaluations of algorithms. We demonstrate the utility of LensKit by replicating and extending a set of prior comparative studies of recommender algorithms - showing limitations in some of the original results - and by investigating a question recently raised by a leader in the recommender systems community on problems with error-based prediction evaluation.

Original languageEnglish (US)
Title of host publicationRecSys'11 - Proceedings of the 5th ACM Conference on Recommender Systems
Pages133-140
Number of pages8
DOIs
StatePublished - Dec 6 2011
Event5th ACM Conference on Recommender Systems, RecSys 2011 - Chicago, IL, United States
Duration: Oct 23 2011Oct 27 2011

Publication series

NameRecSys'11 - Proceedings of the 5th ACM Conference on Recommender Systems

Other

Other5th ACM Conference on Recommender Systems, RecSys 2011
CountryUnited States
CityChicago, IL
Period10/23/1110/27/11

Keywords

  • evaluation
  • implementation
  • recommender systems

Fingerprint Dive into the research topics of 'Rethinking the recommender research ecosystem: Reproducibility, openness, and LensKit'. Together they form a unique fingerprint.

Cite this