Explicit or implicit feedback? engagement or satisfaction? A field experiment on machine-learning-based recommender systems

Qian Zhao, Max Harper, Gediminas Adomavicius, Joseph A Konstan

Research output: Chapter in Book/Report/Conference proceedingConference contribution

2 Citations (Scopus)

Abstract

Recommender systems algorithms are generally evaluated primarily on machine learning criteria such as recommendation accuracy or top-n precision. In this work, we evaluate six recommendation algorithms from a user-centric perspective, collecting both objective user activity data and subjective user perceptions. In a field experiment involving 1508 users who participated for at least a month, we compare six algorithms built using machine learning techniques, ranging from supervised matrix factorization, contextual bandit learning to Q learning. We found that the objective design in machine-learning-based recommender systems significantly affects user experience. Specifically, a recommender optimizing for implicit action prediction error engages users more than optimizing for explicit rating prediction error when modeled with the classical matrix factorization algorithms, which empirically explains the historical transition of recommender system research from modeling explicit feedback data to implicit feedback data. However, the action-based recommender is not as precise as the rating-based recommender in that it increases not only positive engagement but also negative engagement, e.g., negative action rate and user browsing effort which are negatively correlated with user satisfaction. We show that blending both explicit and implicit feedback from users through an online learning algorithm can gain the benefits of engagement and mitigate one of the possible costs (i.e., the increased browsing effort).

Original languageEnglish (US)
Title of host publicationProceedings of the 33rd Annual ACM Symposium on Applied Computing, SAC 2018
PublisherAssociation for Computing Machinery
Pages1331-1340
Number of pages10
ISBN (Electronic)9781450351911
DOIs
StatePublished - Apr 9 2018
Event33rd Annual ACM Symposium on Applied Computing, SAC 2018 - Pau, France
Duration: Apr 9 2018Apr 13 2018

Other

Other33rd Annual ACM Symposium on Applied Computing, SAC 2018
CountryFrance
CityPau
Period4/9/184/13/18

Fingerprint

Recommender systems
Learning systems
Feedback
Factorization
Experiments
Learning algorithms
Costs

Keywords

  • Contextual bandit
  • Machine learning
  • Q learning
  • Recommender systems
  • User experiment
  • User-centric evaluation

Cite this

Zhao, Q., Harper, M., Adomavicius, G., & Konstan, J. A. (2018). Explicit or implicit feedback? engagement or satisfaction? A field experiment on machine-learning-based recommender systems. In Proceedings of the 33rd Annual ACM Symposium on Applied Computing, SAC 2018 (pp. 1331-1340). Association for Computing Machinery. https://doi.org/10.1145/3167132.3167275

Explicit or implicit feedback? engagement or satisfaction? A field experiment on machine-learning-based recommender systems. / Zhao, Qian; Harper, Max; Adomavicius, Gediminas; Konstan, Joseph A.

Proceedings of the 33rd Annual ACM Symposium on Applied Computing, SAC 2018. Association for Computing Machinery, 2018. p. 1331-1340.

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Zhao, Q, Harper, M, Adomavicius, G & Konstan, JA 2018, Explicit or implicit feedback? engagement or satisfaction? A field experiment on machine-learning-based recommender systems. in Proceedings of the 33rd Annual ACM Symposium on Applied Computing, SAC 2018. Association for Computing Machinery, pp. 1331-1340, 33rd Annual ACM Symposium on Applied Computing, SAC 2018, Pau, France, 4/9/18. https://doi.org/10.1145/3167132.3167275
Zhao Q, Harper M, Adomavicius G, Konstan JA. Explicit or implicit feedback? engagement or satisfaction? A field experiment on machine-learning-based recommender systems. In Proceedings of the 33rd Annual ACM Symposium on Applied Computing, SAC 2018. Association for Computing Machinery. 2018. p. 1331-1340 https://doi.org/10.1145/3167132.3167275
Zhao, Qian ; Harper, Max ; Adomavicius, Gediminas ; Konstan, Joseph A. / Explicit or implicit feedback? engagement or satisfaction? A field experiment on machine-learning-based recommender systems. Proceedings of the 33rd Annual ACM Symposium on Applied Computing, SAC 2018. Association for Computing Machinery, 2018. pp. 1331-1340
@inproceedings{72902d2afabf464f93e9e7ad1c289b4c,
title = "Explicit or implicit feedback? engagement or satisfaction?: A field experiment on machine-learning-based recommender systems",
abstract = "Recommender systems algorithms are generally evaluated primarily on machine learning criteria such as recommendation accuracy or top-n precision. In this work, we evaluate six recommendation algorithms from a user-centric perspective, collecting both objective user activity data and subjective user perceptions. In a field experiment involving 1508 users who participated for at least a month, we compare six algorithms built using machine learning techniques, ranging from supervised matrix factorization, contextual bandit learning to Q learning. We found that the objective design in machine-learning-based recommender systems significantly affects user experience. Specifically, a recommender optimizing for implicit action prediction error engages users more than optimizing for explicit rating prediction error when modeled with the classical matrix factorization algorithms, which empirically explains the historical transition of recommender system research from modeling explicit feedback data to implicit feedback data. However, the action-based recommender is not as precise as the rating-based recommender in that it increases not only positive engagement but also negative engagement, e.g., negative action rate and user browsing effort which are negatively correlated with user satisfaction. We show that blending both explicit and implicit feedback from users through an online learning algorithm can gain the benefits of engagement and mitigate one of the possible costs (i.e., the increased browsing effort).",
keywords = "Contextual bandit, Machine learning, Q learning, Recommender systems, User experiment, User-centric evaluation",
author = "Qian Zhao and Max Harper and Gediminas Adomavicius and Konstan, {Joseph A}",
year = "2018",
month = "4",
day = "9",
doi = "10.1145/3167132.3167275",
language = "English (US)",
pages = "1331--1340",
booktitle = "Proceedings of the 33rd Annual ACM Symposium on Applied Computing, SAC 2018",
publisher = "Association for Computing Machinery",

}

TY - GEN

T1 - Explicit or implicit feedback? engagement or satisfaction?

T2 - A field experiment on machine-learning-based recommender systems

AU - Zhao, Qian

AU - Harper, Max

AU - Adomavicius, Gediminas

AU - Konstan, Joseph A

PY - 2018/4/9

Y1 - 2018/4/9

N2 - Recommender systems algorithms are generally evaluated primarily on machine learning criteria such as recommendation accuracy or top-n precision. In this work, we evaluate six recommendation algorithms from a user-centric perspective, collecting both objective user activity data and subjective user perceptions. In a field experiment involving 1508 users who participated for at least a month, we compare six algorithms built using machine learning techniques, ranging from supervised matrix factorization, contextual bandit learning to Q learning. We found that the objective design in machine-learning-based recommender systems significantly affects user experience. Specifically, a recommender optimizing for implicit action prediction error engages users more than optimizing for explicit rating prediction error when modeled with the classical matrix factorization algorithms, which empirically explains the historical transition of recommender system research from modeling explicit feedback data to implicit feedback data. However, the action-based recommender is not as precise as the rating-based recommender in that it increases not only positive engagement but also negative engagement, e.g., negative action rate and user browsing effort which are negatively correlated with user satisfaction. We show that blending both explicit and implicit feedback from users through an online learning algorithm can gain the benefits of engagement and mitigate one of the possible costs (i.e., the increased browsing effort).

AB - Recommender systems algorithms are generally evaluated primarily on machine learning criteria such as recommendation accuracy or top-n precision. In this work, we evaluate six recommendation algorithms from a user-centric perspective, collecting both objective user activity data and subjective user perceptions. In a field experiment involving 1508 users who participated for at least a month, we compare six algorithms built using machine learning techniques, ranging from supervised matrix factorization, contextual bandit learning to Q learning. We found that the objective design in machine-learning-based recommender systems significantly affects user experience. Specifically, a recommender optimizing for implicit action prediction error engages users more than optimizing for explicit rating prediction error when modeled with the classical matrix factorization algorithms, which empirically explains the historical transition of recommender system research from modeling explicit feedback data to implicit feedback data. However, the action-based recommender is not as precise as the rating-based recommender in that it increases not only positive engagement but also negative engagement, e.g., negative action rate and user browsing effort which are negatively correlated with user satisfaction. We show that blending both explicit and implicit feedback from users through an online learning algorithm can gain the benefits of engagement and mitigate one of the possible costs (i.e., the increased browsing effort).

KW - Contextual bandit

KW - Machine learning

KW - Q learning

KW - Recommender systems

KW - User experiment

KW - User-centric evaluation

UR - http://www.scopus.com/inward/record.url?scp=85050554283&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85050554283&partnerID=8YFLogxK

U2 - 10.1145/3167132.3167275

DO - 10.1145/3167132.3167275

M3 - Conference contribution

AN - SCOPUS:85050554283

SP - 1331

EP - 1340

BT - Proceedings of the 33rd Annual ACM Symposium on Applied Computing, SAC 2018

PB - Association for Computing Machinery

ER -