What Are We Optimizing For? A Human-centric Evaluation of Deep Learning-based Movie Recommenders

In the past decade, deep learning (DL) models have gained prominence for their exceptional accuracy on benchmark datasets in recommender systems (RecSys). However, their evaluation has primarily relied on offline metrics, overlooking direct user perception and experience. To address this gap, we conduct a human-centric evaluation case study for four leading DL-RecSys models in the movie domain. We test how different DL-RecSys models perform in personalized recommendation generation by conducting a survey study with 445 real active users. We find some DL-RecSys models to be superior in recommending novel and unexpected items but weaker in diversity, trustworthiness, transparency, accuracy, and overall user satisfaction compared to classic collaborative filtering (CF) methods. Qualitatively, we confirm with real user quotes that accuracy plus at least one other attribute is necessary to ensure good user experience, while their demands for transparency and trust cannot be neglected. Based on our findings, we discuss future human-centric DL-RecSys design and optimization strategies.