Multi-Prompting Scenario-based Movie Recommendation with Large Language Models: Real User Case Study

This paper investigates the potential of using large language models (LLMs) for personalized movie recommendations in an online field experiment. We assess the performance of LLM recommenders using a combination of between-subject prompts, historical consumption patterns, and within-subject recommendation scenarios. Analyzing conversation and survey data from 160 active users, we find that while LLMs excel in providing explainable recommendations, they lack in personalization, diversity, and user trust. Interestingly, personalized prompting techniques do not significantly affect user-perceived recommendation quality, while the number of movies a user has watched plays a more significant role. Furthermore, LLMs demonstrate a stronger ability to recommend lesser-known or niche movies. Through qualitative analysis, we identify key conversational patterns linked to positive and negative user interaction experiences and conclude that providing personal context and examples is crucial for obtaining high-quality recommendations from LLMs. These insights offer practical implications for improving LLM-based RecSys in real-world applications.