Why They Come And Go: A Case Study of Productive Flyby Users and Their Rating Integrity Challenge in Movie Recommenders

We present a case study of productive flyby users (PFB users) on a recommendation website. These users exhibit counterintuitive behavior: they input a large amount of data during their first visit but never return. This phenomenon can have both positive and negative impacts on the system. On the positive side, their high productivity contributes a substantial amount of data. On the negative side, they may input inappropriate ratings that violate the assumptions of recommendation algorithms, potentially undermining system performance. To better understand the nature and causes of this behavior, we investigated their motivations, expectations, reasons for leaving, and the potential risks associated with their ratings using a mixed-methods approach. Specifically, we conducted interviews with 11 users, surveyed 41 users, and analyzed the impact of 1,000 PFB users on the performance of recommendation algorithms for regular users. Our findings revealed diverse motivations among PFB users. Some engaged with the system merely to pass the time, while others had unrealistic expectations of the recommender system. Regarding rating quality, 27% of surveyed users admitted to rating movies they had not seen, citing reasons such as browsing too quickly or attempting to manipulate the algorithm. Notably, users who reported leaving because they were "just killing time and forgot about the website" were the most likely to rate unseen movies. Overall, PFB users significantly influence recommendation algorithms and their performance for regular users. While some subgroups negatively affect prediction accuracy, others provide valuable data contributions. We discuss strategies for recommendation websites to better support these users or mitigate the impact of inappropriate ratings.