Related papers: Measuring Recommender System Effects with Simulated Users

Measuring Recommender System Effects with Simulated Users

URL: http://arxiv.org/abs/2101.04526v1
Date: Tue, 12 Jan 2021 14:51:11 GMT
Title: Measuring Recommender System Effects with Simulated Users
Authors: Sirui Yao and Yoni Halpern and Nithum Thain and Xuezhi Wang and Kang Lee and Flavien Prost and Ed H. Chi and Jilin Chen and Alex Beutel
Abstract summary: Popularity bias and filter bubbles are two of the most well-studied recommender system biases. We offer a simulation framework for measuring the impact of a recommender system under different types of user behavior.
Score: 19.09065424910035
License: http://creativecommons.org/licenses/by/4.0/
Abstract: Imagine a food recommender system -- how would we check if it is \emph{causing} and fostering unhealthy eating habits or merely reflecting users' interests? How much of a user's experience over time with a recommender is caused by the recommender system's choices and biases, and how much is based on the user's preferences and biases? Popularity bias and filter bubbles are two of the most well-studied recommender system biases, but most of the prior research has focused on understanding the system behavior in a single recommendation step. How do these biases interplay with user behavior, and what types of user experiences are created from repeated interactions? In this work, we offer a simulation framework for measuring the impact of a recommender system under different types of user behavior. Using this simulation framework, we can (a) isolate the effect of the recommender system from the user preferences, and (b) examine how the system performs not just on average for an "average user" but also the extreme experiences under atypical user behavior. As part of the simulation framework, we propose a set of evaluation metrics over the simulations to understand the recommender system's behavior. Finally, we present two empirical case studies -- one on traditional collaborative filtering in MovieLens and one on a large-scale production recommender system -- to understand how popularity bias manifests over time.