Measuring Recommender System Effects with Simulated Users

• URL: http://arxiv.org/abs/2101.04526v1
• Date: Tue, 12 Jan 2021 14:51:11 GMT
• Title: Measuring Recommender System Effects with Simulated Users
• Authors: Sirui Yao and Yoni Halpern and Nithum Thain and Xuezhi Wang and Kang Lee and Flavien Prost and Ed H. Chi and Jilin Chen and Alex Beutel
• Abstract summary: Popularity bias and filter bubbles are two of the most well-studied recommender system biases. We offer a simulation framework for measuring the impact of a recommender system under different types of user behavior.
• Score: 19.09065424910035
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Imagine a food recommender system -- how would we check if it is \emph{causing} and fostering unhealthy eating habits or merely reflecting users' interests? How much of a user's experience over time with a recommender is caused by the recommender system's choices and biases, and how much is based on the user's preferences and biases? Popularity bias and filter bubbles are two of the most well-studied recommender system biases, but most of the prior research has focused on understanding the system behavior in a single recommendation step. How do these biases interplay with user behavior, and what types of user experiences are created from repeated interactions? In this work, we offer a simulation framework for measuring the impact of a recommender system under different types of user behavior. Using this simulation framework, we can (a) isolate the effect of the recommender system from the user preferences, and (b) examine how the system performs not just on average for an "average user" but also the extreme experiences under atypical user behavior. As part of the simulation framework, we propose a set of evaluation metrics over the simulations to understand the recommender system's behavior. Finally, we present two empirical case studies -- one on traditional collaborative filtering in MovieLens and one on a large-scale production recommender system -- to understand how popularity bias manifests over time.

Related papers

• Algorithmic Drift: A Simulation Framework to Study the Effects of Recommender Systems on User Preferences [7.552217586057245]
  We propose a simulation framework that mimics user-recommender system interactions in a long-term scenario. We introduce two novel metrics for quantifying the algorithm's impact on user preferences, specifically in terms of drift over time.
  arXiv  Detail & Related papers  (2024-09-24T21:54:22Z)
• Cold-start Recommendation by Personalized Embedding Region Elicitation [14.78070264373809]
  Existing elicitation methods employ a fixed set of items to learn the user's preference. This paper addresses this challenge using a 2-phase, personalized elicitation scheme. We show that each subproblem in the elicitation scheme can be efficiently implemented.
  arXiv  Detail & Related papers  (2024-06-03T04:03:24Z)
• System-2 Recommenders: Disentangling Utility and Engagement in Recommendation Systems via Temporal Point-Processes [80.97898201876592]
  We propose a generative model in which past content interactions impact the arrival rates of users based on a self-exciting Hawkes process. We show analytically that given samples it is possible to disentangle System-1 and System-2 and allow content optimization based on user utility.
  arXiv  Detail & Related papers  (2024-05-29T18:19:37Z)
• Learning from Negative User Feedback and Measuring Responsiveness for Sequential Recommenders [13.762960304406016]
  We introduce explicit and implicit negative user feedback into the training objective of sequential recommenders. We demonstrate the effectiveness of this approach using live experiments on a large-scale industrial recommender system.
  arXiv  Detail & Related papers  (2023-08-23T17:16:07Z)
• Latent User Intent Modeling for Sequential Recommenders [92.66888409973495]
  Sequential recommender models learn to predict the next items a user is likely to interact with based on his/her interaction history on the platform. Most sequential recommenders however lack a higher-level understanding of user intents, which often drive user behaviors online. Intent modeling is thus critical for understanding users and optimizing long-term user experience.
  arXiv  Detail & Related papers  (2022-11-17T19:00:24Z)
• Breaking Feedback Loops in Recommender Systems with Causal Inference [99.22185950608838]
  Recent work has shown that feedback loops may compromise recommendation quality and homogenize user behavior. We propose the Causal Adjustment for Feedback Loops (CAFL), an algorithm that provably breaks feedback loops using causal inference. We show that CAFL improves recommendation quality when compared to prior correction methods.
  arXiv  Detail & Related papers  (2022-07-04T17:58:39Z)
• Estimating and Penalizing Induced Preference Shifts in Recommender Systems [10.052697877248601]
  We argue that system designers should: estimate the shifts a recommender would induce; evaluate whether such shifts would be undesirable; and even actively optimize to avoid problematic shifts. We do this by using historical user interaction data to train predictive user model which implicitly contains their preference dynamics. In simulated experiments, we show that our learned preference dynamics model is effective in estimating user preferences and how they would respond to new recommenders.
  arXiv  Detail & Related papers  (2022-04-25T21:04:46Z)
• Modeling Attrition in Recommender Systems with Departing Bandits [84.85560764274399]
  We propose a novel multi-armed bandit setup that captures policy-dependent horizons. We first address the case where all users share the same type, demonstrating that a recent UCB-based algorithm is optimal. We then move forward to the more challenging case, where users are divided among two types.
  arXiv  Detail & Related papers  (2022-03-25T02:30:54Z)
• Correcting the User Feedback-Loop Bias for Recommendation Systems [34.44834423714441]
  We propose a systematic and dynamic way to correct user feedback-loop bias in recommendation systems. Our method includes a deep-learning component to learn each user's dynamic rating history embedding. We empirically validated the existence of such user feedback-loop bias in real world recommendation systems.
  arXiv  Detail & Related papers  (2021-09-13T15:02:55Z)
• PURS: Personalized Unexpected Recommender System for Improving User Satisfaction [76.98616102965023]
  We describe a novel Personalized Unexpected Recommender System (PURS) model that incorporates unexpectedness into the recommendation process. Extensive offline experiments on three real-world datasets illustrate that the proposed PURS model significantly outperforms the state-of-the-art baseline approaches.
  arXiv  Detail & Related papers  (2021-06-05T01:33:21Z)
• Controllable Multi-Interest Framework for Recommendation [64.30030600415654]
  We formalize the recommender system as a sequential recommendation problem. We propose a novel controllable multi-interest framework for the sequential recommendation, called ComiRec. Our framework has been successfully deployed on the offline Alibaba distributed cloud platform.
  arXiv  Detail & Related papers  (2020-05-19T10:18:43Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.