Online Evaluation Methods for the Causal Effect of Recommendations

• URL: http://arxiv.org/abs/2107.06630v2
• Date: Thu, 15 Jul 2021 14:02:04 GMT
• Title: Online Evaluation Methods for the Causal Effect of Recommendations
• Authors: Masahiro Sato
• Abstract summary: We propose the first interleaving methods that can efficiently compare recommendation models in terms of causal effects. We measure the outcomes of both items on an interleaved list and items not on the interleaved list, since the causal effect is the difference between outcomes with and without recommendations. We then verify the unbiasedness and efficiency of online evaluation methods through simulated online experiments.
• Score: 0.20305676256390934
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Evaluating the causal effect of recommendations is an important objective because the causal effect on user interactions can directly leads to an increase in sales and user engagement. To select an optimal recommendation model, it is common to conduct A/B testing to compare model performance. However, A/B testing of causal effects requires a large number of users, making such experiments costly and risky. We therefore propose the first interleaving methods that can efficiently compare recommendation models in terms of causal effects. In contrast to conventional interleaving methods, we measure the outcomes of both items on an interleaved list and items not on the interleaved list, since the causal effect is the difference between outcomes with and without recommendations. To ensure that the evaluations are unbiased, we either select items with equal probability or weight the outcomes using inverse propensity scores. We then verify the unbiasedness and efficiency of online evaluation methods through simulated online experiments. The results indicate that our proposed methods are unbiased and that they have superior efficiency to A/B testing.

Related papers

• Causal Distillation for Alleviating Performance Heterogeneity in Recommender Systems [142.3424649008479]
  We show the uneven distribution of historical interactions and the biased training of recommender models. The key to debiased training lies in eliminating the effect of confounders that influence both the user's historical behaviors and the next behavior. We propose a causal multi-teacher distillation framework (CausalD) to address unobserved confounders.
  arXiv  Detail & Related papers  (2024-05-31T05:31:00Z)
• A First Look at Selection Bias in Preference Elicitation for Recommendation [64.44255178199846]
  We study the effect of selection bias in preference elicitation on the resulting recommendations. A big hurdle is the lack of any publicly available dataset that has preference elicitation interactions. We propose a simulation of a topic-based preference elicitation process.
  arXiv  Detail & Related papers  (2024-05-01T14:56:56Z)
• Be Aware of the Neighborhood Effect: Modeling Selection Bias under Interference [50.95521705711802]
  Previous studies have focused on addressing selection bias to achieve unbiased learning of the prediction model. This paper formally formulates the neighborhood effect as an interference problem from the perspective of causal inference. We propose a novel ideal loss that can be used to deal with selection bias in the presence of neighborhood effect.
  arXiv  Detail & Related papers  (2024-04-30T15:20:41Z)
• Going Beyond Popularity and Positivity Bias: Correcting for Multifactorial Bias in Recommender Systems [74.47680026838128]
  Two typical forms of bias in user interaction data with recommender systems (RSs) are popularity bias and positivity bias. We consider multifactorial selection bias affected by both item and rating value factors. We propose smoothing and alternating gradient descent techniques to reduce variance and improve the robustness of its optimization.
  arXiv  Detail & Related papers  (2024-04-29T12:18:21Z)
• Effect Size Estimation for Duration Recommendation in Online Experiments: Leveraging Hierarchical Models and Objective Utility Approaches [13.504353263032359]
  The selection of the assumed effect size (AES) critically determines the duration of an experiment, and hence its accuracy and efficiency. Traditionally, experimenters determine AES based on domain knowledge, but this method becomes impractical for online experimentation services managing numerous experiments. We propose two solutions for data-driven AES selection in for online experimentation services.
  arXiv  Detail & Related papers  (2023-12-20T09:34:28Z)
• Fair Effect Attribution in Parallel Online Experiments [57.13281584606437]
  A/B tests serve the purpose of reliably identifying the effect of changes introduced in online services. It is common for online platforms to run a large number of simultaneous experiments by splitting incoming user traffic randomly. Despite a perfect randomization between different groups, simultaneous experiments can interact with each other and create a negative impact on average population outcomes.
  arXiv  Detail & Related papers  (2022-10-15T17:15:51Z)
• Cross Pairwise Ranking for Unbiased Item Recommendation [57.71258289870123]
  We develop a new learning paradigm named Cross Pairwise Ranking (CPR) CPR achieves unbiased recommendation without knowing the exposure mechanism. We prove in theory that this way offsets the influence of user/item propensity on the learning.
  arXiv  Detail & Related papers  (2022-04-26T09:20:27Z)
• A Two-Stage Feature Selection Approach for Robust Evaluation of Treatment Effects in High-Dimensional Observational Data [1.4710887888397084]
  We propose a novel two-stage feature selection technique called, Outcome Adaptive Elastic Net (OAENet) OAENet is explicitly designed for making robust causal inference decisions using matching techniques. Numerical experiments on simulated data demonstrate that OAENet significantly outperforms state-of-the-art methods.
  arXiv  Detail & Related papers  (2021-11-27T02:54:30Z)
• Causality-Aware Neighborhood Methods for Recommender Systems [3.0919302844782717]
  Business objectives of recommenders, such as increasing sales, are aligned with the causal effect of recommendations. Previous recommenders employ the inverse propensity scoring (IPS) in causal inference. We develop robust ranking methods for the causal effect of recommendations.
  arXiv  Detail & Related papers  (2020-12-17T08:23:17Z)
• Unbiased Learning for the Causal Effect of Recommendation [8.849159720632612]
  This paper proposes an unbiased learning framework for the causal effect of recommendation. We develop an unbiased learning method for the causal effect extension of a ranking metric.
  arXiv  Detail & Related papers  (2020-08-11T07:30:44Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.