Online Evaluation Methods for the Causal Effect of Recommendations
- URL: http://arxiv.org/abs/2107.06630v2
- Date: Thu, 15 Jul 2021 14:02:04 GMT
- Title: Online Evaluation Methods for the Causal Effect of Recommendations
- Authors: Masahiro Sato
- Abstract summary: We propose the first interleaving methods that can efficiently compare recommendation models in terms of causal effects.
We measure the outcomes of both items on an interleaved list and items not on the interleaved list, since the causal effect is the difference between outcomes with and without recommendations.
We then verify the unbiasedness and efficiency of online evaluation methods through simulated online experiments.
- Score: 0.20305676256390934
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: Evaluating the causal effect of recommendations is an important objective
because the causal effect on user interactions can directly leads to an
increase in sales and user engagement. To select an optimal recommendation
model, it is common to conduct A/B testing to compare model performance.
However, A/B testing of causal effects requires a large number of users, making
such experiments costly and risky. We therefore propose the first interleaving
methods that can efficiently compare recommendation models in terms of causal
effects. In contrast to conventional interleaving methods, we measure the
outcomes of both items on an interleaved list and items not on the interleaved
list, since the causal effect is the difference between outcomes with and
without recommendations. To ensure that the evaluations are unbiased, we either
select items with equal probability or weight the outcomes using inverse
propensity scores. We then verify the unbiasedness and efficiency of online
evaluation methods through simulated online experiments. The results indicate
that our proposed methods are unbiased and that they have superior efficiency
to A/B testing.
Related papers
- Be Aware of the Neighborhood Effect: Modeling Selection Bias under Interference [50.95521705711802]
Previous studies have focused on addressing selection bias to achieve unbiased learning of the prediction model.
This paper formally formulates the neighborhood effect as an interference problem from the perspective of causal inference.
We propose a novel ideal loss that can be used to deal with selection bias in the presence of neighborhood effect.
arXiv Detail & Related papers (2024-04-30T15:20:41Z) - Effect Size Estimation for Duration Recommendation in Online Experiments: Leveraging Hierarchical Models and Objective Utility Approaches [13.504353263032359]
The selection of the assumed effect size (AES) critically determines the duration of an experiment, and hence its accuracy and efficiency.
Traditionally, experimenters determine AES based on domain knowledge, but this method becomes impractical for online experimentation services managing numerous experiments.
We propose two solutions for data-driven AES selection in for online experimentation services.
arXiv Detail & Related papers (2023-12-20T09:34:28Z) - Fair Effect Attribution in Parallel Online Experiments [57.13281584606437]
A/B tests serve the purpose of reliably identifying the effect of changes introduced in online services.
It is common for online platforms to run a large number of simultaneous experiments by splitting incoming user traffic randomly.
Despite a perfect randomization between different groups, simultaneous experiments can interact with each other and create a negative impact on average population outcomes.
arXiv Detail & Related papers (2022-10-15T17:15:51Z) - Cross Pairwise Ranking for Unbiased Item Recommendation [57.71258289870123]
We develop a new learning paradigm named Cross Pairwise Ranking (CPR)
CPR achieves unbiased recommendation without knowing the exposure mechanism.
We prove in theory that this way offsets the influence of user/item propensity on the learning.
arXiv Detail & Related papers (2022-04-26T09:20:27Z) - A Two-Stage Feature Selection Approach for Robust Evaluation of
Treatment Effects in High-Dimensional Observational Data [1.4710887888397084]
We propose a novel two-stage feature selection technique called, Outcome Adaptive Elastic Net (OAENet)
OAENet is explicitly designed for making robust causal inference decisions using matching techniques.
Numerical experiments on simulated data demonstrate that OAENet significantly outperforms state-of-the-art methods.
arXiv Detail & Related papers (2021-11-27T02:54:30Z) - Reenvisioning Collaborative Filtering vs Matrix Factorization [65.74881520196762]
Collaborative filtering models based on matrix factorization and learned similarities using Artificial Neural Networks (ANNs) have gained significant attention in recent years.
Announcement of ANNs within the recommendation ecosystem has been recently questioned, raising several comparisons in terms of efficiency and effectiveness.
We show the potential these techniques may have on beyond-accuracy evaluation while analyzing effect on complementary evaluation dimensions.
arXiv Detail & Related papers (2021-07-28T16:29:38Z) - Probabilistic and Variational Recommendation Denoising [56.879165033014026]
Learning from implicit feedback is one of the most common cases in the application of recommender systems.
We propose probabilistic and variational recommendation denoising for implicit feedback.
We employ the proposed DPI and DVAE on four state-of-the-art recommendation models and conduct experiments on three datasets.
arXiv Detail & Related papers (2021-05-20T08:59:44Z) - Scalable Personalised Item Ranking through Parametric Density Estimation [53.44830012414444]
Learning from implicit feedback is challenging because of the difficult nature of the one-class problem.
Most conventional methods use a pairwise ranking approach and negative samplers to cope with the one-class problem.
We propose a learning-to-rank approach, which achieves convergence speed comparable to the pointwise counterpart.
arXiv Detail & Related papers (2021-05-11T03:38:16Z) - Causality-Aware Neighborhood Methods for Recommender Systems [3.0919302844782717]
Business objectives of recommenders, such as increasing sales, are aligned with the causal effect of recommendations.
Previous recommenders employ the inverse propensity scoring (IPS) in causal inference.
We develop robust ranking methods for the causal effect of recommendations.
arXiv Detail & Related papers (2020-12-17T08:23:17Z) - Unbiased Learning for the Causal Effect of Recommendation [8.849159720632612]
This paper proposes an unbiased learning framework for the causal effect of recommendation.
We develop an unbiased learning method for the causal effect extension of a ranking metric.
arXiv Detail & Related papers (2020-08-11T07:30:44Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.