Estimating Treatment Effects under Recommender Interference: A Structured Neural Networks Approach

• URL: http://arxiv.org/abs/2406.14380v3
• Date: Fri, 5 Jul 2024 13:40:48 GMT
• Title: Estimating Treatment Effects under Recommender Interference: A Structured Neural Networks Approach
• Authors: Ruohan Zhan, Shichao Han, Yuchen Hu, Zhenling Jiang,
• Abstract summary: We show that the standard difference-in-means estimator can lead to biased estimates due to recommender interference. We propose a "recommender choice model" that describes which item gets exposed from a pool containing both treated and control items. We show that the proposed estimator yields results comparable to the benchmark, whereas the standard difference-in-means estimator can exhibit significant bias and even produce reversed signs.
• Score: 13.208141830901845
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Recommender systems are essential for content-sharing platforms by curating personalized content. To evaluate updates to recommender systems targeting content creators, platforms frequently rely on creator-side randomized experiments. The treatment effect measures the change in outcomes when a new algorithm is implemented compared to the status quo. We show that the standard difference-in-means estimator can lead to biased estimates due to recommender interference that arises when treated and control creators compete for exposure. We propose a "recommender choice model" that describes which item gets exposed from a pool containing both treated and control items. By combining a structural choice model with neural networks, this framework directly models the interference pathway while accounting for rich viewer-content heterogeneity. We construct a debiased estimator of the treatment effect and prove it is $\sqrt n$-consistent and asymptotically normal with potentially correlated samples. We validate our estimator's empirical performance with a field experiment on Weixin short-video platform. In addition to the standard creator-side experiment, we conduct a costly double-sided randomization design to obtain a benchmark estimate free from interference bias. We show that the proposed estimator yields results comparable to the benchmark, whereas the standard difference-in-means estimator can exhibit significant bias and even produce reversed signs.

Related papers

• Adaptive Experimentation When You Can't Experiment [55.86593195947978]
  This paper introduces the emphconfounded pure exploration transductive linear bandit (textttCPET-LB) problem. Online services can employ a properly randomized encouragement that incentivizes users toward a specific treatment.
  arXiv  Detail & Related papers  (2024-06-15T20:54:48Z)
• Optimal Baseline Corrections for Off-Policy Contextual Bandits [61.740094604552475]
  We aim to learn decision policies that optimize an unbiased offline estimate of an online reward metric. We propose a single framework built on their equivalence in learning scenarios. Our framework enables us to characterize the variance-optimal unbiased estimator and provide a closed-form solution for it.
  arXiv  Detail & Related papers  (2024-05-09T12:52:22Z)
• Exploring Popularity Bias in Session-based Recommendation [0.6798775532273751]
  We extend the analysis to session-based setup and adapted propensity calculation to the unique characteristics of session-based recommendation tasks. We study the distributions of propensity and different stratification techniques on different datasets and find that propensity-related traits are actually dataset-specific.
  arXiv  Detail & Related papers  (2023-12-13T02:48:35Z)
• Likelihood Ratio Confidence Sets for Sequential Decision Making [51.66638486226482]
  We revisit the likelihood-based inference principle and propose to use likelihood ratios to construct valid confidence sequences. Our method is especially suitable for problems with well-specified likelihoods. We show how to provably choose the best sequence of estimators and shed light on connections to online convex optimization.
  arXiv  Detail & Related papers  (2023-11-08T00:10:21Z)
• Deep Causal Reasoning for Recommendations [47.83224399498504]
  A new trend in recommender system research is to negate the influence of confounders from a causal perspective. We model the recommendation as a multi-cause multi-outcome (MCMO) inference problem. We show that MCMO modeling may lead to high variance due to scarce observations associated with the high-dimensional causal space.
  arXiv  Detail & Related papers  (2022-01-06T15:00:01Z)
• Control Variates for Slate Off-Policy Evaluation [112.35528337130118]
  We study the problem of off-policy evaluation from batched contextual bandit data with multidimensional actions. We obtain new estimators with risk improvement guarantees over both the PI and self-normalized PI estimators.
  arXiv  Detail & Related papers  (2021-06-15T06:59:53Z)
• Deconfounding Scores: Feature Representations for Causal Effect Estimation with Weak Overlap [140.98628848491146]
  We introduce deconfounding scores, which induce better overlap without biasing the target of estimation. We show that deconfounding scores satisfy a zero-covariance condition that is identifiable in observed data. In particular, we show that this technique could be an attractive alternative to standard regularizations.
  arXiv  Detail & Related papers  (2021-04-12T18:50:11Z)
• Counterfactual Evaluation of Slate Recommendations with Sequential Reward Interactions [18.90946044396516]
  Music streaming, video streaming, news recommendation, and e-commerce services often engage with content in a sequential manner. Providing and evaluating good sequences of recommendations is therefore a central problem for these services. We propose a new counterfactual estimator that allows for sequential interactions in the rewards with lower variance in anally unbiased manner.
  arXiv  Detail & Related papers  (2020-07-25T17:58:01Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.