Correcting for Interference in Experiments: A Case Study at Douyin
- URL: http://arxiv.org/abs/2305.02542v1
- Date: Thu, 4 May 2023 04:30:30 GMT
- Title: Correcting for Interference in Experiments: A Case Study at Douyin
- Authors: Vivek F. Farias, Hao Li, Tianyi Peng, Xinyuyang Ren, Huawei Zhang,
Andrew Zheng
- Abstract summary: Interference is a ubiquitous problem in experiments conducted on two-sided content marketplaces, such as Douyin (China's analog of TikTok)
We introduce a novel Monte-Carlo estimator, based on "Differences-in-Qs" (DQ) techniques, which achieves bias that is second-order in the treatment effect, while remaining sample-efficient to estimate.
We implement our estimator on Douyin's experimentation platform, and in the process develop DQ into a truly "plug-and-play" estimator for interference in real-world settings.
- Score: 9.586075896428177
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: Interference is a ubiquitous problem in experiments conducted on two-sided
content marketplaces, such as Douyin (China's analog of TikTok). In many cases,
creators are the natural unit of experimentation, but creators interfere with
each other through competition for viewers' limited time and attention. "Naive"
estimators currently used in practice simply ignore the interference, but in
doing so incur bias on the order of the treatment effect. We formalize the
problem of inference in such experiments as one of policy evaluation.
Off-policy estimators, while unbiased, are impractically high variance. We
introduce a novel Monte-Carlo estimator, based on "Differences-in-Qs" (DQ)
techniques, which achieves bias that is second-order in the treatment effect,
while remaining sample-efficient to estimate. On the theoretical side, our
contribution is to develop a generalized theory of Taylor expansions for policy
evaluation, which extends DQ theory to all major MDP formulations. On the
practical side, we implement our estimator on Douyin's experimentation
platform, and in the process develop DQ into a truly "plug-and-play" estimator
for interference in real-world settings: one which provides robust, low-bias,
low-variance treatment effect estimates; admits computationally cheap,
asymptotically exact uncertainty quantification; and reduces MSE by 99\%
compared to the best existing alternatives in our applications.
Related papers
- Targeted Machine Learning for Average Causal Effect Estimation Using the
Front-Door Functional [3.0232957374216953]
evaluating the average causal effect (ACE) of a treatment on an outcome often involves overcoming the challenges posed by confounding factors in observational studies.
Here, we introduce novel estimation strategies for the front-door criterion based on the targeted minimum loss-based estimation theory.
We demonstrate the applicability of these estimators to analyze the effect of early stage academic performance on future yearly income.
arXiv Detail & Related papers (2023-12-15T22:04:53Z) - Semiparametric Efficient Inference in Adaptive Experiments [29.43493007296859]
We consider the problem of efficient inference of the Average Treatment Effect in a sequential experiment where the policy governing the assignment of subjects to treatment or control can change over time.
We first provide a central limit theorem for the Adaptive Augmented Inverse-Probability Weighted estimator, which is semi efficient, under weaker assumptions than those previously made in the literature.
We then consider sequential inference setting, deriving both propensity and nonasymptotic confidence sequences that are considerably tighter than previous methods.
arXiv Detail & Related papers (2023-11-30T06:25:06Z) - A Double Machine Learning Approach to Combining Experimental and Observational Data [59.29868677652324]
We propose a double machine learning approach to combine experimental and observational studies.
Our framework tests for violations of external validity and ignorability under milder assumptions.
arXiv Detail & Related papers (2023-07-04T02:53:11Z) - Improved Policy Evaluation for Randomized Trials of Algorithmic Resource
Allocation [54.72195809248172]
We present a new estimator leveraging our proposed novel concept, that involves retrospective reshuffling of participants across experimental arms at the end of an RCT.
We prove theoretically that such an estimator is more accurate than common estimators based on sample means.
arXiv Detail & Related papers (2023-02-06T05:17:22Z) - Neighborhood Adaptive Estimators for Causal Inference under Network
Interference [152.4519491244279]
We consider the violation of the classical no-interference assumption, meaning that the treatment of one individuals might affect the outcomes of another.
To make interference tractable, we consider a known network that describes how interference may travel.
We study estimators for the average direct treatment effect on the treated in such a setting.
arXiv Detail & Related papers (2022-12-07T14:53:47Z) - Data-Driven Influence Functions for Optimization-Based Causal Inference [105.5385525290466]
We study a constructive algorithm that approximates Gateaux derivatives for statistical functionals by finite differencing.
We study the case where probability distributions are not known a priori but need to be estimated from data.
arXiv Detail & Related papers (2022-08-29T16:16:22Z) - Markovian Interference in Experiments [7.426870925611945]
We consider experiments in dynamical systems where interventions on some experimental units impact other units through a limiting constraint.
Despite outsize practical importance, the best estimators for this problem are largely in nature, and their bias is not well understood.
Off-policy estimators, while unbiased, apparently incur a large penalty in variance relative to state-of-the-art alternatives.
We introduce an on-policy estimator: the Differences-In-Q's (DQ) estimator.
arXiv Detail & Related papers (2022-06-06T05:53:36Z) - A New Central Limit Theorem for the Augmented IPW Estimator: Variance
Inflation, Cross-Fit Covariance and Beyond [0.9172870611255595]
Cross-fit inverse probability weighting (AIPW) with cross-fitting is a popular choice in practice.
We study this cross-fit AIPW estimator under well-specified outcome regression and propensity score models in a high-dimensional regime.
Our work utilizes a novel interplay between three distinct tools--approximate message passing theory, the theory of deterministic equivalents, and the leave-one-out approach.
arXiv Detail & Related papers (2022-05-20T14:17:53Z) - Learning to Estimate Without Bias [57.82628598276623]
Gauss theorem states that the weighted least squares estimator is a linear minimum variance unbiased estimation (MVUE) in linear models.
In this paper, we take a first step towards extending this result to non linear settings via deep learning with bias constraints.
A second motivation to BCE is in applications where multiple estimates of the same unknown are averaged for improved performance.
arXiv Detail & Related papers (2021-10-24T10:23:51Z) - Scalable Personalised Item Ranking through Parametric Density Estimation [53.44830012414444]
Learning from implicit feedback is challenging because of the difficult nature of the one-class problem.
Most conventional methods use a pairwise ranking approach and negative samplers to cope with the one-class problem.
We propose a learning-to-rank approach, which achieves convergence speed comparable to the pointwise counterpart.
arXiv Detail & Related papers (2021-05-11T03:38:16Z) - Valid Causal Inference with (Some) Invalid Instruments [24.794879633855373]
We show how to perform consistent IV estimation despite violations of the exclusion assumption.
We achieve accurate estimates of conditional average treatment effects using an ensemble of deep network-based estimators.
arXiv Detail & Related papers (2020-06-19T21:09:26Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.