Correcting for Interference in Experiments: A Case Study at Douyin
- URL: http://arxiv.org/abs/2305.02542v1
- Date: Thu, 4 May 2023 04:30:30 GMT
- Title: Correcting for Interference in Experiments: A Case Study at Douyin
- Authors: Vivek F. Farias, Hao Li, Tianyi Peng, Xinyuyang Ren, Huawei Zhang,
Andrew Zheng
- Abstract summary: Interference is a ubiquitous problem in experiments conducted on two-sided content marketplaces, such as Douyin (China's analog of TikTok)
We introduce a novel Monte-Carlo estimator, based on "Differences-in-Qs" (DQ) techniques, which achieves bias that is second-order in the treatment effect, while remaining sample-efficient to estimate.
We implement our estimator on Douyin's experimentation platform, and in the process develop DQ into a truly "plug-and-play" estimator for interference in real-world settings.
- Score: 9.586075896428177
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: Interference is a ubiquitous problem in experiments conducted on two-sided
content marketplaces, such as Douyin (China's analog of TikTok). In many cases,
creators are the natural unit of experimentation, but creators interfere with
each other through competition for viewers' limited time and attention. "Naive"
estimators currently used in practice simply ignore the interference, but in
doing so incur bias on the order of the treatment effect. We formalize the
problem of inference in such experiments as one of policy evaluation.
Off-policy estimators, while unbiased, are impractically high variance. We
introduce a novel Monte-Carlo estimator, based on "Differences-in-Qs" (DQ)
techniques, which achieves bias that is second-order in the treatment effect,
while remaining sample-efficient to estimate. On the theoretical side, our
contribution is to develop a generalized theory of Taylor expansions for policy
evaluation, which extends DQ theory to all major MDP formulations. On the
practical side, we implement our estimator on Douyin's experimentation
platform, and in the process develop DQ into a truly "plug-and-play" estimator
for interference in real-world settings: one which provides robust, low-bias,
low-variance treatment effect estimates; admits computationally cheap,
asymptotically exact uncertainty quantification; and reduces MSE by 99\%
compared to the best existing alternatives in our applications.
Related papers
- In-Context Parametric Inference: Point or Distribution Estimators? [66.22308335324239]
We show that amortized point estimators generally outperform posterior inference, though the latter remain competitive in some low-dimensional problems.
Our experiments indicate that amortized point estimators generally outperform posterior inference, though the latter remain competitive in some low-dimensional problems.
arXiv Detail & Related papers (2025-02-17T10:00:24Z) - Model-free Methods for Event History Analysis and Efficient Adjustment (PhD Thesis) [55.2480439325792]
This thesis is a series of independent contributions to statistics unified by a model-free perspective.
The first chapter elaborates on how a model-free perspective can be used to formulate flexible methods that leverage prediction techniques from machine learning.
The second chapter studies the concept of local independence, which describes whether the evolution of one process is directly influenced by another.
arXiv Detail & Related papers (2025-02-11T19:24:09Z) - Doubly Robust Inference on Causal Derivative Effects for Continuous Treatments [5.151880096713011]
We investigate nonparametric inference on the derivative of the dose-response curve with and without the positivity condition.
We propose a doubly robust (DR) inference method for estimating the derivative of the dose-response curve using kernel smoothing.
In all settings, our DR estimators achieves normality at the standard nonparametric rate of convergence.
arXiv Detail & Related papers (2025-01-12T23:00:16Z) - Semiparametric Efficient Inference in Adaptive Experiments [29.43493007296859]
We consider the problem of efficient inference of the Average Treatment Effect in a sequential experiment where the policy governing the assignment of subjects to treatment or control can change over time.
We first provide a central limit theorem for the Adaptive Augmented Inverse-Probability Weighted estimator, which is semi efficient, under weaker assumptions than those previously made in the literature.
We then consider sequential inference setting, deriving both propensity and nonasymptotic confidence sequences that are considerably tighter than previous methods.
arXiv Detail & Related papers (2023-11-30T06:25:06Z) - A Double Machine Learning Approach to Combining Experimental and Observational Data [59.29868677652324]
We propose a double machine learning approach to combine experimental and observational studies.
Our framework tests for violations of external validity and ignorability under milder assumptions.
arXiv Detail & Related papers (2023-07-04T02:53:11Z) - Neighborhood Adaptive Estimators for Causal Inference under Network
Interference [152.4519491244279]
We consider the violation of the classical no-interference assumption, meaning that the treatment of one individuals might affect the outcomes of another.
To make interference tractable, we consider a known network that describes how interference may travel.
We study estimators for the average direct treatment effect on the treated in such a setting.
arXiv Detail & Related papers (2022-12-07T14:53:47Z) - Data-Driven Influence Functions for Optimization-Based Causal Inference [105.5385525290466]
We study a constructive algorithm that approximates Gateaux derivatives for statistical functionals by finite differencing.
We study the case where probability distributions are not known a priori but need to be estimated from data.
arXiv Detail & Related papers (2022-08-29T16:16:22Z) - Markovian Interference in Experiments [7.426870925611945]
We consider experiments in dynamical systems where interventions on some experimental units impact other units through a limiting constraint.
Despite outsize practical importance, the best estimators for this problem are largely in nature, and their bias is not well understood.
Off-policy estimators, while unbiased, apparently incur a large penalty in variance relative to state-of-the-art alternatives.
We introduce an on-policy estimator: the Differences-In-Q's (DQ) estimator.
arXiv Detail & Related papers (2022-06-06T05:53:36Z) - A New Central Limit Theorem for the Augmented IPW Estimator: Variance
Inflation, Cross-Fit Covariance and Beyond [0.9172870611255595]
Cross-fit inverse probability weighting (AIPW) with cross-fitting is a popular choice in practice.
We study this cross-fit AIPW estimator under well-specified outcome regression and propensity score models in a high-dimensional regime.
Our work utilizes a novel interplay between three distinct tools--approximate message passing theory, the theory of deterministic equivalents, and the leave-one-out approach.
arXiv Detail & Related papers (2022-05-20T14:17:53Z) - Learning to Estimate Without Bias [57.82628598276623]
Gauss theorem states that the weighted least squares estimator is a linear minimum variance unbiased estimation (MVUE) in linear models.
In this paper, we take a first step towards extending this result to non linear settings via deep learning with bias constraints.
A second motivation to BCE is in applications where multiple estimates of the same unknown are averaged for improved performance.
arXiv Detail & Related papers (2021-10-24T10:23:51Z) - Valid Causal Inference with (Some) Invalid Instruments [24.794879633855373]
We show how to perform consistent IV estimation despite violations of the exclusion assumption.
We achieve accurate estimates of conditional average treatment effects using an ensemble of deep network-based estimators.
arXiv Detail & Related papers (2020-06-19T21:09:26Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.