Unifying On- and Off-Policy Variance Reduction Methods

• URL: http://arxiv.org/abs/2603.08370v1
• Date: Mon, 09 Mar 2026 13:32:39 GMT
• Title: Unifying On- and Off-Policy Variance Reduction Methods
• Abstract summary: We prove that the standard online Difference-in-Means estimator is mathematically identical to an off-policy Inverse Propensity Scoring estimator.<n> Extending this unification, we demonstrate that widespread regression adjustment methods are structurally equivalent to Doubly Robust estimation.
• Score: 8.291484471359633
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Continuous and efficient experimentation is key to the practical success of user-facing applications on the web, both through online A/B-tests and off-policy evaluation. Despite their shared objective -- estimating the incremental value of a treatment -- these domains often operate in isolation, utilising distinct terminologies and statistical toolkits. This paper bridges that divide by establishing a formal equivalence between their canonical variance reduction methods. We prove that the standard online Difference-in-Means estimator is mathematically identical to an off-policy Inverse Propensity Scoring estimator equipped with an optimal (variance-minimising) additive control variate. Extending this unification, we demonstrate that widespread regression adjustment methods (such as CUPED, CUPAC, and ML-RATE) are structurally equivalent to Doubly Robust estimation. This unified view extends our understanding of commonly used approaches, and can guide practitioners and researchers working on either class of problems.

Related papers

• Accelerating A/B-Tests with Counterfactual Estimation: Reducing Variance through Policy Overlap [8.291484471359633]
  We identify a critical inefficiency in the standard A/B-testing protocol.<n>When a treatment and control policy agree on an action, the resulting outcome contributes noise but no signal regarding the treatment effect.<n>We propose a novel experimental protocol that exploits this policy overlap to accelerate experimentation.
  arXiv  Detail & Related papers  (2026-07-16T06:08:52Z)
• Practical Improvements of A/B Testing with Off-Policy Estimation [51.25970890274447]
  We introduce a family of unbiased off-policy estimators that achieves lower variance than the standard approach.<n>Our theoretical analysis and experimental results validate the effectiveness and practicality of the proposed method.
  arXiv  Detail & Related papers  (2025-06-12T13:11:01Z)
• Achieving $\widetilde{\mathcal{O}}(\sqrt{T})$ Regret in Average-Reward POMDPs with Known Observation Models [69.1820058966619]
  We tackle average-reward infinite-horizon POMDPs with an unknown transition model.<n>We present a novel and simple estimator that overcomes this barrier.
  arXiv  Detail & Related papers  (2025-01-30T22:29:41Z)
• $Δ\ ext{-}{\ m OPE}$: Off-Policy Estimation with Pairs of Policies [13.528097424046823]
  We introduce $Deltatext-rm OPE$ methods based on the widely used Inverse Propensity Scoring estimator. Simulated, offline, and online experiments show that our methods significantly improve performance for both evaluation and learning tasks.
  arXiv  Detail & Related papers  (2024-05-16T12:04:55Z)
• Optimal Baseline Corrections for Off-Policy Contextual Bandits [61.740094604552475]
  We aim to learn decision policies that optimize an unbiased offline estimate of an online reward metric. We propose a single framework built on their equivalence in learning scenarios. Our framework enables us to characterize the variance-optimal unbiased estimator and provide a closed-form solution for it.
  arXiv  Detail & Related papers  (2024-05-09T12:52:22Z)
• Policy Gradient with Active Importance Sampling [55.112959067035916]
  Policy gradient (PG) methods significantly benefit from IS, enabling the effective reuse of previously collected samples. However, IS is employed in RL as a passive tool for re-weighting historical samples. We look for the best behavioral policy from which to collect samples to reduce the policy gradient variance.
  arXiv  Detail & Related papers  (2024-05-09T09:08:09Z)
• A Semiparametric Instrumented Difference-in-Differences Approach to Policy Learning [2.1989182578668243]
  We propose a general instrumented difference-in-differences (DiD) approach for learning the optimal treatment policy. Specifically, we establish identification results using a binary instrumental variable (IV) when the parallel trends assumption fails to hold. We also construct a Wald estimator, novel inverse probability estimators, and a class of semi efficient and multiply robust estimators.
  arXiv  Detail & Related papers  (2023-10-14T09:38:32Z)
• Scalable Personalised Item Ranking through Parametric Density Estimation [53.44830012414444]
  Learning from implicit feedback is challenging because of the difficult nature of the one-class problem. Most conventional methods use a pairwise ranking approach and negative samplers to cope with the one-class problem. We propose a learning-to-rank approach, which achieves convergence speed comparable to the pointwise counterpart.
  arXiv  Detail & Related papers  (2021-05-11T03:38:16Z)
• GenDICE: Generalized Offline Estimation of Stationary Values [108.17309783125398]
  We show that effective estimation can still be achieved in important applications. Our approach is based on estimating a ratio that corrects for the discrepancy between the stationary and empirical distributions. The resulting algorithm, GenDICE, is straightforward and effective.
  arXiv  Detail & Related papers  (2020-02-21T00:27:52Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.