Variance Reduction for Heavy-Tailed Monetization Metrics in Ranking Experiments via Post-Stratification

• URL: http://arxiv.org/abs/2606.04110v1
• Date: Tue, 02 Jun 2026 18:14:14 GMT
• Title: Variance Reduction for Heavy-Tailed Monetization Metrics in Ranking Experiments via Post-Stratification
• Abstract summary: We present a practical framework for variance reduction in online experiments by combining post-stratification with CUPED.<n> Deployed at ShareChat across ranking-driven monetization experiments, the method substantially reduces variance and improves decision stability.
• Score: 16.57321202143624
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Online evaluation of ranking and retrieval systems often relies on downstream monetization metrics such as app revenue or creator earnings. These metrics are typically heavy-tailed, with a small fraction of users dominating both mean and variance, leading to low statistical power and unreliable conclusions in A/B experiments -- especially under limited traffic. We present a practical framework for variance reduction in online experiments by combining post-stratification with CUPED. Our approach leverages pre-experiment covariates to improve the sensitivity of monetization experiments without requiring additional traffic. Deployed at ShareChat across ranking-driven monetization experiments, the method substantially reduces variance and improves decision stability, achieving equivalent statistical confidence with ~45\% less traffic than standard metrics. We further discuss practical design choices, guardrails, and limitations, providing guidance on when post-stratification is appropriate for real-world information retrieval and Recommendation systems.

Related papers

• Accelerating A/B-Tests with Counterfactual Estimation: Reducing Variance through Policy Overlap [8.291484471359633]
  We identify a critical inefficiency in the standard A/B-testing protocol.<n>When a treatment and control policy agree on an action, the resulting outcome contributes noise but no signal regarding the treatment effect.<n>We propose a novel experimental protocol that exploits this policy overlap to accelerate experimentation.
  arXiv  Detail & Related papers  (2026-07-16T06:08:52Z)
• Offline Preference-Based Trajectory Evaluation [54.93316418589433]
  We propose preference-based trajectory evaluation, which compares trajectories directly through temporal preferences over progress and time-to-return profiles.<n>We find that, across diverse agentic and interactive benchmarks, standard success-based metrics produce tied comparisons on roughly 75% of instances, whereas trajectory-aware preferences reduce ties to roughly 35%.
  arXiv  Detail & Related papers  (2026-06-16T05:42:19Z)
• Benchmarking Few-shot Transferability of Pre-trained Models with Improved Evaluation Protocols [123.73663884421272]
  Few-shot transfer has been revolutionized by stronger pre-trained models and improved adaptation algorithms.<n>We establish FEWTRANS, a comprehensive benchmark containing 10 diverse datasets.<n>By releasing FEWTRANS, we aim to provide a rigorous "ruler" to streamline reproducible advances in few-shot transfer learning research.
  arXiv  Detail & Related papers  (2026-02-28T05:41:57Z)
• Profit over Proxies: A Scalable Bayesian Decision Framework for Optimizing Multi-Variant Online Experiments [0.0352925259310339]
  Online controlled experiments (A/B tests) are fundamental to data-driven decision-making in the digital economy.<n>"p-value" inflates false positive rates, and an over-reliance on proxy metrics like conversion rates can lead to decisions that inadvertently harm core business profitability.<n>This paper introduces a comprehensive and scalable Bayesian decision framework designed for profit optimization in multi-variant (A/B/n) experiments.
  arXiv  Detail & Related papers  (2025-09-16T02:24:20Z)
• Practical Improvements of A/B Testing with Off-Policy Estimation [51.25970890274447]
  We introduce a family of unbiased off-policy estimators that achieves lower variance than the standard approach.<n>Our theoretical analysis and experimental results validate the effectiveness and practicality of the proposed method.
  arXiv  Detail & Related papers  (2025-06-12T13:11:01Z)
• Benchmarking Transferability: A Framework for Fair and Robust Evaluation [6.9052557953336295]
  Transferability scores aim to quantify how well a model trained on one domain generalizes to a target domain.<n>Despite numerous methods proposed for measuring transferability, their reliability and practical usefulness remain inconclusive.<n>We introduce a comprehensive benchmarking framework designed to systematically evaluate transferability scores across diverse settings.
  arXiv  Detail & Related papers  (2025-04-28T11:01:43Z)
• STATE: A Robust ATE Estimator of Heavy-Tailed Metrics for Variance Reduction in Online Controlled Experiments [22.32661807469984]
  We develop a novel framework that integrates the Student's t-distribution with machine learning tools to fit heavy-tailed metrics. By adopting a variational EM method to optimize the loglikehood function, we can infer a robust solution that greatly eliminates the negative impact of outliers. Both simulations on synthetic data and long-term empirical results on Meituan experiment platform demonstrate the effectiveness of our method.
  arXiv  Detail & Related papers  (2024-07-23T09:35:59Z)
• Optimal Baseline Corrections for Off-Policy Contextual Bandits [61.740094604552475]
  We aim to learn decision policies that optimize an unbiased offline estimate of an online reward metric. We propose a single framework built on their equivalence in learning scenarios. Our framework enables us to characterize the variance-optimal unbiased estimator and provide a closed-form solution for it.
  arXiv  Detail & Related papers  (2024-05-09T12:52:22Z)
• Variance Reduction in Ratio Metrics for Efficient Online Experiments [12.036747050794135]
  We apply variance reduction techniques to ratio metrics on a large-scale short-video platform: ShareChat. Our results show that we can either improve A/B-test confidence in 77% of cases, or can retain the same level of confidence with 30% fewer data points.
  arXiv  Detail & Related papers  (2024-01-08T18:01:09Z)
• Uncertainty-Aware Instance Reweighting for Off-Policy Learning [63.31923483172859]
  We propose a Uncertainty-aware Inverse Propensity Score estimator (UIPS) for improved off-policy learning. Experiment results on synthetic and three real-world recommendation datasets demonstrate the advantageous sample efficiency of the proposed UIPS estimator.
  arXiv  Detail & Related papers  (2023-03-11T11:42:26Z)
• Improved Policy Evaluation for Randomized Trials of Algorithmic Resource Allocation [54.72195809248172]
  We present a new estimator leveraging our proposed novel concept, that involves retrospective reshuffling of participants across experimental arms at the end of an RCT. We prove theoretically that such an estimator is more accurate than common estimators based on sample means.
  arXiv  Detail & Related papers  (2023-02-06T05:17:22Z)
• Counterfactual Evaluation of Slate Recommendations with Sequential Reward Interactions [18.90946044396516]
  Music streaming, video streaming, news recommendation, and e-commerce services often engage with content in a sequential manner. Providing and evaluating good sequences of recommendations is therefore a central problem for these services. We propose a new counterfactual estimator that allows for sequential interactions in the rewards with lower variance in anally unbiased manner.
  arXiv  Detail & Related papers  (2020-07-25T17:58:01Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.