Policy-Aware Design of Large-Scale Factorial Experiments

• URL: http://arxiv.org/abs/2604.08804v1
• Date: Thu, 09 Apr 2026 22:39:18 GMT
• Title: Policy-Aware Design of Large-Scale Factorial Experiments
• Abstract summary: Overlapping experiments can generate interaction effects that are poorly handled by decentralized A/B testing.<n>We study how to design large-scale factorial experiments when the objective is not to estimate every treatment effect.<n>We propose a two-stage design that centralizes overlapping experiments into a single factorial problem and models expected outcomes as a low-rank tensor.
• Score: 10.530820072207
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Digital firms routinely run many online experiments on shared user populations. When product decisions are compositional, such as combinations of interface elements, flows, messages, or incentives, the number of feasible interventions grows combinatorially, while available traffic remains limited. Overlapping experiments can therefore generate interaction effects that are poorly handled by decentralized A/B testing. We study how to design large-scale factorial experiments when the objective is not to estimate every treatment effect, but to identify a high-performing policy under a fixed experimentation budget. We propose a two-stage design that centralizes overlapping experiments into a single factorial problem and models expected outcomes as a low-rank tensor. In the first stage, the platform samples a subset of intervention combinations, uses tensor completion to infer performance on untested combinations, and eliminates weak factor levels using estimated marginal contributions. In the second stage, it applies sequential halving to the surviving combinations to select a final policy. We establish gap-independent simple-regret bounds and gap-dependent identification guarantees showing that the relevant complexity scales with the degrees of freedom of the low-rank tensor and the separation structure across factor levels, rather than the full factorial size. In an offline evaluation based on a product-bundling problem constructed from 100 million Taobao interactions, the proposed method substantially outperforms one-shot tensor completion and unstructured best-arm benchmarks, especially in low-budget and high-noise settings. These results show how centralized, policy-aware experimentation can make combinatorial product design operationally feasible at platform scale.

Related papers

• Before Thinking, Learn to Decide: Proactive Routing for Efficient Visual Reasoning [87.18590662029432]
  We propose PRP, a Proactive Routing Paradigm that enables early decision-making by jointly evaluating the competence of both the draft and target models.<n>Our draft Rating Learning (DRL) equips the draft model with an internal confidence estimator, while Joint Rating Learning (JRL) predicts how well the target model can handle a given query.<n>These ratings enable fine-grained, instance-level textbfProactive Routing and substantially accelerate inference without compromising overall performance.
  arXiv  Detail & Related papers  (2026-06-29T12:30:24Z)
• Towards Reliable Social A/B Testing: Spillover-Contained Clustering with Robust Post-Experiment Analysis [11.30339991179317]
  A/B testing is the foundation of decision-making in online platforms, but social products often suffer from network interference.<n>We propose a spillover-contained experimentation framework with two stages.<n>We validate our approach through large-scale social sharing experiments on Kuaishou, a platform serving hundreds of millions of users.
  arXiv  Detail & Related papers  (2026-02-09T12:08:29Z)
• Hierarchical Clustering With Confidence [6.479319856992936]
  Agglomerative hierarchical clustering is highly sensitive to small perturbations in the data.<n>We show how randomizing hierarchical clustering can be useful not just for measuring stability but also for designing valid hypothesis testing procedures.
  arXiv  Detail & Related papers  (2025-12-06T18:18:20Z)
• Q-function Decomposition with Intervention Semantics with Factored Action Spaces [51.01244229483353]
  We consider Q-functions defined over a lower dimensional projected subspace of the original action space, and study the condition for the unbiasedness of decomposed Q-functions.<n>This leads to a general scheme which we call action decomposed reinforcement learning that uses the projected Q-functions to approximate the Q-function in standard model-free reinforcement learning algorithms.
  arXiv  Detail & Related papers  (2025-04-30T05:26:51Z)
• Differences-in-Neighbors for Network Interference in Experiments [5.079602839359523]
  We propose a new estimator, dubbed Differences-in-Neighbors (DN), designed explicitly to mitigate network interference.<n>Compared to DM estimators, DN bias second order in the magnitude of the interference effect, while its variance is exponentially smaller than that of HT estimators.<n> Empirical evaluations on a large-scale social network and a city-level ride-sharing simulator demonstrate DN's superior performance.
  arXiv  Detail & Related papers  (2025-03-04T04:40:12Z)
• Efficient Biological Data Acquisition through Inference Set Design [3.9633147697178996]
  In this work, we aim to select the smallest set of candidates in order to achieve some desired level of accuracy for the system as a whole.<n>We call this mechanism inference set design, and propose the use of a confidence-based active learning solution to prune out challenging examples.
  arXiv  Detail & Related papers  (2024-10-25T15:34:03Z)
• Data-light Uncertainty Set Merging with Admissibility [5.140740197135575]
  This article introduces a Synthetics, Aggregation, and Test inversion (SAT) approach for merging diverse and potentially dependent uncertainty sets into a single unified set.<n>SAT is motivated by the challenge of integrating uncertainty sets when only the initial sets and their control levels are available.<n>Key theoretical contribution is a rigorous analysis of SAT's properties, including its admissibility in the context of deterministic set merging.
  arXiv  Detail & Related papers  (2024-10-16T03:52:47Z)
• Adaptive Experimentation When You Can't Experiment [55.86593195947978]
  This paper introduces the emphconfounded pure exploration transductive linear bandit (textttCPET-LB) problem. Online services can employ a properly randomized encouragement that incentivizes users toward a specific treatment.
  arXiv  Detail & Related papers  (2024-06-15T20:54:48Z)
• Probabilistic Contrastive Learning for Long-Tailed Visual Recognition [78.70453964041718]
  Longtailed distributions frequently emerge in real-world data, where a large number of minority categories contain a limited number of samples. Recent investigations have revealed that supervised contrastive learning exhibits promising potential in alleviating the data imbalance. We propose a novel probabilistic contrastive (ProCo) learning algorithm that estimates the data distribution of the samples from each class in the feature space.
  arXiv  Detail & Related papers  (2024-03-11T13:44:49Z)
• On the Complexity of Multi-Agent Decision Making: From Learning in Games to Partial Monitoring [105.13668993076801]
  A central problem in the theory of multi-agent reinforcement learning (MARL) is to understand what structural conditions and algorithmic principles lead to sample-efficient learning guarantees. We study this question in a general framework for interactive decision making with multiple agents. We show that characterizing the statistical complexity for multi-agent decision making is equivalent to characterizing the statistical complexity of single-agent decision making.
  arXiv  Detail & Related papers  (2023-05-01T06:46:22Z)
• Leveraging Ensembles and Self-Supervised Learning for Fully-Unsupervised Person Re-Identification and Text Authorship Attribution [77.85461690214551]
  Learning from fully-unlabeled data is challenging in Multimedia Forensics problems, such as Person Re-Identification and Text Authorship Attribution. Recent self-supervised learning methods have shown to be effective when dealing with fully-unlabeled data in cases where the underlying classes have significant semantic differences. We propose a strategy to tackle Person Re-Identification and Text Authorship Attribution by enabling learning from unlabeled data even when samples from different classes are not prominently diverse.
  arXiv  Detail & Related papers  (2022-02-07T13:08:11Z)
• Enhanced Principal Component Analysis under A Collaborative-Robust Framework [89.28334359066258]
  We introduce a general collaborative-robust weight learning framework that combines weight learning and robust loss in a non-trivial way. Under the proposed framework, only a part of well-fitting samples are activated which indicates more importance during training, and others, whose errors are large, will not be ignored. In particular, the negative effects of inactivated samples are alleviated by the robust loss function.
  arXiv  Detail & Related papers  (2021-03-22T15:17:37Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.