Beyond ATE: Multi-Criteria Design for A/B Testing

• URL: http://arxiv.org/abs/2509.05864v1
• Date: Sat, 06 Sep 2025 23:42:22 GMT
• Title: Beyond ATE: Multi-Criteria Design for A/B Testing
• Authors: Jiachun Li, Kaining Shi, David Simchi-Levi,
• Abstract summary: A/B testing is a widely adopted methodology for estimating conditional average treatment effects (CATEs) in both clinical trials and online platforms.<n>In this paper, we investigate the fundamental trade-off between social welfare loss and statistical accuracy in (adaptive) experiments with heterogeneous treatment effects.
• Score: 16.67400872891851
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: A/B testing is a widely adopted methodology for estimating conditional average treatment effects (CATEs) in both clinical trials and online platforms. While most existing research has focused primarily on maximizing estimation accuracy, practical applications must also account for additional objectives-most notably welfare or revenue loss. In many settings, it is critical to administer treatments that improve patient outcomes or to implement plans that generate greater revenue from customers. Within a machine learning framework, such objectives are naturally captured through the notion of cumulative regret. In this paper, we investigate the fundamental trade-off between social welfare loss and statistical accuracy in (adaptive) experiments with heterogeneous treatment effects. We establish matching upper and lower bounds for the resulting multi-objective optimization problem and employ the concept of Pareto optimality to characterize the necessary and sufficient conditions for optimal experimental designs. Beyond estimating CATEs, practitioners often aim to deploy treatment policies that maximize welfare across the entire population. We demonstrate that our Pareto-optimal adaptive design achieves optimal post-experiment welfare, irrespective of the in-experiment trade-off between accuracy and welfare. Furthermore, since clinical and commercial data are often highly sensitive, it is essential to incorporate robust privacy guarantees into any treatment-allocation mechanism. To this end, we develop differentially private algorithms that continue to achieve our established lower bounds, showing that privacy can be attained at negligible cost.

Related papers

• Treatment Effect Estimation for Optimal Decision-Making [65.30942348196443]
  We study optimal decision-making based on two-stage CATE estimators.<n>We propose a novel two-stage learning objective that retargets the CATE to balance CATE estimation error and decision performance.
  arXiv  Detail & Related papers  (2025-05-19T13:24:57Z)
• Enhancing Treatment Effect Estimation via Active Learning: A Counterfactual Covering Perspective [61.284843894545475]
  Complex algorithms for treatment effect estimation are ineffective when handling insufficiently labeled training sets.<n>We propose FCCM, which transforms the optimization objective into the textitFactual and textitCounterfactual Coverage Maximization to ensure effective radius reduction during data acquisition.<n> benchmarking FCCM against other baselines demonstrates its superiority across both fully synthetic and semi-synthetic datasets.
  arXiv  Detail & Related papers  (2025-05-08T13:42:00Z)
• Optimal Adaptive Experimental Design for Estimating Treatment Effect [14.088972921434761]
  This paper addresses the fundamental question of determining the optimal accuracy in estimating the treatment effect. By incorporating the concept of doubly robust method into sequential experimental design, we frame the optimal estimation problem as an online bandit learning problem. Using tools and ideas from both bandit algorithm design and adaptive statistical estimation, we propose a general low switching adaptive experiment framework.
  arXiv  Detail & Related papers  (2024-10-07T23:22:51Z)
• Are causal effect estimations enough for optimal recommendations under multitreatment scenarios? [2.4578723416255754]
  It is essential to include a causal effect estimation analysis to compare potential outcomes under different treatments or controls. We propose a comprehensive methodology for multitreatment selection.
  arXiv  Detail & Related papers  (2024-10-07T16:37:35Z)
• Reduced-Rank Multi-objective Policy Learning and Optimization [57.978477569678844]
  In practice, causal researchers do not have a single outcome in mind a priori. In government-assisted social benefit programs, policymakers collect many outcomes to understand the multidimensional nature of poverty. We present a data-driven dimensionality-reduction methodology for multiple outcomes in the context of optimal policy learning.
  arXiv  Detail & Related papers  (2024-04-29T08:16:30Z)
• Privacy Preserving Adaptive Experiment Design [13.839525385976303]
  We investigate the tradeoff between loss of social welfare and statistical power in contextual bandit experiment. We propose differentially private algorithms which still matches the lower bound, showing that privacy is "almost free"
  arXiv  Detail & Related papers  (2024-01-16T09:22:12Z)
• Pruning the Way to Reliable Policies: A Multi-Objective Deep Q-Learning Approach to Critical Care [46.2482873419289]
  We introduce a deep Q-learning approach to obtain more reliable critical care policies. We evaluate our method in off-policy and offline settings using simulated environments and real health records from intensive care units.
  arXiv  Detail & Related papers  (2023-06-13T18:02:57Z)
• Task-specific experimental design for treatment effect estimation [59.879567967089145]
  Large randomised trials (RCTs) are the standard for causal inference. Recent work has proposed more sample-efficient alternatives to RCTs, but these are not adaptable to the downstream application for which the causal effect is sought. We develop a task-specific approach to experimental design and derive sampling strategies customised to particular downstream applications.
  arXiv  Detail & Related papers  (2023-06-08T18:10:37Z)
• Policy Optimization for Personalized Interventions in Behavioral Health [8.10897203067601]
  Behavioral health interventions, delivered through digital platforms, have the potential to significantly improve health outcomes. We study the problem of optimizing personalized interventions for patients to maximize a long-term outcome. We present a new approach for this problem that we dub DecompPI, which decomposes the state space for a system of patients to the individual level.
  arXiv  Detail & Related papers  (2023-03-21T21:42:03Z)
• An Efficient Approach for Optimizing the Cost-effective Individualized Treatment Rule Using Conditional Random Forest [5.406112598028401]
  We use the concept of net-monetary-benefit (NMB) to assess the trade-off between health benefits and related costs. We identify the optimal CE-ITR using NMB-based classification algorithms. We apply our top-performing algorithm to the NIH-funded Systolic Blood Pressure Intervention Trial.
  arXiv  Detail & Related papers  (2022-04-23T01:36:24Z)
• Constrained multi-objective optimization of process design parameters in settings with scarce data: an application to adhesive bonding [48.7576911714538]
  Finding the optimal process parameters for an adhesive bonding process is challenging. Traditional evolutionary approaches (such as genetic algorithms) are then ill-suited to solve the problem. In this research, we successfully applied specific machine learning techniques to emulate the objective and constraint functions.
  arXiv  Detail & Related papers  (2021-12-16T10:14:39Z)
• Resource Planning for Hospitals Under Special Consideration of the COVID-19 Pandemic: Optimization and Sensitivity Analysis [87.31348761201716]
  Crises like the COVID-19 pandemic pose a serious challenge to health-care institutions. BaBSim.Hospital is a tool for capacity planning based on discrete event simulation. We aim to investigate and optimize these parameters to improve BaBSim.Hospital.
  arXiv  Detail & Related papers  (2021-05-16T12:38:35Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.