Balancing Immediate Revenue and Future Off-Policy Evaluation in Coupon Allocation

• URL: http://arxiv.org/abs/2407.11039v2
• Date: Wed, 17 Jul 2024 07:02:41 GMT
• Title: Balancing Immediate Revenue and Future Off-Policy Evaluation in Coupon Allocation
• Authors: Naoki Nishimura, Ken Kobayashi, Kazuhide Nakata,
• Abstract summary: A/B testing can validate new policies, but it risks compromising short-term revenue. We propose a novel approach that combines a model-based revenue policy and a randomized exploration policy for data collection. We empirically verified the effectiveness of the proposed mixed policy using both synthetic and real-world data.
• Score: 3.4562366784858685
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Coupon allocation drives customer purchases and boosts revenue. However, it presents a fundamental trade-off between exploiting the current optimal policy to maximize immediate revenue and exploring alternative policies to collect data for future policy improvement via off-policy evaluation (OPE). While online A/B testing can validate new policies, it risks compromising short-term revenue. Conversely, relying solely on an exploitative policy hinders the ability to reliably estimate and enhance future policies. To balance this trade-off, we propose a novel approach that combines a model-based revenue maximization policy and a randomized exploration policy for data collection. Our framework enables flexibly adjusting the mixture ratio between these two policies to optimize the balance between short-term revenue and future policy improvement. We formulate the problem of determining the optimal mixture ratio between a model-based revenue maximization policy and a randomized exploration policy for data collection. We empirically verified the effectiveness of the proposed mixed policy using both synthetic and real-world data. Our main contributions are: (1) Demonstrating a mixed policy combining deterministic and probabilistic policies, flexibly adjusting the data collection vs. revenue trade-off. (2) Formulating the optimal mixture ratio problem as multi-objective optimization, enabling quantitative evaluation of this trade-off. By optimizing the mixture ratio, businesses can maximize revenue while ensuring reliable future OPE and policy improvement. This framework is applicable in any context where the exploration-exploitation trade-off is relevant.

Related papers

• Reflective Policy Optimization [20.228281670899204]
  Reflective Policy Optimization (RPO) amalgamates past and future state-action information for policy optimization. RPO empowers the agent for introspection, allowing modifications to its actions within the current state. Empirical results demonstrate RPO's feasibility and efficacy in two reinforcement learning benchmarks.
  arXiv  Detail & Related papers  (2024-06-06T01:46:49Z)
• Off-Policy Evaluation for Large Action Spaces via Policy Convolution [60.6953713877886]
  Policy Convolution family of estimators uses latent structure within actions to strategically convolve the logging and target policies. Experiments on synthetic and benchmark datasets demonstrate remarkable mean squared error (MSE) improvements when using PC.
  arXiv  Detail & Related papers  (2023-10-24T01:00:01Z)
• Local Policy Improvement for Recommender Systems [8.617221361305901]
  We show how to train a new policy given data collected from a previously-deployed policy. We suggest an alternative approach of local policy improvement without off-policy correction. This local policy improvement paradigm is ideal for recommender systems, as previous policies are typically of decent quality and policies are updated frequently.
  arXiv  Detail & Related papers  (2022-12-22T00:47:40Z)
• A Data Science Approach to Risk Assessment for Automobile Insurance Policies [1.0660480034605242]
  We focus on risk assessment using a Data Science approach. We predict the total claims that will be made by a new customer using historical data of current and past policies.
  arXiv  Detail & Related papers  (2022-09-06T18:32:27Z)
• Latent-Variable Advantage-Weighted Policy Optimization for Offline RL [70.01851346635637]
  offline reinforcement learning methods hold the promise of learning policies from pre-collected datasets without the need to query the environment for new transitions. In practice, offline datasets are often heterogeneous, i.e., collected in a variety of scenarios. We propose to leverage latent-variable policies that can represent a broader class of policy distributions. Our method improves the average performance of the next best-performing offline reinforcement learning methods by 49% on heterogeneous datasets.
  arXiv  Detail & Related papers  (2022-03-16T21:17:03Z)
• Understanding the Effect of Stochasticity in Policy Optimization [86.7574122154668]
  We show that the preferability of optimization methods depends critically on whether exact gradients are used. Second, to explain these findings we introduce the concept of committal rate for policy optimization. Third, we show that in the absence of external oracle information, there is an inherent trade-off between exploiting geometry to accelerate convergence versus achieving optimality almost surely.
  arXiv  Detail & Related papers  (2021-10-29T06:35:44Z)
• Supervised Off-Policy Ranking [145.3039527243585]
  Off-policy evaluation (OPE) leverages data generated by other policies to evaluate a target policy. We propose supervised off-policy ranking that learns a policy scoring model by correctly ranking training policies with known performance. Our method outperforms strong baseline OPE methods in terms of both rank correlation and performance gap between the truly best and the best of the ranked top three policies.
  arXiv  Detail & Related papers  (2021-07-03T07:01:23Z)
• Active Offline Policy Selection [19.18251239758809]
  This paper addresses the problem of policy selection in domains with abundant logged data, but with a very restricted interaction budget. Several off-policy evaluation (OPE) techniques have been proposed to assess the value of policies using only logged data. We introduce a novel emphactive offline policy selection problem formulation, which combined logged data and limited online interactions to identify the best policy.
  arXiv  Detail & Related papers  (2021-06-18T17:33:13Z)
• Offline Policy Selection under Uncertainty [113.57441913299868]
  We consider offline policy selection as learning preferences over a set of policy prospects given a fixed experience dataset. Access to the full distribution over one's belief of the policy value enables more flexible selection algorithms under a wider range of downstream evaluation metrics. We show how BayesDICE may be used to rank policies with respect to any arbitrary downstream policy selection metric.
  arXiv  Detail & Related papers  (2020-12-12T23:09:21Z)
• Stable Policy Optimization via Off-Policy Divergence Regularization [50.98542111236381]
  Trust Region Policy Optimization (TRPO) and Proximal Policy Optimization (PPO) are among the most successful policy gradient approaches in deep reinforcement learning (RL) We propose a new algorithm which stabilizes the policy improvement through a proximity term that constrains the discounted state-action visitation distribution induced by consecutive policies to be close to one another. Our proposed method can have a beneficial effect on stability and improve final performance in benchmark high-dimensional control tasks.
  arXiv  Detail & Related papers  (2020-03-09T13:05:47Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.