Multi-Objective Recommendation via Multivariate Policy Learning

• URL: http://arxiv.org/abs/2405.02141v2
• Date: Mon, 16 Sep 2024 09:21:15 GMT
• Title: Multi-Objective Recommendation via Multivariate Policy Learning
• Authors: Olivier Jeunen, Jatin Mandav, Ivan Potapov, Nakul Agarwal, Sourabh Vaid, Wenzhe Shi, Aleksei Ustimenko,
• Abstract summary: Real-world recommender systems often need to balance multiple objectives when deciding which recommendations to present to users. These include behavioural signals (e.g. clicks, shares, dwell time), as well as broader objectives (e.g. diversity, fairness)
• Score: 10.494676556696213
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Real-world recommender systems often need to balance multiple objectives when deciding which recommendations to present to users. These include behavioural signals (e.g. clicks, shares, dwell time), as well as broader objectives (e.g. diversity, fairness). Scalarisation methods are commonly used to handle this balancing task, where a weighted average of per-objective reward signals determines the final score used for ranking. Naturally, how these weights are computed exactly, is key to success for any online platform. We frame this as a decision-making task, where the scalarisation weights are actions taken to maximise an overall North Star reward (e.g. long-term user retention or growth). We extend existing policy learning methods to the continuous multivariate action domain, proposing to maximise a pessimistic lower bound on the North Star reward that the learnt policy will yield. Typical lower bounds based on normal approximations suffer from insufficient coverage, and we propose an efficient and effective policy-dependent correction for this. We provide guidance to design stochastic data collection policies, as well as highly sensitive reward signals. Empirical observations from simulations, offline and online experiments highlight the efficacy of our deployed approach.

Related papers

• An Efficient Continuous Control Perspective for Reinforcement-Learning-based Sequential Recommendation [14.506332665769746]
  We propose an underlinetextbfEfficient underlinetextbfContinuous underlinetextbfControl framework (ECoC) Based on a statistically tested assumption, we first propose the novel unified action representation abstracted from normalized user and item spaces. During this process, strategic exploration and directional control in terms of unified actions are carefully designed and crucial to final recommendation decisions.
  arXiv  Detail & Related papers  (2024-08-15T09:26:26Z)
• $Δ\text{-}{\rm OPE}$: Off-Policy Estimation with Pairs of Policies [13.528097424046823]
  We introduce $Deltatext-rm OPE$ methods based on the widely used Inverse Propensity Scoring estimator. Simulated, offline, and online experiments show that our methods significantly improve performance for both evaluation and learning tasks.
  arXiv  Detail & Related papers  (2024-05-16T12:04:55Z)
• Optimal Baseline Corrections for Off-Policy Contextual Bandits [61.740094604552475]
  We aim to learn decision policies that optimize an unbiased offline estimate of an online reward metric. We propose a single framework built on their equivalence in learning scenarios. Our framework enables us to characterize the variance-optimal unbiased estimator and provide a closed-form solution for it.
  arXiv  Detail & Related papers  (2024-05-09T12:52:22Z)
• Learning Optimal Deterministic Policies with Stochastic Policy Gradients [62.81324245896716]
  Policy gradient (PG) methods are successful approaches to deal with continuous reinforcement learning (RL) problems. In common practice, convergence (hyper)policies are learned only to deploy their deterministic version. We show how to tune the exploration level used for learning to optimize the trade-off between the sample complexity and the performance of the deployed deterministic policy.
  arXiv  Detail & Related papers  (2024-05-03T16:45:15Z)
• Off-Policy Evaluation for Large Action Spaces via Policy Convolution [60.6953713877886]
  Policy Convolution family of estimators uses latent structure within actions to strategically convolve the logging and target policies. Experiments on synthetic and benchmark datasets demonstrate remarkable mean squared error (MSE) improvements when using PC.
  arXiv  Detail & Related papers  (2023-10-24T01:00:01Z)
• Goal-conditioned Offline Reinforcement Learning through State Space Partitioning [9.38848713730931]
  offline reinforcement learning (RL) aims to infer sequential decision policies using only offline datasets. We argue that, despite its benefits, this approach is still insufficient to fully address the distribution shift and multi-modality problems. We propose a complementary advantage-based weighting scheme that introduces an additional source of inductive bias.
  arXiv  Detail & Related papers  (2023-03-16T14:52:53Z)
• Latent-Variable Advantage-Weighted Policy Optimization for Offline RL [70.01851346635637]
  offline reinforcement learning methods hold the promise of learning policies from pre-collected datasets without the need to query the environment for new transitions. In practice, offline datasets are often heterogeneous, i.e., collected in a variety of scenarios. We propose to leverage latent-variable policies that can represent a broader class of policy distributions. Our method improves the average performance of the next best-performing offline reinforcement learning methods by 49% on heterogeneous datasets.
  arXiv  Detail & Related papers  (2022-03-16T21:17:03Z)
• Improving Long-Term Metrics in Recommendation Systems using Short-Horizon Offline RL [56.20835219296896]
  We study session-based recommendation scenarios where we want to recommend items to users during sequential interactions to improve their long-term utility. We develop a new batch RL algorithm called Short Horizon Policy Improvement (SHPI) that approximates policy-induced distribution shifts across sessions.
  arXiv  Detail & Related papers  (2021-06-01T15:58:05Z)
• Risk-Sensitive Deep RL: Variance-Constrained Actor-Critic Provably Finds Globally Optimal Policy [95.98698822755227]
  We make the first attempt to study risk-sensitive deep reinforcement learning under the average reward setting with the variance risk criteria. We propose an actor-critic algorithm that iteratively and efficiently updates the policy, the Lagrange multiplier, and the Fenchel dual variable.
  arXiv  Detail & Related papers  (2020-12-28T05:02:26Z)
• Optimal Mixture Weights for Off-Policy Evaluation with Multiple Behavior Policies [3.855085732184416]
  Off-policy evaluation is a key component of reinforcement learning which evaluates a target policy with offline data collected from behavior policies. This paper discusses how to correctly mix estimators produced by different behavior policies. Experiments on simulated recommender systems show that our methods are effective in reducing the Mean-Square Error of estimation.
  arXiv  Detail & Related papers  (2020-11-29T12:57:54Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.