Impatient Bandits: Optimizing Recommendations for the Long-Term Without Delay

• URL: http://arxiv.org/abs/2307.09943v2
• Date: Thu, 20 Jul 2023 16:11:39 GMT
• Title: Impatient Bandits: Optimizing Recommendations for the Long-Term Without Delay
• Abstract summary: We study a content exploration task, which we formalize as a multi-armed bandit problem with delayed rewards. We develop a predictive model of delayed rewards that incorporates all information obtained to date. We apply our approach to a podcast recommendation problem, where we seek to identify shows that users engage with repeatedly over two months.
• Score: 29.23091752603817
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Recommender systems are a ubiquitous feature of online platforms. Increasingly, they are explicitly tasked with increasing users' long-term satisfaction. In this context, we study a content exploration task, which we formalize as a multi-armed bandit problem with delayed rewards. We observe that there is an apparent trade-off in choosing the learning signal: Waiting for the full reward to become available might take several weeks, hurting the rate at which learning happens, whereas measuring short-term proxy rewards reflects the actual long-term goal only imperfectly. We address this challenge in two steps. First, we develop a predictive model of delayed rewards that incorporates all information obtained to date. Full observations as well as partial (short or medium-term) outcomes are combined through a Bayesian filter to obtain a probabilistic belief. Second, we devise a bandit algorithm that takes advantage of this new predictive model. The algorithm quickly learns to identify content aligned with long-term success by carefully balancing exploration and exploitation. We apply our approach to a podcast recommendation problem, where we seek to identify shows that users engage with repeatedly over two months. We empirically validate that our approach results in substantially better performance compared to approaches that either optimize for short-term proxies, or wait for the long-term outcome to be fully realized.

Related papers

• Long-term User Engagement Optimization through Model-agnostic Downstream Rewards Learning [2.66348673166416]
  We present a unified, model-agnostic downstream reward framework for optimizing long-term user value in large-scale recommendation systems.<n>We propose several model-agnostic downstream rewards signals derived from observed user action patterns across multiple sources.<n>Online A/B experiments demonstrate consistent improvements in engagement and retention-related metrics.
  arXiv  Detail & Related papers  (2026-07-15T16:17:58Z)
• LatSearch: Latent Reward-Guided Search for Faster Inference-Time Scaling in Video Diffusion [87.42285185305813]
  We introduce a novel inference-time search mechanism that performs Reward-Guided Resampling and Pruning.<n>LatSearch consistently improves video generation across multiple evaluation dimensions compared to the baseline Wan2.1 model.
  arXiv  Detail & Related papers  (2026-03-15T18:07:29Z)
• Think Longer to Explore Deeper: Learn to Explore In-Context via Length-Incentivized Reinforcement Learning [53.58654277639939]
  In-context exploration is the intrinsic ability to generate, verify, and refine hypotheses within a single continuous context.<n>We propose Length-Incentivized Exploration, which explicitly encourages models to explore more.<n>Our method achieves an average improvement of 4.4% on in-domain tasks and a 2.7% gain on out-of-domain benchmarks.
  arXiv  Detail & Related papers  (2026-02-12T09:24:32Z)
• Impatient Bandits: Optimizing for the Long-Term Without Delay [20.753692240620644]
  We study a content exploration task, which we formalize as a bandit problem with delayed rewards. We develop a predictive model of delayed rewards that incorporates all information obtained to date. We devise a bandit algorithm that quickly learns to identify content aligned with long-term success.
  arXiv  Detail & Related papers  (2025-01-14T00:28:26Z)
• Bidirectional Decoding: Improving Action Chunking via Guided Test-Time Sampling [51.38330727868982]
  We show how action chunking impacts the divergence between a learner and a demonstrator. We propose Bidirectional Decoding (BID), a test-time inference algorithm that bridges action chunking with closed-loop adaptation. Our method boosts the performance of two state-of-the-art generative policies across seven simulation benchmarks and two real-world tasks.
  arXiv  Detail & Related papers  (2024-08-30T15:39:34Z)
• Long-term Off-Policy Evaluation and Learning [21.047613223586794]
  Short- and long-term outcomes of an algorithm often differ, with damaging downstream effects. It takes months or even longer to observe the long-term outcomes of interest, making the algorithm selection process unacceptably slow. We propose a new framework called Long-term Off-Policy Evaluation (LOPE), which is based on reward function decomposition.
  arXiv  Detail & Related papers  (2024-04-24T06:59:59Z)
• Multi-Armed Bandits with Abstention [62.749500564313834]
  We introduce a novel extension of the canonical multi-armed bandit problem that incorporates an additional strategic element: abstention. In this enhanced framework, the agent is not only tasked with selecting an arm at each time step, but also has the option to abstain from accepting the instantaneous reward before observing it.
  arXiv  Detail & Related papers  (2024-02-23T06:27:12Z)
• Are We Wasting Time? A Fast, Accurate Performance Evaluation Framework for Knowledge Graph Link Predictors [4.31947784387967]
  In Knowledge Graphs on a larger scale, the ranking process rapidly becomes heavy. Previous approaches used random sampling of entities to assess the quality of links predicted or suggested by a method. We show that this approach has serious limitations since the ranking metrics produced do not properly reflect true outcomes. We propose a framework that uses relational recommenders to guide the selection of candidates for evaluation.
  arXiv  Detail & Related papers  (2024-01-25T15:44:46Z)
• Accelerating Exploration with Unlabeled Prior Data [66.43995032226466]
  We study how prior data without reward labels may be used to guide and accelerate exploration for an agent solving a new sparse reward task. We propose a simple approach that learns a reward model from online experience, labels the unlabeled prior data with optimistic rewards, and then uses it concurrently alongside the online data for downstream policy and critic optimization.
  arXiv  Detail & Related papers  (2023-11-09T00:05:17Z)
• Bandit Social Learning: Exploration under Myopic Behavior [54.767961587919075]
  We study social learning dynamics motivated by reviews on online platforms. Agents collectively follow a simple multi-armed bandit protocol, but each agent acts myopically, without regards to exploration. We derive stark learning failures for any such behavior, and provide matching positive results.
  arXiv  Detail & Related papers  (2023-02-15T01:57:57Z)
• ZigZag: Universal Sampling-free Uncertainty Estimation Through Two-Step Inference [54.17205151960878]
  We introduce a sampling-free approach that is generic and easy to deploy. We produce reliable uncertainty estimates on par with state-of-the-art methods at a significantly lower computational cost.
  arXiv  Detail & Related papers  (2022-11-21T13:23:09Z)
• Meta-Wrapper: Differentiable Wrapping Operator for User Interest Selection in CTR Prediction [97.99938802797377]
  Click-through rate (CTR) prediction, whose goal is to predict the probability of the user to click on an item, has become increasingly significant in recommender systems. Recent deep learning models with the ability to automatically extract the user interest from his/her behaviors have achieved great success. We propose a novel approach under the framework of the wrapper method, which is named Meta-Wrapper.
  arXiv  Detail & Related papers  (2022-06-28T03:28:15Z)
• Learning to Bid Long-Term: Multi-Agent Reinforcement Learning with Long-Term and Sparse Reward in Repeated Auction Games [5.865500664175491]
  We propose a multi-agent distributed reinforcement learning algorithm that balances between potentially conflicting short-term reward and sparse, delayed long-term reward. We compare different long-term rewards to incentivize the algorithm to maximize individual payoff and overall social welfare.
  arXiv  Detail & Related papers  (2022-04-05T15:02:28Z)
• Learning Guidance Rewards with Trajectory-space Smoothing [22.456737935789103]
  Long-term temporal credit assignment is an important challenge in deep reinforcement learning. Existing policy-gradient and Q-learning algorithms rely on dense environmental rewards that provide rich short-term supervision. Recent works have proposed algorithms to learn dense "guidance" rewards that could be used in place of the sparse or delayed environmental rewards.
  arXiv  Detail & Related papers  (2020-10-23T23:55:06Z)
• Sequential Recommender via Time-aware Attentive Memory Network [67.26862011527986]
  We propose a temporal gating methodology to improve attention mechanism and recurrent units. We also propose a Multi-hop Time-aware Attentive Memory network to integrate long-term and short-term preferences. Our approach is scalable for candidate retrieval tasks and can be viewed as a non-linear generalization of latent factorization for dot-product based Top-K recommendation.
  arXiv  Detail & Related papers  (2020-05-18T11:29:38Z)
• Planning to Explore via Self-Supervised World Models [120.31359262226758]
  Plan2Explore is a self-supervised reinforcement learning agent. We present a new approach to self-supervised exploration and fast adaptation to new tasks. Without any training supervision or task-specific interaction, Plan2Explore outperforms prior self-supervised exploration methods.
  arXiv  Detail & Related papers  (2020-05-12T17:59:45Z)
• Long-Term Visitation Value for Deep Exploration in Sparse Reward Reinforcement Learning [34.38011902445557]
  Reinforcement learning with sparse rewards is still an open challenge. We present a novel approach that plans exploration actions far into the future by using a long-term visitation count. Contrary to existing methods which use models of reward and dynamics, our approach is off-policy and model-free.
  arXiv  Detail & Related papers  (2020-01-01T01:01:15Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.