GuideBoot: Guided Bootstrap for Deep Contextual Bandits

• URL: http://arxiv.org/abs/2107.08383v1
• Date: Sun, 18 Jul 2021 07:53:04 GMT
• Title: GuideBoot: Guided Bootstrap for Deep Contextual Bandits
• Abstract summary: We introduce Guided Bootstrap (GuideBoot) for contextual bandit algorithms. GuideBoot provides explicit guidance to the exploration behavior by training multiple models over both real samples and noisy samples with fake labels. The proposed method is efficient as it can make decisions on-the-fly by utilizing only one randomly chosen model.
• Score: 14.983677719946666
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: The exploration/exploitation (E&E) dilemma lies at the core of interactive systems such as online advertising, for which contextual bandit algorithms have been proposed. Bayesian approaches provide guided exploration with principled uncertainty estimation, but the applicability is often limited due to over-simplified assumptions. Non-Bayesian bootstrap methods, on the other hand, can apply to complex problems by using deep reward models, but lacks clear guidance to the exploration behavior. It still remains largely unsolved to develop a practical method for complex deep contextual bandits. In this paper, we introduce Guided Bootstrap (GuideBoot for short), combining the best of both worlds. GuideBoot provides explicit guidance to the exploration behavior by training multiple models over both real samples and noisy samples with fake labels, where the noise is added according to the predictive uncertainty. The proposed method is efficient as it can make decisions on-the-fly by utilizing only one randomly chosen model, but is also effective as we show that it can be viewed as a non-Bayesian approximation of Thompson sampling. Moreover, we extend it to an online version that can learn solely from streaming data, which is favored in real applications. Extensive experiments on both synthetic task and large-scale advertising environments show that GuideBoot achieves significant improvements against previous state-of-the-art methods.

Related papers

• RIE-Greedy: Regularization-Induced Exploration for Contextual Bandits [4.282654316889406]
  Real-world contextual bandit problems with complex reward models are often tackled with iteratively trained models, such as boosting trees.<n>Existing approaches rely on sophisticated assumptions or intractable procedures that are hard to verify and implement in practice.<n>In this work, we explore the use of an exploration-free (pure-greedy) action selection strategy, that exploits the randomness inherent in model fitting process as an intrinsic source of exploration.
  arXiv  Detail & Related papers  (2026-03-11T20:11:01Z)
• ReBaPL: Repulsive Bayesian Prompt Learning [8.744783207923211]
  This paper introduces Repulsive Bayesian Prompt Learning (ReBaPL), a novel method for Bayesian prompt learning.<n>Our method integrates a cyclical step-size schedule with a posterior gradient Hamiltonian Monte Carlo algorithm.<n>We demonstrate the efficacy of ReBaPL on several benchmark datasets, showing superior performance over state-of-the-art methods for prompt learning.
  arXiv  Detail & Related papers  (2025-11-21T16:00:13Z)
• Epinet for Content Cold Start [14.018820788546535]
  epinets enables efficient approximations of Thompson sampling even when the learning model is a complex neural network. Our experiments demonstrate improvements in both user traffic and engagement efficiency on the Facebook Reels online video platform.
  arXiv  Detail & Related papers  (2024-11-20T19:43:27Z)
• Adaptive Experimentation When You Can't Experiment [55.86593195947978]
  This paper introduces the emphconfounded pure exploration transductive linear bandit (textttCPET-LB) problem. Online services can employ a properly randomized encouragement that incentivizes users toward a specific treatment.
  arXiv  Detail & Related papers  (2024-06-15T20:54:48Z)
• An Online Bootstrap for Time Series [0.27195102129094995]
  We propose a novel bootstrap method that is designed to account for data dependencies and can be executed online. We prove the theoretical validity of the proposed bootstrap scheme under general conditions. Our work bridges the gap between classical resampling techniques and the demands of modern data analysis.
  arXiv  Detail & Related papers  (2023-10-30T16:03:11Z)
• Dynamic Exploration-Exploitation Trade-Off in Active Learning Regression with Bayesian Hierarchical Modeling [4.132882666134921]
  Methods that consider exploration-exploitation simultaneously employ fixed or ad-hoc measures to control the trade-off that may not be optimal. We develop a Bayesian hierarchical approach, referred as BHEEM, to dynamically balance the exploration-exploitation trade-off.
  arXiv  Detail & Related papers  (2023-04-16T01:40:48Z)
• Thompson Sampling with Virtual Helping Agents [0.0]
  We address the problem of online sequential decision making, i.e., balancing the trade-off between exploiting current knowledge to maximize immediate performance and exploring the new information to gain long-term benefits. We propose two algorithms for the multi-armed bandit problem and provide theoretical bounds on the cumulative regret.
  arXiv  Detail & Related papers  (2022-09-16T23:34:44Z)
• Incentivizing Combinatorial Bandit Exploration [87.08827496301839]
  Consider a bandit algorithm that recommends actions to self-interested users in a recommendation system. Users are free to choose other actions and need to be incentivized to follow the algorithm's recommendations. While the users prefer to exploit, the algorithm can incentivize them to explore by leveraging the information collected from the previous users.
  arXiv  Detail & Related papers  (2022-06-01T13:46:25Z)
• Residual Overfit Method of Exploration [78.07532520582313]
  We propose an approximate exploration methodology based on fitting only two point estimates, one tuned and one overfit. The approach drives exploration towards actions where the overfit model exhibits the most overfitting compared to the tuned model. We compare ROME against a set of established contextual bandit methods on three datasets and find it to be one of the best performing.
  arXiv  Detail & Related papers  (2021-10-06T17:05:33Z)
• Learning Salient Boundary Feature for Anchor-free Temporal Action Localization [81.55295042558409]
  Temporal action localization is an important yet challenging task in video understanding. We propose the first purely anchor-free temporal localization method. Our model includes (i) an end-to-end trainable basic predictor, (ii) a saliency-based refinement module, and (iii) several consistency constraints.
  arXiv  Detail & Related papers  (2021-03-24T12:28:32Z)
• Scalable Bayesian Inverse Reinforcement Learning [93.27920030279586]
  We introduce Approximate Variational Reward Imitation Learning (AVRIL) Our method addresses the ill-posed nature of the inverse reinforcement learning problem. Applying our method to real medical data alongside classic control simulations, we demonstrate Bayesian reward inference in environments beyond the scope of current methods.
  arXiv  Detail & Related papers  (2021-02-12T12:32:02Z)
• Deep Bayesian Bandits: Exploring in Online Personalized Recommendations [4.845576821204241]
  We formulate a display advertising recommender as a contextual bandit. We implement exploration techniques that require sampling from the posterior distribution of click-through-rates. We test our proposed deep Bayesian bandits algorithm in the offline simulation and online AB setting.
  arXiv  Detail & Related papers  (2020-08-03T08:58:18Z)
• Latent Bandits Revisited [55.88616813182679]
  A latent bandit problem is one in which the learning agent knows the arm reward distributions conditioned on an unknown discrete latent state. We propose general algorithms for this setting, based on both upper confidence bounds (UCBs) and Thompson sampling. We provide a unified theoretical analysis of our algorithms, which have lower regret than classic bandit policies when the number of latent states is smaller than actions.
  arXiv  Detail & Related papers  (2020-06-15T19:24:02Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.