Sample-Mean Anchored Thompson Sampling for Offline-to-Online Learning with Distribution Shift

• URL: http://arxiv.org/abs/2605.10289v2
• Date: Thu, 14 May 2026 09:25:47 GMT
• Title: Sample-Mean Anchored Thompson Sampling for Offline-to-Online Learning with Distribution Shift
• Abstract summary: A central challenge in offline-to-online learning is the distribution shift between offline and online data.<n>We propose a novel median-based anchoring rule that defines the arm index as the median of an online posterior sample, a hybrid posterior sample, and the online sample mean.<n>We establish theoretical guarantees showing that the proposed algorithm safely leverages offline data to accelerate online learning.
• Score: 24.048629084196904
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Offline-to-online learning aims to improve online decision-making by leveraging offline logged data. A central challenge in this setting is the distribution shift between offline and online environments. While some existing works attempt to leverage shifted offline data, they largely rely on UCB-type algorithms. Thompson sampling (TS) represents another canonical class of bandit algorithms, well known for its strong empirical performance and naturally suited to offline-to-online learning through its Bayesian formulation. However, unlike UCB indices, posterior samples in TS are not guaranteed to be optimistic with respect to the true arm means. This makes indices constructed from purely online and hybrid data difficult to compare and complicates their use. To address this issue, we propose sample-mean anchored TS (Anchor-TS), which introduces a novel median-based anchoring rule that defines the arm index as the median of an online posterior sample, a hybrid posterior sample, and the online sample mean. The median anchoring systematically corrects bias induced by distribution shift by mitigating over-estimation for suboptimal arms and under-estimation for optimal arms, while exploiting offline information to obtain more accurate estimates when the shift is small. We establish theoretical guarantees showing that the proposed algorithm safely leverages offline data to accelerate online learning, and quantifying how the degree of distribution shift and the size of offline data affect the resulting regret reduction. Extensive experiments demonstrate consistent improvements of our algorithm over baselines.

Related papers

• Geometry-Aware Offline-to-Online Learning in Linear Contextual Bandits [5.0861449841476984]
  We study offline-to-online learning in linear contextual bandits with biased offline regression data.<n>We propose emphEllipsoidal-MINUCB, which combines a standard online branch with an offline-informed pooled branch.<n>We show how to learn a data-driven certificate from data at finitely many refresh times and establish a high-probability regret bound for Ellipsoidal-MINUCB.
  arXiv  Detail & Related papers  (2026-04-27T03:53:30Z)
• Hybrid Combinatorial Multi-armed Bandits with Probabilistically Triggered Arms [10.146314852311638]
  We propose hybrid CMAB-T, a new framework that integrates offline data with online interaction in a principled manner.<n>Our proposed hybrid CUCB algorithm leverages offline data to guide exploration and accelerate convergence.<n>We provide theoretical guarantees on the algorithm's regret, demonstrating that hybrid CUCB significantly outperforms purely online approaches.
  arXiv  Detail & Related papers  (2025-12-26T08:42:12Z)
• Best Arm Identification with Possibly Biased Offline Data [56.965938201853625]
  We study the best arm identification problem with potentially biased offline data in the fixed confidence setting.<n>We propose the LUCB-H algorithm, which introduces adaptive confidence bounds by incorporating an auxiliary bias correction.
  arXiv  Detail & Related papers  (2025-05-29T06:58:49Z)
• Understanding the performance gap between online and offline alignment algorithms [63.137832242488926]
  We show that offline algorithms train policy to become good at pairwise classification, while online algorithms are good at generations. This hints at a unique interplay between discriminative and generative capabilities, which is greatly impacted by the sampling process. Our study sheds light on the pivotal role of on-policy sampling in AI alignment, and hints at certain fundamental challenges of offline alignment algorithms.
  arXiv  Detail & Related papers  (2024-05-14T09:12:30Z)
• Online Feature Updates Improve Online (Generalized) Label Shift Adaptation [51.328801874640675]
  Our novel method, Online Label Shift adaptation with Online Feature Updates (OLS-OFU), leverages self-supervised learning to refine the feature extraction process. By carefully designing the algorithm, OLS-OFU maintains the similar online regret convergence to the results in the literature while taking the improved features into account.
  arXiv  Detail & Related papers  (2024-02-05T22:03:25Z)
• Noisy Correspondence Learning with Self-Reinforcing Errors Mitigation [63.180725016463974]
  Cross-modal retrieval relies on well-matched large-scale datasets that are laborious in practice. We introduce a novel noisy correspondence learning framework, namely textbfSelf-textbfReinforcing textbfErrors textbfMitigation (SREM)
  arXiv  Detail & Related papers  (2023-12-27T09:03:43Z)
• Low-rank extended Kalman filtering for online learning of neural networks from streaming data [71.97861600347959]
  We propose an efficient online approximate Bayesian inference algorithm for estimating the parameters of a nonlinear function from a potentially non-stationary data stream. The method is based on the extended Kalman filter (EKF), but uses a novel low-rank plus diagonal decomposition of the posterior matrix. In contrast to methods based on variational inference, our method is fully deterministic, and does not require step-size tuning.
  arXiv  Detail & Related papers  (2023-05-31T03:48:49Z)
• Rapid Adaptation in Online Continual Learning: Are We Evaluating It Right? [135.71855998537347]
  We revisit the common practice of evaluating adaptation of Online Continual Learning (OCL) algorithms through the metric of online accuracy. We show that this metric is unreliable, as even vacuous blind classifiers can achieve unrealistically high online accuracy. Existing OCL algorithms can also achieve high online accuracy, but perform poorly in retaining useful information.
  arXiv  Detail & Related papers  (2023-05-16T08:29:33Z)
• Bilevel Online Deep Learning in Non-stationary Environment [4.565872584112864]
  Bilevel Online Deep Learning (BODL) framework combines bilevel optimization strategy and online ensemble classifier. When the concept drift is detected, our BODL algorithm can adaptively update the model parameters via bilevel optimization and then circumvent the large drift and encourage positive transfer.
  arXiv  Detail & Related papers  (2022-01-25T11:05:51Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.