Geometry-Aware Offline-to-Online Learning in Linear Contextual Bandits

• URL: http://arxiv.org/abs/2604.24016v1
• Date: Mon, 27 Apr 2026 03:53:30 GMT
• Title: Geometry-Aware Offline-to-Online Learning in Linear Contextual Bandits
• Abstract summary: We study offline-to-online learning in linear contextual bandits with biased offline regression data.<n>We propose emphEllipsoidal-MINUCB, which combines a standard online branch with an offline-informed pooled branch.<n>We show how to learn a data-driven certificate from data at finitely many refresh times and establish a high-probability regret bound for Ellipsoidal-MINUCB.
• Score: 5.0861449841476984
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: We study offline-to-online learning in linear contextual bandits with biased offline regression data: the offline parameter need not match the online one, so history should not be treated as a single warm start. We model directional transfer with a shift certificate $(M_{\mathrm{shift}},ρ)$ and offline ridge estimation, yielding a geometry-aware confidence region for the online parameter rather than an isotropic radius. We propose \emph{Ellipsoidal-MINUCB}, which combines a standard online branch with an offline-informed pooled branch and uses offline information only when it tightens uncertainty. With high probability, regret is bounded by the minimum of a standard SupLinUCB-style fallback and a pooled term that separates statistical width from a certificate-weighted shift penalty. Under a simple alignment condition, the pooled term further simplifies to a rate governed by an effective dimension induced by the offline geometry. We also show that a purely Euclidean (scalar) shift bound, by itself, does not determine which feature directions are transferable. Beyond this fixed certificate, we show how to learn a data-driven certificate from data at finitely many refresh times and establish a high-probability regret bound for Ellipsoidal-MINUCB with epoch-wise learned certificates. Experiments match the main prediction: gains are strongest at intermediate horizons when offline coverage and transferability align, while the method otherwise tracks the safe online baseline.

Related papers

• Offline-to-Online Learning in Linear Bandits [2.372134377778215]
  We study online learning with an additional offline dataset in the linear bandit setting.<n>We propose a linear bandit algorithm that balances this tradeoff: it relies on offline data during early rounds, and increasingly favors exploration as the horizon grows.
  arXiv  Detail & Related papers  (2026-06-03T00:18:46Z)
• Information-Directed Offline-to-Online Reinforcement Learning [0.0]
  Decision-making from offline datasets typically warm-starts a policy or score model from fixed offline data and then refines it with limited online interaction.<n>We formalise this residual uncertainty by the conditional mutual information $I(;_1:TmidmathcalD_N)$ between a learning target $$ and the online trajectories after conditioning on the offline dataset.<n>This view leads naturally to information-directed sampling (IDS), a family parameterised by $ge 0$ that selects actions by trading off instantaneous regret against information gain.
  arXiv  Detail & Related papers  (2026-05-28T05:58:15Z)
• Sample-Mean Anchored Thompson Sampling for Offline-to-Online Learning with Distribution Shift [24.048629084196904]
  A central challenge in offline-to-online learning is the distribution shift between offline and online data.<n>We propose a novel median-based anchoring rule that defines the arm index as the median of an online posterior sample, a hybrid posterior sample, and the online sample mean.<n>We establish theoretical guarantees showing that the proposed algorithm safely leverages offline data to accelerate online learning.
  arXiv  Detail & Related papers  (2026-05-11T09:50:58Z)
• Offline-Online Reinforcement Learning for Linear Mixture MDPs [0.5538445885586951]
  We study offline-online reinforcement learning in linear mixture Markov decision processes (MDPs) under environment shift.<n>In the offline phase, data are collected by an unknown behavior policy and may come from a mismatched environment.<n>We propose an algorithm that adaptively leverages offline data.
  arXiv  Detail & Related papers  (2026-04-13T19:34:30Z)
• Hybrid Combinatorial Multi-armed Bandits with Probabilistically Triggered Arms [10.146314852311638]
  We propose hybrid CMAB-T, a new framework that integrates offline data with online interaction in a principled manner.<n>Our proposed hybrid CUCB algorithm leverages offline data to guide exploration and accelerate convergence.<n>We provide theoretical guarantees on the algorithm's regret, demonstrating that hybrid CUCB significantly outperforms purely online approaches.
  arXiv  Detail & Related papers  (2025-12-26T08:42:12Z)
• Online Learning and Unlearning [56.770023668379615]
  We present two online learner-unlearner (OLU) algorithms, both built upon online gradient descent (OGD)<n>The first, passive OLU, leverages OGD's contractive property and injects noise when unlearning occurs, incurring no additional computation.<n>The second, active OLU, uses an offline unlearning algorithm that shifts the model toward a solution excluding the deleted data.
  arXiv  Detail & Related papers  (2025-05-13T13:33:36Z)
• Efficient Online Learning with Offline Datasets for Infinite Horizon MDPs: A Bayesian Approach [25.77911741149966]
  We show that if the learning agent models the behavioral policy used by the expert, it can do substantially better in terms of minimizing cumulative regret. We then propose the Informed RLSVI algorithm to efficiently approximate the iPSRL algorithm.
  arXiv  Detail & Related papers  (2023-10-17T19:01:08Z)
• Online Label Shift: Optimal Dynamic Regret meets Practical Algorithms [33.61487362513345]
  This paper focuses on supervised and unsupervised online label shift, where the class marginals $Q(y)$ varies but the class-conditionals $Q(x|y)$ remain invariant. In the unsupervised setting, our goal is to adapt a learner, trained on some offline labeled data, to changing label distributions given unlabeled online data. We develop novel algorithms that reduce the adaptation problem to online regression and guarantee optimal dynamic regret without any prior knowledge of the extent of drift in the label distribution.
  arXiv  Detail & Related papers  (2023-05-31T05:39:52Z)
• Cal-QL: Calibrated Offline RL Pre-Training for Efficient Online Fine-Tuning [104.05522247411018]
  offline reinforcement learning (RL) methods tend to behave poorly during fine-tuning. We show that offline RL algorithms that learn such calibrated value functions lead to effective online fine-tuning. In practice, Cal-QL can be implemented on top of the conservative Q learning (CQL) for offline RL within a one-line code change.
  arXiv  Detail & Related papers  (2023-03-09T18:31:13Z)
• Adaptive Behavior Cloning Regularization for Stable Offline-to-Online Reinforcement Learning [80.25648265273155]
  Offline reinforcement learning, by learning from a fixed dataset, makes it possible to learn agent behaviors without interacting with the environment. During online fine-tuning, the performance of the pre-trained agent may collapse quickly due to the sudden distribution shift from offline to online data. We propose to adaptively weigh the behavior cloning loss during online fine-tuning based on the agent's performance and training stability. Experiments show that the proposed method yields state-of-the-art offline-to-online reinforcement learning performance on the popular D4RL benchmark.
  arXiv  Detail & Related papers  (2022-10-25T09:08:26Z)
• Curriculum Offline Imitation Learning [72.1015201041391]
  offline reinforcement learning tasks require the agent to learn from a pre-collected dataset with no further interactions with the environment. We propose textitCurriculum Offline Learning (COIL), which utilizes an experience picking strategy for imitating from adaptive neighboring policies with a higher return. On continuous control benchmarks, we compare COIL against both imitation-based and RL-based methods, showing that it not only avoids just learning a mediocre behavior on mixed datasets but is also even competitive with state-of-the-art offline RL methods.
  arXiv  Detail & Related papers  (2021-11-03T08:02:48Z)
• Online Continual Adaptation with Active Self-Training [69.5815645379945]
  We propose an online setting where the learner aims to continually adapt to changing distributions using both unlabeled samples and active queries of limited labels. Online Self-Adaptive Mirror Descent (OSAMD) adopts an online teacher-student structure to enable online self-training from unlabeled data. We show that OSAMD achieves favorable regrets under changing environments with limited labels on both simulated and real-world data.
  arXiv  Detail & Related papers  (2021-06-11T17:51:25Z)
• OPAL: Offline Primitive Discovery for Accelerating Offline Reinforcement Learning [107.6943868812716]
  In many practical applications, the situation is reversed: an agent may have access to large amounts of undirected offline experience data, while access to the online environment is severely limited. Our main insight is that, when presented with offline data composed of a variety of behaviors, an effective way to leverage this data is to extract a continuous space of recurring and temporally extended primitive behaviors. In addition to benefiting offline policy optimization, we show that performing offline primitive learning in this way can also be leveraged for improving few-shot imitation learning.
  arXiv  Detail & Related papers  (2020-10-26T14:31:08Z)
• AliExpress Learning-To-Rank: Maximizing Online Model Performance without Going Online [60.887637616379926]
  This paper proposes an evaluator-generator framework for learning-to-rank. It consists of an evaluator that generalizes to evaluate recommendations involving the context, and a generator that maximizes the evaluator score by reinforcement learning. Our method achieves a significant improvement in terms of Conversion Rate (CR) over the industrial-level fine-tuned model in online A/B tests.
  arXiv  Detail & Related papers  (2020-03-25T10:27:44Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.