Don't Let Bandit Feedback Pull Continual LLM-Recommender Updates Off Target

• URL: http://arxiv.org/abs/2605.18899v1
• Date: Sun, 17 May 2026 11:10:44 GMT
• Title: Don't Let Bandit Feedback Pull Continual LLM-Recommender Updates Off Target
• Abstract summary: Generative recommenders (LLM-Rec) require continual post-deployment updates.<n> deployment logs provide only policy-shaped contextual bandit feedback.<n>We propose an Anchored Bandit Policy Optimization framework for continual LLM-Rec updates.
• Score: 42.681980014826536
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: Generative LLM-based recommenders (LLM-Rec) require continual post-deployment updates, yet deployment logs provide only policy-shaped contextual bandit feedback: outcomes are observed solely for items exposed by a prior serving policy, inducing exposure bias and yielding partial, asymmetric signals consisting of relatively reliable positive responses and ambiguous no-responses. We propose an Anchored Bandit Policy Optimization (ABPO) framework for continual LLM-Rec updates that combines group-relative policy optimization (GRPO) with explicit treatment of exposure bias and feedback ambiguity. Specifically, we insert the exposed recommendation as a logged anchor into each GRPO rollout group, so that group-relative normalization is calibrated against the action actually exposed by the prior policy rather than against newly sampled rollouts alone. Because both positive- and no-responses are observed only through prior-policy exposure, we apply self-normalized inverse propensity scoring to the fixed anchor for both feedback types to correct for policy mismatch. At the same time, we treat the two feedback types asymmetrically in reliability: positive responses provide relatively direct endorsement signals, whereas no-responses remain ambiguous because they may reflect either true disinterest or unobserved external factors. To avoid overly aggressive updates from ambiguous no-responses, we temper their penalties with self-certainty, using the model's output-token confidence as a verifier-free reliability signal. Across five domains from Amazon Reviews and MovieLens, our method yields consistent post-update gains in recommendation accuracy while mitigating prior-policy-induced exposure bias more effectively than prior baselines.

Related papers

• User Feedback Provides a Unique Signal that LLMs Can not Detect [25.849928795741672]
  We demonstrate that user feedback is a highly actionable signal for improvement.<n>We show that feedback-informed revisions resolve targeted issues at significantly higher rates than baseline revisions.
  arXiv  Detail & Related papers  (2026-09-02T17:42:44Z)
• Are We Really Making Progress in Group Recommendation? Unmasking the Tie-Breaking Illusion [4.153834414190214]
  Group recommendation methods have reported strong improvements on standard benchmarks.<n>But it remains unclear whether these gains always reflect genuine advances in modeling group preferences.<n>We show that several recent methods are affected by a systematic evaluation bias caused by the interaction between training-time score compression and evaluation-time deterministic tie-breaking.
  arXiv  Detail & Related papers  (2026-08-11T17:50:08Z)
• RRPO: Reference-Relative Policy Optimization with Stratified Conditional Rollouts [84.20747697259326]
  Group Relative Policy Optimization (GRPO) has shown strong effectiveness in reinforcement learning from verifiable feedback.<n>We propose textbfReference-Relative Policy Optimization (RRPO), which generalizes GRPO by replacing direct correctness-based advantage construction with reference-relative contrastive comparisons.<n>We evaluate RRPO using anchor-based contrastive advantages throughout policy optimization, without relying on task ground-truth verifiers.
  arXiv  Detail & Related papers  (2026-07-20T19:40:25Z)
• Reward Bias Substitution: Single-Axis Bias Mitigations Redirect Optimization Pressure [23.021647176206972]
  Single-axis mitigations of reward-model biases can rotate optimization pressure onto correlated proxies rather than eliminate it.<n>The failure is enabled by a measurement-versus-optimization gap between audit and policy-induced distributions.<n>We formalize mitigation outcomes into a regime taxonomy and prove that successful mitigation, bias substitution, and overcorrection produce identical observables.
  arXiv  Detail & Related papers  (2026-05-27T05:40:22Z)
• One-Way Policy Optimization for Self-Evolving LLMs [63.8638342097375]
  Reinforcement Learning with Verifiable Rewards (RLVR) has become a promising paradigm for scaling reasoning capabilities of Large Language Models (LLMs)<n>We propose One-Way Policy Optimization (OWPO), a method based on the principle of decoupling optimization direction from update magnitude.<n> Experimental results demonstrate that OWPO outperforms strong baselines, including DAPO, OPD, and MOPD.
  arXiv  Detail & Related papers  (2026-05-21T08:25:27Z)
• Revisiting Reinforcement Learning with Verifiable Rewards from a Contrastive Perspective [10.958642517467721]
  We present a framework for Contrastive Sequence-level Policy Optimization in RLVR.<n>ConSPO replaces GRPO's clipped ratio-based scores with length-normalized sequence log-probabilities.<n>ConSPO consistently outperforms several strong RLVR baselines on challenging mathematical reasoning benchmarks.
  arXiv  Detail & Related papers  (2026-05-13T04:02:36Z)
• The Reciprocity Gradient [52.35929743862925]
  Communication is fundamental to sustaining reciprocity and cooperation in strategic interactions.<n>We identify and formulate the influence attribution problem as the central optimization difficulty inherent in such dynamics for a learning agent.<n>To address this, we introduce the reciprocity gradient, which explicitly backpropagates reward gradients through private estimators of opponents' policies trained from public observations.
  arXiv  Detail & Related papers  (2026-05-08T16:29:52Z)
• SiMPO: Measure Matching for Online Diffusion Reinforcement Learning [52.46919717963149]
  We introduce SiMPO, a simple and unified framework that generalizes reweighting scheme in diffusion RL with general monotonic functions.<n>SiMPO revisits diffusion RL via a two-stage measure matching lens.<n>We provide geometric interpretations to illustrate how negative reweighting actively repels the policy from suboptimal actions.
  arXiv  Detail & Related papers  (2026-03-10T22:01:13Z)
• Mitigating Mismatch within Reference-based Preference Optimization [55.07698254211876]
  Direct Preference Optimization (DPO) has become the de facto standard for offline preference alignment of large language models.<n>DPO weighs each update relative to a reference, which stabilizes the training by regularizing the updates within a trusted region.<n>This reliance becomes problematic for pessimistic pairs, where the reference model prefers the rejected response.<n>We modify DPO to treat the reference as neutral when it is pessimistic by replacing $_-_mathrmref$ with $_-max0,_mathrmref$.
  arXiv  Detail & Related papers  (2026-02-12T12:55:51Z)
• Coverage Improvement and Fast Convergence of On-policy Preference Learning [67.36750525893514]
  Online on-policy preference learning algorithms for language model alignment can significantly outperform their offline counterparts.<n>We analyze how the sampling policy's coverage evolves throughout on-policy training.<n>We develop principled on-policy schemes for reward distillation in the general function class setting.
  arXiv  Detail & Related papers  (2026-01-13T10:46:06Z)
• CARE What Fails: Contrastive Anchored-REflection for Verifiable Multimodal [84.71254539482369]
  Group-relative reinforcement learning with verifiable rewards (RLVR) often wastes the most informative data it already has the failures.<n>We present CARE, a failure-centric post-training framework for multimodal reasoning that turns errors into supervision.<n> CARE improves accuracy and training smoothness while explicitly increasing the share of learning signal that comes from failures.
  arXiv  Detail & Related papers  (2025-12-22T16:34:21Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.