Exponential Reward Weighting for Fine-Tuning Generative Recommenders under Sparse and Noisy Feedback

• URL: http://arxiv.org/abs/2608.00816v1
• Date: Sat, 01 Aug 2026 18:48:39 GMT
• Title: Exponential Reward Weighting for Fine-Tuning Generative Recommenders under Sparse and Noisy Feedback
• Abstract summary: In recommendation systems, users interact with only a small fraction of a vast item catalog, producing feedback that is both sparse and noisy.<n>We propose Exponential reward-weighted fine-tuning (Exp-RSFT)<n>We show that Exp-RSFT's suboptimality decomposes into two costs: a coverage cost arising from limitations of the logging policy and a noise cost from imperfect feedback.
• Score: 7.2858507889096815
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: In recommendation systems, users interact with only a small fraction of a vast item catalog, producing feedback that is both sparse and noisy. This challenges post-training generative recommenders: reward models trained from logged interactions often fail to generalize, while directly optimizing imperfect rewards can lead to reward over-optimization. We propose Exponential reward-weighted fine-tuning (Exp-RSFT), where each logged interaction is weighted by $\exp(r/λ)$, avoids this failure by optimizing directly on the logged rewards, with the temperature $λ$ regularizing against their noise. We theoretically show that Exp-RSFT's suboptimality decomposes into two costs: a coverage cost arising from limitations of the logging policy and a noise cost from imperfect feedback. The temperature $λ$ balances these competing effects, yielding an optimal tradeoff between exploiting high-reward behavior and robustness to noise. Across three public benchmarks and a large-scale industrial dataset, we verify this theoretical prediction: performance follows an inverted-U trend as a function of $λ$, while PPO and DPO often over-optimize unreliable reward models and degrade recommendation quality. Exp-RSFT consistently improves ranking performance without requiring online exploration or preference data.

Related papers

• ODRPO: Ordinal Decompositions of Discrete Rewards for Robust Policy Optimization [14.900223489465683]
  Large Language Models (LLMs) utilize Reinforcement Learning from AI Feedback (RLAIF) for non-verifiable domains.<n>These domains often rely on based auto-raters to provide granular, multi-tier discrete rewards.<n>We introduce $textbfO$ptimization ($textbfOOO$), a framework that structurally isolates evaluation noise by decomposing discrete rewards into a sequence of ordinal binary indicators.
  arXiv  Detail & Related papers  (2026-05-12T19:17:14Z)
• MARBLE: Multi-Aspect Reward Balance for Diffusion RL [71.6241143519038]
  Reinforcement learning has become the dominant approach for aligning diffusion models with human preferences.<n>Existing practice deal with multiple rewards by training one specialist model per reward.<n>We propose MARBLE, a gradient-space optimization framework that maintains independent advantage estimators for each reward.
  arXiv  Detail & Related papers  (2026-05-07T16:20:42Z)
• Objective Shaping with Hard Negatives: Windowed Partial AUC Optimization for RL-based LLM Recommenders [74.55181072260713]
  Training with beam-search negatives consistently outperforms random negatives.<n>We introduce Windowed Partial AUC (WPAUC), which constrains the false positive rate (FPR) to a window to more directly align with Top-$K$ metrics.<n>Experiments on four real-world datasets validate the theory and deliver consistent state-of-the-art performance.
  arXiv  Detail & Related papers  (2026-04-24T12:31:57Z)
• Reinforcement Learning from Multi-Source Imperfect Preferences: Best-of-Both-Regimes Regret [71.69884486156359]
  We study episodic RL from emphmulti-source imperfect preferences through a cumulative imperfection budget.<n>We propose a unified algorithm with regret $tildeO(sqrtK/M+)$, which exhibits a best-of-both-regimes behavior.
  arXiv  Detail & Related papers  (2026-03-20T19:34:53Z)
• Robust Post-Training for Generative Recommenders: Why Exponential Reward-Weighted SFT Outperforms RLHF [7.2858507889096815]
  Existing post-training methods reward hack due to noisy user feedback and unreliable reward models.<n>We identify exponential reward-weighted SFT with weights $w = exp(r/)$ as uniquely suited to this setting.<n>We prove the first policy improvement guarantees for this setting under noisy rewards.
  arXiv  Detail & Related papers  (2026-03-10T23:48:25Z)
• Fine-Tuning Diffusion-Based Recommender Systems via Reinforcement Learning with Reward Function Optimization [21.769717387197943]
  Diffusion models offer state-of-the-art performance by modeling the generative process of user-item interactions.<n>We propose ReFiT, a new framework that integrates Reinforcement learning (RL)-based Fine-Tuning into diffusion-based recommender systems.
  arXiv  Detail & Related papers  (2025-11-10T10:38:16Z)
• RLHF Fine-Tuning of LLMs for Alignment with Implicit User Feedback in Conversational Recommenders [0.8246494848934447]
  We propose a fine-tuning solution using human feedback reinforcement learning (RLHF) to maximize implied user feedback (IUF) in a multi-turn recommendation context.<n>We show that our RLHF-fine-tuned models can perform better in terms of top-$k$ recommendation accuracy, coherence, and user satisfaction compared to (arrow-zero-cmwrquca-teja-falset ensuite 2Round group-deca States penalty give up.
  arXiv  Detail & Related papers  (2025-08-07T11:36:55Z)
• Supervised Optimism Correction: Be Confident When LLMs Are Sure [91.7459076316849]
  We establish a novel theoretical connection between supervised fine-tuning and offline reinforcement learning.<n>We show that the widely used beam search method suffers from unacceptable over-optimism.<n>We propose Supervised Optimism Correction, which introduces a simple yet effective auxiliary loss for token-level $Q$-value estimations.
  arXiv  Detail & Related papers  (2025-04-10T07:50:03Z)
• SPRec: Self-Play to Debias LLM-based Recommendation [23.875509546540904]
  Large language models (LLMs) have attracted significant attention in recommendation systems.<n>We propose SPRec, a novel self-play framework designed to mitigate over-recommendation and improve fairness without requiring additional data or manual intervention.
  arXiv  Detail & Related papers  (2024-12-12T12:53:30Z)
• Multi-Preference Optimization: Generalizing DPO via Set-Level Contrasts [17.243429150450886]
  We propose $textbfMulti-Preference Optimization (MPO) to optimize over entire sets of responses.<n>MPO employs deviation-based weighting, which emphasizes outlier responses that deviate most from the mean reward.<n>We theoretically prove that MPO reduces alignment bias at a rate of $mathcalOleft(frac1sqrtnright)$ with respect to the number of responses per query.
  arXiv  Detail & Related papers  (2024-12-05T21:50:22Z)
• ASFT: Aligned Supervised Fine-Tuning through Absolute Likelihood [14.512464277772194]
  Aligned Supervised Fine-Tuning (ASFT) is an effective approach that better aligns Large Language Models with pair-wise datasets. ASFT mitigates the issue where the DPO loss function decreases the probability of generating human-dispreferred data. Extensive experiments demonstrate that ASFT is an effective alignment approach, consistently outperforming existing methods.
  arXiv  Detail & Related papers  (2024-09-14T11:39:13Z)
• Provably Mitigating Overoptimization in RLHF: Your SFT Loss is Implicitly an Adversarial Regularizer [52.09480867526656]
  We identify the source of misalignment as a form of distributional shift and uncertainty in learning human preferences.<n>To mitigate overoptimization, we first propose a theoretical algorithm that chooses the best policy for an adversarially chosen reward model.<n>Using the equivalence between reward models and the corresponding optimal policy, the algorithm features a simple objective that combines a preference optimization loss and a supervised learning loss.
  arXiv  Detail & Related papers  (2024-05-26T05:38:50Z)
• ROPO: Robust Preference Optimization for Large Language Models [59.10763211091664]
  We propose an iterative alignment approach that integrates noise-tolerance and filtering of noisy samples without the aid of external models. Experiments on three widely-used datasets with Mistral-7B and Llama-2-7B demonstrate that ROPO significantly outperforms existing preference alignment methods.
  arXiv  Detail & Related papers  (2024-04-05T13:58:51Z)
• Overcoming Reward Overoptimization via Adversarial Policy Optimization with Lightweight Uncertainty Estimation [46.61909578101735]
  Adversarial Policy Optimization (AdvPO) is a novel solution to the pervasive issue of reward over-optimization in Reinforcement Learning from Human Feedback. In this paper, we introduce a lightweight way to quantify uncertainties in rewards, relying solely on the last layer embeddings of the reward model.
  arXiv  Detail & Related papers  (2024-03-08T09:20:12Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.