ImplicitRM: Unbiased Reward Modeling from Implicit Preference Data for LLM alignment

• URL: http://arxiv.org/abs/2603.23184v1
• Date: Tue, 24 Mar 2026 13:32:14 GMT
• Title: ImplicitRM: Unbiased Reward Modeling from Implicit Preference Data for LLM alignment
• Abstract summary: Current reward modeling is heavily contingent upon experimental feedback data with high collection costs.<n>ImplicitRM aims to learn unbiased reward models from implicit preference data.<n> Experiments demonstrate that ImplicitRM learns accurate reward models across implicit preference datasets.
• Score: 49.93516092785756
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Reward modeling represents a long-standing challenge in reinforcement learning from human feedback (RLHF) for aligning language models. Current reward modeling is heavily contingent upon experimental feedback data with high collection costs. In this work, we study \textit{implicit reward modeling} -- learning reward models from implicit human feedback (e.g., clicks and copies) -- as a cost-effective alternative. We identify two fundamental challenges in implicit reward modeling: (1) Implicit preference data lacks definitive negative samples, which makes standard positive-negative classification methods inapplicable; (2) Implicit preference data suffers from user preference bias, where different responses have different propensities to elicit user feedback actions, which exacerbates the difficulty of distinguishing definitive negative samples. To address these challenges, we propose ImplicitRM, which aims to learn unbiased reward models from implicit preference data. ImplicitRM stratifies training samples into four latent groups via a stratification model. Building on this, it derives a learning objective through likelihood maximization, which we prove is theoretically unbiased, effectively resolving both challenges. Experiments demonstrate that ImplicitRM learns accurate reward models across implicit preference datasets. Code is available on our project website.

Related papers

• CausalRM: Causal-Theoretic Reward Modeling for RLHF from Observational User Feedbacks [65.44788139573144]
  We introduce observational reward modeling -- learning reward models with observational user feedback -- as a scalable and cost-effective alternative.<n>CaulRM aims to learn unbiased reward models from observational feedback.<n>Experiments validate that CausalRM effectively learns accurate reward signals from noisy and biased observational feedback.
  arXiv  Detail & Related papers  (2026-03-19T10:37:34Z)
• Debiasing Reward Models by Representation Learning with Guarantees [20.170532007501883]
  We propose a principled framework that mitigates spurious correlation issues in reward models.<n>We show that non-spurious latent variables can be theoretically identified from data, regardless of whether a surrogate for the spurious latent variables is available.<n>Experiments on synthetic and real-world datasets demonstrate that our method effectively mitigates spurious correlation issues and yields more robust reward models.
  arXiv  Detail & Related papers  (2025-10-27T18:37:57Z)
• Fake it till You Make it: Reward Modeling as Discriminative Prediction [49.31309674007382]
  GAN-RM is an efficient reward modeling framework that eliminates manual preference annotation and explicit quality dimension engineering.<n>Our method trains the reward model through discrimination between a small set of representative, unpaired target samples.<n>Experiments demonstrate our GAN-RM's effectiveness across multiple key applications.
  arXiv  Detail & Related papers  (2025-06-16T17:59:40Z)
• Detecting Prefix Bias in LLM-based Reward Models [4.596249232904721]
  We introduce novel methods to detect and evaluate prefix bias in reward models trained on preference datasets.<n>We leverage these metrics to reveal significant biases in preference models across racial and gender dimensions.<n>Our findings highlight the critical need for bias-aware dataset design and evaluation in developing fair and reliable reward models.
  arXiv  Detail & Related papers  (2025-05-13T21:50:03Z)
• RewardBench: Evaluating Reward Models for Language Modeling [100.28366840977966]
  We present RewardBench, a benchmark dataset and code-base for evaluation of reward models. The dataset is a collection of prompt-chosen-rejected trios spanning chat, reasoning, and safety. On the RewardBench leaderboard, we evaluate reward models trained with a variety of methods.
  arXiv  Detail & Related papers  (2024-03-20T17:49:54Z)
• Secrets of RLHF in Large Language Models Part II: Reward Modeling [134.97964938009588]
  We introduce a series of novel methods to mitigate the influence of incorrect and ambiguous preferences in the dataset. We also introduce contrastive learning to enhance the ability of reward models to distinguish between chosen and rejected responses.
  arXiv  Detail & Related papers  (2024-01-11T17:56:59Z)
• Towards Robust Visual Question Answering: Making the Most of Biased Samples via Contrastive Learning [54.61762276179205]
  We propose a novel contrastive learning approach, MMBS, for building robust VQA models by Making the Most of Biased Samples. Specifically, we construct positive samples for contrastive learning by eliminating the information related to spurious correlation from the original training samples. We validate our contributions by achieving competitive performance on the OOD dataset VQA-CP v2 while preserving robust performance on the ID dataset VQA v2.
  arXiv  Detail & Related papers  (2022-10-10T11:05:21Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.