Three Models of RLHF Annotation: Extension, Evidence, and Authority

• URL: http://arxiv.org/abs/2604.25895v1
• Date: Tue, 28 Apr 2026 17:39:14 GMT
• Title: Three Models of RLHF Annotation: Extension, Evidence, and Authority
• Abstract summary: I argue that these models have implications for how RLHF pipelines should solicit, validate and aggregate annotations.<n>My central recommendation is that RLHF pipeline designers should decompose annotation into separable dimensions.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Preference-based alignment methods, most prominently Reinforcement Learning with Human Feedback (RLHF), use the judgments of human annotators to shape large language model behaviour. However, the normative role of these judgments is rarely made explicit. I distinguish three conceptual models of that role. The first is extension: annotators extend the system designers' own judgments about what outputs should be. The second is evidence: annotators provide independent evidence about some facts, whether moral, social or otherwise. The third is authority: annotators have some independent authority (as representatives of the broader population) to determine system outputs. I argue that these models have implications for how RLHF pipelines should solicit, validate and aggregate annotations. I survey landmark papers in the literature on RLHF and related methods to illustrate how they implicitly draw on these models, describe failure modes that come from unintentionally or intentionally conflating them, and offer normative criteria for choosing among them. My central recommendation is that RLHF pipeline designers should decompose annotation into separable dimensions and tailor each pipeline to the model most appropriate for that dimension, rather than seeking a single unified pipeline.

Related papers

• The Neutral Mask: How RLHF Provides Shallow Alignment while Leaving Partisan Structure Intact in a Large Language Model [0.0]
  We show that RLHF does not remove the structured partisan direction in the base model.<n>We show that RLHF encodes a norm of political neutrality, not by erasing the model's knowledge of partisanship, but by severing the causal pathway from partisan geometry to output generation.
  arXiv  Detail & Related papers  (2026-06-08T17:00:31Z)
• Small Reward Models via Backward Inference [100.59075794599768]
  FLIP (FLipped Inference for Prompt Reconstruction) is a reference-free and rubric-free reward modeling approach.<n>It reformulates reward modeling through backward inference: inferring the instruction that would most plausibly produce a given response.
  arXiv  Detail & Related papers  (2026-02-14T01:55:39Z)
• Dependence-Aware Label Aggregation for LLM-as-a-Judge via Ising Models [55.94503936470247]
  Large-scale AI evaluation increasingly relies on aggregating binary judgments from $K$ annotators, including judges.<n>Most classical methods assume annotators are conditionally independent given the true label $Yin0,1$, an assumption often violated by LLM judges.<n>We study label aggregation through a hierarchy of dependence-aware models based on Ising graphical models and latent factors.
  arXiv  Detail & Related papers  (2026-01-29T21:26:50Z)
• JudgeRLVR: Judge First, Generate Second for Efficient Reasoning [20.448286296459344]
  Reinforcement Learning with Verifiable Rewards (RLVR) has become a standard paradigm for reasoning in Large Language Models.<n>In this paper, we argue that discriminative capability is a prerequisite for efficient generation.<n>We propose JudgeRLVR, a two-stage judge-then-generate paradigm.
  arXiv  Detail & Related papers  (2026-01-13T11:47:42Z)
• Mitigating Length Bias in RLHF through a Causal Lens [8.334918207379173]
  Reinforcement learning from human feedback (RLHF) is widely used to align large language models with human preferences.<n>We propose a causal framework for analyzing and mitigating length bias in RLHF reward modeling.
  arXiv  Detail & Related papers  (2025-11-16T12:25:10Z)
• Think Twice: Branch-and-Rethink Reasoning Reward Model [32.70732791642558]
  We introduce branch-and-rethink (BR-RM), a two-turn RM that transfers the think-twice principle to reward modeling.<n>We train with GRPO-style reinforcement learning over structured two-turn traces using a simple binary outcome reward with strict format checks.<n>By converting all-at-oncescoringintofocused, second-lookreasoning, BR-RMreducesjudgmentdiffusionandimproves sensitivity to subtle yet consequential errors.
  arXiv  Detail & Related papers  (2025-10-27T17:58:07Z)
• Interactive Reasoning: Visualizing and Controlling Chain-of-Thought Reasoning in Large Language Models [54.85405423240165]
  We introduce Interactive Reasoning, an interaction design that visualizes chain-of-thought outputs as a hierarchy of topics.<n>We implement interactive reasoning in Hippo, a prototype for AI-assisted decision making in the face of uncertain trade-offs.
  arXiv  Detail & Related papers  (2025-06-30T10:00:43Z)
• Bias Fitting to Mitigate Length Bias of Reward Model in RLHF [81.44256822500257]
  Reinforcement Learning from Human Feedback relies on reward models to align large language models with human preferences.<n>We propose FiMi-RM, a framework that autonomously learns and corrects underlying bias patterns.<n> Experimental results demonstrate that FiMi-RM achieves a more balanced length-reward distribution.
  arXiv  Detail & Related papers  (2025-05-19T08:29:28Z)
• How to Evaluate Reward Models for RLHF [51.31240621943791]
  We introduce a new benchmark for reward models that quantifies their ability to produce strong language models through RLHF (Reinforcement Learning from Human Feedback) We build a predictive model of downstream LLM performance by evaluating the reward model on proxy tasks. We launch an end-to-end RLHF experiment on a large-scale crowdsourced human preference platform to view real reward model downstream performance as ground truth.
  arXiv  Detail & Related papers  (2024-10-18T21:38:21Z)
• Predicting vs. Acting: A Trade-off Between World Modeling & Agent Modeling [34.3520410136648]
  RLHF-aligned LMs have shown unprecedented ability on both benchmarks and long-form text generation. As RLHF models become agent models aimed at interacting with humans, they seem to lose their world modeling. We propose a potential explanation: to perform coherent long-form generation, RLHF models restrict randomness via implicit blueprints.
  arXiv  Detail & Related papers  (2024-07-02T17:22:54Z)
• A Long Way to Go: Investigating Length Correlations in RLHF [59.49656695716066]
  This paper demonstrates, on three diverse settings, that optimizing for response length is a significant factor behind RLHF. We find improvements in reward to largely be driven by increasing response length, instead of other features. Even a purely length-based reward reproduces most downstream RLHF improvements over supervised fine-tuned models.
  arXiv  Detail & Related papers  (2023-10-05T17:38:28Z)
• Did the Models Understand Documents? Benchmarking Models for Language Understanding in Document-Level Relation Extraction [2.4665182280122577]
  Document-level relation extraction (DocRE) attracts more research interest recently. While models achieve consistent performance gains in DocRE, their underlying decision rules are still understudied. In this paper, we take the first step toward answering this question and then introduce a new perspective on comprehensively evaluating a model.
  arXiv  Detail & Related papers  (2023-06-20T08:52:05Z)
• Why do you think that? Exploring Faithful Sentence-Level Rationales Without Supervision [60.62434362997016]
  We propose a differentiable training-framework to create models which output faithful rationales on a sentence level. Our model solves the task based on each rationale individually and learns to assign high scores to those which solved the task best.
  arXiv  Detail & Related papers  (2020-10-07T12:54:28Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.