Hidden Consensus:Preference-Validity Compression in Human Feedback

• URL: http://arxiv.org/abs/2606.10569v1
• Date: Tue, 09 Jun 2026 08:32:11 GMT
• Title: Hidden Consensus:Preference-Validity Compression in Human Feedback
• Abstract summary: We analyze RLHF-style feedback aggregation through preference events linking prompts, responses, and acceptability judgments.<n>Findings show that majority aggregation in this corpus measures argmax acceptability rather than plural alignment.
• Score: 5.277861091249917
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Standard RLHF pipelines often reduce heterogeneous human judgments into a single scalar reward target. We argue that this reduction can mis-measure alignment in structurally plural societies, where disagreement may reflect culturally, historically, linguistically, regionally, or normatively grounded interpretations rather than annotation noise. We call this failure Preference-Validity Compression, the collapse of multiple plural-valid response options into a single optimization target. Using Malaysia as a diagnostic setting, we analyze RLHF-style feedback aggregation through preference events linking prompts, responses, and acceptability judgments across interpretive frames. Across 321 preference events from 20 participants and 107 trio-annotated prompts, 79% of prompts contain more than one majority-supported response that single-winner aggregation would discard, and apparent dominance gaps between top responses diminish when all majority-supported options are considered. Participants frequently select multiple acceptable responses, and discarded responses demonstrably reflect coherent local, practical, or cultural frames. These findings show that majority aggregation in this corpus measures argmax acceptability rather than plural alignment. We treat this as a measurement-validity issue and argue that future alignment methods should satisfy Validity-Preserving Consistency, remaining stable across plural-valid interpretive frames rather than collapsing them into a single reward target.

Related papers

• Inference-Time Steering for Cross-Lingual Factual Consistency in LLMs [51.56484100374058]
  Large Language Models (LLMs) demonstrate remarkable multilingual fluency, but their internal knowledge representations remain disproportionately biased toward high-resource languages.<n>We investigate whether these biases can be mitigated at inference time, forcing an English-prompted model to answer as if it were queried in target languages.<n>Experiments on Gemma 3 12B Instruct reveal persona prompting to be the strongest overall intervention, balancing efficacy, safety, and out-of-domain generalization.
  arXiv  Detail & Related papers  (2026-07-21T16:15:05Z)
• BALTO: Balanced Token-Level Policy Optimization for Hallucination Mitigation [51.22170603236523]
  We propose a Balanced Token-level Policy Optimization framework for hallucination mitigation.<n>BalTO extracts checkable factual claims, verifies them against the reference context, and projects claim-level judgments to token-level labels.<n>Experiments on ConFiQA, RAGTruth, and FinLLM-Eval show that BALTO achieves the highest faithfulness across all six model--benchmark settings.
  arXiv  Detail & Related papers  (2026-06-14T16:25:59Z)
• CQC-RAG: Robust Retrieval-Augmented Generation via Cross-Query Consistency [9.50454647525075]
  We introduce CQC-RAG, a framework that co-designs query-level diversity injection with cross-Query Consistency evaluation.<n>CQC-RAG rewrites the original question into diverse but meaning-preserving queries.<n>It applies an evidence-grounded protocol to extract answer-evidence pairs and selects answers according to confidence stability.
  arXiv  Detail & Related papers  (2026-06-11T15:01:49Z)
• Escaping the Mode Lottery: Multi-Response Training Improves Language Model Generalization [14.039980020878815]
  Modern language-model fine-tuning pairs each prompt with a single response, even though many prompts admit multiple valid completions.<n>This effectively reduces a multi-modal conditional distribution to a one-sample view, where training emphasizes a subset of plausible modes.<n>We study multi-response training (MRT), which retains multiple responses per prompt, and develop a principled account of when and why it helps.
  arXiv  Detail & Related papers  (2026-05-30T05:30:39Z)
• From Sycophantic Consensus to Pluralistic Repair: Why AI Alignment Must Surface Disagreement [8.459329029609602]
  We argue that aggregation alone is an incomplete primitive for deployed pluralistic alignment.<n>We formalise a metric, the Pluralistic Repair Score (PRS), distinguishing principled revision from capitulation.
  arXiv  Detail & Related papers  (2026-05-14T14:47:06Z)
• Response Time Enhances Alignment with Heterogeneous Preferences [49.69696266152175]
  We show that augmenting preference datasets with a simple, secondary signal can restore identifiability of the population's average preference.<n>Our results bring promises and open up new opportunities for future data-collection pipelines.
  arXiv  Detail & Related papers  (2026-05-07T22:05:23Z)
• Pushing the Boundaries of Multiple Choice Evaluation to One Hundred Options [4.902225285395898]
  We propose a massive option evaluation protocol that scales the candidate set to one hundred options.<n>We apply this framework to a Korean orthography error detection task.<n>Results indicate that strong performance in low option settings can overstate model competence.
  arXiv  Detail & Related papers  (2026-04-16T05:22:12Z)
• A Single Revision Step Improves Token-Efficient LLM Reasoning [3.344806691289323]
  We introduce Packet-Conditioned Revision (PACER), a training-free, inference-only framework for large language models.<n>PACER enables reasoning traces to revise their conclusions through a structured coordination step.<n>On challenging competitive math benchmarks, PACER matches or exceeds the accuracy of 256-sample majority voting.
  arXiv  Detail & Related papers  (2026-02-02T21:28:42Z)
• Critique to Verify: Accurate and Honest Test-Time Scaling with RL-Trained Verifiers [63.99316853136304]
  Mirror-Critique is a framework that trains a verifier with informative critiques.<n>We deploy a small instruction-tuned model to synthesize high-quality critique data.<n>The resulting Mirror-Verifier is deployed to evaluate candidate solutions by generating multiple critiques per solution.
  arXiv  Detail & Related papers  (2025-09-27T06:50:24Z)
• Pairwise Calibrated Rewards for Pluralistic Alignment [32.82521299417802]
  Current alignment pipelines presume a single, universal notion of desirable behavior.<n>We propose reflecting diverse human preferences through a distribution over multiple reward functions.<n>We prove that even a small outlier-free ensemble can accurately represent diverse preference distributions.
  arXiv  Detail & Related papers  (2025-05-17T18:38:24Z)
• Unintentional Unalignment: Likelihood Displacement in Direct Preference Optimization [60.176008034221404]
  Direct Preference Optimization (DPO) and its variants are increasingly used for aligning language models with human preferences.<n>Prior work has observed that the likelihood of preferred responses often decreases during training.<n>We demonstrate that likelihood displacement can be catastrophic, shifting probability mass from preferred responses to responses with an opposite meaning.
  arXiv  Detail & Related papers  (2024-10-11T14:22:44Z)
• From Chaos to Clarity: Claim Normalization to Empower Fact-Checking [57.024192702939736]
  Claim Normalization (aka ClaimNorm) aims to decompose complex and noisy social media posts into more straightforward and understandable forms. We propose CACN, a pioneering approach that leverages chain-of-thought and claim check-worthiness estimation. Our experiments demonstrate that CACN outperforms several baselines across various evaluation measures.
  arXiv  Detail & Related papers  (2023-10-22T16:07:06Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.