Pairwise Calibrated Rewards for Pluralistic Alignment

• URL: http://arxiv.org/abs/2506.06298v1
• Date: Sat, 17 May 2025 18:38:24 GMT
• Title: Pairwise Calibrated Rewards for Pluralistic Alignment
• Abstract summary: Current alignment pipelines presume a single, universal notion of desirable behavior.<n>We propose reflecting diverse human preferences through a distribution over multiple reward functions.<n>We prove that even a small outlier-free ensemble can accurately represent diverse preference distributions.
• Score: 32.82521299417802
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Current alignment pipelines presume a single, universal notion of desirable behavior. However, human preferences often diverge across users, contexts, and cultures. As a result, disagreement collapses into the majority signal and minority perspectives are discounted. To address this, we propose reflecting diverse human preferences through a distribution over multiple reward functions, each inducing a distinct aligned policy. The distribution is learned directly from pairwise preference without annotator identifiers or predefined groups. Instead, annotator disagreements are treated as informative soft labels. Our central criterion is pairwise calibration: for every pair of candidate responses, the proportion of reward functions preferring one response matches the fraction of annotators with that preference. We prove that even a small outlier-free ensemble can accurately represent diverse preference distributions. Empirically, we introduce and validate a practical training heuristic to learn such ensembles, and demonstrate its effectiveness through improved calibration, implying a more faithful representation of pluralistic values.

Related papers

• Hidden Consensus:Preference-Validity Compression in Human Feedback [5.277861091249917]
  We analyze RLHF-style feedback aggregation through preference events linking prompts, responses, and acceptability judgments.<n>Findings show that majority aggregation in this corpus measures argmax acceptability rather than plural alignment.
  arXiv  Detail & Related papers  (2026-06-09T08:32:11Z)
• Response Time Enhances Alignment with Heterogeneous Preferences [49.69696266152175]
  We show that augmenting preference datasets with a simple, secondary signal can restore identifiability of the population's average preference.<n>Our results bring promises and open up new opportunities for future data-collection pipelines.
  arXiv  Detail & Related papers  (2026-05-07T22:05:23Z)
• Minority-Aware Satisfaction Estimation in Dialogue Systems via Preference-Adaptive Reinforcement Learning [19.994184617064395]
  We propose a unified framework that models both individual- and group-level preferences for user satisfaction estimation.<n>Experiments on the Emotional Support Conversation dataset demonstrate consistent improvements in user satisfaction estimation.
  arXiv  Detail & Related papers  (2025-11-07T16:34:03Z)
• Judging with Confidence: Calibrating Autoraters to Preference Distributions [56.17041629492863]
  We argue that a reliable autorater must learn to model the full distribution of preferences defined by a target population.<n>We present two learning methods tailored to different data conditions.<n>Our results show that finetuning autoraters with a distribution-matching objective leads to verbalized probability predictions that are better aligned with the target preference distribution.
  arXiv  Detail & Related papers  (2025-09-30T20:36:41Z)
• Population-Proportional Preference Learning from Human Feedback: An Axiomatic Approach [6.578074497549894]
  We develop a novel preference learning framework capable of aligning aggregate opinions and policies proportionally with the true population distribution of evaluator preferences.<n>Our approach infers the feasible set of evaluator population distributions directly from pairwise comparison data.<n>We propose a soft-max relaxation method that smoothly trade-offs population-proportional representation with the selection of the Condorcet winner.
  arXiv  Detail & Related papers  (2025-06-05T22:15:07Z)
• Direct Alignment with Heterogeneous Preferences [11.693372619696683]
  We show that aligning to heterogeneous preferences with a single policy is best achieved using the average reward across user types.<n>We find that minimal information can yield first-order improvements, while full feedback from each user type leads to consistent learning of the optimal policy.
  arXiv  Detail & Related papers  (2025-02-22T18:46:33Z)
• Probably Approximately Precision and Recall Learning [60.00180898830079]
  A key challenge in machine learning is the prevalence of one-sided feedback.<n>We introduce a Probably Approximately Correct (PAC) framework in which hypotheses are set functions that map each input to a set of labels.<n>We develop new algorithms that learn from positive data alone, achieving optimal sample complexity in the realizable case.
  arXiv  Detail & Related papers  (2024-11-20T04:21:07Z)
• Not All Preference Pairs Are Created Equal: A Recipe for Annotation-Efficient Iterative Preference Learning [81.69044784288005]
  Iterative preference learning requires online annotated preference labels. We study strategies to select worth-annotating response pairs for cost-efficient annotation.
  arXiv  Detail & Related papers  (2024-06-25T06:49:16Z)
• Whose Preferences? Differences in Fairness Preferences and Their Impact on the Fairness of AI Utilizing Human Feedback [8.04095222893591]
  We find significant gaps in fairness preferences depending on the race, age, political stance, educational level, and LGBTQ+ identity of annotators. We also demonstrate that demographics mentioned in text have a strong influence on how users perceive individual fairness in moderation.
  arXiv  Detail & Related papers  (2024-06-09T19:42:25Z)
• Provable Multi-Party Reinforcement Learning with Diverse Human Feedback [63.830731470186855]
  Reinforcement learning with human feedback (RLHF) is an emerging paradigm to align models with human preferences. We show how traditional RLHF approaches can fail since learning a single reward function cannot capture and balance the preferences of multiple individuals. We incorporate meta-learning to learn multiple preferences and adopt different social welfare functions to aggregate the preferences across multiple parties.
  arXiv  Detail & Related papers  (2024-03-08T03:05:11Z)
• MaxMin-RLHF: Alignment with Diverse Human Preferences [101.57443597426374]
  Reinforcement Learning from Human Feedback (RLHF) aligns language models to human preferences by employing a singular reward model derived from preference data.<n>We learn a mixture of preference distributions via an expectation-maximization algorithm to better represent diverse human preferences.<n>Our algorithm achieves an average improvement of more than 16% in win-rates over conventional RLHF algorithms.
  arXiv  Detail & Related papers  (2024-02-14T03:56:27Z)
• Dealing with Disagreements: Looking Beyond the Majority Vote in Subjective Annotations [6.546195629698355]
  We investigate the efficacy of multi-annotator models for subjective tasks. We show that this approach yields same or better performance than aggregating labels in the data prior to training. Our approach also provides a way to estimate uncertainty in predictions, which we demonstrate better correlate with annotation disagreements than traditional methods.
  arXiv  Detail & Related papers  (2021-10-12T03:12:34Z)
• Consensus-Guided Correspondence Denoising [67.35345850146393]
  We propose to denoise correspondences with a local-to-global consensus learning framework to robustly identify correspondence. A novel "pruning" block is introduced to distill reliable candidates from initial matches according to their consensus scores estimated by dynamic graphs from local to global regions. Our method outperforms state-of-the-arts on robust line fitting, wide-baseline image matching and image localization benchmarks by noticeable margins.
  arXiv  Detail & Related papers  (2021-01-03T09:10:00Z)
• Towards Model-Agnostic Post-Hoc Adjustment for Balancing Ranking Fairness and Algorithm Utility [54.179859639868646]
  Bipartite ranking aims to learn a scoring function that ranks positive individuals higher than negative ones from labeled data. There have been rising concerns on whether the learned scoring function can cause systematic disparity across different protected groups. We propose a model post-processing framework for balancing them in the bipartite ranking scenario.
  arXiv  Detail & Related papers  (2020-06-15T10:08:39Z)
• SetRank: A Setwise Bayesian Approach for Collaborative Ranking from Implicit Feedback [50.13745601531148]
  We propose a novel setwise Bayesian approach for collaborative ranking, namely SetRank, to accommodate the characteristics of implicit feedback in recommender system. Specifically, SetRank aims at maximizing the posterior probability of novel setwise preference comparisons. We also present the theoretical analysis of SetRank to show that the bound of excess risk can be proportional to $sqrtM/N$.
  arXiv  Detail & Related papers  (2020-02-23T06:40:48Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.