Deployable Human Preference Alignment in Robotics: Learning Representative Rewards from Diverse Human Preferences

• URL: http://arxiv.org/abs/2607.12466v1
• Date: Tue, 14 Jul 2026 07:50:43 GMT
• Title: Deployable Human Preference Alignment in Robotics: Learning Representative Rewards from Diverse Human Preferences
• Abstract summary: In per-user alignment approach, preference feedback is often sparse, so learning becomes unstable and vulnerable to human preference noise.<n>A single shared policy approach to user alignment avoids this cost but fails to capture heterogeneous preferences and often neglects minority preferences.<n>We introduce Preference-based REward Clustering (PREC), a novel framework that learns a compact set of policies from binary preference labels provided by diverse users.
• Score: 2.0531872026357383
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Aligning robot policies with human preferences is essential for deployment to diverse end users. In per-user alignment approach, preference feedback is often sparse, so learning becomes unstable and vulnerable to human preference noise, and a growing number of individualized policies makes validation difficult before deployment. A single shared policy approach to user alignment avoids this cost but fails to capture heterogeneous preferences and often neglects minority preferences. To address these challenges, we introduce Preference-based REward Clustering (PREC), a novel framework that learns a compact set of policies from binary preference labels provided by diverse users. From a dataset of user trajectories and their preference labels, PREC first sets the labels aside and aggregates trajectories across users to learn a population-level shared trajectory encoder, alleviating limited per-user coverage and avoiding label noise during representation learning. Using this representation, PREC jointly assigns users to preference-coherent clusters and learns a representative reward model per cluster using preference labels, from which a policy is optimized for each cluster. Clustering similar users compensates for the limited number of labels available from each user and mitigates the effect of label noise. At the same time, maintaining a manageable number of reward models reduces the validation burden at deployment. Experiments across diverse simulated locomotion environments show that PREC groups users who label different trajectory subsets into preference-coherent clusters more accurately than baseline methods. Under sparse and noisy feedback, policies trained with PREC improve all three social welfare metrics over an existing single shared-policy user-alignment approach and even outperform per-user alignment approaches.

Related papers

• Rethinking Personalized Reward Modeling for LLMs under Preference Heterogeneity via Group-Debiased Federated Learning [31.693764625836305]
  We propose FedGD (Federated Learning with Group Debiasing), which discovers latent preference groups during federated training and learns a single reward model using group-debiased client sampling.
  arXiv  Detail & Related papers  (2026-08-03T00:25:52Z)
• Personalizing LLMs with Binary Feedback: A Preference-Corrected Optimization Framework [17.343792633458254]
  Large Language Model (LLM) personalization aims to align model behaviors with individual user preferences.<n>Existing methods often focus on isolated user histories, neglecting the essential role of inter-user differences.<n>We propose C-BPO, a framework that personalizes LLMs via preference-calibrated binary signals.
  arXiv  Detail & Related papers  (2026-05-11T06:12:24Z)
• Response Time Enhances Alignment with Heterogeneous Preferences [49.69696266152175]
  We show that augmenting preference datasets with a simple, secondary signal can restore identifiability of the population's average preference.<n>Our results bring promises and open up new opportunities for future data-collection pipelines.
  arXiv  Detail & Related papers  (2026-05-07T22:05:23Z)
• Replacing Parameters with Preferences: Federated Alignment of Heterogeneous Vision-Language Models [63.70401095689976]
  Vision-Language Models (VLMs) have broad potential in privacy-sensitive domains such as healthcare and finance.<n>We propose MoR, a federated alignment framework that combines GRPO with Mixture-of-Rewards for heterogeneousVLMs.<n>MoR consistently outperforms federated alignment baselines in generalization and cross-client adaptability.
  arXiv  Detail & Related papers  (2026-05-05T07:02:50Z)
• FB-NLL: A Feature-Based Approach to Tackle Noisy Labels in Personalized Federated Learning [2.750124853532831]
  We propose FB-NLL, a feature-centric framework that decouples user clustering from iterative training dynamics.<n>Our framework consistently outperforms state-of-the-art baselines in terms of average accuracy and performance stability.
  arXiv  Detail & Related papers  (2026-04-21T17:51:58Z)
• RCC-PFL: Robust Client Clustering under Noisy Labels in Personalized Federated Learning [8.37314799155978]
  We propose a label-agnostic data similarity-based clustering algorithm, coined RCC-PFL.<n>We show that it outperforms multiple baselines in terms of average accuracy and variance reduction.
  arXiv  Detail & Related papers  (2025-03-25T17:50:54Z)
• Few-shot Steerable Alignment: Adapting Rewards and LLM Policies with Neural Processes [50.544186914115045]
  Large language models (LLMs) are increasingly embedded in everyday applications.<n> Ensuring their alignment with the diverse preferences of individual users has become a critical challenge.<n>We present a novel framework for few-shot steerable alignment.
  arXiv  Detail & Related papers  (2024-12-18T16:14:59Z)
• Improved Diversity-Promoting Collaborative Metric Learning for Recommendation [127.08043409083687]
  Collaborative Metric Learning (CML) has recently emerged as a popular method in recommendation systems. This paper focuses on a challenging scenario where a user has multiple categories of interests. We propose a novel method called textitDiversity-Promoting Collaborative Metric Learning (DPCML)
  arXiv  Detail & Related papers  (2024-09-02T07:44:48Z)
• The Minority Matters: A Diversity-Promoting Collaborative Metric Learning Algorithm [154.47590401735323]
  Collaborative Metric Learning (CML) has recently emerged as a popular method in recommendation systems. This paper focuses on a challenging scenario where a user has multiple categories of interests. We propose a novel method called textitDiversity-Promoting Collaborative Metric Learning (DPCML)
  arXiv  Detail & Related papers  (2022-09-30T08:02:18Z)
• Overcoming Data Sparsity in Group Recommendation [52.00998276970403]
  Group recommender systems should be able to accurately learn not only users' personal preferences but also preference aggregation strategy. In this paper, we take Bipartite Graphding Model (BGEM), the self-attention mechanism and Graph Convolutional Networks (GCNs) as basic building blocks to learn group and user representations in a unified way.
  arXiv  Detail & Related papers  (2020-10-02T07:11:19Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.