Rethinking Personalized Reward Modeling for LLMs under Preference Heterogeneity via Group-Debiased Federated Learning

• URL: http://arxiv.org/abs/2608.01556v1
• Date: Mon, 03 Aug 2026 00:25:52 GMT
• Title: Rethinking Personalized Reward Modeling for LLMs under Preference Heterogeneity via Group-Debiased Federated Learning
• Abstract summary: We propose FedGD (Federated Learning with Group Debiasing), which discovers latent preference groups during federated training and learns a single reward model using group-debiased client sampling.
• Score: 31.693764625836305
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Large language models are increasingly aligned to human preferences via reward modeling, but user preference data are sensitive and often cannot be centralized. Federated learning keeps such data local while learning a shared initial reward model, which is later personalized for each client through local fine-tuning. Because users often assign opposite labels to the same pair of responses, existing federated methods address preference heterogeneity by clustering similar clients and training one reward model per group, assuming that each group requires its own initialization. We show that this assumption is unnecessary. Under balanced preference groups, a single FedAvg model, despite starting at nearly random accuracy, surpasses reward models trained separately for each ground-truth group after only a few local optimization steps. We attribute this phenomenon to the flatness of the shared initialization: averaging across all clients learns richer shared representations that distinguish responses while canceling conflicting preference directions, leaving the model near a decision boundary that can be rapidly adapted. Group imbalance breaks this effect as the cancellation becomes asymmetric and leaves minority clients too far from the boundary to recover. Motivated by this observation, we propose FedGD (Federated Learning with Group Debiasing), which discovers latent preference groups during federated training and learns a single reward model using group-debiased client sampling. By counteracting the effect of group imbalance, FedGD learns an initialization that remains highly adaptable, enabling effective personalization without prior knowledge of the underlying groups.

Related papers

• PAFO: Pareto Fairness Optimization for Personalized Reward Modeling [43.95197844656587]
  Large language models (LLMs) increasingly rely on reward models to align their outputs with diverse user preferences.<n> personalized reward models are often trained on imbalanced user preference data and may therefore favor users whose preferences are more common in the training population.
  arXiv  Detail & Related papers  (2026-06-06T05:35:31Z)
• Enhancing Federated Quadruplet Learning: Stochastic Client Selection and Embedding Stability Analysis [2.298932494750101]
  Federated Learning (FL) enables decentralised model training across distributed clients without requiring data centralisation.<n>We propose FedQuad, a novel method that explicitly enforces minimising intra-class representations while enabling inter-class splits across clients.
  arXiv  Detail & Related papers  (2026-05-08T15:34:41Z)
• Replacing Parameters with Preferences: Federated Alignment of Heterogeneous Vision-Language Models [63.70401095689976]
  Vision-Language Models (VLMs) have broad potential in privacy-sensitive domains such as healthcare and finance.<n>We propose MoR, a federated alignment framework that combines GRPO with Mixture-of-Rewards for heterogeneousVLMs.<n>MoR consistently outperforms federated alignment baselines in generalization and cross-client adaptability.
  arXiv  Detail & Related papers  (2026-05-05T07:02:50Z)
• FedDES: Graph-Based Dynamic Ensemble Selection for Personalized Federated Learning [0.07161783472741748]
  FedDES is a decentralized pFL framework that achieves instance-level personalization through dynamic ensemble selection.<n>A Graph Neural Network (GNN) meta-learner is trained on a heterogeneous graph modeling interactions between data samples and candidate classifiers.<n>Experiments on CIFAR-10 and real-world ICU healthcare data demonstrate that FedDES outperforms state-of-the-art pFL baselines in non-IID settings.
  arXiv  Detail & Related papers  (2026-03-30T03:57:52Z)
• Interaction-Aware Gaussian Weighting for Clustered Federated Learning [58.92159838586751]
  Federated Learning (FL) emerged as a decentralized paradigm to train models while preserving privacy.<n>We propose a novel clustered FL method, FedGWC (Federated Gaussian Weighting Clustering), which groups clients based on their data distribution.<n>Our experiments on benchmark datasets show that FedGWC outperforms existing FL algorithms in cluster quality and classification accuracy.
  arXiv  Detail & Related papers  (2025-02-05T16:33:36Z)
• ComPO: Community Preferences for Language Model Personalization [122.54846260663922]
  ComPO is a method to personalize preference optimization in language models. We collect and release ComPRed, a question answering dataset with community-level preferences from Reddit.
  arXiv  Detail & Related papers  (2024-10-21T14:02:40Z)
• Mitigating Group Bias in Federated Learning for Heterogeneous Devices [1.181206257787103]
  Federated Learning is emerging as a privacy-preserving model training approach in distributed edge applications. Our work proposes a group-fair FL framework that minimizes group-bias while preserving privacy and without resource utilization overhead.
  arXiv  Detail & Related papers  (2023-09-13T16:53:48Z)
• Federated Two Stage Decoupling With Adaptive Personalization Layers [5.69361786082969]
  Federated learning has gained significant attention due to its ability to enable distributed learning while maintaining privacy constraints. It inherently experiences significant learning degradation and slow convergence speed. It is natural to employ the concept of clustering homogeneous clients into the same group, allowing only the model weights within each group to be aggregated.
  arXiv  Detail & Related papers  (2023-08-30T07:46:32Z)
• Fed-CBS: A Heterogeneity-Aware Client Sampling Mechanism for Federated Learning via Class-Imbalance Reduction [76.26710990597498]
  We show that the class-imbalance of the grouped data from randomly selected clients can lead to significant performance degradation. Based on our key observation, we design an efficient client sampling mechanism, i.e., Federated Class-balanced Sampling (Fed-CBS) In particular, we propose a measure of class-imbalance and then employ homomorphic encryption to derive this measure in a privacy-preserving way.
  arXiv  Detail & Related papers  (2022-09-30T05:42:56Z)
• On the Convergence of Clustered Federated Learning [57.934295064030636]
  In a federated learning system, the clients, e.g. mobile devices and organization participants, usually have different personal preferences or behavior patterns. This paper proposes a novel weighted client-based clustered FL algorithm to leverage the client's group and each client in a unified optimization framework.
  arXiv  Detail & Related papers  (2022-02-13T02:39:19Z)
• Overcoming Data Sparsity in Group Recommendation [52.00998276970403]
  Group recommender systems should be able to accurately learn not only users' personal preferences but also preference aggregation strategy. In this paper, we take Bipartite Graphding Model (BGEM), the self-attention mechanism and Graph Convolutional Networks (GCNs) as basic building blocks to learn group and user representations in a unified way.
  arXiv  Detail & Related papers  (2020-10-02T07:11:19Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.