FedPDPO: Federated Personalized Direct Preference Optimization for Large Language Model Alignment

• URL: http://arxiv.org/abs/2603.19741v1
• Date: Fri, 20 Mar 2026 08:24:49 GMT
• Title: FedPDPO: Federated Personalized Direct Preference Optimization for Large Language Model Alignment
• Abstract summary: Fed PDPO (Federated Personalized Direct Preference Optimization) is a personalized framework for preference alignment of large language models (LLMs)<n>It adopts a parameter-efficient fine-tuning architecture where each client maintains a frozen pretrained LLM backbone augmented with a Low-Rank Adaptation (LoRA) adapter, enabling communication-efficient aggregation.<n>Experiments on multiple preference datasets demonstrate state-of-the-art performance, achieving up to 4.80% average accuracy improvements in federated intra-domain and cross-domain settings.
• Score: 55.97027207627
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Aligning large language models (LLMs) with human preferences in federated learning (FL) is challenging due to decentralized, privacy-sensitive, and highly non-IID preference data. Direct Preference Optimization (DPO) offers an efficient alternative to reinforcement learning with human feedback (RLHF), but its direct application in FL suffers from severe performance degradation under non-IID data and limited generalization of implicit rewards. To bridge this gap, we propose FedPDPO (Federated Personalized Direct Preference Optimization), a personalized federated framework for preference alignment of LLMs. It adopts a parameter-efficient fine-tuning architecture where each client maintains a frozen pretrained LLM backbone augmented with a Low-Rank Adaptation (LoRA) adapter, enabling communication-efficient aggregation. To address non-IID heterogeneity, we devise (1) the globally shared LoRA adapter with the personalized client-specific LLM head. Moreover, we introduce (2) a personalized DPO training strategy with a client-specific explicit reward head to complement implicit rewards and further alleviate non-IID heterogeneity, and (3) a bottleneck adapter to balance global and local features. We provide theoretical analysis establishing the probabilistic foundation and soundness. Extensive experiments on multiple preference datasets demonstrate state-of-the-art performance, achieving up to 4.80% average accuracy improvements in federated intra-domain and cross-domain settings.

Related papers

• Breaking the Structural Identity: Personalized Federated LoRA Fine-tuning under Rank Heterogeneity [12.733972494875713]
  FedRoRA is a novel framework that enables fine-grained personalization within rank-heterogeneous federations.<n>It decouples adaptation into shared global directions and personalized rank-wise magnitudes governed by learnable diagonal scales.<n>Experiments on NLU and NLG benchmarks demonstrate that FedRoRA consistently outperforms state-of-the-art methods.
  arXiv  Detail & Related papers  (2026-09-01T03:09:20Z)
• Cooperative Multi-Agent Reinforcement Learning for Adaptive Aggregation in Semi-Supervised Federated Learning with non-IID Data [4.9856574965240545]
  Federated Learning (FL) enables distributed training of machine learning models while preserving data privacy.<n>We propose pFedMARL, a novel approach leveraging Multi-Agent Reinforcement Learning (MARL) with Twin Delayed Deep Deterministic Policy Gradient (TD3)<n>Our method employs a server-side agent adjusting client contributions to optimize global model robustness and client-side agents balancing global and local updates to personalize models effectively without pre-training.
  arXiv  Detail & Related papers  (2026-08-26T13:46:43Z)
• Spectral Souping: A Unified Framework for Online Preference Alignment [39.263620139543505]
  Reinforcement Learning from Human Feedback (RLHF) effectively aligns Large Language Models with aggregate human preferences but often fails to address the diverse and conflicting needs of individual users.<n>We introduce Spectral Souping, a unified framework for efficient, online preference alignment.<n> Experiments on online preference alignment benchmarks demonstrate that our method achieves significant performance improvements over existing state-of-the-art approaches.
  arXiv  Detail & Related papers  (2026-05-19T19:04:47Z)
• Replacing Parameters with Preferences: Federated Alignment of Heterogeneous Vision-Language Models [63.70401095689976]
  Vision-Language Models (VLMs) have broad potential in privacy-sensitive domains such as healthcare and finance.<n>We propose MoR, a federated alignment framework that combines GRPO with Mixture-of-Rewards for heterogeneousVLMs.<n>MoR consistently outperforms federated alignment baselines in generalization and cross-client adaptability.
  arXiv  Detail & Related papers  (2026-05-05T07:02:50Z)
• FeDecider: An LLM-Based Framework for Federated Cross-Domain Recommendation [75.50721642765994]
  Large language model (LLM)-based recommendation models have demonstrated impressive performance.<n>We propose an LLM-based framework for Federated cross-domain recommendation, FeDecider.<n>Extensive experiments across diverse datasets validate the effectiveness of our proposed FeDecider.
  arXiv  Detail & Related papers  (2026-02-17T21:42:28Z)
• CO-PFL: Contribution-Oriented Personalized Federated Learning for Heterogeneous Networks [51.43780477302533]
  Contribution-Oriented PFL (CO-PFL) is a novel algorithm that dynamically estimates each client's contribution for global aggregation.<n>CO-PFL consistently surpasses state-of-the-art methods in robustness in personalization accuracy, robustness, scalability and convergence stability.
  arXiv  Detail & Related papers  (2025-10-23T05:10:06Z)
• FedDuA: Doubly Adaptive Federated Learning [2.6108066206600555]
  Federated learning is a distributed learning framework where clients collaboratively train a global model without sharing their raw data.<n>We formalize the central server optimization procedure through the lens of mirror descent and propose a novel framework, called FedDuA.<n>We prove that our proposed doubly adaptive step-size rule is minimax optimal and provide a convergence analysis for convex objectives.
  arXiv  Detail & Related papers  (2025-05-16T11:15:27Z)
• Personalized Federated Fine-tuning for Heterogeneous Data: An Automatic Rank Learning Approach via Two-Level LoRA [14.786030311860145]
  PF2LoRA is a new personalized federated fine-tuning algorithm built on a novel emphautomatic rank learning approach via two-level LoRA<n>Our experiments on natural language understanding and generation tasks demonstrate that PF2LoRA significantly outperforms existing federated fine-tuning methods.
  arXiv  Detail & Related papers  (2025-03-05T21:41:03Z)
• Client-Centric Federated Adaptive Optimization [78.30827455292827]
  Federated Learning (FL) is a distributed learning paradigm where clients collaboratively train a model while keeping their own data private.<n>We propose Federated-Centric Adaptive Optimization, which is a class of novel federated optimization approaches.
  arXiv  Detail & Related papers  (2025-01-17T04:00:50Z)
• AlphaDPO: Adaptive Reward Margin for Direct Preference Optimization [45.46582930202524]
  $alpha$-DPO is an adaptive preference optimization algorithm for large language models.<n>It balances the policy model and the reference model to achieve personalized reward margins.<n>It consistently outperforms DPO and SimPO across various model settings.
  arXiv  Detail & Related papers  (2024-10-14T04:29:57Z)
• Adaptive Preference Scaling for Reinforcement Learning with Human Feedback [103.36048042664768]
  Reinforcement learning from human feedback (RLHF) is a prevalent approach to align AI systems with human values. We propose a novel adaptive preference loss, underpinned by distributionally robust optimization (DRO) Our method is versatile and can be readily adapted to various preference optimization frameworks.
  arXiv  Detail & Related papers  (2024-06-04T20:33:22Z)
• FedMAP: Personalised Federated Learning for Real Large-Scale Healthcare Systems [14.378330471182586]
  Federated learning (FL) promises to enable machine learning across healthcare sites whilst preserving data privacy.<n>We introduce FedMAP, a personalised FL framework that addresses statistical heterogeneity through local Maximum a Posteriori (MAP) estimation.<n>We demonstrate improved performance compared to local training, FedAvg, and several PFL methods across three large-scale clinical datasets.
  arXiv  Detail & Related papers  (2024-05-29T11:28:06Z)
• Unified Preference Optimization: Language Model Alignment Beyond the Preference Frontier [0.5120567378386615]
  We propose a unified approach to aligning large language models (LLMs)<n>Based on a simple decomposition of preference and auxiliary objectives, we allow for tuning LLMs to optimize user and designer preferences.
  arXiv  Detail & Related papers  (2024-05-28T08:35:48Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.