FIRM: Federated In-client Regularized Multi-objective Alignment for Large Language Models

• URL: http://arxiv.org/abs/2511.16992v1
• Date: Fri, 21 Nov 2025 06:59:28 GMT
• Title: FIRM: Federated In-client Regularized Multi-objective Alignment for Large Language Models
• Authors: Fatemeh, Nourzad, Amirhossein Roknilamouki, Eylem Ekici, Jia, Liu, Ness B. Shroff,
• Abstract summary: We introduce FIRM (Federated In-client Regularized Multi-objective alignment), a novel algorithm that achieves both client disagreement drift mitigation and communication efficiency.<n>We show that FIRM leads to smoother training dynamics, reduced client disagreement drift, and improved reward trade-offs compared to baselines.
• Score: 19.266791462067655
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Aligning Large Language Models (LLMs) with human values often involves balancing multiple, conflicting objectives such as helpfulness and harmlessness. Training these models is computationally intensive, and centralizing the process raises significant data privacy concerns. Federated Learning (FL) offers a compelling alternative, but existing Federated Multi-Objective Optimization (FMOO) methods face severe communication bottlenecks as their reliance on transmitting multiple gradients to a server is unscalable for large models. We introduce FIRM (Federated In-client Regularized Multi-objective alignment), a novel algorithm that achieves both client disagreement drift mitigation and communication efficiency. In FIRM, each client locally solves a regularized multi-objective optimization problem. By directly mitigating client disagreement drift through in-client regularization, our method eliminates the need for the multi-gradient transmissions common in prior works. Consequently, clients need only to transmit a single set of adapted parameters, maintaining high communication efficiency. We prove that our algorithm converges to Pareto-stationary points and, to our knowledge, provide the first finite-time convergence guarantees for this federated multi-objective alignment setting. Empirically, we show that FIRM leads to smoother training dynamics, reduced client disagreement drift, and improved reward trade-offs compared to baselines. We further propose a method to incorporate a preference over the objectives and report empirical Pareto plots, demonstrating that FIRM can smoothly adapt trade-offs between objectives in response to specified preferences.

Related papers

• FeDecider: An LLM-Based Framework for Federated Cross-Domain Recommendation [75.50721642765994]
  Large language model (LLM)-based recommendation models have demonstrated impressive performance.<n>We propose an LLM-based framework for Federated cross-domain recommendation, FeDecider.<n>Extensive experiments across diverse datasets validate the effectiveness of our proposed FeDecider.
  arXiv  Detail & Related papers  (2026-02-17T21:42:28Z)
• Federated Multi-Task Clustering [44.73672172790804]
  This paper proposes a novel framework named Federated Multi-Task Clustering (i.e.,FMTC)<n>It is composed of two main components: client-side personalized clustering module and server-side tensorial correlation module.<n>We derive an efficient, privacy-preserving distributed algorithm based on the Alternating Direction Method of Multipliers.
  arXiv  Detail & Related papers  (2025-12-28T12:02:32Z)
• Federated Multi-Objective Learning with Controlled Pareto Frontiers [10.818539304970935]
  Federated learning (FL) is a widely adopted paradigm for privacy-preserving model training.<n>Existing methods such as federated multi-objective learning (FMOL) attempts to import multi-objective optimisation (MOO) into FL.<n>We introduce Conically-Regularised FMOL (CR-FMOL), the first federated MOO framework that enforces client-wise optimality through a preference-cone constraint.
  arXiv  Detail & Related papers  (2025-08-07T14:15:12Z)
• Federated Multimodal Learning with Dual Adapters and Selective Pruning for Communication and Computational Efficiency [6.0860246234554545]
  Federated Learning (FL) enables collaborative learning across distributed clients while preserving data privacy.<n>We propose a novel framework designed to tackle these challenges by introducing a dual-adapter approach.
  arXiv  Detail & Related papers  (2025-03-10T17:21:33Z)
• UC-MOA: Utility-Conditioned Multi-Objective Alignment for Distributional Pareto-Optimality [52.49062565901046]
  Reinforcement Learning from Human Feedback (RLHF) has become a cornerstone for aligning large language models with human values.<n>Existing approaches struggle to capture the multi-dimensional, distributional nuances of human preferences.<n>We introduce Utility-Conditioned Multi-Objective Alignment (UC-MOA), a novel framework that overcomes these limitations.
  arXiv  Detail & Related papers  (2025-03-10T09:52:42Z)
• Client-Centric Federated Adaptive Optimization [78.30827455292827]
  Federated Learning (FL) is a distributed learning paradigm where clients collaboratively train a model while keeping their own data private.<n>We propose Federated-Centric Adaptive Optimization, which is a class of novel federated optimization approaches.
  arXiv  Detail & Related papers  (2025-01-17T04:00:50Z)
• Few-shot Steerable Alignment: Adapting Rewards and LLM Policies with Neural Processes [50.544186914115045]
  Large language models (LLMs) are increasingly embedded in everyday applications.<n> Ensuring their alignment with the diverse preferences of individual users has become a critical challenge.<n>We present a novel framework for few-shot steerable alignment.
  arXiv  Detail & Related papers  (2024-12-18T16:14:59Z)
• Federated Communication-Efficient Multi-Objective Optimization [27.492821176616815]
  We propose FedCMOO, a novel federated multiobjective communication (F) model.<n>CMOO does not scale with the number of Fed objectives, as each communication aggregated to the central server.<n>We demonstrate the superiority of our proposed method baseline approaches.
  arXiv  Detail & Related papers  (2024-10-21T18:09:22Z)
• Re-Weighted Softmax Cross-Entropy to Control Forgetting in Federated Learning [14.196701066823499]
  In Federated Learning, a global model is learned by aggregating model updates computed at a set of independent client nodes. We show that individual client models experience a catastrophic forgetting with respect to data from other clients. We propose an efficient approach that modifies the cross-entropy objective on a per-client basis by re-weighting the softmax logits prior to computing the loss.
  arXiv  Detail & Related papers  (2023-04-11T14:51:55Z)
• Improving Privacy-Preserving Vertical Federated Learning by Efficient Communication with ADMM [62.62684911017472]
  Federated learning (FL) enables devices to jointly train shared models while keeping the training data local for privacy purposes. We introduce a VFL framework with multiple heads (VIM), which takes the separate contribution of each client into account. VIM achieves significantly higher performance and faster convergence compared with the state-of-the-art.
  arXiv  Detail & Related papers  (2022-07-20T23:14:33Z)
• Federated Multi-Target Domain Adaptation [99.93375364579484]
  Federated learning methods enable us to train machine learning models on distributed user data while preserving its privacy. We consider a more practical scenario where the distributed client data is unlabeled, and a centralized labeled dataset is available on the server. We propose an effective DualAdapt method to address the new challenges.
  arXiv  Detail & Related papers  (2021-08-17T17:53:05Z)
• Low-Latency Federated Learning over Wireless Channels with Differential Privacy [142.5983499872664]
  In federated learning (FL), model training is distributed over clients and local models are aggregated by a central server. In this paper, we aim to minimize FL training delay over wireless channels, constrained by overall training performance as well as each client's differential privacy (DP) requirement.
  arXiv  Detail & Related papers  (2021-06-20T13:51:18Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.