Federated Customization of Large Models: Approaches, Experiments, and Insights

• URL: http://arxiv.org/abs/2601.00526v1
• Date: Fri, 02 Jan 2026 01:45:52 GMT
• Title: Federated Customization of Large Models: Approaches, Experiments, and Insights
• Abstract summary: We review several popular large model customization techniques, including full fine-tuning, efficient fine-tuning, prompt engineering, prefix-tuning, knowledge distillation, and retrieval-augmented generation.<n>We conduct experiments on federated prefix-tuning, which, to the best of our knowledge, is the first trial to apply prefix-tuning in the federated learning setting.
• Score: 50.70334261194185
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: In this article, we explore federated customization of large models and highlight the key challenges it poses within the federated learning framework. We review several popular large model customization techniques, including full fine-tuning, efficient fine-tuning, prompt engineering, prefix-tuning, knowledge distillation, and retrieval-augmented generation. Then, we discuss how these techniques can be implemented within the federated learning framework. Moreover, we conduct experiments on federated prefix-tuning, which, to the best of our knowledge, is the first trial to apply prefix-tuning in the federated learning setting. The conducted experiments validate its feasibility with performance close to centralized approaches. Further comparison with three other federated customization methods demonstrated its competitive performance, satisfactory efficiency, and consistent robustness.

Related papers

• Towards Physics of Multimodal Pretraining: Knowledge Flow, Modality Synergy, Early Unification, and Recipes [57.55143519213221]
  We provide empirical clarity through a systematic exploration of multimodal pretraining.<n>Our experiments on both synthetic and large-scale real-world datasets yield four key insights into the physics of multimodal pretraining.
  arXiv  Detail & Related papers  (2026-08-05T16:09:25Z)
• High-Performance Few-Shot Segmentation with Foundation Models: An Empirical Study [64.06777376676513]
  We develop a few-shot segmentation (FSS) framework based on foundation models. To be specific, we propose a simple approach to extract implicit knowledge from foundation models to construct coarse correspondence. Experiments on two widely used datasets demonstrate the effectiveness of our approach.
  arXiv  Detail & Related papers  (2024-09-10T08:04:11Z)
• Skywork-MoE: A Deep Dive into Training Techniques for Mixture-of-Experts Language Models [57.582219834039506]
  We introduce the training methodologies implemented in the development of Skywork-MoE, a high-performance mixture-of-experts (MoE) large language model (LLM) with 146 billion parameters and 16 experts. It is based on the pre-existing dense checkpoints of our Skywork-13B model.
  arXiv  Detail & Related papers  (2024-06-03T03:58:41Z)
• Exploring Federated Unlearning: Review, Comparison, and Insights [101.64910079905566]
  federated unlearning enables the selective removal of data from models trained in federated systems.<n>This paper examines existing federated unlearning approaches, examining their algorithmic efficiency, impact on model accuracy, and effectiveness in preserving privacy.<n>We propose the OpenFederatedUnlearning framework, a unified benchmark for evaluating federated unlearning methods.
  arXiv  Detail & Related papers  (2023-10-30T01:34:33Z)
• UNIDEAL: Curriculum Knowledge Distillation Federated Learning [17.817181326740698]
  Federated Learning (FL) has emerged as a promising approach to enable collaborative learning among multiple clients. In this paper, we present UNI, a novel FL algorithm specifically designed to tackle the challenges of cross-domain scenarios. Our results demonstrate that UNI achieves superior performance in terms of both model accuracy and communication efficiency.
  arXiv  Detail & Related papers  (2023-09-16T11:30:29Z)
• Personalizing Federated Learning with Over-the-Air Computations [84.8089761800994]
  Federated edge learning is a promising technology to deploy intelligence at the edge of wireless networks in a privacy-preserving manner. Under such a setting, multiple clients collaboratively train a global generic model under the coordination of an edge server. This paper presents a distributed training paradigm that employs analog over-the-air computation to address the communication bottleneck.
  arXiv  Detail & Related papers  (2023-02-24T08:41:19Z)
• An Empirical Study of Personalized Federated Learning [8.641606056228675]
  Federated learning is a distributed machine learning approach in which a single server and multiple clients collaboratively build machine learning models without sharing datasets on clients. To cope with this issue, numerous federated learning methods aim at personalized federated learning and build optimized models for clients. It is unclear which personalized federate learning method achieves the best performance and how much progress can be made by using these methods instead of standard (i.e., non-personalized) federated learning.
  arXiv  Detail & Related papers  (2022-06-27T11:08:16Z)
• Fine-tuning is Fine in Federated Learning [3.222802562733787]
  We study the performance of federated learning algorithms and their variants in an framework. This multi-criterion approach naturally models the high-dimensional, many-tuned nature of federated learning.
  arXiv  Detail & Related papers  (2021-08-16T18:59:24Z)
• Learning From Multiple Experts: Self-paced Knowledge Distillation for Long-tailed Classification [106.08067870620218]
  We propose a self-paced knowledge distillation framework, termed Learning From Multiple Experts (LFME) We refer to these models as 'Experts', and the proposed LFME framework aggregates the knowledge from multiple 'Experts' to learn a unified student model. We conduct extensive experiments and demonstrate that our method is able to achieve superior performances compared to state-of-the-art methods.
  arXiv  Detail & Related papers  (2020-01-06T12:57:36Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.