Large-Small Model Collaborative Framework for Federated Continual Learning

• URL: http://arxiv.org/abs/2508.09489v1
• Date: Wed, 13 Aug 2025 04:49:50 GMT
• Title: Large-Small Model Collaborative Framework for Federated Continual Learning
• Authors: Hao Yu, Xin Yang, Boyang Fan, Xuemei Cao, Hanlin Gu, Lixin Fan, Qiang Yang,
• Abstract summary: Continual learning (CL) for Foundation Models (FMs) is an essential yet underexplored challenge.<n>We propose the first collaborative framework in Federated Continual Learning (FCL), where lightweight local models act as a dynamic bridge.<n>Two novel components are also included: Small Model Continual Fine-tuning is for preventing small models from temporal forgetting; One-by-One Distillation performs personalized fusion of heterogeneous local knowledge on the server.
• Score: 20.05022827987955
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: Continual learning (CL) for Foundation Models (FMs) is an essential yet underexplored challenge, especially in Federated Continual Learning (FCL), where each client learns from a private, evolving task stream under strict data and communication constraints. Despite their powerful generalization abilities, FMs often exhibit suboptimal performance on local downstream tasks, as they are unable to utilize private local data. Furthermore, enabling FMs to learn new tasks without forgetting prior knowledge is inherently a challenging problem, primarily due to their immense parameter count and high model complexity. In contrast, small models can be trained locally under resource-constrained conditions and benefit from more mature CL techniques. To bridge the gap between small models and FMs, we propose the first collaborative framework in FCL, where lightweight local models act as a dynamic bridge, continually adapting to new tasks while enhancing the utility of the large model. Two novel components are also included: Small Model Continual Fine-tuning is for preventing small models from temporal forgetting; One-by-One Distillation performs personalized fusion of heterogeneous local knowledge on the server. Experimental results demonstrate its superior performance, even when clients utilize heterogeneous small models.

Related papers

• EoS-FM: Can an Ensemble of Specialist Models act as a Generalist Feature Extractor? [8.178030486012437]
  We present an Ensemble-of-Specialists framework for building Remote Sensing Foundation Models (RSFMs)<n>Our method decomposes the training process into lightweight, task-specific ConvNeXtV2 specialists that can be frozen and reused.<n>Our framework sets a new direction for building scalable and efficient RSFMs.
  arXiv  Detail & Related papers  (2025-11-26T15:52:56Z)
• SVTime: Small Time Series Forecasting Models Informed by "Physics" of Large Vision Model Forecasters [86.38433605933515]
  Time series AI is crucial for analyzing dynamic web content.<n>Given their energy-intensive training, inference, and hardware demands, using large models as a one-fits-all solution raises serious concerns about carbon footprint and sustainability.<n>This paper introduces SVTime, a novel Small model inspired by large Vision model (LVM) forecasters for long-term Time series forecasting (LTSF)
  arXiv  Detail & Related papers  (2025-10-10T18:42:23Z)
• GeFL: Model-Agnostic Federated Learning with Generative Models [3.4546761246181696]
  Federated learning (FL) is a distributed training paradigm that enables collaborative learning across clients without sharing local data, thereby preserving privacy.<n>We propose Generative Model-Aided Federated Learning (GeFL), a framework that enables cross-client knowledge sharing via a generative model trained in a federated manner.
  arXiv  Detail & Related papers  (2024-12-24T14:39:47Z)
• Specialized Foundation Models Struggle to Beat Supervised Baselines [60.23386520331143]
  We look at three modalities -- genomics, satellite imaging, and time series -- with multiple recent FMs and compare them to a standard supervised learning workflow.<n>We find that it is consistently possible to train simple supervised models that match or even outperform the latest foundation models.
  arXiv  Detail & Related papers  (2024-11-05T04:10:59Z)
• A Survey on Efficient Federated Learning Methods for Foundation Model Training [62.473245910234304]
  Federated Learning (FL) has become an established technique to facilitate privacy-preserving collaborative training across a multitude of clients. In the wake of Foundation Models (FM), the reality is different for many deep learning applications. We discuss the benefits and drawbacks of parameter-efficient fine-tuning (PEFT) for FL applications.
  arXiv  Detail & Related papers  (2024-01-09T10:22:23Z)
• Efficient Personalized Federated Learning via Sparse Model-Adaptation [47.088124462925684]
  Federated Learning (FL) aims to train machine learning models for multiple clients without sharing their own private data. We propose pFedGate for efficient personalized FL by adaptively and efficiently learning sparse local models. We show that pFedGate achieves superior global accuracy, individual accuracy and efficiency simultaneously over state-of-the-art methods.
  arXiv  Detail & Related papers  (2023-05-04T12:21:34Z)
• Exploring Parameter-Efficient Fine-Tuning to Enable Foundation Models in Federated Learning [12.839398408791778]
  Federated learning (FL) has emerged as a promising paradigm for enabling the collaborative training of models without centralized access to the raw data on local devices.<n>Recent state-of-the-art pre-trained models are getting more capable but also have more parameters, known as the "Foundation Models"<n>Can we find a solution to enable those strong and readily available pre-trained models in FL to achieve excellent performance while simultaneously reducing the communication burden?<n>Specifically, we systemically evaluate the performance of FedPEFT across a variety of client stability, data distribution, and differential privacy settings.
  arXiv  Detail & Related papers  (2022-10-04T16:08:54Z)
• No One Left Behind: Inclusive Federated Learning over Heterogeneous Devices [79.16481453598266]
  We propose InclusiveFL, a client-inclusive federated learning method to handle this problem. The core idea of InclusiveFL is to assign models of different sizes to clients with different computing capabilities. We also propose an effective method to share the knowledge among multiple local models with different sizes.
  arXiv  Detail & Related papers  (2022-02-16T13:03:27Z)
• Federated Mutual Learning [65.46254760557073]
  Federated Mutual Leaning (FML) allows clients training a generalized model collaboratively and a personalized model independently. The experiments show that FML can achieve better performance than alternatives in typical Federated learning setting.
  arXiv  Detail & Related papers  (2020-06-27T09:35:03Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.