Breaking the Memory Wall for Heterogeneous Federated Learning via Progressive Training

• URL: http://arxiv.org/abs/2404.13349v2
• Date: Wed, 21 May 2025 06:38:20 GMT
• Title: Breaking the Memory Wall for Heterogeneous Federated Learning via Progressive Training
• Abstract summary: ProFL is a new framework that effectively addresses the memory constraints in FL.<n>It partitions the model into blocks based on its original architecture and trains each block in a progressive fashion.<n>ProFL effectively reduces the peak memory footprint by up to 57.4% and improves model accuracy by up to 82.4%.
• Score: 17.885679388567887
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: This paper presents ProFL, a new framework that effectively addresses the memory constraints in FL. Rather than updating the full model during local training, ProFL partitions the model into blocks based on its original architecture and trains each block in a progressive fashion. It first trains the front blocks and safely freezes them after convergence. Training of the next block is then triggered. This process progressively grows the model to be trained until the training of the full model is completed. In this way, the peak memory footprint is effectively reduced for feasible deployment on heterogeneous devices. In order to preserve the feature representation of each block, the training process is divided into two stages: model shrinking and model growing. During the model shrinking stage, we meticulously design corresponding output modules to assist each block in learning the expected feature representation and obtain the initialization model parameters. Subsequently, the obtained output modules and initialization model parameters are utilized in the corresponding model growing stage, which progressively trains the full model. Additionally, a novel metric from the scalar perspective is proposed to assess the learning status of each block, enabling us to securely freeze it after convergence and initiate the training of the next one. Finally, we theoretically prove the convergence of ProFL and conduct extensive experiments on representative models and datasets to evaluate its effectiveness. The results demonstrate that ProFL effectively reduces the peak memory footprint by up to 57.4% and improves model accuracy by up to 82.4%.

Related papers

• Chain-of-Models Pre-Training: Rethinking Training Acceleration of Vision Foundation Models [16.42169130504086]
  Chain-of-Models Pre-Training (CoM-PT) is a performance-lossless training acceleration method for vision foundation models.<n>CoM-PT is designed to accelerate the training pipeline at the model family level, scaling efficiently as the model family expands.<n>It is extensively validated across 45 datasets spanning zero-shot and fine-tuning tasks.
  arXiv  Detail & Related papers  (2026-04-14T07:26:23Z)
• MERGETUNE: Continued fine-tuning of vision-language models [77.8627788911249]
  Fine-tuning vision-language models (VLMs) often leads to catastrophic forgetting of pretrained knowledge.<n>We introduce a novel paradigm, continued fine-tuning (CFT), which seeks to recover pretrained knowledge after a zero-shot model has already been adapted.
  arXiv  Detail & Related papers  (2026-01-15T15:15:53Z)
• Model Inversion with Layer-Specific Modeling and Alignment for Data-Free Continual Learning [19.12792297140574]
  Continual learning aims to incrementally train a model on a sequence of tasks while retaining performance on prior ones.<n> storing and replaying data is often infeasible due to privacy or security constraints.<n>We propose Per-layer Model Inversion (PMI), inspired by faster convergence in single-layer optimization.
  arXiv  Detail & Related papers  (2025-10-30T09:58:48Z)
• Grokking ExPLAIND: Unifying Model, Data, and Training Attribution to Study Model Behavior [25.975757048963413]
  Post-hoc interpretability methods typically attribute a model's behavior to its components, data, or training trajectory in isolation.<n>We present ExPLAIND, a unified framework that integrates all three perspectives.
  arXiv  Detail & Related papers  (2025-05-26T14:53:11Z)
• IDEA Prune: An Integrated Enlarge-and-Prune Pipeline in Generative Language Model Pretraining [50.53912352342753]
  We propose an integrated enlarge-and-prune pipeline, which combines enlarge model training, pruning, and recovery. We conduct experiments on compressing 2.8B models to 1.3B with up to 2T tokens in pretraining. It demonstrates the integrated approach not only provides insights into the token efficiency of enlarged model pretraining but also achieves superior performance of pruned models.
  arXiv  Detail & Related papers  (2025-03-07T20:35:31Z)
• LESA: Learnable LLM Layer Scaling-Up [57.0510934286449]
  Training Large Language Models (LLMs) from scratch requires immense computational resources, making it prohibitively expensive. Model scaling-up offers a promising solution by leveraging the parameters of smaller models to create larger ones. We propose textbfLESA, a novel learnable method for depth scaling-up.
  arXiv  Detail & Related papers  (2025-02-19T14:58:48Z)
• Accelerate High-Quality Diffusion Models with Inner Loop Feedback [50.00066451431194]
  Inner Loop Feedback (ILF) is a novel approach to accelerate diffusion models' inference. ILF trains a lightweight module to predict future features in the denoising process. ILF achieves strong performance for both class-to-image generation with diffusion transformer (DiT) and text-to-image generation with DiT-based PixArt-alpha and PixArt-sigma.
  arXiv  Detail & Related papers  (2025-01-22T18:59:58Z)
• Transferable Post-training via Inverse Value Learning [83.75002867411263]
  We propose modeling changes at the logits level during post-training using a separate neural network (i.e., the value network) After training this network on a small base model using demonstrations, this network can be seamlessly integrated with other pre-trained models during inference. We demonstrate that the resulting value network has broad transferability across pre-trained models of different parameter sizes.
  arXiv  Detail & Related papers  (2024-10-28T13:48:43Z)
• FedRepOpt: Gradient Re-parametrized Optimizers in Federated Learning [17.642826635563946]
  Federated learning (FL) has emerged as a privacy-preserving method for training machine learning models in a distributed manner on edge devices. As the model's size increases, the frequency of gradient updates on edge devices decreases, leading to suboptimal training outcomes. We propose FedRepOpt, a gradient re- parameterized for FL, to address this issue.
  arXiv  Detail & Related papers  (2024-09-24T09:17:08Z)
• OmniBal: Towards Fast Instruct-tuning for Vision-Language Models via Omniverse Computation Balance [65.48009829137824]
  Large-scale 3D parallel training on vision-language instruct-tuning models leads to an imbalanced computation load across different devices. We rebalanced the computational loads from data, model, and memory perspectives to address this issue. Our method's efficacy and generalizability were further demonstrated across various models and datasets.
  arXiv  Detail & Related papers  (2024-07-30T12:02:58Z)
• SHERL: Synthesizing High Accuracy and Efficient Memory for Resource-Limited Transfer Learning [63.93193829913252]
  We propose an innovative METL strategy called SHERL for resource-limited scenarios. In the early route, intermediate outputs are consolidated via an anti-redundancy operation. In the late route, utilizing minimal late pre-trained layers could alleviate the peak demand on memory overhead.
  arXiv  Detail & Related papers  (2024-07-10T10:22:35Z)
• Save It All: Enabling Full Parameter Tuning for Federated Large Language Models via Cycle Block Gradient Descent [15.463595798992621]
  Large language models (LLMs) have revolutionized the deep learning paradigm, yielding impressive results across a wide array of tasks. Existing solutions make the unrealistic assumption that the entire model is exchanged for training. We introduce a novel method for the efficient training and fine-tuning of LLMs in FL, with minimal resource consumption.
  arXiv  Detail & Related papers  (2024-06-17T03:49:44Z)
• Causal Estimation of Memorisation Profiles [58.20086589761273]
  Understanding memorisation in language models has practical and societal implications. Memorisation is the causal effect of training with an instance on the model's ability to predict that instance. This paper proposes a new, principled, and efficient method to estimate memorisation based on the difference-in-differences design from econometrics.
  arXiv  Detail & Related papers  (2024-06-06T17:59:09Z)
• Minusformer: Improving Time Series Forecasting by Progressively Learning Residuals [14.741951369068877]
  We find that ubiquitous time series (TS) forecasting models are prone to severe overfitting. We introduce a dual-stream and subtraction mechanism, which is a deep Boosting ensemble learning method. The proposed method outperform existing state-of-the-art methods, yielding an average performance improvement of 11.9% across various datasets.
  arXiv  Detail & Related papers  (2024-02-04T03:54:31Z)
• A-SDM: Accelerating Stable Diffusion through Redundancy Removal and Performance Optimization [54.113083217869516]
  In this work, we first explore the computational redundancy part of the network. We then prune the redundancy blocks of the model and maintain the network performance. Thirdly, we propose a global-regional interactive (GRI) attention to speed up the computationally intensive attention part.
  arXiv  Detail & Related papers  (2023-12-24T15:37:47Z)
• The Languini Kitchen: Enabling Language Modelling Research at Different Scales of Compute [66.84421705029624]
  We introduce an experimental protocol that enables model comparisons based on equivalent compute, measured in accelerator hours. We pre-process an existing large, diverse, and high-quality dataset of books that surpasses existing academic benchmarks in quality, diversity, and document length. This work also provides two baseline models: a feed-forward model derived from the GPT-2 architecture and a recurrent model in the form of a novel LSTM with ten-fold throughput.
  arXiv  Detail & Related papers  (2023-09-20T10:31:17Z)
• Latent Iterative Refinement for Modular Source Separation [44.78689915209527]
  Traditional source separation approaches train deep neural network models end-to-end with all the data available at once. We argue that we can significantly increase resource efficiency during both training and inference stages.
  arXiv  Detail & Related papers  (2022-11-22T00:02:57Z)
• FedOBD: Opportunistic Block Dropout for Efficiently Training Large-scale Neural Networks through Federated Learning [18.357577491590686]
  We propose the Federated Opportunistic Block Dropout (FedOBD) approach to train large-scale neural networks. FedOBD decomposes large-scale models into semantic blocks so that FL participants can opportunistically upload quantized blocks. Experiments show that FedOBD reduces the overall communication overhead by more than 88% compared to the best performing baseline approach.
  arXiv  Detail & Related papers  (2022-08-10T06:36:49Z)
• FOSTER: Feature Boosting and Compression for Class-Incremental Learning [52.603520403933985]
  Deep neural networks suffer from catastrophic forgetting when learning new categories. We propose a novel two-stage learning paradigm FOSTER, empowering the model to learn new categories adaptively.
  arXiv  Detail & Related papers  (2022-04-10T11:38:33Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.