ACME: Adaptive Customization of Large Models via Distributed Systems

• URL: http://arxiv.org/abs/2507.14802v1
• Date: Sun, 20 Jul 2025 03:30:24 GMT
• Title: ACME: Adaptive Customization of Large Models via Distributed Systems
• Authors: Ziming Dai, Chao Qiu, Fei Gao, Yunfeng Zhao, Xiaofei Wang,
• Abstract summary: We propose ACME, an adaptive customization approach of Transformer-based large models via distributed systems.<n>ACME achieves cost-efficient models under model size constraints.<n>Average accuracy improves by 10 percent compared to the baseline.
• Score: 7.358399967930416
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Pre-trained Transformer-based large models have revolutionized personal virtual assistants, but their deployment in cloud environments faces challenges related to data privacy and response latency. Deploying large models closer to the data and users has become a key research area to address these issues. However, applying these models directly often entails significant difficulties, such as model mismatching, resource constraints, and energy inefficiency. Automated design of customized models is necessary, but it faces three key challenges, namely, the high cost of centralized model customization, imbalanced performance from user heterogeneity, and suboptimal performance from data heterogeneity. In this paper, we propose ACME, an adaptive customization approach of Transformer-based large models via distributed systems. To avoid the low cost-efficiency of centralized methods, ACME employs a bidirectional single-loop distributed system to progressively achieve fine-grained collaborative model customization. In order to better match user heterogeneity, it begins by customizing the backbone generation and identifying the Pareto Front under model size constraints to ensure optimal resource utilization. Subsequently, it performs header generation and refines the model using data distribution-based personalized architecture aggregation to match data heterogeneity. Evaluation on different datasets shows that ACME achieves cost-efficient models under model size constraints. Compared to centralized systems, data transmission volume is reduced to 6 percent. Additionally, the average accuracy improves by 10 percent compared to the baseline, with the trade-off metrics increasing by nearly 30 percent.

Related papers

• Boosting Generalization Performance in Model-Heterogeneous Federated Learning Using Variational Transposed Convolution [0.27309692684728615]
  Federated learning (FL) is a pioneering machine learning paradigm that enables distributed clients to process local data effectively.<n>Traditional model-homogeneous approaches mainly involve debiasing the local training procedures with regularization or dynamically adjusting client weights in aggregation.<n>We propose a model-heterogeneous FL framework that can improve clients' generalization performance over unseen data without model aggregation.
  arXiv  Detail & Related papers  (2025-08-03T08:55:18Z)
• Efficient Federated Learning with Heterogeneous Data and Adaptive Dropout [62.73150122809138]
  Federated Learning (FL) is a promising distributed machine learning approach that enables collaborative training of a global model using multiple edge devices.<n>We propose the FedDHAD FL framework, which comes with two novel methods: Dynamic Heterogeneous model aggregation (FedDH) and Adaptive Dropout (FedAD)<n>The combination of these two methods makes FedDHAD significantly outperform state-of-the-art solutions in terms of accuracy (up to 6.7% higher), efficiency (up to 2.02 times faster), and cost (up to 15.0% smaller)
  arXiv  Detail & Related papers  (2025-07-14T16:19:00Z)
• Not All Clients Are Equal: Personalized Federated Learning on Heterogeneous Multi-Modal Clients [52.14230635007546]
  Foundation models have shown remarkable capabilities across diverse multi-modal tasks, but their centralized training raises privacy concerns and induces high transmission costs.<n>For the growing demand for personalizing AI models for different user purposes, personalized federated learning (PFL) has emerged.<n>PFL allows each client to leverage the knowledge of other clients for further adaptation to individual user preferences, again without the need to share data.
  arXiv  Detail & Related papers  (2025-05-20T09:17:07Z)
• Communication-Efficient and Personalized Federated Foundation Model Fine-Tuning via Tri-Matrix Adaptation [47.82423317739088]
  This paper introduces communication-efficient federated LoRA adaption (CE-LoRA), a method that employs a tri-factorization low-rank adaptation approach with personalized model parameter aggregation.<n>Experiments on various LLM and VLM fine-tuning tasks demonstrate that CE-LoRA not only significantly reduces communication overhead but also improves performance under not independently and identically distributed data conditions.
  arXiv  Detail & Related papers  (2025-03-31T09:18:42Z)
• Optimizing Sequential Recommendation Models with Scaling Laws and Approximate Entropy [104.48511402784763]
  Performance Law for SR models aims to theoretically investigate and model the relationship between model performance and data quality.<n>We propose Approximate Entropy (ApEn) to assess data quality, presenting a more nuanced approach compared to traditional data quantity metrics.
  arXiv  Detail & Related papers  (2024-11-30T10:56:30Z)
• Selective Knowledge Sharing for Personalized Federated Learning Under Capacity Heterogeneity [12.333226301343029]
  Pa3dFL is a novel framework designed to enhance local model performance by decoupling and selectively sharing knowledge among capacity-heterogeneous models. We conduct extensive experiments on three datasets to evaluate the effectiveness of Pa3dFL.
  arXiv  Detail & Related papers  (2024-05-31T02:59:25Z)
• Stable Training of Probabilistic Models Using the Leave-One-Out Maximum Log-Likelihood Objective [0.7373617024876725]
  Kernel density estimation (KDE) based models are popular choices for this task, but they fail to adapt to data regions with varying densities. An adaptive KDE model is employed to circumvent this, where each kernel in the model has an individual bandwidth. A modified expectation-maximization algorithm is employed to accelerate the optimization speed reliably.
  arXiv  Detail & Related papers  (2023-10-05T14:08:42Z)
• Towards Personalized Federated Learning via Heterogeneous Model Reassembly [84.44268421053043]
  pFedHR is a framework that leverages heterogeneous model reassembly to achieve personalized federated learning. pFedHR dynamically generates diverse personalized models in an automated manner.
  arXiv  Detail & Related papers  (2023-08-16T19:36:01Z)
• OSOA: One-Shot Online Adaptation of Deep Generative Models for Lossless Compression [49.10945855716001]
  We propose a novel setting that starts from a pretrained deep generative model and compresses the data batches while adapting the model with a dynamical system for only one epoch. Experimental results show that vanilla OSOA can save significant time versus training bespoke models and space versus using one model for all targets.
  arXiv  Detail & Related papers  (2021-11-02T15:18:25Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.