Hybrid SD: Edge-Cloud Collaborative Inference for Stable Diffusion Models

• URL: http://arxiv.org/abs/2408.06646v2
• Date: Wed, 30 Oct 2024 03:41:42 GMT
• Title: Hybrid SD: Edge-Cloud Collaborative Inference for Stable Diffusion Models
• Authors: Chenqian Yan, Songwei Liu, Hongjian Liu, Xurui Peng, Xiaojian Wang, Fangmin Chen, Lean Fu, Xing Mei,
• Abstract summary: We introduce Hybrid SD, a training-free SDMs inference framework for edge-cloud collaborative inference. We show that our compressed models achieve state-of-the-art parameter efficiency (225.8M) on edge devices with competitive image quality. Hybrid SD reduces the cloud cost by 66% with edge-cloud collaborative inference.
• Score: 6.015486729281141
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Stable Diffusion Models (SDMs) have shown remarkable proficiency in image synthesis. However, their broad application is impeded by their large model sizes and intensive computational requirements, which typically require expensive cloud servers for deployment. On the flip side, while there are many compact models tailored for edge devices that can reduce these demands, they often compromise on semantic integrity and visual quality when compared to full-sized SDMs. To bridge this gap, we introduce Hybrid SD, an innovative, training-free SDMs inference framework designed for edge-cloud collaborative inference. Hybrid SD distributes the early steps of the diffusion process to the large models deployed on cloud servers, enhancing semantic planning. Furthermore, small efficient models deployed on edge devices can be integrated for refining visual details in the later stages. Acknowledging the diversity of edge devices with differing computational and storage capacities, we employ structural pruning to the SDMs U-Net and train a lightweight VAE. Empirical evaluations demonstrate that our compressed models achieve state-of-the-art parameter efficiency (225.8M) on edge devices with competitive image quality. Additionally, Hybrid SD reduces the cloud cost by 66% with edge-cloud collaborative inference.

Related papers

• EC-Diff: Fast and High-Quality Edge-Cloud Collaborative Inference for Diffusion Models [57.059991285047296]
  hybrid edge-cloud collaborative framework was recently proposed to realize fast inference and high-quality generation.<n>Excessive cloud denoising prolongs inference time, while insufficient steps cause semantic ambiguity, leading to inconsistency in edge model output.<n>We propose EC-Diff that accelerates cloud inference through gradient-based noise estimation.<n>Our method significantly enhances generation quality compared to edge inference, while achieving up to an average $2times$ speedup in inference compared to cloud inference.
  arXiv  Detail & Related papers  (2025-07-16T07:23:14Z)
• Fast and Cost-effective Speculative Edge-Cloud Decoding with Early Exits [11.398891065175686]
  Large Language Models (LLMs) enable various applications on edge devices such as smartphones, wearables, and embodied robots.<n>LLMs can be deployed on-device, offering a cost-effective solution with reduced latency and improved privacy.<n>We propose a fast and cost-effective speculative edge-cloud decoding framework with a large target model on the server and a small draft model on the device.
  arXiv  Detail & Related papers  (2025-05-27T14:55:16Z)
• Edge-First Language Model Inference: Models, Metrics, and Tradeoffs [0.7980273012483663]
  This work examines the interplay between edge and cloud deployments, starting from detailed benchmarking of SLM capabilities on single edge devices.<n>We identify scenarios where edge inference offers comparable performance with lower costs, and others where cloud fallback becomes essential due to limits in scalability or model capacity.<n>Rather than proposing a one-size-fits-all solution, we present platform-level comparisons and design insights for building efficient, adaptive LM inference systems.
  arXiv  Detail & Related papers  (2025-05-22T10:43:00Z)
• Efficient Hybrid Language Model Compression through Group-Aware SSM Pruning [54.584665518334035]
  Hybrid architectures that combine Attention and State Space Models (SSMs) achieve state-of-the-art accuracy and runtime performance. Recent work has demonstrated that applying compression and distillation to Attention-only models yields smaller, more accurate models at a fraction of the training cost. We introduce a novel group-aware pruning strategy that preserves the structural integrity of SSM blocks and their sequence modeling capabilities.
  arXiv  Detail & Related papers  (2025-04-15T17:26:29Z)
• A Hybrid Swarm Intelligence Approach for Optimizing Multimodal Large Language Models Deployment in Edge-Cloud-based Federated Learning Environments [10.72166883797356]
  Federated Learning (FL), Multimodal Large Language Models (MLLMs), and edge-cloud computing enables distributed and real-time data processing. We propose a novel hybrid framework wherein MLLMs are deployed on edge devices equipped with sufficient resources and battery life, while the majority of training occurs in the cloud. Our experimental results show that the proposed method significantly improves system performance, achieving an accuracy of 92%, reducing communication cost by 30%, and enhancing client participation.
  arXiv  Detail & Related papers  (2025-02-04T03:03:24Z)
• Dual-Model Distillation for Efficient Action Classification with Hybrid Edge-Cloud Solution [1.8029479474051309]
  We design a hybrid edge-cloud solution that leverages the efficiency of smaller models for local processing while deferring to larger, more accurate cloud-based models when necessary. Specifically, we propose a novel unsupervised data generation method, Dual-Model Distillation (DMD), to train a lightweight switcher model that can predict when the edge model's output is uncertain. Experimental results on the action classification task show that our framework not only requires less computational overhead, but also improves accuracy compared to using a large model alone.
  arXiv  Detail & Related papers  (2024-10-16T02:06:27Z)
• Fine-Tuning and Deploying Large Language Models Over Edges: Issues and Approaches [64.42735183056062]
  Large language models (LLMs) have transitioned from specialized models to versatile foundation models. LLMs exhibit impressive zero-shot ability, however, require fine-tuning on local datasets and significant resources for deployment.
  arXiv  Detail & Related papers  (2024-08-20T09:42:17Z)
• IDF-CR: Iterative Diffusion Process for Divide-and-Conquer Cloud Removal in Remote-sensing Images [55.40601468843028]
  We present an iterative diffusion process for cloud removal (IDF-CR) IDF-CR is divided into two-stage models that address pixel space and latent space. In the latent space stage, the diffusion model transforms low-quality cloud removal into high-quality clean output.
  arXiv  Detail & Related papers  (2024-03-18T15:23:48Z)
• Towards Robust and Efficient Cloud-Edge Elastic Model Adaptation via Selective Entropy Distillation [56.79064699832383]
  We establish a Cloud-Edge Elastic Model Adaptation (CEMA) paradigm in which the edge models only need to perform forward propagation. In our CEMA, to reduce the communication burden, we devise two criteria to exclude unnecessary samples from uploading to the cloud.
  arXiv  Detail & Related papers  (2024-02-27T08:47:19Z)
• Cloud-Device Collaborative Learning for Multimodal Large Language Models [24.65882336700547]
  We introduce a Cloud-Device Collaborative Continual Adaptation framework to enhance the performance of compressed, device-deployed MLLMs. Our framework is structured into three key components: a device-to-cloud uplink for efficient data transmission, cloud-based knowledge adaptation, and an optimized cloud-to-device downlink for model deployment.
  arXiv  Detail & Related papers  (2023-12-26T18:46:14Z)
• ECLM: Efficient Edge-Cloud Collaborative Learning with Continuous Environment Adaptation [47.35179593006409]
  We propose ECLM, an edge-cloud collaborative learning framework for rapid model adaptation for dynamic edge environments. We show that ECLM significantly improves model performance (e.g., 18.89% accuracy increase) and resource efficiency (e.g. 7.12x communication cost reduction) in adapting models to dynamic edge environments.
  arXiv  Detail & Related papers  (2023-11-18T14:10:09Z)
• Auto-Split: A General Framework of Collaborative Edge-Cloud AI [49.750972428032355]
  This paper describes the techniques and engineering practice behind Auto-Split, an edge-cloud collaborative prototype of Huawei Cloud. To the best of our knowledge, there is no existing industry product that provides the capability of Deep Neural Network (DNN) splitting.
  arXiv  Detail & Related papers  (2021-08-30T08:03:29Z)
• Cocktail: Leveraging Ensemble Learning for Optimized Model Serving in Public Cloud [9.149566952446058]
  We proposeCocktail, a costeffective ensembling-based model serving framework. A prototype implementation ofCocktailon the AWS EC2 platform and exhaustive evalua-tions using a variety of workloads demonstrate thatCocktailcan reduce deployment cost by 1.45x.
  arXiv  Detail & Related papers  (2021-06-09T19:23:58Z)
• Device-Cloud Collaborative Learning for Recommendation [50.01289274123047]
  We propose a novel MetaPatch learning approach on the device side to efficiently achieve "thousands of people with thousands of models" given a centralized cloud model. With billions of updated personalized device models, we propose a "model-over-models" distillation algorithm, namely MoMoDistill, to update the centralized cloud model.
  arXiv  Detail & Related papers  (2021-04-14T05:06:59Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.