Heterogeneous Decentralized Diffusion Models

• URL: http://arxiv.org/abs/2603.06741v1
• Date: Fri, 06 Mar 2026 08:43:43 GMT
• Title: Heterogeneous Decentralized Diffusion Models
• Abstract summary: Training frontier-scale diffusion models often requires substantial computational resources concentrated in tightly coupled clusters.<n>We present an efficient framework that reduces resource requirements while supporting heterogeneous training objectives.
• Score: 11.120199309935435
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Training frontier-scale diffusion models often requires substantial computational resources concentrated in tightly coupled clusters, limiting participation to well-resourced institutions. While Decentralized Diffusion Models (DDM) enable training multiple experts in isolation, existing approaches require 1176 GPU-days and homogeneous training objectives across all experts. We present an efficient framework that reduces resource requirements while supporting heterogeneous training objectives. Our approach combines three contributions: (1) a heterogeneous decentralized training paradigm that allows experts to use different objectives (DDPM and Flow Matching), unified at inference time via a deterministic schedule-aware conversion into a common velocity space without retraining; (2) pretrained checkpoint conversion from ImageNet-DDPM to Flow Matching objectives, accelerating convergence and enabling initialization without objective-specific pretraining; and (3) PixArt-alpha's efficient AdaLN-Single architecture, reducing parameters while maintaining quality. Experiments on LAION-Aesthetics show that, relative to the training scale reported for prior DDM work, our approach reduces compute from 1176 to 72 GPU-days (16x) and data from 158M to 11M (14x). Under aligned inference settings, our heterogeneous 2DDPM:6FM configuration achieves better FID (11.88 vs. 12.45) and higher intra-prompt diversity (LPIPS 0.631 vs. 0.617) than the homogeneous 8FM baseline. By eliminating synchronization requirements and enabling mixed DDPM/FM objectives, our framework lowers infrastructure requirements for decentralized generative model training.

Related papers

• DiffusionOPD: A Unified Perspective of On-Policy Distillation in Diffusion Models [55.01951088768769]
  DiffusionOPD is a new multi-task training paradigm for diffusion models based on Online Policy Distillation (OPD)<n>We show that DiffusionOPD consistently surpasses both multi-reward RL and cascade RL baselines in training efficiency and final performance.
  arXiv  Detail & Related papers  (2026-05-14T16:49:09Z)
• $R_\ ext{dm}$: Re-conceptualizing Distribution Matching as a Reward for Diffusion Distillation [9.105357939499683]
  Diffusion models achieve state-of-the-art generative performance but are bottlenecked by their slow, iterative sampling process.<n>Recent approaches have attempted to break this ceiling by integrating Reinforcement Learning (RL)<n>We propose a novel paradigm by re-conceptualizing distribution matching as a reward, denoted as $R_textdm$.
  arXiv  Detail & Related papers  (2026-03-30T14:01:31Z)
• Decentralized Diffusion Models [53.89995588977048]
  Large-scale AI model training divides work across thousands of GPU, then synchronizes gradients across them at each step.<n>This incurs a significant network burden that only centralized, monolithic clusters can support.<n>We propose Decentralized Diffusion Models, a scalable framework for distributing diffusion model training across independent clusters.
  arXiv  Detail & Related papers  (2025-01-09T18:59:56Z)
• FedDM: Enhancing Communication Efficiency and Handling Data Heterogeneity in Federated Diffusion Models [9.646249924704021]
  FedDM is a novel training framework designed for the federated training of diffusion models. We propose a suite of training algorithms that leverage the U-Net architecture as the backbone for our diffusion models. We evaluate our algorithms on FashionMNIST (28x28 resolution), CIFAR-10 (32x32 resolution), and CelebA (64x64 resolution) for DDPMs, and LSUN Church Outdoors (256x256 resolution) for LDMs.
  arXiv  Detail & Related papers  (2024-07-20T02:54:41Z)
• Adaptive Training Meets Progressive Scaling: Elevating Efficiency in Diffusion Models [52.1809084559048]
  We propose a novel two-stage divide-and-conquer training strategy termed TDC Training.<n>It groups timesteps based on task similarity and difficulty, assigning highly customized denoising models to each group, thereby enhancing the performance of diffusion models.<n>While two-stage training avoids the need to train each model separately, the total training cost is even lower than training a single unified denoising model.
  arXiv  Detail & Related papers  (2023-12-20T03:32:58Z)
• Towards More Suitable Personalization in Federated Learning via Decentralized Partial Model Training [67.67045085186797]
  Almost all existing systems have to face large communication burdens if the central FL server fails. It personalizes the "right" in the deep models by alternately updating the shared and personal parameters. To further promote the shared parameters aggregation process, we propose DFed integrating the local Sharpness Miniization.
  arXiv  Detail & Related papers  (2023-05-24T13:52:18Z)
• Simplifying Distributed Neural Network Training on Massive Graphs: Randomized Partitions Improve Model Aggregation [23.018715954992352]
  We present a simplified framework for distributed GNN training that does not rely on the aforementioned costly operations. Specifically, our framework assembles independent trainers, each of which asynchronously learns a local model on locally-available parts of the training graph. In experiments on social and e-commerce networks with up to 1.3 billion edges, our proposed RandomTMA and SuperTMA approaches achieve state-of-the-art performance and 2.31x speedup compared to the fastest baseline.
  arXiv  Detail & Related papers  (2023-05-17T01:49:44Z)
• Asynchronous Decentralized Federated Lifelong Learning for Landmark Localization in Medical Imaging [13.985136866888379]
  We propose a novel method of asynchronous decentralized federated lifelong learning (ADFLL) ADFLL can train on multiple tasks simultaneously without need for a central node or synchronous training. We demonstrate excellent performance on the brain tumor segmentation dataset for localizing the left ventricle.
  arXiv  Detail & Related papers  (2023-03-12T23:51:51Z)
• Decentralized Training of Foundation Models in Heterogeneous Environments [77.47261769795992]
  Training foundation models, such as GPT-3 and PaLM, can be extremely expensive. We present the first study of training large foundation models with model parallelism in a decentralized regime over a heterogeneous network.
  arXiv  Detail & Related papers  (2022-06-02T20:19:51Z)
• Adaptive Serverless Learning [114.36410688552579]
  We propose a novel adaptive decentralized training approach, which can compute the learning rate from data dynamically. Our theoretical results reveal that the proposed algorithm can achieve linear speedup with respect to the number of workers. To reduce the communication-efficient overhead, we further propose a communication-efficient adaptive decentralized training approach.
  arXiv  Detail & Related papers  (2020-08-24T13:23:02Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.