ElasticDiT: Efficient Diffusion Transformers via Elastic Architecture and Sparse Attention for High-Resolution Image Generation on Mobile Devices

• URL: http://arxiv.org/abs/2605.15684v1
• Date: Fri, 15 May 2026 07:13:27 GMT
• Title: ElasticDiT: Efficient Diffusion Transformers via Elastic Architecture and Sparse Attention for High-Resolution Image Generation on Mobile Devices
• Abstract summary: Diffusion Transformer (DiT) architecture is the state-of-the-art paradigm for high-fidelity image generation.<n>However, deploying these models on resource-constrained mobile devices entails prohibitive computational and memory overhead.<n>We introduce ElasticDiT, which achieves this dynamic trade-off by adjusting spatial compression ratios and DiT block depths.
• Score: 19.789749822094617
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: The Diffusion Transformer (DiT) architecture is the state-of-the-art paradigm for high-fidelity image generation, underpinning models like Stable Diffusion-3 and FLUX.1. However, deploying these models on resource-constrained mobile devices entails prohibitive computational and memory overhead. While efficiency-driven approaches like Linear-DiT and static pruning alleviate bottlenecks, they often incur quality degradation. Unlike cloud environments, mobile constraints require a single-model paradigm that dynamically balances fidelity and latency. We introduce ElasticDiT, which achieves this dynamic trade-off by adjusting spatial compression ratios and DiT block depths. By integrating Shift Sparse Block Attention (SSBA) and a Tiny DWT-Distilled VAE (T-DVAE), ElasticDiT reduces inference latency and memory footprint while maintaining image quality. Experiments confirm that ElasticDiT effectively covers a wide range of fidelity-latency trade-offs within a single set of parameters. By jointly adjusting compression and depth, a single ElasticDiT model can be reconfigured on-the-fly to outperform task-specific baselines. Specifically, our flex lite variant achieves an HPS of 32.87, surpassing the Flux model, while maintaining competitive quality at 84.16 percent average sparsity through SSBA. Furthermore, the plug-and-play T-DVAE provides SD3-level reconstruction with only 1/8x the computational cost of standard VAEs, and Flow-GRPO boosts semantic alignment (GenEval: 66.93 to 73.62). These results demonstrate that ElasticDiT offers a versatile, hardware-adaptive solution that eliminates the need for multiple specialized models, providing a promising path for future high-resolution image generation on mobile devices.

Related papers

• Elastoformer: Enabling Dynamic Adaptivity via Elastic Model Transformation [0.0]
  Elastoformer is a framework that transforms conventional neural networks (NN) into Elastic NN capable of real-time elastic inference.<n>Our framework achieves up to 85% reduction in FLOPs, 50% reduction in latency and 76% reduction in memory overhead.
  arXiv  Detail & Related papers  (2026-09-09T10:52:21Z)
• Diffusion Image Generation with Explicit Modeling of Data Manifold Geometry [64.61931485161833]
  Image generative models aim to sample data points from the underlying data manifold.<n>We propose the Data Manifold-aware Image diffusioN moDel (MIND)<n>MIND integrates discrete patch tokenization into the score function of a continuous diffusion model.
  arXiv  Detail & Related papers  (2026-05-25T08:43:14Z)
• Elastic Diffusion Transformer [32.62353162897611]
  Diffusion Transformers (DiT) have demonstrated remarkable generative capabilities but remain highly computationally expensive.<n>We propose textbfElastic Diffusion Transformer (E-DiT), an adaptive acceleration framework for DiT.
  arXiv  Detail & Related papers  (2026-02-15T05:19:17Z)
• SnapGen++: Unleashing Diffusion Transformers for Efficient High-Fidelity Image Generation on Edge Devices [72.0937240883345]
  Recent advances in diffusion transformers (DiTs) have set new standards in image generation, yet remain impractical for on-device deployment.<n>We present an efficient DiT framework tailored for mobile and edge devices that achieves transformer-level generation quality under strict resource constraints.
  arXiv  Detail & Related papers  (2026-01-13T07:46:46Z)
• Pluggable Pruning with Contiguous Layer Distillation for Diffusion Transformers [10.251154683874033]
  Diffusion Transformers (DiTs) have shown exceptional performance in image generation, yet their large parameter counts incur high computational costs.<n>We propose Pluggable Pruning with Contiguous Layer Distillation (PPCL), a flexible structured pruning framework specifically designed for DiT architectures.
  arXiv  Detail & Related papers  (2025-11-20T08:53:07Z)
• MOBIUS: Big-to-Mobile Universal Instance Segmentation via Multi-modal Bottleneck Fusion and Calibrated Decoder Pruning [91.90342432541138]
  Scaling up model size and training data has advanced foundation models for instance-level perception.<n>High computational cost limits adoption on resource-constrained platforms.<n>We introduce a new benchmark for efficient segmentation on both high-performance computing platforms and mobile devices.
  arXiv  Detail & Related papers  (2025-10-16T18:00:00Z)
• Layer- and Timestep-Adaptive Differentiable Token Compression Ratios for Efficient Diffusion Transformers [55.87192133758051]
  Diffusion Transformers (DiTs) have achieved state-of-the-art (SOTA) image generation quality but suffer from high latency and memory inefficiency.<n>We propose DiffCR, a dynamic DiT inference framework with differentiable compression ratios.
  arXiv  Detail & Related papers  (2024-12-22T02:04:17Z)
• Dynamic Diffusion Transformer [67.13876021157887]
  Diffusion Transformer (DiT) has demonstrated superior performance but suffers from substantial computational costs. We propose Dynamic Diffusion Transformer (DyDiT), an architecture that dynamically adjusts its computation along both timestep and spatial dimensions during generation. With 3% additional fine-tuning, our method reduces the FLOPs of DiT-XL by 51%, accelerates generation by 1.73, and achieves a competitive FID score of 2.07 on ImageNet.
  arXiv  Detail & Related papers  (2024-10-04T14:14:28Z)
• Q-DiT: Accurate Post-Training Quantization for Diffusion Transformers [45.762142897697366]
  Post-Training Quantization (PTQ) emerges as a promising solution, enabling model compression and accelerated inference for pretrained models. Research on DiT quantization remains sparse, and existing PTQ frameworks tend to suffer from biased quantization, leading to notable performance degradation. We propose Q-DiT, a novel approach that seamlessly integrates two key techniques: automatic quantization granularity allocation to handle the significant variance of weights and activations across input channels, and sample-wise dynamic activation quantization to adaptively capture activation changes across both timesteps and samples.
  arXiv  Detail & Related papers  (2024-06-25T07:57:27Z)
• Dynamic Pre-training: Towards Efficient and Scalable All-in-One Image Restoration [100.54419875604721]
  All-in-one image restoration tackles different types of degradations with a unified model instead of having task-specific, non-generic models for each degradation. We propose DyNet, a dynamic family of networks designed in an encoder-decoder style for all-in-one image restoration tasks. Our DyNet can seamlessly switch between its bulkier and lightweight variants, thereby offering flexibility for efficient model deployment.
  arXiv  Detail & Related papers  (2024-04-02T17:58:49Z)
• Global Vision Transformer Pruning with Hessian-Aware Saliency [93.33895899995224]
  This work challenges the common design philosophy of the Vision Transformer (ViT) model with uniform dimension across all the stacked blocks in a model stage. We derive a novel Hessian-based structural pruning criteria comparable across all layers and structures, with latency-aware regularization for direct latency reduction. Performing iterative pruning on the DeiT-Base model leads to a new architecture family called NViT (Novel ViT), with a novel parameter that utilizes parameters more efficiently.
  arXiv  Detail & Related papers  (2021-10-10T18:04:59Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.