HybridStitch: Pixel and Timestep Level Model Stitching for Diffusion Acceleration

• URL: http://arxiv.org/abs/2603.07815v1
• Date: Sun, 08 Mar 2026 21:34:50 GMT
• Title: HybridStitch: Pixel and Timestep Level Model Stitching for Diffusion Acceleration
• Abstract summary: We propose HybridStitch, a new T2I generation paradigm that treats generation like editing.<n>HybridStitch separates the entire image into two regions: one that is relatively easy to render, and another that is more complex and therefore requires refinement by the large model.<n> HybridStitch achieves 1.83$times$ speedup on Stable Diffusion 3, which is faster than all existing mixture of model methods.
• Score: 6.3289777476721625
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Diffusion models have demonstrated a remarkable ability in Text-to-Image (T2I) generation applications. Despite the advanced generation output, they suffer from heavy computation overhead, especially for large models that contain tens of billions of parameters. Prior work has illustrated that replacing part of the denoising steps with a smaller model still maintains the generation quality. However, these methods only focus on saving computation for some timesteps, ignoring the difference in compute demand within one timestep. In this work, we propose HybridStitch, a new T2I generation paradigm that treats generation like editing. Specifically, we introduce a hybrid stage that jointly incorporates both the large model and the small model. HybridStitch separates the entire image into two regions: one that is relatively easy to render, enabling an early transition to the smaller model, and another that is more complex and therefore requires refinement by the large model. HybridStitch employs the small model to construct a coarse sketch while exploiting the large model to edit and refine the complex regions. According to our evaluation, HybridStitch achieves 1.83$\times$ speedup on Stable Diffusion 3, which is faster than all existing mixture of model methods.

Related papers

• Explorative Modeling: Unlocking a Third Pretraining Axis and End-to-End Generation [89.51408170909548]
  We introduce Explorative Modeling, a new paradigm that factors the training loop, exploring K candidate matches between model generations and data, and training on the best.<n>First, increasing exploration adds a third pretraining axis beyond parameters and data for existing generative models.<n>Second, XMs enable end-to-end reconstructive generative modeling, matching diffusion on control tasks with 16-256x fewer inference steps.
  arXiv  Detail & Related papers  (2026-07-29T18:25:17Z)
• Expressivity-Efficiency Tradeoffs for Hybrid Sequence Models [50.45915413315706]
  We prove the existence of fundamental limitations for non-hybrid models.<n>We construct hybrid models of small size and working memory that provably solve these tasks.<n>We additionally demonstrate that hybrid models exhibit stronger length generalization and out-of-distribution robustness than non-hybrids.
  arXiv  Detail & Related papers  (2026-03-09T19:20:01Z)
• Navigating the Accuracy-Size Trade-Off with Flexible Model Merging [15.497612580389479]
  We propose FlexMerge, a novel data-free model merging framework.<n>It flexibly generates merged models of varying sizes, spanning the full spectrum from a single merged model to retaining all fine-tuned models.<n>Using FlexMerge, we systematically characterize the accuracy-size trade-off of different algorithms.
  arXiv  Detail & Related papers  (2025-05-29T07:50:32Z)
• DMM: Building a Versatile Image Generation Model via Distillation-Based Model Merging [32.97010533998294]
  We introduce a style-promptable image generation pipeline which can accurately generate arbitrary-style images under the control of style vectors.<n>Based on this design, we propose the score distillation based model merging paradigm (DMM), compressing multiple models into a single versatile T2I model.<n>Our experiments demonstrate that DMM can compactly reorganize the knowledge from multiple teacher models and achieve controllable arbitrary-style generation.
  arXiv  Detail & Related papers  (2025-04-16T15:09:45Z)
• SnapGen: Taming High-Resolution Text-to-Image Models for Mobile Devices with Efficient Architectures and Training [77.681908636429]
  Text-to-image (T2I) models face several limitations, including large model sizes, slow, and low-quality generation on mobile devices.<n>This paper aims to develop an extremely small and fast T2I model that generates high-resolution and high-quality images on mobile platforms.
  arXiv  Detail & Related papers  (2024-12-12T18:59:53Z)
• M-VAR: Decoupled Scale-wise Autoregressive Modeling for High-Quality Image Generation [39.97174784206976]
  We show that this scale-wise autoregressive framework can be effectively decoupled into textitintra-scale modeling We apply linear-complexity mechanisms like Mamba to substantially reduce computational overhead. Experiments demonstrate that our method outperforms existing models in both image quality and generation speed.
  arXiv  Detail & Related papers  (2024-11-15T18:54:42Z)
• LinFusion: 1 GPU, 1 Minute, 16K Image [71.44735417472043]
  We introduce a low-rank approximation of a wide spectrum of popular linear token mixers. We find that the distilled model, termed LinFusion, achieves performance on par with or superior to the original SD. Experiments on SD-v1.5, SD-v2.1, and SD-XL demonstrate that LinFusion enables satisfactory and efficient zero-shot cross-resolution generation.
  arXiv  Detail & Related papers  (2024-09-03T17:54:39Z)
• MaxFusion: Plug&Play Multi-Modal Generation in Text-to-Image Diffusion Models [34.611309081801345]
  Large diffusion-based Text-to-Image (T2I) models have shown impressive generative powers for text-to-image generation. In this paper, we propose a novel strategy to scale a generative model across new tasks with minimal compute.
  arXiv  Detail & Related papers  (2024-04-15T17:55:56Z)
• A-SDM: Accelerating Stable Diffusion through Redundancy Removal and Performance Optimization [54.113083217869516]
  In this work, we first explore the computational redundancy part of the network. We then prune the redundancy blocks of the model and maintain the network performance. Thirdly, we propose a global-regional interactive (GRI) attention to speed up the computationally intensive attention part.
  arXiv  Detail & Related papers  (2023-12-24T15:37:47Z)
• Consistency Models [89.68380014789861]
  We propose a new family of models that generate high quality samples by directly mapping noise to data. They support fast one-step generation by design, while still allowing multistep sampling to trade compute for sample quality. They also support zero-shot data editing, such as image inpainting, colorization, and super-resolution, without requiring explicit training.
  arXiv  Detail & Related papers  (2023-03-02T18:30:16Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.