Denoising Diffusion Generative Models Secretly Calculate Attentions

• URL: http://arxiv.org/abs/2609.00885v1
• Date: Tue, 01 Sep 2026 08:17:57 GMT
• Title: Denoising Diffusion Generative Models Secretly Calculate Attentions
• Abstract summary: We show that diffusion models inherently use an attention mechanism very similar to that of transformers.<n>We also show similarities in basic functional principle of auto-encoders and attention-based models.<n>As an example, we can reformulate the diffusion framework to reduce the lengthy training process and computation-intensive image generation.
• Score: 1.016315800665405
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Denoising diffusion models are the dominant architecture for image generation, whereas most natural language generation and modeling are primarily handled by well-known transformer architectures employing attention mechanism. Here, we show that diffusion models also inherently use an attention mechanism very similar to that of transformers. Therefore, attention emerges as a universal machine learning principle, based on a general training objective. We also show similarities in basic functional principle of auto-encoders and attention-based models. These equivalences allows us to interchange these designs based on practical requirements. As an example, we can reformulate the diffusion framework to reduce the lengthy training process and computation-intensive image generation. Using this approach, a simplified algorithm is proposed for image generation which is based on attention mechanism. Results show that the attention-based implementation achieves comparable performance with significantly less effort and computational resources.

Related papers

• Beyond Self-Attention: Sub-Quadratic Vision Transformers for Fast Image Captioning [2.33474141211381]
  The proposed model focuses on improving computational efficiency by restructuring the vision transformer architecture.<n>The model is evaluated on the Flickr 30K dataset, demonstrating competitive and significant improvement over existing works.
  arXiv  Detail & Related papers  (2026-06-07T16:57:24Z)
• Foundations and Models in Modern Computer Vision: Key Building Blocks in Landmark Architectures [34.542592986038265]
  This report analyzes the evolution of key design patterns in computer vision by examining six influential papers.<n>We review ResNet, which introduced residual connections to overcome the vanishing gradient problem.<n>We examine the Vision Transformer (ViT), which established a new paradigm by applying the Transformer architecture to sequences of image patches.
  arXiv  Detail & Related papers  (2025-07-31T09:08:11Z)
• RestoreVAR: Visual Autoregressive Generation for All-in-One Image Restoration [51.77917733024544]
  latent diffusion models (LDMs) have improved the perceptual quality of All-in-One image Restoration (AiOR) methods.<n>LDMs suffer from slow inference due to their iterative denoising process, rendering them impractical for time-sensitive applications.<n>Visual autoregressive modeling ( VAR) performs scale-space autoregression and achieves comparable performance to that of state-of-the-art diffusion transformers.
  arXiv  Detail & Related papers  (2025-05-23T15:52:26Z)
• Can We Achieve Efficient Diffusion without Self-Attention? Distilling Self-Attention into Convolutions [94.21989689001848]
  We propose (Delta)ConvFusion to replace conventional self-attention modules with Pyramid Convolution Blocks ((Delta)ConvBlocks)<n>By distilling attention patterns into localized convolutional operations while keeping other components frozen, (Delta)ConvFusion achieves performance comparable to transformer-based counterparts while reducing computational cost by 6929$times$ and surpassing LinFusion by 5.42$times$ in efficiency--all without compromising generative fidelity.
  arXiv  Detail & Related papers  (2025-04-30T03:57:28Z)
• HRR: Hierarchical Retrospection Refinement for Generated Image Detection [16.958383381415445]
  We propose a diffusion model-based generative image detection framework termed Hierarchical Retrospection Refinement(HRR)<n>The HRR framework consistently delivers significant performance improvements, outperforming state-of-the-art methods in generated image detection task.
  arXiv  Detail & Related papers  (2025-02-25T05:13:44Z)
• Graph-Aware Isomorphic Attention for Adaptive Dynamics in Transformers [0.0]
  We reformulate the Transformer's attention mechanism as a graph operation.<n>We introduce Sparse GIN-Attention, a fine-tuning approach that employs sparse GINs.
  arXiv  Detail & Related papers  (2025-01-04T22:30:21Z)
• Oscillation Inversion: Understand the structure of Large Flow Model through the Lens of Inversion Method [60.88467353578118]
  We show that a fixed-point-inspired iterative approach to invert real-world images does not achieve convergence, instead oscillating between distinct clusters. We introduce a simple and fast distribution transfer technique that facilitates image enhancement, stroke-based recoloring, as well as visual prompt-guided image editing.
  arXiv  Detail & Related papers  (2024-11-17T17:45:37Z)
• A Primal-Dual Framework for Transformers and Neural Networks [52.814467832108875]
  Self-attention is key to the remarkable success of transformers in sequence modeling tasks. We show that the self-attention corresponds to the support vector expansion derived from a support vector regression problem. We propose two new attentions: Batch Normalized Attention (Attention-BN) and Attention with Scaled Head (Attention-SH)
  arXiv  Detail & Related papers  (2024-06-19T19:11:22Z)
• Active Generation for Image Classification [45.93535669217115]
  We propose to address the efficiency of image generation by focusing on the specific needs and characteristics of the model. With a central tenet of active learning, our method, named ActGen, takes a training-aware approach to image generation.
  arXiv  Detail & Related papers  (2024-03-11T08:45:31Z)
• Denoising Autoregressive Representation Learning [13.185567468951628]
  Our method, DARL, employs a decoder-only Transformer to predict image patches autoregressively. We show that the learned representation can be improved by using tailored noise schedules and longer training in larger models.
  arXiv  Detail & Related papers  (2024-03-08T10:19:00Z)
• Diffusion Models Without Attention [110.5623058129782]
  Diffusion State Space Model (DiffuSSM) is an architecture that supplants attention mechanisms with a more scalable state space model backbone. Our focus on FLOP-efficient architectures in diffusion training marks a significant step forward.
  arXiv  Detail & Related papers  (2023-11-30T05:15:35Z)
• Counterfactual Generative Networks [59.080843365828756]
  We propose to decompose the image generation process into independent causal mechanisms that we train without direct supervision. By exploiting appropriate inductive biases, these mechanisms disentangle object shape, object texture, and background. We show that the counterfactual images can improve out-of-distribution with a marginal drop in performance on the original classification task.
  arXiv  Detail & Related papers  (2021-01-15T10:23:12Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.