Related papers: I2AM: Interpreting Image-to-Image Latent Diffusion Models via Attribution Maps

I2AM: Interpreting Image-to-Image Latent Diffusion Models via Attribution Maps

URL: http://arxiv.org/abs/2407.12331v1
Date: Wed, 17 Jul 2024 06:15:05 GMT
Title: I2AM: Interpreting Image-to-Image Latent Diffusion Models via Attribution Maps
Authors: Junseo Park, Hyeryung Jang,
Abstract summary: This paper introduces the Image-to-Image Maps I2AM method, which aggregates patch-level cross-attention scores to enhance the interpretability of latent diffusion models. I2AM facilitates detailed image-to-image attribution analysis, enabling observation of how diffusion models prioritize key features over time and head. To further assess our understanding, we introduce a new evaluation metric tailored for reference-based image inpainting tasks.
Score: 8.195126516665914
License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
Abstract: Large-scale diffusion models have made significant advancements in the field of image generation, especially through the use of cross-attention mechanisms that guide image formation based on textual descriptions. While the analysis of text-guided cross-attention in diffusion models has been extensively studied in recent years, its application in image-to-image diffusion models remains underexplored. This paper introduces the Image-to-Image Attribution Maps I2AM method, which aggregates patch-level cross-attention scores to enhance the interpretability of latent diffusion models across time steps, heads, and attention layers. I2AM facilitates detailed image-to-image attribution analysis, enabling observation of how diffusion models prioritize key features over time and head during the image generation process from reference images. Through extensive experiments, we first visualize the attribution maps of both generated and reference images, verifying that critical information from the reference image is effectively incorporated into the generated image, and vice versa. To further assess our understanding, we introduce a new evaluation metric tailored for reference-based image inpainting tasks. This metric, measuring the consistency between the attribution maps of generated and reference images, shows a strong correlation with established performance metrics for inpainting tasks, validating the potential use of I2AM in future research endeavors.