How to Blend Concepts in Diffusion Models

• URL: http://arxiv.org/abs/2407.14280v2
• Date: Sun, 22 Sep 2024 07:02:35 GMT
• Title: How to Blend Concepts in Diffusion Models
• Authors: Lorenzo Olearo, Giorgio Longari, Simone Melzi, Alessandro Raganato, Rafael Peñaloza,
• Abstract summary: Recent methods exploit multiple latent representations and their connection, making this research question even more entangled. Our goal is to understand how operations in the latent space affect the underlying concepts. Our conclusion is that concept blending through space manipulation is possible, although the best strategy depends on the context of the blend.
• Score: 48.68800153838679
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: For the last decade, there has been a push to use multi-dimensional (latent) spaces to represent concepts; and yet how to manipulate these concepts or reason with them remains largely unclear. Some recent methods exploit multiple latent representations and their connection, making this research question even more entangled. Our goal is to understand how operations in the latent space affect the underlying concepts. To that end, we explore the task of concept blending through diffusion models. Diffusion models are based on a connection between a latent representation of textual prompts and a latent space that enables image reconstruction and generation. This task allows us to try different text-based combination strategies, and evaluate easily through a visual analysis. Our conclusion is that concept blending through space manipulation is possible, although the best strategy depends on the context of the blend.

Related papers

• Scaling Concept With Text-Guided Diffusion Models [53.80799139331966]
  Instead of replacing a concept, can we enhance or suppress the concept itself? We introduce ScalingConcept, a simple yet effective method to scale decomposed concepts up or down in real input without introducing new elements. More importantly, ScalingConcept enables a variety of novel zero-shot applications across image and audio domains.
  arXiv  Detail & Related papers  (2024-10-31T17:09:55Z)
• Implicit Concept Removal of Diffusion Models [92.55152501707995]
  Text-to-image (T2I) diffusion models often inadvertently generate unwanted concepts such as watermarks and unsafe images. We present the Geom-Erasing, a novel concept removal method based on the geometric-driven control.
  arXiv  Detail & Related papers  (2023-10-09T17:13:10Z)
• The Hidden Language of Diffusion Models [70.03691458189604]
  We present Conceptor, a novel method to interpret the internal representation of a textual concept by a diffusion model. We find surprising visual connections between concepts, that transcend their textual semantics. We additionally discover concepts that rely on mixtures of exemplars, biases, renowned artistic styles, or a simultaneous fusion of multiple meanings.
  arXiv  Detail & Related papers  (2023-06-01T17:57:08Z)
• Break-A-Scene: Extracting Multiple Concepts from a Single Image [80.47666266017207]
  We introduce the task of textual scene decomposition. We propose augmenting the input image with masks that indicate the presence of target concepts. We then present a novel two-phase customization process.
  arXiv  Detail & Related papers  (2023-05-25T17:59:04Z)
• When are Post-hoc Conceptual Explanations Identifiable? [18.85180188353977]
  When no human concept labels are available, concept discovery methods search trained embedding spaces for interpretable concepts. We argue that concept discovery should be identifiable, meaning that a number of known concepts can be provably recovered to guarantee reliability of the explanations. Our results highlight the strict conditions under which reliable concept discovery without human labels can be guaranteed.
  arXiv  Detail & Related papers  (2022-06-28T10:21:17Z)
• FALCON: Fast Visual Concept Learning by Integrating Images, Linguistic descriptions, and Conceptual Relations [99.54048050189971]
  We present a framework for learning new visual concepts quickly, guided by multiple naturally occurring data streams. The learned concepts support downstream applications, such as answering questions by reasoning about unseen images. We demonstrate the effectiveness of our model on both synthetic and real-world datasets.
  arXiv  Detail & Related papers  (2022-03-30T19:45:00Z)
• Sparse Subspace Clustering for Concept Discovery (SSCCD) [1.7319807100654885]
  Concepts are key building blocks of higher level human understanding. Local attribution methods do not allow to identify coherent model behavior across samples. We put forward a new definition of concepts as low-dimensional subspaces of hidden feature layers.
  arXiv  Detail & Related papers  (2022-03-11T16:15:48Z)
• Discovering Concepts in Learned Representations using Statistical Inference and Interactive Visualization [0.76146285961466]
  Concept discovery is important for bridging the gap between non-deep learning experts and model end-users. Current approaches include hand-crafting concept datasets and then converting them to latent space directions. In this study, we offer another two approaches to guide user discovery of meaningful concepts, one based on multiple hypothesis testing, and another on interactive visualization.
  arXiv  Detail & Related papers  (2022-02-09T22:29:48Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.