Concept Decomposition for Visual Exploration and Inspiration
- URL: http://arxiv.org/abs/2305.18203v2
- Date: Wed, 31 May 2023 16:04:24 GMT
- Title: Concept Decomposition for Visual Exploration and Inspiration
- Authors: Yael Vinker, Andrey Voynov, Daniel Cohen-Or, Ariel Shamir
- Abstract summary: We propose a method to decompose a visual concept into different visual aspects encoded in a hierarchical tree structure.
We utilize large vision-language models and their rich latent space for concept decomposition and generation.
- Score: 53.06983340652571
- License: http://creativecommons.org/licenses/by-nc-sa/4.0/
- Abstract: A creative idea is often born from transforming, combining, and modifying
ideas from existing visual examples capturing various concepts. However, one
cannot simply copy the concept as a whole, and inspiration is achieved by
examining certain aspects of the concept. Hence, it is often necessary to
separate a concept into different aspects to provide new perspectives. In this
paper, we propose a method to decompose a visual concept, represented as a set
of images, into different visual aspects encoded in a hierarchical tree
structure. We utilize large vision-language models and their rich latent space
for concept decomposition and generation. Each node in the tree represents a
sub-concept using a learned vector embedding injected into the latent space of
a pretrained text-to-image model. We use a set of regularizations to guide the
optimization of the embedding vectors encoded in the nodes to follow the
hierarchical structure of the tree. Our method allows to explore and discover
new concepts derived from the original one. The tree provides the possibility
of endless visual sampling at each node, allowing the user to explore the
hidden sub-concepts of the object of interest. The learned aspects in each node
can be combined within and across trees to create new visual ideas, and can be
used in natural language sentences to apply such aspects to new designs.
Related papers
- CusConcept: Customized Visual Concept Decomposition with Diffusion Models [13.95568624067449]
We propose a two-stage framework, CusConcept, to extract customized visual concept embedding vectors.
In the first stage, CusConcept employs a vocabularies-guided concept decomposition mechanism.
In the second stage, joint concept refinement is performed to enhance the fidelity and quality of generated images.
arXiv Detail & Related papers (2024-10-01T04:41:44Z) - Language-Informed Visual Concept Learning [22.911347501969857]
We train a set of concept encoders to encode the information pertinent to a set of language-informed concept axes.
We then anchor the concept embeddings to a set of text embeddings obtained from a pre-trained Visual Question Answering (VQA) model.
At inference time, the model extracts concept embeddings along various axes from new test images, which can be remixed to generate images with novel compositions of visual concepts.
arXiv Detail & Related papers (2023-12-06T16:24:47Z) - Unsupervised Compositional Concepts Discovery with Text-to-Image
Generative Models [80.75258849913574]
In this paper, we consider the inverse problem -- given a collection of different images, can we discover the generative concepts that represent each image?
We present an unsupervised approach to discover generative concepts from a collection of images, disentangling different art styles in paintings, objects, and lighting from kitchen scenes, and discovering image classes given ImageNet images.
arXiv Detail & Related papers (2023-06-08T17:02:15Z) - The Hidden Language of Diffusion Models [70.03691458189604]
We present Conceptor, a novel method to interpret the internal representation of a textual concept by a diffusion model.
We find surprising visual connections between concepts, that transcend their textual semantics.
We additionally discover concepts that rely on mixtures of exemplars, biases, renowned artistic styles, or a simultaneous fusion of multiple meanings.
arXiv Detail & Related papers (2023-06-01T17:57:08Z) - Visual Superordinate Abstraction for Robust Concept Learning [80.15940996821541]
Concept learning constructs visual representations that are connected to linguistic semantics.
We ascribe the bottleneck to a failure of exploring the intrinsic semantic hierarchy of visual concepts.
We propose a visual superordinate abstraction framework for explicitly modeling semantic-aware visual subspaces.
arXiv Detail & Related papers (2022-05-28T14:27:38Z) - SegDiscover: Visual Concept Discovery via Unsupervised Semantic
Segmentation [29.809900593362844]
SegDiscover is a novel framework that discovers semantically meaningful visual concepts from imagery datasets with complex scenes without supervision.
Our method generates concept primitives from raw images, discovering concepts by clustering in the latent space of a self-supervised pretrained encoder, and concept refinement via neural network smoothing.
arXiv Detail & Related papers (2022-04-22T20:44:42Z) - FALCON: Fast Visual Concept Learning by Integrating Images, Linguistic
descriptions, and Conceptual Relations [99.54048050189971]
We present a framework for learning new visual concepts quickly, guided by multiple naturally occurring data streams.
The learned concepts support downstream applications, such as answering questions by reasoning about unseen images.
We demonstrate the effectiveness of our model on both synthetic and real-world datasets.
arXiv Detail & Related papers (2022-03-30T19:45:00Z) - Toward a Visual Concept Vocabulary for GAN Latent Space [74.12447538049537]
This paper introduces a new method for building open-ended vocabularies of primitive visual concepts represented in a GAN's latent space.
Our approach is built from three components: automatic identification of perceptually salient directions based on their layer selectivity; human annotation of these directions with free-form, compositional natural language descriptions.
Experiments show that concepts learned with our approach are reliable and composable -- generalizing across classes, contexts, and observers.
arXiv Detail & Related papers (2021-10-08T17:58:19Z) - Flexible Compositional Learning of Structured Visual Concepts [17.665938343060112]
We study how people learn different types of visual compositions, using abstract visual forms with rich relational structure.
We find that people can make meaningful compositional generalizations from just a few examples in a variety of scenarios.
Unlike past work examining special cases of compositionality, our work shows how a single computational approach can account for many distinct types of compositional generalization.
arXiv Detail & Related papers (2021-05-20T15:48:05Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.