Linguistic Binding in Diffusion Models: Enhancing Attribute
Correspondence through Attention Map Alignment
- URL: http://arxiv.org/abs/2306.08877v3
- Date: Tue, 23 Jan 2024 20:55:48 GMT
- Title: Linguistic Binding in Diffusion Models: Enhancing Attribute
Correspondence through Attention Map Alignment
- Authors: Royi Rassin, Eran Hirsch, Daniel Glickman, Shauli Ravfogel, Yoav
Goldberg, Gal Chechik
- Abstract summary: Text-conditioned image generation models often generate incorrect associations between entities and their visual attributes.
We propose SynGen, an approach which first syntactically analyses the prompt to identify entities and their modifier.
Human evaluation on three datasets, including one new and challenging set, demonstrate significant improvements of SynGen compared with current state of the art methods.
- Score: 87.1732801732059
- License: http://creativecommons.org/licenses/by-sa/4.0/
- Abstract: Text-conditioned image generation models often generate incorrect
associations between entities and their visual attributes. This reflects an
impaired mapping between linguistic binding of entities and modifiers in the
prompt and visual binding of the corresponding elements in the generated image.
As one notable example, a query like "a pink sunflower and a yellow flamingo"
may incorrectly produce an image of a yellow sunflower and a pink flamingo. To
remedy this issue, we propose SynGen, an approach which first syntactically
analyses the prompt to identify entities and their modifiers, and then uses a
novel loss function that encourages the cross-attention maps to agree with the
linguistic binding reflected by the syntax. Specifically, we encourage large
overlap between attention maps of entities and their modifiers, and small
overlap with other entities and modifier words. The loss is optimized during
inference, without retraining or fine-tuning the model. Human evaluation on
three datasets, including one new and challenging set, demonstrate significant
improvements of SynGen compared with current state of the art methods. This
work highlights how making use of sentence structure during inference can
efficiently and substantially improve the faithfulness of text-to-image
generation.
Related papers
- Distractors-Immune Representation Learning with Cross-modal Contrastive Regularization for Change Captioning [71.14084801851381]
Change captioning aims to succinctly describe the semantic change between a pair of similar images.
Most existing methods directly capture the difference between them, which risk obtaining error-prone difference features.
We propose a distractors-immune representation learning network that correlates the corresponding channels of two image representations.
arXiv Detail & Related papers (2024-07-16T13:00:33Z) - Object-Attribute Binding in Text-to-Image Generation: Evaluation and Control [58.37323932401379]
Current diffusion models create images given a text prompt as input but struggle to correctly bind attributes mentioned in the text to the right objects in the image.
We propose focused cross-attention (FCA) that controls the visual attention maps by syntactic constraints found in the input sentence.
We show substantial improvements in T2I generation and especially its attribute-object binding on several datasets.
arXiv Detail & Related papers (2024-04-21T20:26:46Z) - Relation Rectification in Diffusion Model [64.84686527988809]
We introduce a novel task termed Relation Rectification, aiming to refine the model to accurately represent a given relationship it initially fails to generate.
We propose an innovative solution utilizing a Heterogeneous Graph Convolutional Network (HGCN)
The lightweight HGCN adjusts the text embeddings generated by the text encoder, ensuring the accurate reflection of the textual relation in the embedding space.
arXiv Detail & Related papers (2024-03-29T15:54:36Z) - Contrastive Prompts Improve Disentanglement in Text-to-Image Diffusion
Models [68.47333676663312]
We show a simple modification of classifier-free guidance can help disentangle image factors in text-to-image models.
The key idea of our method, Contrastive Guidance, is to characterize an intended factor with two prompts that differ in minimal tokens.
We illustrate whose benefits in three scenarios: (1) to guide domain-specific diffusion models trained on an object class, (2) to gain continuous, rig-like controls for text-to-image generation, and (3) to improve the performance of zero-shot image editors.
arXiv Detail & Related papers (2024-02-21T03:01:17Z) - Predicated Diffusion: Predicate Logic-Based Attention Guidance for Text-to-Image Diffusion Models [9.514940899499752]
Diffusion models have achieved remarkable results in generating high-quality, diverse, and creative images.
However, when it comes to text-based image generation, they often fail to capture the intended meaning presented in the text.
We propose Predicated Diffusion, a unified framework to express users' intentions.
arXiv Detail & Related papers (2023-10-03T15:45:50Z) - Dynamic Prompt Learning: Addressing Cross-Attention Leakage for
Text-Based Image Editing [23.00202969969574]
We propose Dynamic Prompt Learning (DPL) to force cross-attention maps to focus on correct noun words in the text prompt.
We show improved prompt editing results for Word-Swap, Prompt Refinement, and Attention Re-weighting, especially for complex multi-object scenes.
arXiv Detail & Related papers (2023-09-27T13:55:57Z) - Consensus Graph Representation Learning for Better Grounded Image
Captioning [48.208119537050166]
We propose the Consensus Rraph Representation Learning framework (CGRL) for grounded image captioning.
We validate the effectiveness of our model, with a significant decline in object hallucination (-9% CHAIRi) on the Flickr30k Entities dataset.
arXiv Detail & Related papers (2021-12-02T04:17:01Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.