Related papers: Selectively Informative Description can Reduce Undesired Embedding Entanglements in Text-to-Image Personalization

Selectively Informative Description can Reduce Undesired Embedding Entanglements in Text-to-Image Personalization

URL: http://arxiv.org/abs/2403.15330v1
Date: Fri, 22 Mar 2024 16:35:38 GMT
Title: Selectively Informative Description can Reduce Undesired Embedding Entanglements in Text-to-Image Personalization
Authors: Jimyeong Kim, Jungwon Park, Wonjong Rhee,
Abstract summary: We propose SID(Selectively Informative Description), a text description strategy that deviates from the prevalent approach of only characterizing the subject's class identification. We present comprehensive experimental results along with analyses of cross-attention maps, subject-alignment, non-subject-disentanglement, and text-alignment.
Score: 5.141049647900161
License: http://creativecommons.org/licenses/by/4.0/
Abstract: In text-to-image personalization, a timely and crucial challenge is the tendency of generated images overfitting to the biases present in the reference images. We initiate our study with a comprehensive categorization of the biases into background, nearby-object, tied-object, substance (in style re-contextualization), and pose biases. These biases manifest in the generated images due to their entanglement into the subject embedding. This undesired embedding entanglement not only results in the reflection of biases from the reference images into the generated images but also notably diminishes the alignment of the generated images with the given generation prompt. To address this challenge, we propose SID~(Selectively Informative Description), a text description strategy that deviates from the prevalent approach of only characterizing the subject's class identification. SID is generated utilizing multimodal GPT-4 and can be seamlessly integrated into optimization-based models. We present comprehensive experimental results along with analyses of cross-attention maps, subject-alignment, non-subject-disentanglement, and text-alignment.