RealCustom++: Representing Images as Real Textual Word for Real-Time Customization

• URL: http://arxiv.org/abs/2408.09744v3
• Date: Sun, 26 Oct 2025 08:14:10 GMT
• Title: RealCustom++: Representing Images as Real Textual Word for Real-Time Customization
• Authors: Zhendong Mao, Mengqi Huang, Fei Ding, Mingcong Liu, Qian He, Yongdong Zhang,
• Abstract summary: Given a text and an image of a specific subject, text-to-image customization aims to generate new images that align with both the text and the subject's appearance.<n>Existing works follow the pseudo-word paradigm, which represents the subject as a non-existent pseudo word and combines it with other text to generate images.<n>We propose RealCustom++, a novel real-word paradigm that represents the subject with a non-conflicting real word to firstly generate a coherent guidance image and corresponding subject mask.
• Score: 56.30487886962172
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Given a text and an image of a specific subject, text-to-image customization aims to generate new images that align with both the text and the subject's appearance. Existing works follow the pseudo-word paradigm, which represents the subject as a non-existent pseudo word and combines it with other text to generate images. However, the pseudo word causes semantic conflict from its different learning objective and entanglement from overlapping influence scopes with other texts, resulting in a dual-optimum paradox where subject similarity and text controllability cannot be optimal simultaneously. To address this, we propose RealCustom++, a novel real-word paradigm that represents the subject with a non-conflicting real word to firstly generate a coherent guidance image and corresponding subject mask, thereby disentangling the influence scopes of the text and subject for simultaneous optimization. Specifically, RealCustom++ introduces a train-inference decoupled framework: (1) during training, it learns a general alignment between visual conditions and all real words in the text; and (2) during inference, a dual-branch architecture is employed, where the Guidance Branch produces the subject guidance mask and the Generation Branch utilizes this mask to customize the generation of the specific real word exclusively within subject-relevant regions. In contrast to previous methods that excel in either controllability or similarity, RealCustom++ achieves superior performance in both, with improvements of 7.48% in controllability, 3.04% in similarity, and 76.43% in generation quality. For multi-subject customization, RealCustom++ further achieves improvements of 4.6% in controllability and 6.34% in multi-subject similarity. Our work has been applied in JiMeng of ByteDance, and codes are released at https://github.com/bytedance/RealCustom.

Related papers

• Geometric Disentanglement of Text Embeddings for Subject-Consistent Text-to-Image Generation using A Single Prompt [14.734857939203811]
  We propose a training-free approach that addresses semantic entanglement from a subject perspective.<n>Our approach significantly improves both subject consistency and text alignment over existing baselines.
  arXiv  Detail & Related papers  (2025-12-18T11:55:06Z)
• FocusDiff: Advancing Fine-Grained Text-Image Alignment for Autoregressive Visual Generation through RL [78.59912944698992]
  We propose FocusDiff to enhance fine-grained text-image semantic alignment.<n>We construct a new dataset of paired texts and images with similar overall expressions but distinct local semantics.<n>Our approach achieves state-of-the-art performance on existing text-to-image benchmarks and significantly outperforms prior methods on PairComp.
  arXiv  Detail & Related papers  (2025-06-05T18:36:33Z)
• Image-Text Co-Decomposition for Text-Supervised Semantic Segmentation [28.24883865053459]
  This paper aims to learn a model capable of segmenting arbitrary visual concepts within images by using only image-text pairs without dense annotations. Existing methods have demonstrated that contrastive learning on image-text pairs effectively aligns visual segments with the meanings of texts. A text often consists of multiple semantic concepts, whereas semantic segmentation strives to create semantically homogeneous segments.
  arXiv  Detail & Related papers  (2024-04-05T17:25:17Z)
• RealCustom: Narrowing Real Text Word for Real-Time Open-Domain Text-to-Image Customization [57.86083349873154]
  Text-to-image customization aims to synthesize text-driven images for the given subjects. Existing works follow the pseudo-word paradigm, i.e., represent the given subjects as pseudo-words and then compose them with the given text. We present RealCustom that, for the first time, disentangles similarity from controllability by precisely limiting subject influence to relevant parts only.
  arXiv  Detail & Related papers  (2024-03-01T12:12:09Z)
• Training-Free Consistent Text-to-Image Generation [80.4814768762066]
  Text-to-image models can portray the same subject across diverse prompts. Existing approaches fine-tune the model to teach it new words that describe specific user-provided subjects. We present ConsiStory, a training-free approach that enables consistent subject generation by sharing the internal activations of the pretrained model.
  arXiv  Detail & Related papers  (2024-02-05T18:42:34Z)
• FASTER: A Font-Agnostic Scene Text Editing and Rendering Framework [19.564048493848272]
  Scene Text Editing (STE) is a challenging research problem, that primarily aims towards modifying existing texts in an image. Existing style-transfer-based approaches have shown sub-par editing performance due to complex image backgrounds, diverse font attributes, and varying word lengths within the text. We propose a novel font-agnostic scene text editing and rendering framework, named FASTER, for simultaneously generating text in arbitrary styles and locations.
  arXiv  Detail & Related papers  (2023-08-05T15:54:06Z)
• Text-guided Image Restoration and Semantic Enhancement for Text-to-Image Person Retrieval [11.798006331912056]
  The goal of Text-to-Image Person Retrieval (TIPR) is to retrieve specific person images according to the given textual descriptions. We propose a novel TIPR framework to build fine-grained interactions and alignment between person images and the corresponding texts.
  arXiv  Detail & Related papers  (2023-07-18T08:23:46Z)
• IMAGINATOR: Pre-Trained Image+Text Joint Embeddings using Word-Level Grounding of Images [2.9174297412129957]
  We introduce a pre-trained joint embedding (JE), named IMAGINATOR, trained on 21K distinct image objects level from 1M image+text pairs. IMAGINATOR encapsulates three individual representations: (i) object-object co-location, (ii) word-object co-location, and (iii) word-object correlation. We also evaluate pre-trained IMAGINATOR JEs on three downstream tasks: (i) image captioning, (ii) Image2Tweet, and (iii) text-based image retrieval.
  arXiv  Detail & Related papers  (2023-05-12T05:34:52Z)
• CoBIT: A Contrastive Bi-directional Image-Text Generation Model [72.1700346308106]
  CoBIT employs a novel unicoder-decoder structure, which attempts to unify three pre-training objectives in one framework. CoBIT achieves superior performance in image understanding, image-text understanding (Retrieval, Captioning, VQA, SNLI-VE) and text-based content creation, particularly in zero-shot scenarios.
  arXiv  Detail & Related papers  (2023-03-23T17:24:31Z)
• Plug-and-Play Diffusion Features for Text-Driven Image-to-Image Translation [10.39028769374367]
  We present a new framework that takes text-to-image synthesis to the realm of image-to-image translation. Our method harnesses the power of a pre-trained text-to-image diffusion model to generate a new image that complies with the target text.
  arXiv  Detail & Related papers  (2022-11-22T20:39:18Z)
• Fine-Grained Semantically Aligned Vision-Language Pre-Training [151.7372197904064]
  Large-scale vision-language pre-training has shown impressive advances in a wide range of downstream tasks. Existing methods mainly model the cross-modal alignment by the similarity of the global representations of images and texts. We introduce LO, a fine-grained semantically aLigned visiOn-langUage PrE-training framework, which learns fine-grained semantic alignment from the novel perspective of game-theoretic interactions.
  arXiv  Detail & Related papers  (2022-08-04T07:51:48Z)
• CRIS: CLIP-Driven Referring Image Segmentation [71.56466057776086]
  We propose an end-to-end CLIP-Driven Referring Image framework (CRIS) CRIS resorts to vision-language decoding and contrastive learning for achieving the text-to-pixel alignment. Our proposed framework significantly outperforms the state-of-the-art performance without any post-processing.
  arXiv  Detail & Related papers  (2021-11-30T07:29:08Z)
• FILIP: Fine-grained Interactive Language-Image Pre-Training [106.19474076935363]
  Fine-grained Interactive Language-Image Pre-training achieves finer-level alignment through a cross-modal late interaction mechanism. We construct a new large-scale image-text pair dataset called FILIP300M for pre-training. Experiments show that FILIP achieves state-of-the-art performance on multiple downstream vision-language tasks.
  arXiv  Detail & Related papers  (2021-11-09T17:15:38Z)
• Talk-to-Edit: Fine-Grained Facial Editing via Dialog [79.8726256912376]
  Talk-to-Edit is an interactive facial editing framework that performs fine-grained attribute manipulation through dialog between the user and the system. Our key insight is to model a continual "semantic field" in the GAN latent space. Our system generates language feedback by considering both the user request and the current state of the semantic field.
  arXiv  Detail & Related papers  (2021-09-09T17:17:59Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.