Disentangling to Re-couple: Resolving the Similarity-Controllability Paradox in Subject-Driven Text-to-Image Generation

• URL: http://arxiv.org/abs/2604.00849v1
• Date: Wed, 01 Apr 2026 13:03:11 GMT
• Title: Disentangling to Re-couple: Resolving the Similarity-Controllability Paradox in Subject-Driven Text-to-Image Generation
• Abstract summary: We propose DisCo, a framework that first Disntangles and then re-Couples visual and textual information.<n>Our approach effectively resolves the paradox, enabling simultaneous high-fidelity subject preservation and precise textual control.
• Score: 38.61976178933174
• License: http://creativecommons.org/licenses/by-sa/4.0/
• Abstract: Subject-Driven Text-to-Image (T2I) Generation aims to preserve a subject's identity while editing its context based on a text prompt. A core challenge in this task is the "similarity-controllability paradox", where enhancing textual control often degrades the subject's fidelity, and vice-versa. We argue this paradox stems from the ambiguous role of text prompts, which are often tasked with describing both the subject and the desired modifications, leading to conflicting signals for the model. To resolve this, we propose DisCo, a novel framework that first Disntangles and then re-Couples visual and textual information. First, our textual-visual decoupling module isolates the sources of information: subject identity is extracted exclusively from the reference image with the entity word of the subject, while the text prompt is simplified to contain only the modification command, where the subject refers to general pronouns, eliminating descriptive ambiguity. However, this strict separation can lead to unnatural compositions between the subject and its contexts. We address this by designing a dedicated reward signal and using reinforcement learning to seamlessly recouple the visually-defined subject and the textually-generated context. Our approach effectively resolves the paradox, enabling simultaneous high-fidelity subject preservation and precise textual control. Extensive experiments demonstrate that our method achieves state-of-the-art performance, producing highly realistic and coherent images.

Related papers

• Image Generation from Contextually-Contradictory Prompts [50.999420029656214]
  We propose a stage-aware prompt decomposition framework that guides the denoising process using a sequence of proxy prompts.<n>Our method enables fine-grained semantic control and accurate image generation in the presence of contextual contradictions.
  arXiv  Detail & Related papers  (2025-06-02T17:48:12Z)
• One Image is Worth a Thousand Words: A Usability Preservable Text-Image Collaborative Erasing Framework [127.07102988701092]
  We introduce the first text-image Collaborative Concept Erasing (Co-Erasing) framework.<n>Co-Erasing describes the concept jointly by text prompts and the corresponding undesirable images induced by the prompts.<n>We design a text-guided image concept refinement strategy that directs the model to focus on visual features most relevant to the specified text concept.
  arXiv  Detail & Related papers  (2025-05-16T11:25:50Z)
• RealCustom++: Representing Images as Real Textual Word for Real-Time Customization [56.30487886962172]
  Given a text and an image of a specific subject, text-to-image customization aims to generate new images that align with both the text and the subject's appearance.<n>Existing works follow the pseudo-word paradigm, which represents the subject as a non-existent pseudo word and combines it with other text to generate images.<n>We propose RealCustom++, a novel real-word paradigm that represents the subject with a non-conflicting real word to firstly generate a coherent guidance image and corresponding subject mask.
  arXiv  Detail & Related papers  (2024-08-19T07:15:44Z)
• Text Guided Image Editing with Automatic Concept Locating and Forgetting [27.70615803908037]
  We propose a novel method called Locate and Forget (LaF) to locate potential target concepts in the image for modification. Compared to the baselines, our method demonstrates its superiority in text-guided image editing tasks both qualitatively and quantitatively.
  arXiv  Detail & Related papers  (2024-05-30T05:36:32Z)
• RealCustom: Narrowing Real Text Word for Real-Time Open-Domain Text-to-Image Customization [57.86083349873154]
  Text-to-image customization aims to synthesize text-driven images for the given subjects. Existing works follow the pseudo-word paradigm, i.e., represent the given subjects as pseudo-words and then compose them with the given text. We present RealCustom that, for the first time, disentangles similarity from controllability by precisely limiting subject influence to relevant parts only.
  arXiv  Detail & Related papers  (2024-03-01T12:12:09Z)
• Contrastive Prompts Improve Disentanglement in Text-to-Image Diffusion Models [68.47333676663312]
  We show a simple modification of classifier-free guidance can help disentangle image factors in text-to-image models. The key idea of our method, Contrastive Guidance, is to characterize an intended factor with two prompts that differ in minimal tokens. We illustrate whose benefits in three scenarios: (1) to guide domain-specific diffusion models trained on an object class, (2) to gain continuous, rig-like controls for text-to-image generation, and (3) to improve the performance of zero-shot image editors.
  arXiv  Detail & Related papers  (2024-02-21T03:01:17Z)
• DreamInpainter: Text-Guided Subject-Driven Image Inpainting with Diffusion Models [37.133727797607676]
  This study introduces Text-Guided Subject-Driven Image Inpainting. We compute dense subject features to ensure accurate subject replication. We employ a discriminative token selection module to eliminate redundant subject details.
  arXiv  Detail & Related papers  (2023-12-05T22:23:19Z)
• Relational Contrastive Learning for Scene Text Recognition [22.131554868199782]
  We argue that prior contextual information can be interpreted as relations of textual primitives due to the heterogeneous text and background. We propose to enrich the textual relations via rearrangement, hierarchy and interaction, and design a unified framework called RCLSTR: Contrastive Learning for Scene Text Recognition.
  arXiv  Detail & Related papers  (2023-08-01T12:46:58Z)
• Paste, Inpaint and Harmonize via Denoising: Subject-Driven Image Editing with Pre-Trained Diffusion Model [22.975965453227477]
  We introduce a new framework called textitPaste, Inpaint and Harmonize via Denoising (PhD) In our experiments, we apply PhD to both subject-driven image editing tasks and explore text-driven scene generation given a reference subject.
  arXiv  Detail & Related papers  (2023-06-13T07:43:10Z)
• DisenBooth: Identity-Preserving Disentangled Tuning for Subject-Driven Text-to-Image Generation [50.39533637201273]
  We propose DisenBooth, an identity-preserving disentangled tuning framework for subject-driven text-to-image generation. By combining the identity-preserved embedding and identity-irrelevant embedding, DisenBooth demonstrates more generation flexibility and controllability.
  arXiv  Detail & Related papers  (2023-05-05T09:08:25Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.