Decomposing Subject-Driven Image Generation via Intermediate Structural Prediction

• URL: http://arxiv.org/abs/2605.20807v1
• Date: Wed, 20 May 2026 06:58:52 GMT
• Title: Decomposing Subject-Driven Image Generation via Intermediate Structural Prediction
• Abstract summary: We propose a two-stage framework that decouples structure from appearance by first predicting a Canny map and then rendering the final image conditioned on both the source appearance and the predicted structure.<n>To improve text handling, we introduce a fully automatic pipeline that constructs a 100k-pair text-aware dataset with cross-view textual consistency.
• Score: 43.32803972800948
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Subject-driven text-to-image generation still struggles to preserve high-frequency identity details such as logos, patterns, and text. Existing methods typically operate directly in RGB space, which often leads to detail degradation under substantial edits. We propose a two-stage framework that decouples structure from appearance by first predicting a Canny map and then rendering the final image conditioned on both the source appearance and the predicted structure. To improve text handling, we further introduce a fully automatic pipeline that constructs a 100k-pair text-aware dataset with cross-view textual consistency. Experiments, including GPT-4.1-based evaluation and a knowledge distillation study, show clear gains over selected baselines and suggest that intermediate structural prediction is an effective route for high-fidelity subject-driven generation. Our dataset and code will be made publicly available.

Related papers

• Generative Retrieval for Unsupervised Text-Based Person Search [77.46429603553388]
  Text-based person search (TBPS) aims to retrieve images of a target person from a large image gallery based on a given natural language description.<n>We propose GTR+, a two-stage generation-then-retrieval framework.<n>We also contribute LargeFine-Person, a large-scale TBPS dataset with high-quality, fine-grained, and diverse textual annotations.
  arXiv  Detail & Related papers  (2026-09-11T15:25:00Z)
• Semantic-Structural Alignment for Generative Pictorial Charts [59.45629259524998]
  We present a generative framework for the automated synthesis of pictorial charts.<n>We frame the problem as a dual-conditioned generation task guided by two parallel external control signals.<n>Our method produces pictorial charts that are both artistically compelling and structurally consistent.
  arXiv  Detail & Related papers  (2026-05-05T05:20:46Z)
• JEPA-T: Joint-Embedding Predictive Architecture with Text Fusion for Image Generation [10.00677022779314]
  We propose a unified multimodal framework that encodes images and captions into discrete visual and textual tokens.<n>We incorporate cross-attention after the feature predictor for conditional denoising while maintaining a task-agnostic backbone.<n>Our approach shows that late architectural fusion combined with objective-level alignment offers an effective balance between conditioning strength and backbone generality in token-based T2I.
  arXiv  Detail & Related papers  (2025-10-01T14:51:10Z)
• Beyond Flat Text: Dual Self-inherited Guidance for Visual Text Generation [17.552733309504486]
  In real-world images, slanted or curved texts, especially those on cans, banners, or badges, appear as frequently as flat texts due to artistic design or layout constraints.<n>We introduce a new training-free framework, STGen, which accurately generates visual texts in challenging scenarios.
  arXiv  Detail & Related papers  (2025-01-10T11:44:59Z)
• Leveraging Structure Knowledge and Deep Models for the Detection of Abnormal Handwritten Text [19.05500901000957]
  We propose a two-stage detection algorithm that combines structure knowledge and deep models for handwritten text. A shape regression network trained by a novel semi-supervised contrast training strategy is introduced and the positional relationship between the characters is fully employed. Experiments on two handwritten text datasets show that the proposed method can greatly improve the detection performance.
  arXiv  Detail & Related papers  (2024-10-15T14:57:10Z)
• Generating Intermediate Representations for Compositional Text-To-Image Generation [16.757550214291015]
  We propose a compositional approach for text-to-image generation based on two stages. In the first stage, we design a diffusion-based generative model to produce one or more aligned intermediate representations conditioned on text. In the second stage, we map these representations, together with the text, to the final output image using a separate diffusion-based generative model.
  arXiv  Detail & Related papers  (2024-10-13T10:24:55Z)
• Advancing Visual Grounding with Scene Knowledge: Benchmark and Method [74.72663425217522]
  Visual grounding (VG) aims to establish fine-grained alignment between vision and language. Most existing VG datasets are constructed using simple description texts. We propose a novel benchmark of underlineScene underlineKnowledge-guided underlineVisual underlineGrounding.
  arXiv  Detail & Related papers  (2023-07-21T13:06:02Z)
• TextFormer: A Query-based End-to-End Text Spotter with Mixed Supervision [61.186488081379]
  We propose TextFormer, a query-based end-to-end text spotter with Transformer architecture. TextFormer builds upon an image encoder and a text decoder to learn a joint semantic understanding for multi-task modeling. It allows for mutual training and optimization of classification, segmentation, and recognition branches, resulting in deeper feature sharing.
  arXiv  Detail & Related papers  (2023-06-06T03:37:41Z)
• TextDiffuser: Diffusion Models as Text Painters [118.30923824681642]
  We introduce TextDiffuser, focusing on generating images with visually appealing text that is coherent with backgrounds. We contribute the first large-scale text images dataset with OCR annotations, MARIO-10M, containing 10 million image-text pairs. We show that TextDiffuser is flexible and controllable to create high-quality text images using text prompts alone or together with text template images, and conduct text inpainting to reconstruct incomplete images with text.
  arXiv  Detail & Related papers  (2023-05-18T10:16:19Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.