Enhancing MMDiT-Based Text-to-Image Models for Similar Subject Generation

• URL: http://arxiv.org/abs/2411.18301v1
• Date: Wed, 27 Nov 2024 12:47:06 GMT
• Title: Enhancing MMDiT-Based Text-to-Image Models for Similar Subject Generation
• Abstract summary: We find that the latest Multimodal Diffusion Transformer (MMDiT) still suffers from subject ambiguity or mixing when the input text prompt contains multiple subjects of similar semantics or appearance.<n>We identify three possible ambiguities that cause this problem: Inter-block Ambiguity, Text Ambiguity, and Semantic Ambiguity.<n>We propose to repair the ambiguous latent on-the-fly by test-time optimization at early denoising steps.
• Score: 30.99444221104659
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Representing the cutting-edge technique of text-to-image models, the latest Multimodal Diffusion Transformer (MMDiT) largely mitigates many generation issues existing in previous models. However, we discover that it still suffers from subject neglect or mixing when the input text prompt contains multiple subjects of similar semantics or appearance. We identify three possible ambiguities within the MMDiT architecture that cause this problem: Inter-block Ambiguity, Text Encoder Ambiguity, and Semantic Ambiguity. To address these issues, we propose to repair the ambiguous latent on-the-fly by test-time optimization at early denoising steps. In detail, we design three loss functions: Block Alignment Loss, Text Encoder Alignment Loss, and Overlap Loss, each tailored to mitigate these ambiguities. Despite significant improvements, we observe that semantic ambiguity persists when generating multiple similar subjects, as the guidance provided by overlap loss is not explicit enough. Therefore, we further propose Overlap Online Detection and Back-to-Start Sampling Strategy to alleviate the problem. Experimental results on a newly constructed challenging dataset of similar subjects validate the effectiveness of our approach, showing superior generation quality and much higher success rates over existing methods. Our code will be available at https://github.com/wtybest/EnMMDiT.

Related papers

• Visual Prompting Meets Feature Reconstruction-Based Anomaly Detection with Dual-Teacher Supervision [7.548589503544904]
  We introduce a visual prompting pipeline that isolates objects using foreground-background masking.<n>We also introduce a mechanism for unfreezing the teacher in student-teacher models to improve domain adaptability.<n>We achieve a 3.5 percentage point improvement over the previous state-of-the-art on the challenging AeBAD dataset.
  arXiv  Detail & Related papers  (2026-06-08T15:52:05Z)
• Align then Refine: Text-Guided 3D Prostate Lesion Segmentation [5.284052813431228]
  Volumetric methods must combine multiple modalities while ensuring anatomical consistency.<n>Current models struggle to integrate cross-modal information reliably.<n>We propose a new multi-encoder U-Net architecture incorporating three key innovations.
  arXiv  Detail & Related papers  (2026-04-20T18:13:52Z)
• Multimodal Large Language Models for Multi-Subject In-Context Image Generation [56.20395856287325]
  Music is the first MLLM specifically designed for textbfMUlti-textbfSubject textbfIn-textbfContext image generation.<n>We enhance the model's understanding of multi-subject semantic relationships through a vision chain-of-thought mechanism.<n>By incorporating complex subject images during training, we improve the model's capacity for chained reasoning.
  arXiv  Detail & Related papers  (2026-04-08T15:37:42Z)
• Deep Interest Mining with Cross-Modal Alignment for SemanticID Generation in Generative Recommendation [0.0]
  Generative Recommendation (GR) relies on Semantic IDs (SIDs) to compress trillion-scale data into learnable vocabulary sequences.<n>Existing methods suffer from three critical limitations.<n>We propose a novel framework integrating three key innovations: Deep Contextual Interest Mining (DCIM), Cross-Modal Semantic Alignment (CMSA), and Quality-Aware Reinforcement Mechanism (QARM)
  arXiv  Detail & Related papers  (2026-03-03T13:36:22Z)
• Prompt Reinjection: Alleviating Prompt Forgetting in Multimodal Diffusion Transformers [64.4017917917109]
  Multimodal Diffusion Transformers (MMDiTs) for text-to-image generation maintain separate text and image branches.<n>We observe a prompt forgetting phenomenon: the semantics of the prompt representation in the text branch is progressively forgotten as depth increases.<n>Motivated by these findings, we introduce a training-free approach, prompt reinjection, which reinjects prompt representations from early layers into later layers to alleviate this forgetting.
  arXiv  Detail & Related papers  (2026-02-06T17:19:53Z)
• TIP and Polish: Text-Image-Prototype Guided Multi-Modal Generation via Commonality-Discrepancy Modeling and Refinement [6.795398125733378]
  Existing methods suffer from cross-modal mismatch and lack explicit modeling of commonality and discrepancy.<n>We propose textbftextitTIPPo, a framework with explicit input modeling and comprehensive optimization objectives.<n> Experimental results demonstrate the promising performance of textbftextitTIPPo in automatic evaluation and LLM-based criteria for creativity and semantic consistency.
  arXiv  Detail & Related papers  (2025-11-12T07:16:19Z)
• SEPS: Semantic-enhanced Patch Slimming Framework for fine-grained cross-modal alignment [8.657941729790599]
  We introduce the Semantic-Enhanced Patch Slimming (SEPS) framework, which systematically addresses patch redundancy and ambiguity.<n>Our approach employs a two-stage mechanism to integrate unified semantics from both dense and sparse texts, enabling the identification of salient visual patches.<n>Experiments on Flickr30K and MS-COCO datasets validate that SEPS achieves superior performance.
  arXiv  Detail & Related papers  (2025-11-03T09:41:32Z)
• MAGE: Multimodal Alignment and Generation Enhancement via Bridging Visual and Semantic Spaces [23.447713697204225]
  MAGE is a novel framework that bridges the semantic spaces of vision and text through an innovative alignment mechanism.<n>We employ a training strategy that combines cross-entropy and mean squared error, significantly enhancing the alignment effect.<n>Our proposed multimodal large model architecture, MAGE, achieved significantly better performance compared to similar works across various evaluation benchmarks.
  arXiv  Detail & Related papers  (2025-07-29T12:17:46Z)
• Test-time Prompt Refinement for Text-to-Image Models [14.505841027491114]
  We introduce a test-time prompt refinement framework that requires no additional training of the underlying T2I model, termed TIR.<n>In our approach, each generation step is followed by a refinement step, where a pretrained multimodal large language model (MLLM) analyzes the output image and the user's prompt.<n>We demonstrate that this closed-loop strategy improves alignment and visual coherence across multiple benchmark datasets, all while maintaining plug-and-play integration with black-box T2I models.
  arXiv  Detail & Related papers  (2025-07-22T20:30:13Z)
• No Re-Train, More Gain: Upgrading Backbones with Diffusion model for Pixel-Wise and Weakly-Supervised Few-Shot Segmentation [22.263029309151467]
  Few-Shot (FSS) aims to segment novel classes using only a few annotated images. Current FSS methods face three issues: the inflexibility of backbone upgrade without re-training, the inability to uniformly handle various types of annotations. We propose DiffUp, a novel framework that conceptualizes the FSS task as a conditional generative problem using a diffusion process.
  arXiv  Detail & Related papers  (2024-07-23T05:09:07Z)
• Unsupervised Text Style Transfer via LLMs and Attention Masking with Multi-way Interactions [18.64326057581588]
  Unsupervised Text Style Transfer (UTST) has emerged as a critical task within the domain of Natural Language Processing (NLP) We propose four ways of interactions, that are pipeline framework with tuned orders; knowledge distillation from Large Language Models (LLMs) to attention masking model; in-context learning with constructed parallel examples. We empirically show these multi-way interactions can improve the baselines in certain perspective of style strength, content preservation and text fluency.
  arXiv  Detail & Related papers  (2024-02-21T09:28:02Z)
• The Right Losses for the Right Gains: Improving the Semantic Consistency of Deep Text-to-Image Generation with Distribution-Sensitive Losses [0.35898124827270983]
  We propose a contrastive learning approach with a novel combination of two loss functions: fake-to-fake loss and fake-to-real loss. We test this approach on two baseline models: SSAGAN and AttnGAN. Results show that our approach improves the qualitative results on AttnGAN with style blocks on the CUB dataset.
  arXiv  Detail & Related papers  (2023-12-18T00:05:28Z)
• Separate-and-Enhance: Compositional Finetuning for Text2Image Diffusion Models [58.46926334842161]
  This work illuminates the fundamental reasons for such misalignment, pinpointing issues related to low attention activation scores and mask overlaps. We propose two novel objectives, the Separate loss and the Enhance loss, that reduce object mask overlaps and maximize attention scores. Our method diverges from conventional test-time-adaptation techniques, focusing on finetuning critical parameters, which enhances scalability and generalizability.
  arXiv  Detail & Related papers  (2023-12-10T22:07:42Z)
• Towards General Visual-Linguistic Face Forgery Detection [95.73987327101143]
  Deepfakes are realistic face manipulations that can pose serious threats to security, privacy, and trust. Existing methods mostly treat this task as binary classification, which uses digital labels or mask signals to train the detection model. We propose a novel paradigm named Visual-Linguistic Face Forgery Detection(VLFFD), which uses fine-grained sentence-level prompts as the annotation.
  arXiv  Detail & Related papers  (2023-07-31T10:22:33Z)
• Minimally-Supervised Speech Synthesis with Conditional Diffusion Model and Language Model: A Comparative Study of Semantic Coding [57.42429912884543]
  We propose Diff-LM-Speech, Tetra-Diff-Speech and Tri-Diff-Speech to solve high dimensionality and waveform distortion problems. We also introduce a prompt encoder structure based on a variational autoencoder and a prosody bottleneck to improve prompt representation ability. Experimental results show that our proposed methods outperform baseline methods.
  arXiv  Detail & Related papers  (2023-07-28T11:20:23Z)
• Spatial-Temporal Graph Enhanced DETR Towards Multi-Frame 3D Object Detection [54.041049052843604]
  We present STEMD, a novel end-to-end framework that enhances the DETR-like paradigm for multi-frame 3D object detection. First, to model the inter-object spatial interaction and complex temporal dependencies, we introduce the spatial-temporal graph attention network. Finally, it poses a challenge for the network to distinguish between the positive query and other highly similar queries that are not the best match.
  arXiv  Detail & Related papers  (2023-07-01T13:53:14Z)
• GD-MAE: Generative Decoder for MAE Pre-training on LiDAR Point Clouds [72.60362979456035]
  Masked Autoencoders (MAE) are challenging to explore in large-scale 3D point clouds. We propose a textbfGenerative textbfDecoder for MAE (GD-MAE) to automatically merges the surrounding context. We demonstrate the efficacy of the proposed method on several large-scale benchmarks: KITTI, and ONCE.
  arXiv  Detail & Related papers  (2022-12-06T14:32:55Z)
• $\textit{latent}$-GLAT: Glancing at Latent Variables for Parallel Text Generation [65.29170569821093]
  parallel text generation has received widespread attention due to its success in generation efficiency. In this paper, we propose $textitlatent$-GLAT, which employs the discrete latent variables to capture word categorical information. Experiment results show that our method outperforms strong baselines without the help of an autoregressive model.
  arXiv  Detail & Related papers  (2022-04-05T07:34:12Z)
• Exposing Semantic Segmentation Failures via Maximum Discrepancy Competition [102.75463782627791]
  We take steps toward answering the question by exposing failures of existing semantic segmentation methods in the open visual world. Inspired by previous research on model falsification, we start from an arbitrarily large image set, and automatically sample a small image set by MAximizing the Discrepancy (MAD) between two segmentation methods. The selected images have the greatest potential in falsifying either (or both) of the two methods. A segmentation method, whose failures are more difficult to be exposed in the MAD competition, is considered better.
  arXiv  Detail & Related papers  (2021-02-27T16:06:25Z)
• Recurrent Multi-view Alignment Network for Unsupervised Surface Registration [79.72086524370819]
  Learning non-rigid registration in an end-to-end manner is challenging due to the inherent high degrees of freedom and the lack of labeled training data. We propose to represent the non-rigid transformation with a point-wise combination of several rigid transformations. We also introduce a differentiable loss function that measures the 3D shape similarity on the projected multi-view 2D depth images.
  arXiv  Detail & Related papers  (2020-11-24T14:22:42Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.