ELDiff: When Evidential Learning Meets Text-to-Image Diffusion

• URL: http://arxiv.org/abs/2606.20924v1
• Date: Thu, 18 Jun 2026 20:33:20 GMT
• Title: ELDiff: When Evidential Learning Meets Text-to-Image Diffusion
• Abstract summary: Multi-object text-to-image (T2I) diffusion is crucial for image synthesis.<n> consistency constraint is often underemphasized in the denoising process of diffusion models.<n>We propose ELDiff, a new evidential learning-supervised T2I diffusion model.
• Score: 8.411461785334746
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: In multi-object text-to-image (T2I) diffusion, ensuring semantic consistency between textual prompts and generated visual content is crucial for image synthesis. However, such consistency constraint is often underemphasized in the denoising process of diffusion models. Although token supervised diffusion models can mitigate this issue by learning object-wise consistency between the image content and object segmentation maps, it tends to suffer from the problems of segmentation map bias and semantic overlap conflict, especially when involving multiple objects. In this paper, we propose ELDiff, a new evidential learning-supervised T2I diffusion model, which leverages the advantages of uncertainty metric and conflict detection to enhance the fault tolerance of unreliable segmentation maps and suppress semantic conflicts, strengthening object-wise consistency learning. Specifically, a pixel evidence loss is proposed to restrain overconfidence in unreliable labels through evidential regularization, and a token conflict loss is designed to weaken the contradiction between semantics through optimizing a measured conflict factor. Extensive experiments show that our ELDiff outperforms existing training based and train-free based T2I diffusion models on SD v1.4, SD v2.1, SDXL, SD v3.5, and Qwen-Image, without requiring additional inference-time manipulations. Notably, ELDiff can be seamlessly extended to the existing training pipeline of T2I diffusion models. Code can be found at https://github.com/QingtaoPan/ELDiff.

Related papers

• Anchoring and Steering Diffusion: Enhancing the Faithfulness of Text-to-Image Generation at Inference Time [16.04233421396159]
  AnchorSteer is a training-free framework that exerts fine-grained control over.<n>textbfAnchorSteer and its components.<n>Experiments on GenEval and T2I-CompBench++ demonstrate that AnchorSteer consistently outperforms existing baselines in text--image alignment.
  arXiv  Detail & Related papers  (2026-07-29T09:08:57Z)
• Decoupled Residual Denoising Diffusion Models for Unified and Data Efficient Image-to-Image Translation [29.96691004865241]
  We propose Decoupled Residual Denoising Diffusion models (DRDD) for unified and data-efficient image-to-image (I2I) translation.
  arXiv  Detail & Related papers  (2026-05-31T06:38:18Z)
• Making Training-Free Diffusion Segmentors Scale with the Generative Power [118.72472901404814]
  A line of research focuses on adapting a pre-trained diffusion model to semantic segmentation without any further training.<n>We propose two techniques: auto aggregation and per-pixel rescaling, which together enable training-free segmentation to better leverage generative capability.
  arXiv  Detail & Related papers  (2026-03-06T11:35:37Z)
• InfSplign: Inference-Time Spatial Alignment of Text-to-Image Diffusion Models [27.206678799411645]
  InfSplign is a training-free inference-time method for text-to-image models.<n>It improves spatial alignment by adjusting the noise through a compound loss in every denoising step.<n>It achieves substantial performance gains over the strongest existing inference-time baselines.
  arXiv  Detail & Related papers  (2025-12-19T17:52:43Z)
• Contrastive Learning with Diffusion Features for Weakly Supervised Medical Image Segmentation [12.530950480385554]
  Conditional diffusion model (CDM) has been used as an alternative for generating segmentation masks.<n>We introduce Contrastive Learning with Diffusion Features (CLDF) to train a pixel decoder to map the diffusion features from a frozen CDM to a low-dimensional embedding space for segmentation.
  arXiv  Detail & Related papers  (2025-06-30T01:43:50Z)
• Training-Free Layout-to-Image Generation with Marginal Attention Constraints [73.55660250459132]
  We propose a training-free layout-to-image (L2I) approach, which eliminates the need for additional modules or fine-tuning.<n>Specifically, we use text-visual cross-attention feature maps to quantify inconsistencies between the layout of the generated images and the provided instructions.<n>We leverage pixel-to-pixel correlations in the self-attention feature maps to align cross-attention maps and combine three loss functions constrained by boundary attention to update latent features.
  arXiv  Detail & Related papers  (2024-11-15T05:44:45Z)
• Improving Consistency in Diffusion Models for Image Super-Resolution [28.945663118445037]
  We observe two kinds of inconsistencies in diffusion-based methods.<n>We introduce ConsisSR to handle both semantic and training-inference consistencies.<n>Our method demonstrates state-of-the-art performance among existing diffusion models.
  arXiv  Detail & Related papers  (2024-10-17T17:41:52Z)
• Distractors-Immune Representation Learning with Cross-modal Contrastive Regularization for Change Captioning [71.14084801851381]
  Change captioning aims to succinctly describe the semantic change between a pair of similar images. Most existing methods directly capture the difference between them, which risk obtaining error-prone difference features. We propose a distractors-immune representation learning network that correlates the corresponding channels of two image representations.
  arXiv  Detail & Related papers  (2024-07-16T13:00:33Z)
• Fine-grained Image-to-LiDAR Contrastive Distillation with Visual Foundation Models [55.99654128127689]
  Visual Foundation Models (VFMs) are used to generate semantic labels for weakly-supervised pixel-to-point contrastive distillation.<n>We adapt sampling probabilities of points to address imbalances in spatial distribution and category frequency.<n>Our approach consistently surpasses existing image-to-LiDAR contrastive distillation methods in downstream tasks.
  arXiv  Detail & Related papers  (2024-05-23T07:48:19Z)
• DiffDis: Empowering Generative Diffusion Model with Cross-Modal Discrimination Capability [75.9781362556431]
  We propose DiffDis to unify the cross-modal generative and discriminative pretraining into one single framework under the diffusion process. We show that DiffDis outperforms single-task models on both the image generation and the image-text discriminative tasks.
  arXiv  Detail & Related papers  (2023-08-18T05:03:48Z)
• Diffusion Visual Counterfactual Explanations [51.077318228247925]
  Visual Counterfactual Explanations (VCEs) are an important tool to understand the decisions of an image. Current approaches for the generation of VCEs are restricted to adversarially robust models and often contain non-realistic artefacts. In this paper, we overcome this by generating Visual Diffusion Counterfactual Explanations (DVCEs) for arbitrary ImageNet classifiers.
  arXiv  Detail & Related papers  (2022-10-21T09:35:47Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.