EruDiff: Refactoring Knowledge in Diffusion Models for Advanced Text-to-Image Synthesis

• URL: http://arxiv.org/abs/2603.20828v1
• Date: Sat, 21 Mar 2026 14:04:08 GMT
• Title: EruDiff: Refactoring Knowledge in Diffusion Models for Advanced Text-to-Image Synthesis
• Abstract summary: EruDiff aims to rectify the knowledge distribution of intractable implicit prompts with that of well-defined explicit anchors.<n>We develop the Diffusion Knowledge Distribution Matching (DK-DM) to register the knowledge distribution of implicit prompts with that of well-defined explicit anchors.<n>We employ the Negative-Only Reinforcement Learning (NO-RL) strategy for fine-grained correction.
• Score: 49.883192716595026
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Text-to-image diffusion models have achieved remarkable fidelity in synthesizing images from explicit text prompts, yet exhibit a critical deficiency in processing implicit prompts that require deep-level world knowledge, ranging from natural sciences to cultural commonsense, resulting in counter-factual synthesis. This paper traces the root of this limitation to a fundamental dislocation of the underlying knowledge structures, manifesting as a chaotic organization of implicit prompts compared to their explicit counterparts. In this paper, we propose EruDiff, which aims to refactor the knowledge within diffusion models. Specifically, we develop the Diffusion Knowledge Distribution Matching (DK-DM) to register the knowledge distribution of intractable implicit prompts with that of well-defined explicit anchors. Furthermore, to rectify the inherent biases in explicit prompt rendering, we employ the Negative-Only Reinforcement Learning (NO-RL) strategy for fine-grained correction. Rigorous empirical evaluations demonstrate that our method significantly enhances the performance of leading diffusion models, including FLUX and Qwen-Image, across both the scientific knowledge benchmark (i.e., Science-T2I) and the world knowledge benchmark (i.e., WISE), underscoring the effectiveness and generalizability. Our code is available at https://github.com/xiefan-guo/erudiff.

Related papers

• From Memorization to Absorption: Mixed-Policy RL for Continual Knowledge Injection [8.467797533190216]
  We propose Golden-GRPO Injection (GRIN), a three-stage self-learning framework for continual knowledge injection.<n>Golden-GRPO is a mixed-policy reinforcement learning algorithm designed specifically for knowledge injection.<n>GRIN substantially outperforms SFT and mixed-policy RL baselines on the harder question types while matching them on basic fact recall.
  arXiv  Detail & Related papers  (2026-08-26T00:04:46Z)
• Few-Step Distillation for Text-to-Image Generation: A Practical Guide [60.99392100471019]
  Diffusion distillation has dramatically accelerated class-conditional image synthesis, but its applicability to open-ended text-to-image (T2I) generation is still unclear.<n>We present the first systematic study that adapts and compares state-of-the-art distillation techniques on a strong T2I teacher model, FLUX.1-lite.
  arXiv  Detail & Related papers  (2025-12-15T05:58:36Z)
• Diffusion-Classifier Synergy: Reward-Aligned Learning via Mutual Boosting Loop for FSCIL [19.094835780362775]
  Few-Shot Class-Incremental Learning (FSCIL) challenges models to sequentially learn new classes from minimal examples.<n>Current FSCIL methods often struggle with generalization due to their reliance on limited datasets.<n>This paper introduces Diffusion-Classifier Synergy (DCS), a novel framework that establishes a mutual boosting loop between diffusion model and FSCIL classifier.
  arXiv  Detail & Related papers  (2025-10-04T01:48:52Z)
• Causality-aligned Prompt Learning via Diffusion-based Counterfactual Generation [45.395353088233556]
  We introduce a theoretically grounded $textbfDi$ffusion-based $textbfC$ounterf$textbfa$ctual $textbfp$rompt learning framework.<n>Our method performs excellently across tasks such as image classification, image-text retrieval, and visual question answering, with particularly strong advantages in unseen categories.
  arXiv  Detail & Related papers  (2025-07-26T09:27:52Z)
• Localizing Knowledge in Diffusion Transformers [44.27817967554535]
  We propose a model- and knowledge-agnostic method to localize where specific types of knowledge are encoded within the Diffusion Transformer blocks.<n>We show that the identified blocks are both interpretable and causally linked to the expression of knowledge in generated outputs.<n>Our findings offer new insights into the internal structure of DiTs and introduce a practical pathway for more interpretable, efficient, and controllable model editing.
  arXiv  Detail & Related papers  (2025-05-24T19:02:20Z)
• DDAE++: Enhancing Diffusion Models Towards Unified Generative and Discriminative Learning [53.27049077100897]
  generative pre-training has been shown to yield discriminative representations, paving the way towards unified visual generation and understanding.<n>This work introduces self-conditioning, a mechanism that internally leverages the rich semantics inherent in denoising network to guide its own decoding layers.<n>Results are compelling: our method boosts both generation FID and recognition accuracy with 1% computational overhead and generalizes across diverse diffusion architectures.
  arXiv  Detail & Related papers  (2025-05-16T08:47:16Z)
• Score Forgetting Distillation: A Swift, Data-Free Method for Machine Unlearning in Diffusion Models [63.43422118066493]
  Machine unlearning (MU) is a crucial foundation for developing safe, secure, and trustworthy GenAI models.<n>Traditional MU methods often rely on stringent assumptions and require access to real data.<n>This paper introduces Score Forgetting Distillation (SFD), an innovative MU approach that promotes the forgetting of undesirable information in diffusion models.
  arXiv  Detail & Related papers  (2024-09-17T14:12:50Z)
• Distilling Vision-Language Foundation Models: A Data-Free Approach via Prompt Diversification [49.41632476658246]
  We discuss the extension of DFKD to Vision-Language Foundation Models without access to the billion-level image-text datasets. The objective is to customize a student model for distribution-agnostic downstream tasks with given category concepts. We propose three novel Prompt Diversification methods to encourage image synthesis with diverse styles.
  arXiv  Detail & Related papers  (2024-07-21T13:26:30Z)
• DiffDis: Empowering Generative Diffusion Model with Cross-Modal Discrimination Capability [75.9781362556431]
  We propose DiffDis to unify the cross-modal generative and discriminative pretraining into one single framework under the diffusion process. We show that DiffDis outperforms single-task models on both the image generation and the image-text discriminative tasks.
  arXiv  Detail & Related papers  (2023-08-18T05:03:48Z)
• DiffusionSeg: Adapting Diffusion Towards Unsupervised Object Discovery [20.787180028571694]
  DiffusionSeg is a synthesis-exploitation framework containing two-stage strategies. We synthesize abundant images, and propose a novel training-free AttentionCut to obtain masks in the first stage. In the second exploitation stage, to bridge the structural gap, we use the inversion technique, to map the given image back to diffusion features.
  arXiv  Detail & Related papers  (2023-03-17T07:47:55Z)
• Principled Knowledge Extrapolation with GANs [92.62635018136476]
  We study counterfactual synthesis from a new perspective of knowledge extrapolation. We show that an adversarial game with a closed-form discriminator can be used to address the knowledge extrapolation problem. Our method enjoys both elegant theoretical guarantees and superior performance in many scenarios.
  arXiv  Detail & Related papers  (2022-05-21T08:39:42Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.