Relevance Feedback in Text-to-Image Diffusion: A Training-Free And Model-Agnostic Interactive Framework

• URL: http://arxiv.org/abs/2603.14936v1
• Date: Mon, 16 Mar 2026 07:38:23 GMT
• Title: Relevance Feedback in Text-to-Image Diffusion: A Training-Free And Model-Agnostic Interactive Framework
• Abstract summary: We propose RFD, an interactive framework that adapts the relevance feedback mechanism from information retrieval to diffusion models.<n>In RFD, users replace explicit textual dialogue with implicit, multi-select visual feedback to minimize cognitive load.<n>RFD operates entirely within the external text space, making it strictly training-free and model-agnostic as a universal plug-and-play solution.
• Score: 7.254162537916632
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Text-to-image generation using diffusion models has achieved remarkable success. However, users often possess clear visual intents but struggle to express them precisely in language, resulting in ambiguous prompts and misaligned images. Existing methods struggle to bridge this gap, typically relying on high-load textual dialogues, opaque black-box inferences, or expensive fine-tuning. They fail to simultaneously achieve low cognitive load, interpretable preference inference, and remain training-free and model-agnostic. To address this, we propose RFD, an interactive framework that adapts the relevance feedback mechanism from information retrieval to diffusion models. In RFD, users replace explicit textual dialogue with implicit, multi-select visual feedback to minimize cognitive load, easily expressing complex, multi-dimensional preferences. To translate feedback into precise generative guidance, we construct an expert-curated feature repository and introduce an information-theoretic weighted cumulative preference analysis. This white-box method calculates preferences from current-round feedback and incrementally accumulates them, avoiding the concatenation of historical interactions and preventing inference degradation caused by lengthy contexts. Furthermore, RFD employs a probabilistic sampling mechanism for prompt reconstruction to balance exploitation and exploration, preventing output homogenization. Crucially, RFD operates entirely within the external text space, making it strictly training-free and model-agnostic as a universal plug-and-play solution. Extensive experiments demonstrate that RFD effectively captures the user's true visual intent, significantly outperforming baselines in preference alignment.

Related papers

• ExACT: Exemplar-Driven Calibrated Refinement for Training-Free Visual Grounding in Remote Sensing Images [51.53960096934913]
  Remote sensing visual grounding (RSVG) aims to locate specific objects in high-resolution RS imagery using free-form natural language descriptions.<n>Recent advances in multimodal large language models (MLLMs) show great potential for such open-vocabulary RSVG.<n>We propose Exemplar-driven Calibrated Refinement (ExACT), a training-free framework driven by a one-shot visual prompting mechanism.
  arXiv  Detail & Related papers  (2026-06-27T13:50:39Z)
• Focus When Necessary: Adaptive Routing and Collaborative Grounding for Training-Free Visual Grounding [56.38477453373368]
  LazyMCoT is a dynamic and training-free framework that adaptively allocates visual grounding efforts based on sample difficulty.<n>LazyMCoT rivals training-based approaches by simultaneously improving reasoning accuracy and reducing average inference latency.
  arXiv  Detail & Related papers  (2026-06-15T03:17:44Z)
• Guiding Diffusion-based Reconstruction with Contrastive Signals for Balanced Visual Representation [81.40978077888693]
  Contrastive Language-Image Pre-training (CLIP) has become a key bottleneck for downstream performance.<n>Recent solutions use diffusion models to enhance representations by conditioning image reconstruction on CLIP visual tokens.<n>We integrate contrastive signals into diffusion-based reconstruction to pursue more comprehensive visual representations.
  arXiv  Detail & Related papers  (2026-03-05T04:45:49Z)
• Eliminating Hallucination in Diffusion-Augmented Interactive Text-to-Image Retrieval [11.644783748459787]
  Diffusion-Augmented Interactive Text-to-Image Retrieval (DAI-TIR) is a promising paradigm that improves retrieval performance by generating query images via diffusion models.<n>We propose Diffusion-aware Multi-view Contrastive Learning (DMCL), a hallucination-robust training framework that casts DAI-TIR as joint optimization over representations of query intent and the target image.
  arXiv  Detail & Related papers  (2026-01-28T08:58:57Z)
• Continuous-time Discrete-space Diffusion Model for Recommendation [25.432419904462694]
  CDRec is a novel Continuous-time Discrete-space Diffusion Recommendation framework.<n>It is superior in both recommendation accuracy and computational efficiency.<n>Experiments on real-world datasets demonstrate CDRec's superior performance in both recommendation accuracy and computational efficiency.
  arXiv  Detail & Related papers  (2025-11-15T09:06:57Z)
• Implicit Counterfactual Learning for Audio-Visual Segmentation [50.69377287012591]
  We propose the implicit counterfactual framework (ICF) to achieve unbiased cross-modal understanding.<n>Due to the lack of semantics, heterogeneous representations may lead to erroneous matches.<n>We introduce the multi-granularity implicit text (MIT) involving video-, segment- and frame-level as the bridge to establish the modality-shared space.
  arXiv  Detail & Related papers  (2025-07-28T11:46:35Z)
• A Visual Leap in CLIP Compositionality Reasoning through Generation of Counterfactual Sets [26.167194142428475]
  Vision-language models (VLMs) often struggle with compositional reasoning due to insufficient high-quality image-text data.<n>We propose a novel block-based diffusion approach that automatically generates counterfactual datasets without manual annotation.<n>Our approach achieves state-of-the-art results across multiple benchmarks while using substantially less training data than existing methods.
  arXiv  Detail & Related papers  (2025-07-07T06:47:10Z)
• Parallel Rescaling: Rebalancing Consistency Guidance for Personalized Diffusion Models [3.1964044595140217]
  We propose a parallel rescaling technique for personalized diffusion models.<n>Unlike prior personalization methods, our technique does not require additional training data or expensive annotations.
  arXiv  Detail & Related papers  (2025-05-31T15:36:36Z)
• Multimodal LLM-Guided Semantic Correction in Text-to-Image Diffusion [52.315729095824906]
  MLLM Semantic-Corrected Ping-Pong-Ahead Diffusion (PPAD) is a novel framework that introduces a Multimodal Large Language Model (MLLM) as a semantic observer during inference.<n>It performs real-time analysis on intermediate generations, identifies latent semantic inconsistencies, and translates feedback into controllable signals that actively guide the remaining denoising steps.<n>Extensive experiments demonstrate PPAD's significant improvements.
  arXiv  Detail & Related papers  (2025-05-26T14:42:35Z)
• Diffusion Generative Recommendation with Continuous Tokens [21.222713476105195]
  ContRec is a framework that seamlessly integrates continuous tokens into LLM-based RecSys.<n>We show that ContRec consistently outperforms both traditional and SOTA LLM-based recommender systems.<n>Our results highlight the potential of continuous tokenization and generative modeling for advancing the next generation of recommender systems.
  arXiv  Detail & Related papers  (2025-04-16T12:01:03Z)
• Coarse-to-Fine Proposal Refinement Framework for Audio Temporal Forgery Detection and Localization [60.899082019130766]
  We introduce a frame-level detection network (FDN) and a proposal refinement network (PRN) for audio temporal forgery detection and localization. FDN aims to mine informative inconsistency cues between real and fake frames to obtain discriminative features that are beneficial for roughly indicating forgery regions. PRN is responsible for predicting confidence scores and regression offsets to refine the coarse-grained proposals derived from the FDN.
  arXiv  Detail & Related papers  (2024-07-23T15:07:52Z)
• RLVF: Learning from Verbal Feedback without Overgeneralization [94.19501420241188]
  We study the problem of incorporating verbal feedback without such overgeneralization. We develop a new method Contextualized Critiques with Constrained Preference Optimization (C3PO) Our approach effectively applies verbal feedback to relevant scenarios while preserving existing behaviors for other contexts.
  arXiv  Detail & Related papers  (2024-02-16T18:50:24Z)
• Towards General Visual-Linguistic Face Forgery Detection [95.73987327101143]
  Deepfakes are realistic face manipulations that can pose serious threats to security, privacy, and trust. Existing methods mostly treat this task as binary classification, which uses digital labels or mask signals to train the detection model. We propose a novel paradigm named Visual-Linguistic Face Forgery Detection(VLFFD), which uses fine-grained sentence-level prompts as the annotation.
  arXiv  Detail & Related papers  (2023-07-31T10:22:33Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.