ADaFuSE: Adaptive Diffusion-generated Image and Text Fusion for Interactive Text-to-Image Retrieval

• URL: http://arxiv.org/abs/2603.21886v1
• Date: Mon, 23 Mar 2026 12:13:57 GMT
• Title: ADaFuSE: Adaptive Diffusion-generated Image and Text Fusion for Interactive Text-to-Image Retrieval
• Abstract summary: We propose ADaFuSE, a lightweight fusion model designed to align and calibrate multi-modal views for diffusion-augmented I-TIR.<n>We show that ADaFuSE achieves state-of-the-art performance, surpassing DAR by up to 3.49% in Hits@10 with only a 5.29% parameter increase.
• Score: 8.780427902086789
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Recent advances in interactive text-to-image retrieval (I-TIR) use diffusion models to bridge the modality gap between the textual information need and the images to be searched, resulting in increased effectiveness. However, existing frameworks fuse multi-modal views of user feedback by simple embedding addition. In this work, we show that this static and undifferentiated fusion indiscriminately incorporates generative noise produced by the diffusion model, leading to performance degradation for up to 55.62% samples. We further propose ADaFuSE (Adaptive Diffusion-Text Fusion with Semantic-aware Experts), a lightweight fusion model designed to align and calibrate multi-modal views for diffusion-augmented I-TIR, which can be plugged into existing frameworks without modifying the backbone encoder. Specifically, we introduce a dual-branch fusion mechanism that employs an adaptive gating branch to dynamically balance modality reliability, alongside a semantic-aware mixture-of-experts branch to capture fine-grained cross-modal nuances. Via thorough evaluation over four standard I-TIR benchmarks, ADaFuSE achieves state-of-the-art performance, surpassing DAR by up to 3.49% in Hits@10 with only a 5.29% parameter increase, while exhibiting stronger robustness to noisy and longer interactive queries. These results show that generative augmentation coupled with principled fusion provides a simple, generalizable alternative to fine-tuning for interactive retrieval.

Related papers

• P2Fusion: Prompt-based Progressive Infrared-Visible Image Fusion via Dual-Prior Distillation [73.147281449267]
  We propose P2Fusion, a prior-guided distillation-based framework that reformulates infrared-visible image fusion.<n>Instead of imposing hard-coded penalties, we distill image-intrinsic priors, thermal saliency and spatial quality, into learnable dynamic regulators.<n>P2Fusion achieves state-of-the-art results in 14 out of 20 key evaluation metrics across 5 benchmarks.
  arXiv  Detail & Related papers  (2026-08-13T10:09:49Z)
• Combined Dictionary Unfolding Network with Gradient-Adaptive Fidelity for Transferable Multi-Source Fusion [51.66903887088365]
  CDNet is a lightweight Combined Dictionary Unfolding Network for multi-source image fusion.<n>We evaluate CDNet on four tasks, including multi-exposure image fusion, infrared and visible image fusion, medical image fusion, and infrared and visible image fusion for semantic segmentation.
  arXiv  Detail & Related papers  (2026-05-01T06:52:29Z)
• CtrlFuse: Mask-Prompt Guided Controllable Infrared and Visible Image Fusion [51.060328159429154]
  Infrared and visible image fusion generates all-weather perception-capable images by combining complementary modalities.<n>We propose CtrlFuse, a controllable image fusion framework that enables interactive dynamic fusion guided by mask prompts.<n> Experiments demonstrate state-of-the-art results in both fusion controllability and segmentation accuracy, with the adapted task branch even outperforming the original segmentation model.
  arXiv  Detail & Related papers  (2026-01-12T13:36:48Z)
• Towards Unified Semantic and Controllable Image Fusion: A Diffusion Transformer Approach [99.80480649258557]
  DiTFuse is an instruction-driven framework that performs semantics-aware fusion within a single model.<n>Experiments on public IVIF, MFF, and MEF benchmarks confirm superior quantitative and qualitative performance, sharper textures, and better semantic retention.
  arXiv  Detail & Related papers  (2025-12-08T05:04:54Z)
• Generative Editing in the Joint Vision-Language Space for Zero-Shot Composed Image Retrieval [11.724675700368316]
  Composed Image Retrieval (CIR) enables fine-grained visual search by combining a reference image with a textual modification.<n>We propose Fusion-Diff, a novel generative editing framework with high effectiveness and data efficiency designed for multimodal alignment.
  arXiv  Detail & Related papers  (2025-12-01T13:04:55Z)
• VMDiff: Visual Mixing Diffusion for Limitless Cross-Object Synthesis [23.50866105623598]
  We propose a diffusion-based framework that synthesizes a single, coherent object by integrating two input images at both noise and latent levels.<n>Our method outperforms strong baselines in visual quality, semantic consistency, and human-rated creativity.
  arXiv  Detail & Related papers  (2025-09-28T03:17:58Z)
• FAR-Net: Multi-Stage Fusion Network with Enhanced Semantic Alignment and Adaptive Reconciliation for Composed Image Retrieval [36.03123811283016]
  We propose FAR-Net, a multi-stage fusion framework designed with enhanced semantic alignment and adaptive reconciliation.<n>Experiments on CIRR and FashionIQ show consistent performance gains, improving Recall@1 by up to 2.4% and Recall@50 by 1.04% over existing methods.
  arXiv  Detail & Related papers  (2025-07-17T06:30:41Z)
• An Intermediate Fusion ViT Enables Efficient Text-Image Alignment in Diffusion Models [18.184158874126545]
  We investigate how different fusion strategies can affect vision-language alignment. A specially designed intermediate fusion can boost text-to-image alignment with improved generation quality. Our model achieves a higher CLIP Score and lower FID, with 20% reduced FLOPs, and 50% increased training speed.
  arXiv  Detail & Related papers  (2024-03-25T08:16:06Z)
• Unified Frequency-Assisted Transformer Framework for Detecting and Grounding Multi-Modal Manipulation [109.1912721224697]
  We present the Unified Frequency-Assisted transFormer framework, named UFAFormer, to address the DGM4 problem. By leveraging the discrete wavelet transform, we decompose images into several frequency sub-bands, capturing rich face forgery artifacts. Our proposed frequency encoder, incorporating intra-band and inter-band self-attentions, explicitly aggregates forgery features within and across diverse sub-bands.
  arXiv  Detail & Related papers  (2023-09-18T11:06:42Z)
• CDDFuse: Correlation-Driven Dual-Branch Feature Decomposition for Multi-Modality Image Fusion [138.40422469153145]
  We propose a novel Correlation-Driven feature Decomposition Fusion (CDDFuse) network. We show that CDDFuse achieves promising results in multiple fusion tasks, including infrared-visible image fusion and medical image fusion.
  arXiv  Detail & Related papers  (2022-11-26T02:40:28Z)
• A Joint Cross-Attention Model for Audio-Visual Fusion in Dimensional Emotion Recognition [46.443866373546726]
  We focus on dimensional emotion recognition based on the fusion of facial and vocal modalities extracted from videos. We propose a joint cross-attention model that relies on the complementary relationships to extract the salient features. Our proposed A-V fusion model provides a cost-effective solution that can outperform state-of-the-art approaches.
  arXiv  Detail & Related papers  (2022-03-28T14:09:43Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.