FBCIR: Balancing Cross-Modal Focuses in Composed Image Retrieval

• URL: http://arxiv.org/abs/2603.11520v1
• Date: Thu, 12 Mar 2026 04:05:27 GMT
• Title: FBCIR: Balancing Cross-Modal Focuses in Composed Image Retrieval
• Abstract summary: Composed image retrieval (CIR) requires multi-modal models to jointly reason over visual content and semantic modifications presented in text-image input pairs.<n>We propose FBCIR, a multi-modal focus interpretation method that identifies the most crucial visual and textual input components to a model's retrieval decisions.
• Score: 12.652218923459742
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Composed image retrieval (CIR) requires multi-modal models to jointly reason over visual content and semantic modifications presented in text-image input pairs. While current CIR models achieve strong performance on common benchmark cases, their accuracies often degrades in more challenging scenarios where negative candidates are semantically aligned with the query image or text. In this paper, we attribute this degradation to focus imbalances, where models disproportionately attend to one modality while neglecting the other. To validate this claim, we propose FBCIR, a multi-modal focus interpretation method that identifies the most crucial visual and textual input components to a model's retrieval decisions. Using FBCIR, we report that focus imbalances are prevalent in existing CIR models, especially under hard negative settings. Building on the analyses, we further propose a CIR data augmentation workflow that facilitates existing CIR datasets with curated hard negatives designed to encourage balanced cross-modal reasoning. Extensive experiments across multiple CIR models demonstrate that the proposed augmentation consistently improves performance in challenging cases, while maintaining their capabilities on standard benchmarks. Together, our interpretation method and data augmentation workflow provide a new perspective on CIR model diagnosis and robustness improvements.

Related papers

• A Sanity Check on Composed Image Retrieval [91.95275287747499]
  Composed Image Retrieval (CIR) aims to retrieve a target image based on a query composed of a reference image, and a relative caption that specifies the desired modification.<n>We introduce FISD, a Fully-Informed Semantically-Diverse benchmark, which employs generative models to precisely control the variables of reference-target image pairs.<n>We propose an automatic multi-round agentic evaluation framework to probe the potential of the existing models in the interactive scenarios.
  arXiv  Detail & Related papers  (2026-04-14T15:52:22Z)
• WRF4CIR: Weight-Regularized Fine-Tuning Network for Composed Image Retrieval [58.85945491755373]
  Composed Image Retrieval (CIR) task aims to retrieve target images based on reference images and modification texts.<n>Current CIR methods rely on fine-tuning vision-language pre-trained models.<n>We introduce WRF4CIR, a Weight-Regularized Fine-tuning network for CIR.
  arXiv  Detail & Related papers  (2026-04-07T08:23:39Z)
• CDRRM: Contrast-Driven Rubric Generation for Reliable and Interpretable Reward Modeling [61.75914342638658]
  CDRRM (Contrast-Driven Reward Model) is a framework for high-quality rubric generation and guided preference judgment.<n>This work offers a scalable, interpretable, and data-efficient path for reward modeling.
  arXiv  Detail & Related papers  (2026-03-09T07:15:23Z)
• Bias Detection and Rotation-Robustness Mitigation in Vision-Language Models and Generative Image Models [0.0]
  Vision-Language Models (VLMs) and generative image models have achieved remarkable performance across multimodal tasks.<n>This work investigates bias propagation and robustness in state-of-the-art vision-language and generative models.<n>We propose rotation-robust mitigation strategies that combine data augmentation, representation alignment, and model-level regularization.
  arXiv  Detail & Related papers  (2026-01-09T00:36:11Z)
• Generative Editing in the Joint Vision-Language Space for Zero-Shot Composed Image Retrieval [11.724675700368316]
  Composed Image Retrieval (CIR) enables fine-grained visual search by combining a reference image with a textual modification.<n>We propose Fusion-Diff, a novel generative editing framework with high effectiveness and data efficiency designed for multimodal alignment.
  arXiv  Detail & Related papers  (2025-12-01T13:04:55Z)
• Did Models Sufficient Learn? Attribution-Guided Training via Subset-Selected Counterfactual Augmentation [61.248535801314375]
  Subset-Selected Counterfactual Augmentation (SS-CA)<n>We develop Counterfactual LIMA to identify minimal spatial region sets whose removal can selectively alter model predictions.<n>Experiments show that SS-CA improves generalization on in-distribution (ID) test data and achieves superior performance on out-of-distribution (OOD) benchmarks.
  arXiv  Detail & Related papers  (2025-11-15T08:39:22Z)
• CIR-CoT: Towards Interpretable Composed Image Retrieval via End-to-End Chain-of-Thought Reasoning [93.05917922306196]
  Composed Image Retrieval (CIR) aims to find a target image from a reference image and a modification text.<n>CIR-CoT is the first end-to-end retrieval-oriented MLLM designed to integrate explicit Chain-of-Thought (CoT) reasoning.
  arXiv  Detail & Related papers  (2025-10-09T09:41:45Z)
• CoLLM: A Large Language Model for Composed Image Retrieval [76.29725148964368]
  Composed Image Retrieval (CIR) is a complex task that aims to retrieve images based on a multimodal query.<n>We present CoLLM, a one-stop framework that generates triplets on-the-fly from image-caption pairs.<n>We leverage Large Language Models (LLMs) to generate joint embeddings of reference images and modification texts.
  arXiv  Detail & Related papers  (2025-03-25T17:59:50Z)
• A Comprehensive Survey on Composed Image Retrieval [54.54527281731775]
  Composed Image Retrieval (CIR) is an emerging yet challenging task that allows users to search for target images using a multimodal query.<n>There is currently no comprehensive review of CIR to provide a timely overview of this field.<n>We synthesize insights from over 120 publications in top conferences and journals, including ACM TOIS, SIGIR, and CVPR.
  arXiv  Detail & Related papers  (2025-02-19T01:37:24Z)
• Data Roaming and Quality Assessment for Composed Image Retrieval [25.452015862927766]
  Composed Image Retrieval (CoIR) involves queries that combine image and text modalities, allowing users to express their intent more effectively. We introduce the Large Scale Composed Image Retrieval (LaSCo) dataset, a new CoIR dataset which is ten times larger than existing ones. We also introduce a new CoIR baseline, the Cross-Attention driven Shift (CASE)
  arXiv  Detail & Related papers  (2023-03-16T16:02:24Z)
• Correlation Information Bottleneck: Towards Adapting Pretrained Multimodal Models for Robust Visual Question Answering [63.87200781247364]
  Correlation Information Bottleneck (CIB) seeks a tradeoff between compression and redundancy in representations. We derive a tight theoretical upper bound for the mutual information between multimodal inputs and representations.
  arXiv  Detail & Related papers  (2022-09-14T22:04:10Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.