Do Composed Image Retrieval Benchmarks Require Multimodal Composition?

• URL: http://arxiv.org/abs/2605.14787v2
• Date: Fri, 15 May 2026 21:26:27 GMT
• Title: Do Composed Image Retrieval Benchmarks Require Multimodal Composition?
• Abstract summary: We show that strong performance on Composed Image Retrieval benchmarks is assumed to require multimodal composition.<n>Across four widely used CIR benchmarks and eleven Generalist Multimodal Embedding models, a large fraction of queries can be solved using a single modality.<n>High CIR performance can arise from unimodal signals rather than true multimodal composition.
• Score: 86.99911649534795
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Composed Image Retrieval (CIR) is a multimodal retrieval task where a query consists of a reference image and a textual modification, and the goal is to retrieve a target image satisfying both. In principle, strong performance on CIR benchmarks is assumed to require multimodal composition, i.e., combining complementary information from reference image and textual modification. In this work, we show that this assumption does not always hold. Across four widely used CIR benchmarks and eleven Generalist Multimodal Embedding models, a large fraction of queries can be solved using a single modality (from 32.2% to 83.6%), revealing pervasive unimodal shortcuts. Thus, high CIR performance can arise from unimodal signals rather than true multimodal composition. To better understand this issue, we perform a two-stage audit. First, we identify shortcut-solvable queries through cross-model analysis. Second, we conduct human validation on 4,741 shortcut-free queries, of which only 1,689 are well-formed, with common issues including ambiguous edits and mismatched targets. Re-evaluating models on this validated subset reveals qualitatively different behaviour: queries can no longer be solved with a single modality, and successful retrieval requires combining both inputs. While accuracy decreases, reliance on multimodal information increases. Overall, current CIR benchmarks conflate shortcut-solvable, noisy, and genuinely compositional queries, leading to an overestimation of model capability in multimodal composition.

Related papers

• Training-Free Pseudo-Fusion for Composed Image Retrieval with Diffusion Models and Multimodal Large Language Models [10.421702262520775]
  Composed Image Retrieval (CIR) is an emerging paradigm in content-based image retrieval.<n>In this work, we propose PeFuse, a training-free framework that leverages pretrained Diffusion Models and Multimodal Large Language Models.<n>We introduce two novel strategies: uni-directional and bi-directional conversion, which convert CIR into four single-modality retrieval problems.
  arXiv  Detail & Related papers  (2026-08-24T11:08:28Z)
• HINT: Composed Image Retrieval with Dual-path Compositional Contextualized Network [32.657095507841085]
  Composed Image Retrieval (CIR) is a challenging image retrieval paradigm.<n>We propose a dual-patHsItional coNtextualized neTwork (HINT) to perform contextualized encoding and amplify the similarity differences between matching and non-matching samples.<n>Our HINT model achieves optimal performance on all metrics across two CIR benchmark datasets.
  arXiv  Detail & Related papers  (2026-03-27T12:01:40Z)
• Evaluating Cross-Modal Reasoning Ability and Problem Characteristics with Multimodal Item Response Theory [22.63245796446805]
  Benchmarks for Multimodal Large Language Models should measure their ability for cross-modal integration.<n>Current benchmarks are filled with shortcut questions, which can be solved using only a single modality.<n>We introduce a multi-modal and multidimensional item response theory framework (M3IRT) that extends classical IRT.<n>M3IRT estimates cross-modal ability of MLLMs and each question's cross-modal difficulty, enabling compact, high-quality subsets.
  arXiv  Detail & Related papers  (2026-03-03T06:51:08Z)
• UNIDOC-BENCH: A Unified Benchmark for Document-Centric Multimodal RAG [82.84014669683863]
  Multimodal retrieval-augmented generation (MM-RAG) is a key approach for applying large language models to real-world knowledge bases.<n>UniDoc-Bench is the first large-scale, realistic benchmark for MM-RAG built from 70k real-world PDF pages.<n>Our experiments show that multimodal text-image fusion RAG systems consistently outperform both unimodal and jointly multimodal embedding-based retrieval.
  arXiv  Detail & Related papers  (2025-10-04T04:30:13Z)
• MR$^2$-Bench: Going Beyond Matching to Reasoning in Multimodal Retrieval [86.35779264575154]
  Multimodal retrieval is becoming a crucial component of modern AI applications, yet its evaluation lags behind the demands of more realistic and challenging scenarios.<n>We introduce MR$2$-Bench, a reasoning-intensive benchmark for multimodal retrieval.
  arXiv  Detail & Related papers  (2025-09-30T15:09:14Z)
• MM-BrowseComp: A Comprehensive Benchmark for Multimodal Browsing Agents [78.3863007028688]
  MM-BrowseComp is a novel benchmark comprising 224 challenging, hand-crafted questions.<n>These questions often incorporate images in prompts, and crucial information encountered during the search and reasoning process may also be embedded within images or videos on webpages.<n>Our comprehensive evaluation of state-of-the-art models on MM-BrowseComp reveals that even top models like OpenAI o3 with tools achieve only 29.02% accuracy.
  arXiv  Detail & Related papers  (2025-08-14T13:46:47Z)
• CoLLM: A Large Language Model for Composed Image Retrieval [76.29725148964368]
  Composed Image Retrieval (CIR) is a complex task that aims to retrieve images based on a multimodal query.<n>We present CoLLM, a one-stop framework that generates triplets on-the-fly from image-caption pairs.<n>We leverage Large Language Models (LLMs) to generate joint embeddings of reference images and modification texts.
  arXiv  Detail & Related papers  (2025-03-25T17:59:50Z)
• Benchmarking Retrieval-Augmented Generation in Multi-Modal Contexts [56.7225771305861]
  This paper introduces Multi-Modal Retrieval-Augmented Generation (M$2$RAG), a benchmark designed to evaluate the effectiveness of Multi-modal Large Language Models.<n>The benchmark comprises four tasks: image captioning, multi-modal question answering, multi-modal fact verification, and image reranking.<n>To enhance the context utilization capabilities of MLLMs, we also introduce Multi-Modal Retrieval-Augmented Instruction Tuning (MM-RAIT)
  arXiv  Detail & Related papers  (2025-02-24T16:25:25Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.