ConText-CIR: Learning from Concepts in Text for Composed Image Retrieval

• URL: http://arxiv.org/abs/2505.20764v1
• Date: Tue, 27 May 2025 06:09:57 GMT
• Title: ConText-CIR: Learning from Concepts in Text for Composed Image Retrieval
• Authors: Eric Xing, Pranavi Kolouju, Robert Pless, Abby Stylianou, Nathan Jacobs,
• Abstract summary: Composed image retrieval (CIR) is the task of retrieving a target image specified by a query image and a relative text.<n>We introduce a framework, ConText-CIR, trained with a Text Concept-Consistency loss.<n>We show that these components together enable stronger performance on CIR tasks.
• Score: 10.156187875858995
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Composed image retrieval (CIR) is the task of retrieving a target image specified by a query image and a relative text that describes a semantic modification to the query image. Existing methods in CIR struggle to accurately represent the image and the text modification, resulting in subpar performance. To address this limitation, we introduce a CIR framework, ConText-CIR, trained with a Text Concept-Consistency loss that encourages the representations of noun phrases in the text modification to better attend to the relevant parts of the query image. To support training with this loss function, we also propose a synthetic data generation pipeline that creates training data from existing CIR datasets or unlabeled images. We show that these components together enable stronger performance on CIR tasks, setting a new state-of-the-art in composed image retrieval in both the supervised and zero-shot settings on multiple benchmark datasets, including CIRR and CIRCO. Source code, model checkpoints, and our new datasets are available at https://github.com/mvrl/ConText-CIR.

Related papers

• Multimodal Reasoning Agent for Zero-Shot Composed Image Retrieval [52.709090256954276]
  Zero-Shot Composed Image Retrieval (ZS-CIR) aims to retrieve target images given a compositional query.<n>We propose a novel framework by employing a Multimodal Reasoning Agent (MRA) for ZS-CIR.
  arXiv  Detail & Related papers  (2025-05-26T13:17:50Z)
• FineCIR: Explicit Parsing of Fine-Grained Modification Semantics for Composed Image Retrieval [48.63758678240123]
  Composed Image Retrieval (CIR) facilitates image retrieval through a multimodal query consisting of a reference image and modification text.<n>Existing CIR datasets predominantly employ coarse-grained modification text (CoarseMT), which inadequately captures fine-grained retrieval intents.<n>We develop a robust fine-grained CIR data annotation pipeline that minimizes imprecise positive samples and enhances CIR systems' ability to discern modification intents accurately.
  arXiv  Detail & Related papers  (2025-03-27T09:34:21Z)
• Fine-grained Textual Inversion Network for Zero-Shot Composed Image Retrieval [60.20835288280572]
  We propose a novel Fine-grained Textual Inversion Network for ZS-CIR, named FTI4CIR.<n> FTI4CIR comprises two main components: fine-grained pseudo-word token mapping and tri-wise caption-based semantic regularization.
  arXiv  Detail & Related papers  (2025-03-25T02:51:25Z)
• Zero-shot Composed Image Retrieval Considering Query-target Relationship Leveraging Masked Image-text Pairs [44.48400303207482]
  The objective of a zero-shot composed image retrieval (CIR) is to retrieve the target image using a query image and a query text. Existing methods use a textual inversion network to convert the query image into a pseudo word to compose the image and text. We propose a novel zero-shot CIR method that is trained end-to-end using masked image-text pairs.
  arXiv  Detail & Related papers  (2024-06-27T02:10:30Z)
• iSEARLE: Improving Textual Inversion for Zero-Shot Composed Image Retrieval [26.101116761577796]
  Composed Image Retrieval (CIR) aims to retrieve target images visually similar to the reference one while incorporating the changes specified in the relative caption. We introduce a new task, Zero-Shot CIR (ZS-CIR), that addresses CIR without the need for a labeled training dataset. We present an open-domain benchmarking dataset named CIRCO, where each query is labeled with multiple ground truths and a semantic categorization.
  arXiv  Detail & Related papers  (2024-05-05T14:39:06Z)
• Spherical Linear Interpolation and Text-Anchoring for Zero-shot Composed Image Retrieval [43.47770490199544]
  Composed Image Retrieval (CIR) is a complex task that retrieves images using a query, which is configured with an image and a caption. We introduce a novel ZS-CIR method that uses Spherical Linear Interpolation (Slerp) to directly merge image and text representations. We also introduce Text-Anchored-Tuning (TAT), a method that fine-tunes the image encoder while keeping the text encoder fixed.
  arXiv  Detail & Related papers  (2024-05-01T15:19:54Z)
• Language-only Efficient Training of Zero-shot Composed Image Retrieval [46.93446891158521]
  Composed image retrieval (CIR) task takes a composed query of image and text, aiming to search relative images for both conditions. We propose a novel CIR framework, only using language for its training. Our LinCIR (Language-only training for CIR) can be trained only with text datasets by a novel self-supervision named self-masking projection (SMP)
  arXiv  Detail & Related papers  (2023-12-04T16:22:06Z)
• Sentence-level Prompts Benefit Composed Image Retrieval [69.78119883060006]
  Composed image retrieval (CIR) is the task of retrieving specific images by using a query that involves both a reference image and a relative caption. We propose to leverage pretrained V-L models, e.g., BLIP-2, to generate sentence-level prompts. Our proposed method performs favorably against the state-of-the-art CIR methods on the Fashion-IQ and CIRR datasets.
  arXiv  Detail & Related papers  (2023-10-09T07:31:44Z)
• Zero-Shot Composed Image Retrieval with Textual Inversion [28.513594970580396]
  Composed Image Retrieval (CIR) aims to retrieve a target image based on a query composed of a reference image and a relative caption. We propose a new task, Zero-Shot CIR (ZS-CIR), that aims to address CIR without requiring a labeled training dataset.
  arXiv  Detail & Related papers  (2023-03-27T14:31:25Z)
• Pic2Word: Mapping Pictures to Words for Zero-shot Composed Image Retrieval [84.11127588805138]
  Composed Image Retrieval (CIR) combines a query image with text to describe their intended target. Existing methods rely on supervised learning of CIR models using labeled triplets consisting of the query image, text specification, and the target image. We propose Zero-Shot Composed Image Retrieval (ZS-CIR), whose goal is to build a CIR model without requiring labeled triplets for training.
  arXiv  Detail & Related papers  (2023-02-06T19:40:04Z)
• BOSS: Bottom-up Cross-modal Semantic Composition with Hybrid Counterfactual Training for Robust Content-based Image Retrieval [61.803481264081036]
  Content-Based Image Retrieval (CIR) aims to search for a target image by concurrently comprehending the composition of an example image and a complementary text. We tackle this task by a novel underlinetextbfBottom-up crunderlinetextbfOss-modal underlinetextbfSemantic compounderlinetextbfSition (textbfBOSS) with Hybrid Counterfactual Training framework.
  arXiv  Detail & Related papers  (2022-07-09T07:14:44Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.