FineCIR: Explicit Parsing of Fine-Grained Modification Semantics for Composed Image Retrieval

• URL: http://arxiv.org/abs/2503.21309v1
• Date: Thu, 27 Mar 2025 09:34:21 GMT
• Title: FineCIR: Explicit Parsing of Fine-Grained Modification Semantics for Composed Image Retrieval
• Authors: Zixu Li, Zhiheng Fu, Yupeng Hu, Zhiwei Chen, Haokun Wen, Liqiang Nie,
• Abstract summary: Composed Image Retrieval (CIR) facilitates image retrieval through a multimodal query consisting of a reference image and modification text.<n>Existing CIR datasets predominantly employ coarse-grained modification text (CoarseMT), which inadequately captures fine-grained retrieval intents.<n>We develop a robust fine-grained CIR data annotation pipeline that minimizes imprecise positive samples and enhances CIR systems' ability to discern modification intents accurately.
• Score: 48.63758678240123
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Composed Image Retrieval (CIR) facilitates image retrieval through a multimodal query consisting of a reference image and modification text. The reference image defines the retrieval context, while the modification text specifies desired alterations. However, existing CIR datasets predominantly employ coarse-grained modification text (CoarseMT), which inadequately captures fine-grained retrieval intents. This limitation introduces two key challenges: (1) ignoring detailed differences leads to imprecise positive samples, and (2) greater ambiguity arises when retrieving visually similar images. These issues degrade retrieval accuracy, necessitating manual result filtering or repeated queries. To address these limitations, we develop a robust fine-grained CIR data annotation pipeline that minimizes imprecise positive samples and enhances CIR systems' ability to discern modification intents accurately. Using this pipeline, we refine the FashionIQ and CIRR datasets to create two fine-grained CIR datasets: Fine-FashionIQ and Fine-CIRR. Furthermore, we introduce FineCIR, the first CIR framework explicitly designed to parse the modification text. FineCIR effectively captures fine-grained modification semantics and aligns them with ambiguous visual entities, enhancing retrieval precision. Extensive experiments demonstrate that FineCIR consistently outperforms state-of-the-art CIR baselines on both fine-grained and traditional CIR benchmark datasets. Our FineCIR code and fine-grained CIR datasets are available at https://github.com/SDU-L/FineCIR.git.

Related papers

• CoLLM: A Large Language Model for Composed Image Retrieval [76.29725148964368]
  Composed Image Retrieval (CIR) is a complex task that aims to retrieve images based on a multimodal query.<n>We present CoLLM, a one-stop framework that generates triplets on-the-fly from image-caption pairs.<n>We leverage Large Language Models (LLMs) to generate joint embeddings of reference images and modification texts.
  arXiv  Detail & Related papers  (2025-03-25T17:59:50Z)
• Fine-grained Textual Inversion Network for Zero-Shot Composed Image Retrieval [60.20835288280572]
  We propose a novel Fine-grained Textual Inversion Network for ZS-CIR, named FTI4CIR.<n> FTI4CIR comprises two main components: fine-grained pseudo-word token mapping and tri-wise caption-based semantic regularization.
  arXiv  Detail & Related papers  (2025-03-25T02:51:25Z)
• good4cir: Generating Detailed Synthetic Captions for Composed Image Retrieval [10.156187875858995]
  Composed image retrieval (CIR) enables users to search images using a reference image combined with textual modifications.<n>We introduce good4cir, a structured pipeline leveraging vision-language models to generate high-quality synthetic annotations.<n>Results demonstrate improved retrieval accuracy for CIR models trained on our pipeline-generated datasets.
  arXiv  Detail & Related papers  (2025-03-22T22:33:56Z)
• iSEARLE: Improving Textual Inversion for Zero-Shot Composed Image Retrieval [26.101116761577796]
  Composed Image Retrieval (CIR) aims to retrieve target images visually similar to the reference one while incorporating the changes specified in the relative caption. We introduce a new task, Zero-Shot CIR (ZS-CIR), that addresses CIR without the need for a labeled training dataset. We present an open-domain benchmarking dataset named CIRCO, where each query is labeled with multiple ground truths and a semantic categorization.
  arXiv  Detail & Related papers  (2024-05-05T14:39:06Z)
• Spherical Linear Interpolation and Text-Anchoring for Zero-shot Composed Image Retrieval [43.47770490199544]
  Composed Image Retrieval (CIR) is a complex task that retrieves images using a query, which is configured with an image and a caption. We introduce a novel ZS-CIR method that uses Spherical Linear Interpolation (Slerp) to directly merge image and text representations. We also introduce Text-Anchored-Tuning (TAT), a method that fine-tunes the image encoder while keeping the text encoder fixed.
  arXiv  Detail & Related papers  (2024-05-01T15:19:54Z)
• Visual Delta Generator with Large Multi-modal Models for Semi-supervised Composed Image Retrieval [50.72924579220149]
  Composed Image Retrieval (CIR) is a task that retrieves images similar to a query, based on a provided textual modification. Current techniques rely on supervised learning for CIR models using labeled triplets of the reference image, text, target image. We propose a new semi-supervised CIR approach where we search for a reference and its related target images in auxiliary data.
  arXiv  Detail & Related papers  (2024-04-23T21:00:22Z)
• Sentence-level Prompts Benefit Composed Image Retrieval [69.78119883060006]
  Composed image retrieval (CIR) is the task of retrieving specific images by using a query that involves both a reference image and a relative caption. We propose to leverage pretrained V-L models, e.g., BLIP-2, to generate sentence-level prompts. Our proposed method performs favorably against the state-of-the-art CIR methods on the Fashion-IQ and CIRR datasets.
  arXiv  Detail & Related papers  (2023-10-09T07:31:44Z)
• Zero-Shot Composed Image Retrieval with Textual Inversion [28.513594970580396]
  Composed Image Retrieval (CIR) aims to retrieve a target image based on a query composed of a reference image and a relative caption. We propose a new task, Zero-Shot CIR (ZS-CIR), that aims to address CIR without requiring a labeled training dataset.
  arXiv  Detail & Related papers  (2023-03-27T14:31:25Z)
• Pic2Word: Mapping Pictures to Words for Zero-shot Composed Image Retrieval [84.11127588805138]
  Composed Image Retrieval (CIR) combines a query image with text to describe their intended target. Existing methods rely on supervised learning of CIR models using labeled triplets consisting of the query image, text specification, and the target image. We propose Zero-Shot Composed Image Retrieval (ZS-CIR), whose goal is to build a CIR model without requiring labeled triplets for training.
  arXiv  Detail & Related papers  (2023-02-06T19:40:04Z)
• Edge-guided Multi-domain RGB-to-TIR image Translation for Training Vision Tasks with Challenging Labels [12.701191873813583]
  The insufficient number of annotated thermal infrared (TIR) image datasets hinders TIR image-based deep learning networks to have comparable performances to that of RGB. We propose a modified multidomain RGB to TIR image translation model focused on edge preservation to employ annotated RGB images with challenging labels. We have enabled the supervised learning of deep TIR image-based optical flow estimation and object detection that ameliorated in end point error by 56.5% on average and the best object detection mAP of 23.9% respectively.
  arXiv  Detail & Related papers  (2023-01-30T06:44:38Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.