CaLa: Complementary Association Learning for Augmenting Composed Image Retrieval

• URL: http://arxiv.org/abs/2405.19149v2
• Date: Thu, 30 May 2024 13:26:43 GMT
• Title: CaLa: Complementary Association Learning for Augmenting Composed Image Retrieval
• Authors: Xintong Jiang, Yaxiong Wang, Mengjian Li, Yujiao Wu, Bingwen Hu, Xueming Qian,
• Abstract summary: Composed Image Retrieval (CIR) involves searching for target images based on an image-text pair query. We argue that CIR triplets contain additional associations beyond this primary relation. In our paper, we identify two new relations within triplets, treating each triplet as a graph node.
• Score: 15.45550686770835
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Composed Image Retrieval (CIR) involves searching for target images based on an image-text pair query. While current methods treat this as a query-target matching problem, we argue that CIR triplets contain additional associations beyond this primary relation. In our paper, we identify two new relations within triplets, treating each triplet as a graph node. Firstly, we introduce the concept of text-bridged image alignment, where the query text serves as a bridge between the query image and the target image. We propose a hinge-based cross-attention mechanism to incorporate this relation into network learning. Secondly, we explore complementary text reasoning, considering CIR as a form of cross-modal retrieval where two images compose to reason about complementary text. To integrate these perspectives effectively, we design a twin attention-based compositor. By combining these complementary associations with the explicit query pair-target image relation, we establish a comprehensive set of constraints for CIR. Our framework, CaLa (Complementary Association Learning for Augmenting Composed Image Retrieval), leverages these insights. We evaluate CaLa on CIRR and FashionIQ benchmarks with multiple backbones, demonstrating its superiority in composed image retrieval.

Related papers

• Zero-shot Composed Image Retrieval Considering Query-target Relationship Leveraging Masked Image-text Pairs [44.48400303207482]
  The objective of a zero-shot composed image retrieval (CIR) is to retrieve the target image using a query image and a query text. Existing methods use a textual inversion network to convert the query image into a pseudo word to compose the image and text. We propose a novel zero-shot CIR method that is trained end-to-end using masked image-text pairs.
  arXiv  Detail & Related papers  (2024-06-27T02:10:30Z)
• Visual Delta Generator with Large Multi-modal Models for Semi-supervised Composed Image Retrieval [50.72924579220149]
  Composed Image Retrieval (CIR) is a task that retrieves images similar to a query, based on a provided textual modification. Current techniques rely on supervised learning for CIR models using labeled triplets of the reference image, text, target image. We propose a new semi-supervised CIR approach where we search for a reference and its related target images in auxiliary data.
  arXiv  Detail & Related papers  (2024-04-23T21:00:22Z)
• Training-free Zero-shot Composed Image Retrieval with Local Concept Reranking [34.31345844296072]
  Composed image retrieval attempts to retrieve an image of interest from gallery images through a composed query of a reference image and its corresponding modified text. Most current composed image retrieval methods follow a supervised learning approach to training on a costly triplet dataset composed of a reference image, modified text, and a corresponding target image. We present a new training-free zero-shot composed image retrieval method which translates the query into explicit human-understandable text.
  arXiv  Detail & Related papers  (2023-12-14T13:31:01Z)
• Dual Relation Alignment for Composed Image Retrieval [24.812654620141778]
  We argue for the existence of two types of relations in composed image retrieval. The explicit relation pertains to the reference image & complementary text-target image. We propose a new framework for composed image retrieval, termed dual relation alignment.
  arXiv  Detail & Related papers  (2023-09-05T12:16:14Z)
• Collaborative Group: Composed Image Retrieval via Consensus Learning from Noisy Annotations [67.92679668612858]
  We propose the Consensus Network (Css-Net), inspired by the psychological concept that groups outperform individuals. Css-Net comprises two core components: (1) a consensus module with four diverse compositors, each generating distinct image-text embeddings; and (2) a Kullback-Leibler divergence loss that encourages learning of inter-compositor interactions. On benchmark datasets, particularly FashionIQ, Css-Net demonstrates marked improvements. Notably, it achieves significant recall gains, with a 2.77% increase in R@10 and 6.67% boost in R@50, underscoring its
  arXiv  Detail & Related papers  (2023-06-03T11:50:44Z)
• Pic2Word: Mapping Pictures to Words for Zero-shot Composed Image Retrieval [84.11127588805138]
  Composed Image Retrieval (CIR) combines a query image with text to describe their intended target. Existing methods rely on supervised learning of CIR models using labeled triplets consisting of the query image, text specification, and the target image. We propose Zero-Shot Composed Image Retrieval (ZS-CIR), whose goal is to build a CIR model without requiring labeled triplets for training.
  arXiv  Detail & Related papers  (2023-02-06T19:40:04Z)
• BOSS: Bottom-up Cross-modal Semantic Composition with Hybrid Counterfactual Training for Robust Content-based Image Retrieval [61.803481264081036]
  Content-Based Image Retrieval (CIR) aims to search for a target image by concurrently comprehending the composition of an example image and a complementary text. We tackle this task by a novel underlinetextbfBottom-up crunderlinetextbfOss-modal underlinetextbfSemantic compounderlinetextbfSition (textbfBOSS) with Hybrid Counterfactual Training framework.
  arXiv  Detail & Related papers  (2022-07-09T07:14:44Z)
• ARTEMIS: Attention-based Retrieval with Text-Explicit Matching and Implicit Similarity [16.550790981646276]
  Current approaches combine the features of each of the two elements of the query into a single representation. Our work aims at shedding new light on the task by looking at it through the prism of two familiar and related frameworks: text-to-image and image-to-image retrieval.
  arXiv  Detail & Related papers  (2022-03-15T17:29:20Z)
• Two-stream Hierarchical Similarity Reasoning for Image-text Matching [66.43071159630006]
  A hierarchical similarity reasoning module is proposed to automatically extract context information. Previous approaches only consider learning single-stream similarity alignment. A two-stream architecture is developed to decompose image-text matching into image-to-text level and text-to-image level similarity computation.
  arXiv  Detail & Related papers  (2022-03-10T12:56:10Z)
• Consensus-Aware Visual-Semantic Embedding for Image-Text Matching [69.34076386926984]
  Image-text matching plays a central role in bridging vision and language. Most existing approaches only rely on the image-text instance pair to learn their representations. We propose a Consensus-aware Visual-Semantic Embedding model to incorporate the consensus information.
  arXiv  Detail & Related papers  (2020-07-17T10:22:57Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.