Multi-Modal Retrieval using Graph Neural Networks

• URL: http://arxiv.org/abs/2010.01666v1
• Date: Sun, 4 Oct 2020 19:34:20 GMT
• Title: Multi-Modal Retrieval using Graph Neural Networks
• Authors: Aashish Kumar Misraa, Ajinkya Kale, Pranav Aggarwal, Ali Aminian
• Abstract summary: We learn a joint vision and concept embedding in the same high-dimensional space. We model the visual and concept relationships as a graph structure. We also introduce a novel inference time control, based on selective neighborhood connectivity.
• Score: 1.8911962184174562
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Most real world applications of image retrieval such as Adobe Stock, which is a marketplace for stock photography and illustrations, need a way for users to find images which are both visually (i.e. aesthetically) and conceptually (i.e. containing the same salient objects) as a query image. Learning visual-semantic representations from images is a well studied problem for image retrieval. Filtering based on image concepts or attributes is traditionally achieved with index-based filtering (e.g. on textual tags) or by re-ranking after an initial visual embedding based retrieval. In this paper, we learn a joint vision and concept embedding in the same high-dimensional space. This joint model gives the user fine-grained control over the semantics of the result set, allowing them to explore the catalog of images more rapidly. We model the visual and concept relationships as a graph structure, which captures the rich information through node neighborhood. This graph structure helps us learn multi-modal node embeddings using Graph Neural Networks. We also introduce a novel inference time control, based on selective neighborhood connectivity allowing the user control over the retrieval algorithm. We evaluate these multi-modal embeddings quantitatively on the downstream relevance task of image retrieval on MS-COCO dataset and qualitatively on MS-COCO and an Adobe Stock dataset.

Related papers

• Composing Object Relations and Attributes for Image-Text Matching [70.47747937665987]
  This work introduces a dual-encoder image-text matching model, leveraging a scene graph to represent captions with nodes for objects and attributes interconnected by relational edges. Our model efficiently encodes object-attribute and object-object semantic relations, resulting in a robust and fast-performing system.
  arXiv  Detail & Related papers  (2024-06-17T17:56:01Z)
• Enhancing Historical Image Retrieval with Compositional Cues [3.2276097734075426]
  We introduce a crucial factor from computational aesthetics, namely image composition, into this topic. By explicitly integrating composition-related information extracted by CNN into the designed retrieval model, our method considers both the image's composition rules and semantic information.
  arXiv  Detail & Related papers  (2024-03-21T10:51:19Z)
• Masked Contrastive Graph Representation Learning for Age Estimation [44.96502862249276]
  This paper utilizes the property of graph representation learning in dealing with image redundancy information. We propose a novel Masked Contrastive Graph Representation Learning (MCGRL) method for age estimation. Experimental results on real-world face image datasets demonstrate the superiority of our proposed method over other state-of-the-art age estimation approaches.
  arXiv  Detail & Related papers  (2023-06-16T15:53:21Z)
• Graph Neural Networks in Vision-Language Image Understanding: A Survey [6.813036707969848]
  2D image understanding is a complex problem within computer vision. It holds the key to providing human-level scene comprehension. In recent years graph neural networks (GNNs) have become a standard component of many 2D image understanding pipelines.
  arXiv  Detail & Related papers  (2023-03-07T09:56:23Z)
• Deep Image Deblurring: A Survey [165.32391279761006]
  Deblurring is a classic problem in low-level computer vision, which aims to recover a sharp image from a blurred input image. Recent advances in deep learning have led to significant progress in solving this problem.
  arXiv  Detail & Related papers  (2022-01-26T01:31:30Z)
• Boosting Entity-aware Image Captioning with Multi-modal Knowledge Graph [96.95815946327079]
  It is difficult to learn the association between named entities and visual cues due to the long-tail distribution of named entities. We propose a novel approach that constructs a multi-modal knowledge graph to associate the visual objects with named entities.
  arXiv  Detail & Related papers  (2021-07-26T05:50:41Z)
• Exploiting the relationship between visual and textual features in social networks for image classification with zero-shot deep learning [0.0]
  In this work, we propose a classifier ensemble based on the transferable learning capabilities of the CLIP neural network architecture. Our experiments, based on image classification tasks according to the labels of the Places dataset, are performed by first considering only the visual part. Considering the associated texts to the images can help to improve the accuracy depending on the goal.
  arXiv  Detail & Related papers  (2021-07-08T10:54:59Z)
• Tensor Composition Net for Visual Relationship Prediction [115.14829858763399]
  We present a novel Composition Network (TCN) to predict visual relationships in images. The key idea of our TCN is to exploit the low rank property of the visual relationship tensor. We show our TCN's image-level visual relationship prediction provides a simple and efficient mechanism for relation-based image retrieval.
  arXiv  Detail & Related papers  (2020-12-10T06:27:20Z)
• Using Text to Teach Image Retrieval [47.72498265721957]
  We build on the concept of image manifold to represent the feature space of images, learned via neural networks, as a graph. We augment the manifold samples with geometrically aligned text, thereby using a plethora of sentences to teach us about images. The experimental results show that the joint embedding manifold is a robust representation, allowing it to be a better basis to perform image retrieval.
  arXiv  Detail & Related papers  (2020-11-19T16:09:14Z)
• Improving Image Captioning with Better Use of Captions [65.39641077768488]
  We present a novel image captioning architecture to better explore semantics available in captions and leverage that to enhance both image representation and caption generation. Our models first construct caption-guided visual relationship graphs that introduce beneficial inductive bias using weakly supervised multi-instance learning. During generation, the model further incorporates visual relationships using multi-task learning for jointly predicting word and object/predicate tag sequences.
  arXiv  Detail & Related papers  (2020-06-21T14:10:47Z)
• High-Order Information Matters: Learning Relation and Topology for Occluded Person Re-Identification [84.43394420267794]
  We propose a novel framework by learning high-order relation and topology information for discriminative features and robust alignment. Our framework significantly outperforms state-of-the-art by6.5%mAP scores on Occluded-Duke dataset.
  arXiv  Detail & Related papers  (2020-03-18T12:18:35Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.