A Multimodal Approach for Cross-Domain Image Retrieval

• URL: http://arxiv.org/abs/2403.15152v1
• Date: Fri, 22 Mar 2024 12:08:16 GMT
• Title: A Multimodal Approach for Cross-Domain Image Retrieval
• Authors: Lucas Iijima, Tania Stathaki,
• Abstract summary: This paper focuses on Cross-Domain Image Retrieval (CDIR) which can be used as an additional tool to inspect collections of generated images. An ideal retrieval system would be able to generalize to unseen complex images from multiple domains. We propose a novel caption-matching approach that leverages multimodal language-vision architectures pre-trained on large datasets.
• Score: 6.24302896438145
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Image generators are gaining vast amount of popularity and have rapidly changed how digital content is created. With the latest AI technology, millions of high quality images are being generated by the public, which are constantly motivating the research community to push the limits of generative models to create more complex and realistic images. This paper focuses on Cross-Domain Image Retrieval (CDIR) which can be used as an additional tool to inspect collections of generated images by determining the level of similarity between images in a dataset. An ideal retrieval system would be able to generalize to unseen complex images from multiple domains (e.g., photos, drawings and paintings). To address this goal, we propose a novel caption-matching approach that leverages multimodal language-vision architectures pre-trained on large datasets. The method is tested on DomainNet and Office-Home datasets and consistently achieves state-of-the-art performance over the latest approaches in the literature for cross-domain image retrieval. In order to verify the effectiveness with AI-generated images, the method was also put to test with a database composed by samples collected from Midjourney, which is a widely used generative platform for content creation.

Related papers

• Graph-Based Cross-Domain Knowledge Distillation for Cross-Dataset Text-to-Image Person Retrieval [25.760438764541867]
  Video surveillance systems are crucial components for ensuring public safety and management in smart city. Text-to-image person retrieval aims to retrieve the target person from an image gallery that best matches the given text description. Most existing text-to-image person retrieval methods are trained in a supervised manner that requires sufficient labeled data in the target domain.
  arXiv  Detail & Related papers  (2025-01-25T03:24:34Z)
• WIDIn: Wording Image for Domain-Invariant Representation in Single-Source Domain Generalization [63.98650220772378]
  We present WIDIn, Wording Images for Domain-Invariant representation, to disentangle discriminative visual representation. We first estimate the language embedding with fine-grained alignment, which can be used to adaptively identify and then remove domain-specific counterpart. We show that WIDIn can be applied to both pretrained vision-language models like CLIP, and separately trained uni-modal models like MoCo and BERT.
  arXiv  Detail & Related papers  (2024-05-28T17:46:27Z)
• Domain Agnostic Image-to-image Translation using Low-Resolution Conditioning [6.470760375991825]
  We propose a domain-agnostic i2i method for fine-grained problems, where the domains are related. We present a novel approach that relies on training the generative model to produce images that both share distinctive information of the associated source image. We validate our method on the CelebA-HQ and AFHQ datasets by demonstrating improvements in terms of visual quality.
  arXiv  Detail & Related papers  (2023-05-08T19:58:49Z)
• Unsupervised Domain Adaptation for Semantic Segmentation using One-shot Image-to-Image Translation via Latent Representation Mixing [9.118706387430883]
  We propose a new unsupervised domain adaptation method for the semantic segmentation of very high resolution images. An image-to-image translation paradigm is proposed, based on an encoder-decoder principle where latent content representations are mixed across domains. Cross-city comparative experiments have shown that the proposed method outperforms state-of-the-art domain adaptation methods.
  arXiv  Detail & Related papers  (2022-12-07T18:16:17Z)
• Unsupervised Domain Generalization by Learning a Bridge Across Domains [78.855606355957]
  Unsupervised Domain Generalization (UDG) setup has no training supervision in neither source nor target domains. Our approach is based on self-supervised learning of a Bridge Across Domains (BrAD) - an auxiliary bridge domain accompanied by a set of semantics preserving visual (image-to-image) mappings to BrAD from each of the training domains. We show how using an edge-regularized BrAD our approach achieves significant gains across multiple benchmarks and a range of tasks, including UDG, Few-shot UDA, and unsupervised generalization across multi-domain datasets.
  arXiv  Detail & Related papers  (2021-12-04T10:25:45Z)
• Semantic Consistency in Image-to-Image Translation for Unsupervised Domain Adaptation [22.269565708490465]
  Unsupervised Domain Adaptation (UDA) aims to adapt models trained on a source domain to a new target domain where no labelled data is available. We propose a semantically consistent image-to-image translation method in combination with a consistency regularisation method for UDA.
  arXiv  Detail & Related papers  (2021-11-05T14:22:20Z)
• Towards Unsupervised Sketch-based Image Retrieval [126.77787336692802]
  We introduce a novel framework that simultaneously performs unsupervised representation learning and sketch-photo domain alignment. Our framework achieves excellent performance in the new unsupervised setting, and performs comparably or better than state-of-the-art in the zero-shot setting.
  arXiv  Detail & Related papers  (2021-05-18T02:38:22Z)
• Crossing-Domain Generative Adversarial Networks for Unsupervised Multi-Domain Image-to-Image Translation [12.692904507625036]
  We propose a general framework for unsupervised image-to-image translation across multiple domains. Our proposed framework consists of a pair of encoders along with a pair of GANs which learns high-level features across different domains to generate diverse and realistic samples from.
  arXiv  Detail & Related papers  (2020-08-27T01:54:07Z)
• Weakly supervised cross-domain alignment with optimal transport [102.8572398001639]
  Cross-domain alignment between image objects and text sequences is key to many visual-language tasks. This paper investigates a novel approach for the identification and optimization of fine-grained semantic similarities between image and text entities.
  arXiv  Detail & Related papers  (2020-08-14T22:48:36Z)
• Cross-domain Correspondence Learning for Exemplar-based Image Translation [59.35767271091425]
  We present a framework for exemplar-based image translation, which synthesizes a photo-realistic image from the input in a distinct domain. The output has the style (e.g., color, texture) in consistency with the semantically corresponding objects in the exemplar. We show that our method is superior to state-of-the-art methods in terms of image quality significantly.
  arXiv  Detail & Related papers  (2020-04-12T09:10:57Z)
• CrDoCo: Pixel-level Domain Transfer with Cross-Domain Consistency [119.45667331836583]
  Unsupervised domain adaptation algorithms aim to transfer the knowledge learned from one domain to another. We present a novel pixel-wise adversarial domain adaptation algorithm.
  arXiv  Detail & Related papers  (2020-01-09T19:00:35Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.