If At First You Don't Succeed: Test Time Re-ranking for Zero-shot, Cross-domain Retrieval

• URL: http://arxiv.org/abs/2303.17703v2
• Date: Fri, 28 Feb 2025 09:02:21 GMT
• Title: If At First You Don't Succeed: Test Time Re-ranking for Zero-shot, Cross-domain Retrieval
• Authors: Finlay G. C. Hudson, William A. P. Smith,
• Abstract summary: Iterative Cluster-free Re-ranking process leverages gallery-gallery feature information to establish semantic links between query and gallery images.<n>When combined with a carefully chosen Vision Transformer backbone and combination of zero-shot retrieval losses, our approach yields state-of-the-art results.
• Score: 15.272149101494005
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: In this paper, we introduce a novel method for zero-shot, cross-domain image retrieval. Our key contribution is a test-time Iterative Cluster-free Re-ranking process that leverages gallery-gallery feature information to establish semantic links between query and gallery images. This enables the retrieval of relevant images even when they do not exhibit similar visual features but share underlying semantic concepts. This can be combined with any pre-existing cross-domain feature extraction backbone to improve retrieval performance. However, when combined with a carefully chosen Vision Transformer backbone and combination of zero-shot retrieval losses, our approach yields state-of-the-art results on the Sketchy, TU-Berlin and QuickDraw sketch-based retrieval benchmarks. We show that our re-ranking also improves performance with other backbones and outperforms other re-ranking methods applied with our backbone. Importantly, unlike many previous methods, none of the components in our approach are engineered specifically towards the sketch-based image retrieval task - it can be generally applied to any cross-domain, zero-shot retrieval task. We therefore also present new results on zero-shot cartoon-to-photo and art-to-product retrieval using the Office-Home dataset. Project page: finlay-hudson.github.io/icfrr, code available at: github.com/finlay-hudson/ICFRR

Related papers

• Visual Re-Ranking with Non-Visual Side Information [21.7701816159427]
  We propose a graph neural network-based re-ranking method that can leverage other types of available side information. In experiments we show significant improvement not only on image retrieval metrics, but also for the downstream visual localization task.
  arXiv  Detail & Related papers  (2025-04-15T12:37:16Z)
• You'll Never Walk Alone: A Sketch and Text Duet for Fine-Grained Image Retrieval [120.49126407479717]
  We introduce a novel compositionality framework, effectively combining sketches and text using pre-trained CLIP models. Our system extends to novel applications in composed image retrieval, domain transfer, and fine-grained generation.
  arXiv  Detail & Related papers  (2024-03-12T00:27:18Z)
• Zero-shot sketch-based remote sensing image retrieval based on multi-level and attention-guided tokenization [8.678089483952474]
  This study introduces a novel zero-shot, sketch-based retrieval method for remote sensing images. It employs multi-level feature extraction, self-attention-guided tokenization and filtering, and cross-modality attention update. Our method significantly outperforms existing sketch-based remote sensing image retrieval techniques.
  arXiv  Detail & Related papers  (2024-02-03T13:11:14Z)
• Advancing Image Retrieval with Few-Shot Learning and Relevance Feedback [5.770351255180495]
  Image Retrieval with Relevance Feedback (IRRF) involves iterative human interaction during the retrieval process. We propose a new scheme based on a hyper-network, that is tailored to the task and facilitates swift adjustment to user feedback. We show that our method can attain SoTA results in few-shot one-class classification and reach comparable results in binary classification task of few-shot open-set recognition.
  arXiv  Detail & Related papers  (2023-12-18T10:20:28Z)
• Symmetrical Bidirectional Knowledge Alignment for Zero-Shot Sketch-Based Image Retrieval [69.46139774646308]
  This paper studies the problem of zero-shot sketch-based image retrieval (ZS-SBIR) It aims to use sketches from unseen categories as queries to match the images of the same category. We propose a novel Symmetrical Bidirectional Knowledge Alignment for zero-shot sketch-based image retrieval (SBKA)
  arXiv  Detail & Related papers  (2023-12-16T04:50:34Z)
• Sketch-an-Anchor: Sub-epoch Fast Model Adaptation for Zero-shot Sketch-based Image Retrieval [1.52292571922932]
  Sketch-an-Anchor is a novel method to train state-of-the-art Zero-shot Image Retrieval (ZSSBIR) models in under an epoch. Our fast-converging model keeps the single-domain performance while learning to extract similar representations from sketches.
  arXiv  Detail & Related papers  (2023-03-29T15:00:02Z)
• Distribution Aligned Feature Clustering for Zero-Shot Sketch-Based Image Retrieval [18.81230334624234]
  This paper tackles the challenges from a new perspective: utilizing gallery image features. We propose a Cluster-then-Retrieve (ClusterRetri) method that performs clustering on the gallery images and uses the cluster centroids as proxies for retrieval. Despite its simplicity, our proposed method outperforms the state-of-the-art methods by a large margin on popular datasets.
  arXiv  Detail & Related papers  (2023-01-17T03:58:12Z)
• ARTEMIS: Attention-based Retrieval with Text-Explicit Matching and Implicit Similarity [16.550790981646276]
  Current approaches combine the features of each of the two elements of the query into a single representation. Our work aims at shedding new light on the task by looking at it through the prism of two familiar and related frameworks: text-to-image and image-to-image retrieval.
  arXiv  Detail & Related papers  (2022-03-15T17:29:20Z)
• Learning Co-segmentation by Segment Swapping for Retrieval and Discovery [67.6609943904996]
  The goal of this work is to efficiently identify visually similar patterns from a pair of images. We generate synthetic training pairs by selecting object segments in an image and copy-pasting them into another image. We show our approach provides clear improvements for artwork details retrieval on the Brueghel dataset.
  arXiv  Detail & Related papers  (2021-10-29T16:51:16Z)
• Contextual Similarity Aggregation with Self-attention for Visual Re-ranking [96.55393026011811]
  We propose a visual re-ranking method by contextual similarity aggregation with self-attention. We conduct comprehensive experiments on four benchmark datasets to demonstrate the generality and effectiveness of our proposed visual re-ranking method.
  arXiv  Detail & Related papers  (2021-10-26T06:20:31Z)
• Understanding Image Retrieval Re-Ranking: A Graph Neural Network Perspective [52.96911968968888]
  In this paper, we demonstrate that re-ranking can be reformulated as a high-parallelism Graph Neural Network (GNN) function. On the Market-1501 dataset, we accelerate the re-ranking processing from 89.2s to 9.4ms with one K40m GPU, facilitating the real-time post-processing.
  arXiv  Detail & Related papers  (2020-12-14T15:12:36Z)
• Semantically Tied Paired Cycle Consistency for Any-Shot Sketch-based Image Retrieval [55.29233996427243]
  Low-shot sketch-based image retrieval is an emerging task in computer vision. In this paper, we address any-shot, i.e. zero-shot and few-shot, sketch-based image retrieval (SBIR) tasks. For solving these tasks, we propose a semantically aligned cycle-consistent generative adversarial network (SEM-PCYC) Our results demonstrate a significant boost in any-shot performance over the state-of-the-art on the extended version of the Sketchy, TU-Berlin and QuickDraw datasets.
  arXiv  Detail & Related papers  (2020-06-20T22:43:53Z)
• Geometrically Mappable Image Features [85.81073893916414]
  Vision-based localization of an agent in a map is an important problem in robotics and computer vision. We propose a method that learns image features targeted for image-retrieval-based localization.
  arXiv  Detail & Related papers  (2020-03-21T15:36:38Z)
• Image Matching across Wide Baselines: From Paper to Practice [80.9424750998559]
  We introduce a comprehensive benchmark for local features and robust estimation algorithms. Our pipeline's modular structure allows easy integration, configuration, and combination of different methods. We show that with proper settings, classical solutions may still outperform the perceived state of the art.
  arXiv  Detail & Related papers  (2020-03-03T15:20:57Z)
• Sketch Less for More: On-the-Fly Fine-Grained Sketch Based Image Retrieval [203.2520862597357]
  Fine-grained sketch-based image retrieval (FG-SBIR) addresses the problem of retrieving a particular photo instance given a user's query sketch. We reformulate the conventional FG-SBIR framework to tackle these challenges. We propose an on-the-fly design that starts retrieving as soon as the user starts drawing.
  arXiv  Detail & Related papers  (2020-02-24T15:36:02Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.