Revisiting Human-in-the-Loop Object Retrieval with Pre-Trained Vision Transformers

• URL: http://arxiv.org/abs/2604.00809v1
• Date: Wed, 01 Apr 2026 12:18:17 GMT
• Title: Revisiting Human-in-the-Loop Object Retrieval with Pre-Trained Vision Transformers
• Abstract summary: Human-in-the-Loop Object Retrieval is a task that consists of iteratively retrieving images containing objects of a class-of-interest.<n>We revisit the task by leveraging pre-trained ViT representations and addressing key design questions.
• Score: 1.8631279231581521
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Building on existing approaches, we revisit Human-in-the-Loop Object Retrieval, a task that consists of iteratively retrieving images containing objects of a class-of-interest, specified by a user-provided query. Starting from a large unlabeled image collection, the aim is to rapidly identify diverse instances of an object category relying solely on the initial query and the user's Relevance Feedback, with no prior labels. The retrieval process is formulated as a binary classification task, where the system continuously learns to distinguish between relevant and non-relevant images to the query, through iterative user interaction. This interaction is guided by an Active Learning loop: at each iteration, the system selects informative samples for user annotation, thereby refining the retrieval performance. This task is particularly challenging in multi-object datasets, where the object of interest may occupy only a small region of the image within a complex, cluttered scene. Unlike object-centered settings where global descriptors often suffice, multi-object images require more adapted, localized descriptors. In this work, we formulate and revisit the Human-in-the-Loop Object Retrieval task by leveraging pre-trained ViT representations, and addressing key design questions, including which object instances to consider in an image, what form the annotations should take, how Active Selection should be applied, and which representation strategies best capture the object's features. We compare several representation strategies across multi-object datasets highlighting trade-offs between capturing the global context and focusing on fine-grained local object details. Our results offer practical insights for the design of effective interactive retrieval pipelines based on Active Learning for object class retrieval.

Related papers

• VocaDet: Sample-Driven Open-Vocabulary Object Detection and Segmentation via Visual Tokenization and Vector Database Retrieval [4.484170173286332]
  We propose VocaDet, a sample-driven open-vocabulary object detection and segmentation framework.<n>We transform continuous visual representations into discrete visual vocabularies and perform efficient retrieval-based recognition through a scalable vector database.<n> Experiments on the UA-DETRAC dataset demonstrate that VocaDet achieves effective open-vocabulary detection performance without conventional detector training.
  arXiv  Detail & Related papers  (2026-07-09T14:35:37Z)
• Composed Object Retrieval: Object-level Retrieval via Composed Expressions [71.47650333199628]
  Composed Object Retrieval (COR) is a brand-new task that goes beyond image-level retrieval to achieve object-level precision.<n>We construct COR127K, the first large-scale COR benchmark that contains 127,166 retrieval triplets with various semantic transformations in 408 categories.<n>We also present CORE, a unified end-to-end model that integrates reference region encoding, adaptive visual-textual interaction, and region-level contrastive learning.
  arXiv  Detail & Related papers  (2025-08-06T13:11:40Z)
• Multiple Object Stitching for Unsupervised Representation Learning [11.087735229999817]
  We propose a method, Multiple Object Stitching, to refine the unsupervised representation for multi-object images.<n>Our method provides additional object correspondences between multi-object images without human annotations.<n> Experimental results on ImageNet, CIFAR and COCO datasets demonstrate that our proposed method achieves the leading unsupervised representation performance.
  arXiv  Detail & Related papers  (2025-06-09T02:28:21Z)
• Find your Needle: Small Object Image Retrieval via Multi-Object Attention Optimization [17.18636164860152]
  We address the challenge of Small Object Image Retrieval (SoIR), where the goal is to retrieve images containing a specific small object, in a cluttered scene.<n>Key challenge is constructing a single image descriptor, for scalable and efficient search, that effectively represents all objects in the image.<n>We introduce Multi-object Attention Optimization (MaO), a novel retrieval framework which incorporates a dedicated multi-object pre-training phase.
  arXiv  Detail & Related papers  (2025-03-10T08:27:02Z)
• ResVG: Enhancing Relation and Semantic Understanding in Multiple Instances for Visual Grounding [42.10086029931937]
  Visual grounding aims to localize the object referred to in an image based on a natural language query. Existing methods demonstrate a significant performance drop when there are multiple distractions in an image. We propose a novel approach, the Relation and Semantic-sensitive Visual Grounding (ResVG) model, to address this issue.
  arXiv  Detail & Related papers  (2024-08-29T07:32:01Z)
• Object-Centric Open-Vocabulary Image-Retrieval with Aggregated Features [11.112981323262337]
  We present a simple yet effective approach to object-centric open-vocabulary image retrieval.<n>Our approach aggregates dense embeddings extracted from CLIP into a compact representation.<n>We show the effectiveness of our scheme to the task by achieving significantly better results than global feature approaches on three datasets.
  arXiv  Detail & Related papers  (2023-09-26T15:13:09Z)
• Learning-based Relational Object Matching Across Views [63.63338392484501]
  We propose a learning-based approach which combines local keypoints with novel object-level features for matching object detections between RGB images. We train our object-level matching features based on appearance and inter-frame and cross-frame spatial relations between objects in an associative graph neural network.
  arXiv  Detail & Related papers  (2023-05-03T19:36:51Z)
• Universal Instance Perception as Object Discovery and Retrieval [90.96031157557806]
  UNI reformulates diverse instance perception tasks into a unified object discovery and retrieval paradigm. It can flexibly perceive different types of objects by simply changing the input prompts. UNI shows superior performance on 20 challenging benchmarks from 10 instance-level tasks.
  arXiv  Detail & Related papers  (2023-03-12T14:28:24Z)
• Fusing Local Similarities for Retrieval-based 3D Orientation Estimation of Unseen Objects [70.49392581592089]
  We tackle the task of estimating the 3D orientation of previously-unseen objects from monocular images. We follow a retrieval-based strategy and prevent the network from learning object-specific features. Our experiments on the LineMOD, LineMOD-Occluded, and T-LESS datasets show that our method yields a significantly better generalization to unseen objects than previous works.
  arXiv  Detail & Related papers  (2022-03-16T08:53:00Z)
• A Simple and Effective Use of Object-Centric Images for Long-Tailed Object Detection [56.82077636126353]
  We take advantage of object-centric images to improve object detection in scene-centric images. We present a simple yet surprisingly effective framework to do so. Our approach can improve the object detection (and instance segmentation) accuracy of rare objects by 50% (and 33%) relatively.
  arXiv  Detail & Related papers  (2021-02-17T17:27:21Z)
• Tasks Integrated Networks: Joint Detection and Retrieval for Image Search [99.49021025124405]
  In many real-world searching scenarios (e.g., video surveillance), the objects are seldom accurately detected or annotated. We first introduce an end-to-end Integrated Net (I-Net), which has three merits. We further propose an improved I-Net, called DC-I-Net, which makes two new contributions.
  arXiv  Detail & Related papers  (2020-09-03T03:57:50Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.