Personalized Object Identification and Localization via In-Context Inference with Vision-Language Models

• URL: http://arxiv.org/abs/2607.00357v1
• Date: Wed, 01 Jul 2026 02:56:45 GMT
• Title: Personalized Object Identification and Localization via In-Context Inference with Vision-Language Models
• Abstract summary: We formulate a new task, personalized object identification and localization (POIL)<n>POIL aims to localize the target object instance while rejecting query images that do not contain the reference object instance.<n>IPLoc-ID first predicts a candidate bounding box and then determines whether it corresponds to the reference object instance.
• Score: 4.645610212651813
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Personalized object localization (POL) localizes an object instance in a query image based on a few reference images with bounding-box annotations and a target object label. The pioneering method, IPLoc, solves this task through in-context inference with vision-language models (VLMs). However, it assumes that the query image always contains the target object. This assumption severely limits its applicability to real-world scenarios with many irrelevant images. To address this issue, we formulate a new task, personalized object identification and localization (POIL), by positioning POL within the broader few-shot object detection framework. POIL aims to localize the target object instance while rejecting query images that do not contain the reference object instance. We also present POIL datasets constructed from public sources. We further propose an in-context algorithm named IPLoc-ID for solving POIL with VLMs. IPLoc-ID first predicts a candidate bounding box and then determines whether it corresponds to the reference object instance. We introduce a self-posed query to connect these two steps within a single autoregressive generation framework. Through ablation studies and comprehensive experiments, we show that IPLoc-ID substantially suppresses false-positive detections on negative query images while maintaining localization performance comparable to IPLoc. Overall, IPLoc-ID effectively addresses the practical instance-level POIL task, which cannot be sufficiently solved by conventional object detection, few-shot object detection, or the localization-only IPLoc method.

Related papers

• UniRef-UAV: A Multimodal Benchmark for Universal Referring in UAV Imagery [69.87589293875112]
  Unmanned aerial vehicles (UAVs) increasingly rely on visual grounding capabilities to localize task-relevant targets.<n>Existing referring expression comprehension benchmarks and methods are largely built around text-only queries and single-object outputs.<n>We introduce emphUniversal Referring, a generalized UAV referring task that jointly expands the query modality and the output cardinality.
  arXiv  Detail & Related papers  (2026-07-09T09:09:31Z)
• Teaching VLMs to Localize Specific Objects from In-context Examples [56.797110842152]
  We find that present-day Vision-Language Models (VLMs) lack a fundamental cognitive ability: learning to localize specific objects in a scene by taking into account the context.<n>This work is the first to explore and benchmark personalized few-shot localization for VLMs.
  arXiv  Detail & Related papers  (2024-11-20T13:34:22Z)
• Improving Object Detection via Local-global Contrastive Learning [27.660633883387753]
  We present a novel image-to-image translation method that specifically targets cross-domain object detection. We learn to represent objects by contrasting local-global information. This affords investigation of an under-explored challenge: obtaining performant detection, under domain shifts.
  arXiv  Detail & Related papers  (2024-10-07T14:18:32Z)
• Object-Centric Multiple Object Tracking [124.30650395969126]
  This paper proposes a video object-centric model for multiple-object tracking pipelines. It consists of an index-merge module that adapts the object-centric slots into detection outputs and an object memory module. Benefited from object-centric learning, we only require sparse detection labels for object localization and feature binding.
  arXiv  Detail & Related papers  (2023-09-01T03:34:12Z)
• Rethinking the Localization in Weakly Supervised Object Localization [51.29084037301646]
  Weakly supervised object localization (WSOL) is one of the most popular and challenging tasks in computer vision. Recent dividing WSOL into two parts (class-agnostic object localization and object classification) has become the state-of-the-art pipeline for this task. We propose to replace SCR with a binary-class detector (BCD) for localizing multiple objects, where the detector is trained by discriminating the foreground and background.
  arXiv  Detail & Related papers  (2023-08-11T14:38:51Z)
• Weakly-supervised Contrastive Learning for Unsupervised Object Discovery [52.696041556640516]
  Unsupervised object discovery is promising due to its ability to discover objects in a generic manner. We design a semantic-guided self-supervised learning model to extract high-level semantic features from images. We introduce Principal Component Analysis (PCA) to localize object regions.
  arXiv  Detail & Related papers  (2023-07-07T04:03:48Z)
• Referring Camouflaged Object Detection [88.63744004901655]
  Ref-COD aims to segment specified camouflaged objects based on a small set of referring images with salient target objects.<n>We first assemble a large-scale dataset, called R2C7K, which consists of 7K images covering 64 object categories in real-world scenarios.
  arXiv  Detail & Related papers  (2023-06-13T04:15:37Z)
• Image Segmentation-based Unsupervised Multiple Objects Discovery [1.7674345486888503]
  Unsupervised object discovery aims to localize objects in images. We propose a fully unsupervised, bottom-up approach, for multiple objects discovery. We provide state-of-the-art results for both unsupervised class-agnostic object detection and unsupervised image segmentation.
  arXiv  Detail & Related papers  (2022-12-20T09:48:24Z)
• Weakly Supervised Object Localization as Domain Adaption [19.854125742336688]
  Weakly supervised object localization (WSOL) focuses on localizing objects only with the supervision of image-level classification masks. Most previous WSOL methods follow the classification activation map (CAM) that localizes objects based on the classification structure with the multi-instance learning (MIL) mechanism. This work provides a novel perspective that models WSOL as a domain adaption (DA) task, where the score estimator trained on the source/image domain is tested on the target/pixel domain to locate objects.
  arXiv  Detail & Related papers  (2022-03-03T13:50:22Z)
• Sketch-Guided Object Localization in Natural Images [16.982683600384277]
  We introduce the novel problem of localizing all instances of an object (seen or unseen during training) in a natural image via sketch query. We propose a novel cross-modal attention scheme that guides the region proposal network (RPN) to generate object proposals relevant to the sketch query. Our method is effective with as little as a single sketch query.
  arXiv  Detail & Related papers  (2020-08-14T19:35:56Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.