LV-OSD: Language-Vision-Complementary Open-Set Object Detection

• URL: http://arxiv.org/abs/2605.28271v1
• Date: Wed, 27 May 2026 10:17:56 GMT
• Title: LV-OSD: Language-Vision-Complementary Open-Set Object Detection
• Abstract summary: We propose a new problem of language-visual-complementary open-set object detection (LV-OSD)<n>We use the flexible text-based and/or image-based prompts to specify the desired object categories.<n>This setting is more common and practical in real-world applications.
• Score: 35.3832220318018
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Object detection is an important task in computer vision, which aims to detect the objects of interest. through the given category list or query images. In this work, we propose a new problem of language-visual-complementary open-set object detection (LV-OSD), i.e., using the flexible text-based and/or image-based prompts to specify the desired object categories. This setting is more common and practical in real-world applications. For this purpose, we design a dual-branch detection framework, LVDor, which can simultaneously accept both text and image prompts. Specifically, we first build the Multi-modal Prompts (MPr) containing various text descriptions and image samples for each category. Subsequently, to bridge the semantic gap among the input image, text prompts, and image prompts, we design a Target-guided Prompt Dynamic Weighting (TPDW) module. Guided by the prior information of the target image, this module dynamically produces the text and image prompts that best align with the target semantics, achieving precise alignment and effectively reducing the discrepancy between the two modalities, thereby accommodating the LV-OSD setting. We also propose a simple Prompt Random Masking (PRM) mechanism during training to simulate the arbitrary combination of text and/or image prompts in testing. Extensive experimental results verify our problem formulation's reasonability and our method's effectiveness. Prompts and code will be released publicly.

Related papers

• MCoT-MVS: Multi-level Vision Selection by Multi-modal Chain-of-Thought Reasoning for Composed Image Retrieval [23.52156477012636]
  Composed Image Retrieval (CIR) aims to retrieve target images based on a reference image and modified texts.<n>We propose a novel Multi-level Vision Selection by Multi-modal Chain-of-Thought Reasoning (MCoT-MVS) for CIR.
  arXiv  Detail & Related papers  (2026-03-18T04:49:19Z)
• ViLa-MIL: Dual-scale Vision-Language Multiple Instance Learning for Whole Slide Image Classification [52.405499816861635]
  Multiple instance learning (MIL)-based framework has become the mainstream for processing the whole slide image (WSI)<n>We propose a dual-scale vision-language multiple instance learning (ViLa-MIL) framework for whole slide image classification.
  arXiv  Detail & Related papers  (2025-02-12T13:28:46Z)
• Attention Prompting on Image for Large Vision-Language Models [63.794304207664176]
  We propose a new prompting technique named Attention Prompting on Image. We generate an attention heatmap for the input image dependent on the text query with an auxiliary model like CLIP. Experiments on various vison-language benchmarks verify the effectiveness of our technique.
  arXiv  Detail & Related papers  (2024-09-25T17:59:13Z)
• Exploring Fine-Grained Image-Text Alignment for Referring Remote Sensing Image Segmentation [27.13782704236074]
  We propose a new referring remote sensing image segmentation method to fully exploit the visual and linguistic representations.<n>The proposed fine-grained image-text alignment module (FIAM) would simultaneously leverage the features of the input image and the corresponding texts.<n>We evaluate the effectiveness of the proposed method on two public referring remote sensing datasets including RefSegRS and RRSIS-D.
  arXiv  Detail & Related papers  (2024-09-20T16:45:32Z)
• Test-time Contrastive Concepts for Open-world Semantic Segmentation with Vision-Language Models [14.899741072838994]
  Recent CLIP-like Vision-Language Models (VLMs), pre-trained on large amounts of image-text pairs, have paved the way to open-vocabulary semantic segmentation.<n>We propose two different approaches to automatically generate, at test time, query-specific textual contrastive concepts.
  arXiv  Detail & Related papers  (2024-07-06T12:18:43Z)
• Sentence-level Prompts Benefit Composed Image Retrieval [69.78119883060006]
  Composed image retrieval (CIR) is the task of retrieving specific images by using a query that involves both a reference image and a relative caption. We propose to leverage pretrained V-L models, e.g., BLIP-2, to generate sentence-level prompts. Our proposed method performs favorably against the state-of-the-art CIR methods on the Fashion-IQ and CIRR datasets.
  arXiv  Detail & Related papers  (2023-10-09T07:31:44Z)
• Beyond One-to-One: Rethinking the Referring Image Segmentation [117.53010476628029]
  Referring image segmentation aims to segment the target object referred by a natural language expression. We propose a Dual Multi-Modal Interaction (DMMI) Network, which contains two decoder branches. In the text-to-image decoder, text embedding is utilized to query the visual feature and localize the corresponding target. Meanwhile, the image-to-text decoder is implemented to reconstruct the erased entity-phrase conditioned on the visual feature.
  arXiv  Detail & Related papers  (2023-08-26T11:39:22Z)
• Advancing Visual Grounding with Scene Knowledge: Benchmark and Method [74.72663425217522]
  Visual grounding (VG) aims to establish fine-grained alignment between vision and language. Most existing VG datasets are constructed using simple description texts. We propose a novel benchmark of underlineScene underlineKnowledge-guided underlineVisual underlineGrounding.
  arXiv  Detail & Related papers  (2023-07-21T13:06:02Z)
• Contextual Object Detection with Multimodal Large Language Models [66.15566719178327]
  We introduce a novel research problem of contextual object detection. Three representative scenarios are investigated, including the language cloze test, visual captioning, and question answering. We present ContextDET, a unified multimodal model that is capable of end-to-end differentiable modeling of visual-language contexts.
  arXiv  Detail & Related papers  (2023-05-29T17:50:33Z)
• MDETR -- Modulated Detection for End-to-End Multi-Modal Understanding [40.24656027709833]
  We propose MDETR, an end-to-end modulated detector that detects objects in an image conditioned on a raw text query. We use a transformer-based architecture to reason jointly over text and image by fusing the two modalities at an early stage of the model. Our approach can be easily extended for visual question answering, achieving competitive performance on GQA and CLEVR.
  arXiv  Detail & Related papers  (2021-04-26T17:55:33Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.