DETR-ViP: Detection Transformer with Robust Discriminative Visual Prompts

• URL: http://arxiv.org/abs/2604.14684v1
• Date: Thu, 16 Apr 2026 06:40:44 GMT
• Title: DETR-ViP: Detection Transformer with Robust Discriminative Visual Prompts
• Abstract summary: We propose DETR-ViP, a robust object detection framework that yields class-distinguishable visual prompts.<n>On top of basic image-text contrastive learning, DETR-ViP incorporates global prompt integration and visual-textual prompt relation distillation.<n>Experiments on COCO, LVIS, ODinW, and Roboflow100 demonstrate that DETR-ViP achieves substantially higher performance in visual prompt detection.
• Score: 11.577330098443696
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Visual prompted object detection enables interactive and flexible definition of target categories, thereby facilitating open-vocabulary detection. Since visual prompts are derived directly from image features, they often outperform text prompts in recognizing rare categories. Nevertheless, research on visual prompted detection has been largely overlooked, and it is typically treated as a byproduct of training text prompted detectors, which hinders its development. To fully unlock the potential of visual-prompted detection, we investigate the reasons why its performance is suboptimal and reveal that the underlying issue lies in the absence of global discriminability in visual prompts. Motivated by these observations, we propose DETR-ViP, a robust object detection framework that yields class-distinguishable visual prompts. On top of basic image-text contrastive learning, DETR-ViP incorporates global prompt integration and visual-textual prompt relation distillation to learn more discriminative prompt representations. In addition, DETR-ViP employs a selective fusion strategy that ensures stable and robust detection. Extensive experiments on COCO, LVIS, ODinW, and Roboflow100 demonstrate that DETR-ViP achieves substantially higher performance in visual prompt detection compared to other state-of-the-art counterparts. A series of ablation studies and analyses further validate the effectiveness of the proposed improvements and shed light on the underlying reasons for the enhanced detection capability of visual prompts.

Related papers

• DE-FIVE: Detecting Malicious Image Prompts via Fourier Features and Image Vector Embeddings [0.870967746482596]
  We introduce DE-FIVE, a training-free framework for detecting malicious image prompts.<n>We develop a hybrid detection strategy consisting of a black-box detector and a white-box detector.<n>Extensive experiments demonstrate that the proposed framework consistently outperforms state-of-the-art baselines against malicious image prompts.
  arXiv  Detail & Related papers  (2026-06-22T02:42:38Z)
• Unleashing Vision-Language Semantics for Deepfake Video Detection [78.7562836979696]
  Deepfake Video Detection (DFD) studies have demonstrated that pre-trained Vision-Language Models (VLMs) exhibit strong generalization capabilities in detecting artifacts across different identities.<n>We propose VLAForge, a novel DFD framework that unleashes the potential of such cross-modal semantics to enhance model's discriminability in deepfake detection.
  arXiv  Detail & Related papers  (2026-03-25T16:05:35Z)
• VirPro: Visual-referred Probabilistic Prompt Learning for Weakly-Supervised Monocular 3D Detection [12.835071167163607]
  VirPro is an adaptive multi-modal pretraining paradigm that can be seamlessly integrated into diverse weakly supervised monocular 3D detection frameworks.<n>We generate a diverse set of learnable, instance-conditioned prompts across scenes and store them in an Adaptive Prompt Bank (APB)
  arXiv  Detail & Related papers  (2026-03-18T08:23:55Z)
• Focusing by Contrastive Attention: Enhancing VLMs' Visual Reasoning [79.34909830834464]
  Vision-Language Models (VLMs) have demonstrated remarkable success across diverse visual tasks, yet their performance degrades in complex visual environments.<n>We show that visual complexity strongly correlates with attention entropy, negatively impacting reasoning performance.<n>We propose Contrastive Attention Refinement for Visual Enhancement (CARVE), a training-free method that extracts task-relevant visual signals through attention contrasting at the pixel level.
  arXiv  Detail & Related papers  (2025-09-08T09:20:04Z)
• Constrained Prompt Enhancement for Improving Zero-Shot Generalization of Vision-Language Models [57.357091028792325]
  Vision-language models (VLMs) pre-trained on web-scale data exhibit promising zero-shot generalization but often suffer from semantic misalignment.<n>We propose a novel constrained prompt enhancement (CPE) method to improve visual-textual alignment.<n>Our approach consists of two key components: Topology-Guided Synonymous Semantic Generation (TGSSG) and Category-Agnostic Discriminative Region Selection (CADRS)
  arXiv  Detail & Related papers  (2025-08-24T15:45:22Z)
• ViCrit: A Verifiable Reinforcement Learning Proxy Task for Visual Perception in VLMs [98.27348724529257]
  We introduce ViCrit (Visual Caption Hallucination Critic), an RL proxy task that trains VLMs to localize a subtle, synthetic visual hallucination injected into paragraphs of human-written image captions.<n>Models trained with the ViCrit Task exhibit substantial gains across a variety of vision-language models benchmarks.
  arXiv  Detail & Related papers  (2025-06-11T19:16:54Z)
• Learning Knowledge-based Prompts for Robust 3D Mask Presentation Attack Detection [71.60120616284246]
  We propose a novel knowledge-based prompt learning framework to explore the strong generalization capability of vision-language models for 3D mask presentation attack detection.<n> Experimental results demonstrate that the proposed method achieves state-of-the-art intra- and cross-scenario detection performance.
  arXiv  Detail & Related papers  (2025-05-06T15:09:23Z)
• Leveraging Retrieval-Augmented Tags for Large Vision-Language Understanding in Complex Scenes [0.0]
  Vision-Aware Retrieval-Augmented Prompting (VRAP) is a generative approach that enhances Large Vision-Language Models.<n>VRAP achieves state-of-the-art performance in fine-grained reasoning and multimodal understanding.
  arXiv  Detail & Related papers  (2024-12-16T02:52:19Z)
• Exploring Predicate Visual Context in Detecting Human-Object Interactions [44.937383506126274]
  We study how best to re-introduce image features via cross-attention. Our model with enhanced predicate visual context (PViC) outperforms state-of-the-art methods on the HICO-DET and V-COCO benchmarks.
  arXiv  Detail & Related papers  (2023-08-11T15:57:45Z)
• Zero-shot Visual Relation Detection via Composite Visual Cues from Large Language Models [44.60439935450292]
  We propose a novel method for zero-shot visual recognition: RECODE. It decomposes each predicate category into subject, object, and spatial components. Different visual cues enhance the discriminability of similar relation categories from different perspectives.
  arXiv  Detail & Related papers  (2023-05-21T14:40:48Z)
• Open-Vocabulary DETR with Conditional Matching [86.1530128487077]
  OV-DETR is an open-vocabulary detector based on DETR. It can detect any object given its class name or an exemplar image. It achieves non-trivial improvements over current state of the arts.
  arXiv  Detail & Related papers  (2022-03-22T16:54:52Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.