PET-DINO: Unifying Visual Cues into Grounding DINO with Prompt-Enriched Training

• URL: http://arxiv.org/abs/2604.00503v2
• Date: Tue, 07 Apr 2026 03:29:58 GMT
• Title: PET-DINO: Unifying Visual Cues into Grounding DINO with Prompt-Enriched Training
• Abstract summary: Open-Set Object Detection (OSOD) enables recognition of novel categories beyond fixed classes.<n>We propose PET-DINO, a universal detector supporting both text and visual prompts.<n>We introduce two prompt-enriched training strategies: Intra-Batch Parallel Prompting (IBP) at the level and Dynamic Memory-Driven Prompting (DMD) at the overall training level.
• Score: 72.04117115296917
• License: http://creativecommons.org/licenses/by-sa/4.0/
• Abstract: Open-Set Object Detection (OSOD) enables recognition of novel categories beyond fixed classes but faces challenges in aligning text representations with complex visual concepts and the scarcity of image-text pairs for rare categories. This results in suboptimal performance in specialized domains or with complex objects. Recent visual-prompted methods partially address these issues but often involve complex multi-modal designs and multi-stage optimizations, prolonging the development cycle. Additionally, effective training strategies for data-driven OSOD models remain largely unexplored. To address these challenges, we propose PET-DINO, a universal detector supporting both text and visual prompts. Our Alignment-Friendly Visual Prompt Generation (AFVPG) module builds upon an advanced text-prompted detector, addressing the limitations of text representation guidance and reducing the development cycle. We introduce two prompt-enriched training strategies: Intra-Batch Parallel Prompting (IBP) at the iteration level and Dynamic Memory-Driven Prompting (DMD) at the overall training level. These strategies enable simultaneous modeling of multiple prompt routes, facilitating parallel alignment with diverse real-world usage scenarios. Comprehensive experiments demonstrate that PET-DINO exhibits competitive zero-shot object detection capabilities across various prompt-based detection protocols. These strengths can be attributed to inheritance-based philosophy and prompt-enriched training strategies, which play a critical role in building an effective generic object detector. Project page: https://fuweifuvtoo.github.io/pet-dino.

Related papers

• Unleashing Multimodal Large Language Models for Training-free HOI Detection in the Wild [49.77540427384148]
  We present AgentHOI, a training-free, agentic framework that transfers the generalist multimodal reasoning capabilities of foundation models to HOI detection in the wild.<n>To address the challenges of incomplete interaction discovery and ambiguous localization in complex scenes, we introduce two key mechanisms.<n>AgentHOI achieves superior performance over state-of-the-art supervised and weakly supervised methods in real-world settings.
  arXiv  Detail & Related papers  (2026-07-15T14:30:20Z)
• VirPro: Visual-referred Probabilistic Prompt Learning for Weakly-Supervised Monocular 3D Detection [12.835071167163607]
  VirPro is an adaptive multi-modal pretraining paradigm that can be seamlessly integrated into diverse weakly supervised monocular 3D detection frameworks.<n>We generate a diverse set of learnable, instance-conditioned prompts across scenes and store them in an Adaptive Prompt Bank (APB)
  arXiv  Detail & Related papers  (2026-03-18T08:23:55Z)
• Parameterized Prompt for Incremental Object Detection [40.077943384096805]
  Existing prompts pool based approaches assume disjoint class sets across incremental tasks.<n>In co-occurring scenarios, unlabeled objects from previous tasks may appear in current task images, leading to confusion in prompts pool.<n>In this paper, we hold that prompt structures should exhibit adaptive consolidation properties across tasks, with constrained updates to prevent catastrophic forgetting.
  arXiv  Detail & Related papers  (2025-10-31T09:41:49Z)
• AttriPrompt: Dynamic Prompt Composition Learning for CLIP [41.37140060183439]
  AttriPrompt is a novel framework that enhances and refines textual semantic representations.<n>We introduce a Self-Regularization mechanism by applying explicit regularization constraints between the prompted and non-prompted text features.<n>Experiments demonstrate AttriPrompt's superiority over state-of-the-art methods, achieving up to 7.37% improvement in the base-to-novel setting.
  arXiv  Detail & Related papers  (2025-09-07T07:07:59Z)
• ChordPrompt: Orchestrating Cross-Modal Prompt Synergy for Multi-Domain Incremental Learning in CLIP [12.05848395374439]
  Continual learning empowers pre-trained vision-language models to adapt effectively to novel or previously underrepresented data distributions.<n>ChordPrompt introduces cross-modal prompts to leverage interactions between visual and textual information.<n>ChordPrompt outperforms state-of-the-art methods in zero-shot generalization and downstream task performance.
  arXiv  Detail & Related papers  (2025-06-24T13:22:06Z)
• SDVPT: Semantic-Driven Visual Prompt Tuning for Open-World Object Counting [70.49268117587562]
  We propose a plug-and-play Semantic-Driven Visual Prompt Tuning framework (SDVPT) that transfers knowledge from the training set to unseen categories.<n>During inference, we dynamically synthesize the visual prompts for unseen categories based on the semantic correlation between unseen and training categories.
  arXiv  Detail & Related papers  (2025-04-24T09:31:08Z)
• Generative Compositor for Few-Shot Visual Information Extraction [60.663887314625164]
  We propose a novel generative model, named Generative generative spatialtor, to address the challenge of few-shot VIE.<n>Generative generative spatialtor is a hybrid pointer-generator network that emulates the operations of a compositor by retrieving words from the source text.<n>The proposed method achieves highly competitive results in the full-sample training, while notably outperforms the baseline in the 1-shot, 5-shot, and 10-shot settings.
  arXiv  Detail & Related papers  (2025-03-21T04:56:24Z)
• MQADet: A Plug-and-Play Paradigm for Enhancing Open-Vocabulary Object Detection via Multimodal Question Answering [5.503514317063399]
  Existing open-vocabulary detectors are limited by complex visual-textual misalignment and long-tailed category imbalances.<n>We introduce MQADet, a universal paradigm for enhancing existing open-vocabulary detectors by leveraging the cross-modal reasoning capabilities of multimodal large language models.<n>We design a novel three-stage Multimodal Question Answering (MQA) pipeline to guide the MLLMs to precisely localize complex textual and visual targets.
  arXiv  Detail & Related papers  (2025-02-23T07:59:39Z)
• Attend and Enrich: Enhanced Visual Prompt for Zero-Shot Learning [114.59476118365266]
  We propose AENet, which endows semantic information into the visual prompt to distill semantic-enhanced prompt for visual representation enrichment.<n> AENet comprises two key steps: 1) exploring the concept-harmonized tokens for the visual and attribute modalities, grounded on the modal-sharing token that represents consistent visual-semantic concepts; and 2) yielding semantic-enhanced prompt via the visual residual refinement unit with attribute consistency supervision.
  arXiv  Detail & Related papers  (2024-06-05T07:59:48Z)
• Unified Demonstration Retriever for In-Context Learning [56.06473069923567]
  Unified Demonstration Retriever (textbfUDR) is a single model to retrieve demonstrations for a wide range of tasks. We propose a multi-task list-wise ranking training framework, with an iterative mining strategy to find high-quality candidates. Experiments on 30+ tasks across 13 task families and multiple data domains show that UDR significantly outperforms baselines.
  arXiv  Detail & Related papers  (2023-05-07T16:07:11Z)
• Prompting Decision Transformer for Few-Shot Policy Generalization [98.0914217850999]
  We propose a Prompt-based Decision Transformer (Prompt-DT) to achieve few-shot adaptation in offline RL. Prompt-DT is a strong few-shot learner without any extra finetuning on unseen target tasks.
  arXiv  Detail & Related papers  (2022-06-27T17:59:17Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.