OPUS: A Simple yet Effective Unified Framework for Open-Vocabulary Detection

• URL: http://arxiv.org/abs/2608.30247v1
• Date: Mon, 31 Aug 2026 04:57:12 GMT
• Title: OPUS: A Simple yet Effective Unified Framework for Open-Vocabulary Detection
• Abstract summary: We present OPUS, a unified detector supporting text, interactive visual, generic visual, and mixed prompting within one framework.<n> OPUS achieves state-of-the-art Visual-I performance, reaching 68.1/69.2/54.7 AP, while maintaining balanced Text and Visual-G accuracy.
• Score: 9.87006529425909
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Recent unified open-vocabulary detection (OVD) supports heterogeneous prompts, including text queries, visual exemplars, and their combinations, but often rely on increasingly complex designs such as heavy cross-modal fusion, staged training, and iterative annotation pipelines. We revisit whether such complexity is necessary in the era of stronger foundation models. Our finding is that unified OVD can be made substantially simpler with semantic-rich visual representations and scalable grounding supervision. We present OPUS (\textbf{O}pen-vocabulary, \textbf{P}rompt-\textbf{U}nified, \textbf{S}imple), a unified detector supporting text, interactive visual, generic visual, and mixed prompting within one framework. OPUS adopts a simple three-part design. Its model architecture combines a semantic-rich visual encoder, built on a DINOv3-ConvNeXt-B backbone with efficient hybrid encoding, with a prompt-aware decoder that avoids prompt-specific branches for unified prompt reasoning. OPUS is trained with a one-stage text-visual training strategy with Instance-level Contrastive Alignment (ICA), and is supported by a SAM3-based single-pass data engine for heterogeneous grounding supervision. Experiments on COCO, LVIS-minival, and ODinW35 show that OPUS achieves state-of-the-art Visual-I performance, reaching 68.1/69.2/54.7 AP, while maintaining balanced Text and Visual-G accuracy. OPUS also turns mixed prompting from interference into complementarity, improving over text or visual prompt alone. These results show that simplicity and strong unified prompting capability can be achieved together.

Related papers

• Vision-Guided Text Prompt Tuning for Multimodal Sentiment Analysis [15.965349585231932]
  Vision-Guided Text Prompt Tuning (VG-TPT) formulates visual-text sentiment modeling as controllable visual calibration of frozen text representations.<n> Experiments on CMU-MOSEI and CMU-MOSI show that VG-TPT consistently improves over text-only baselines.
  arXiv  Detail & Related papers  (2026-09-06T09:25:32Z)
• APE: Agentic Prompt Enhancer for Image Generation and Editing [58.43341902967105]
  Agentic Prompt Enhancer (APE) is a lightweight framework that post-trains small language models (SLMs) as prompt-enhancement agents.<n>APE supports both single-agent rewriting and role-specialized multi-agent enhancement.<n>Its single-agent instantiation, SAPE, rewrites the prompt in one pass, while its multi-agent instantiation, MAPE, decomposes enhancement into a router--rewriter--composer process.
  arXiv  Detail & Related papers  (2026-05-29T17:59:31Z)
• VL-UniTrack: A Unified Framework with Visual-Language Prompts for UAV-Ground Visual Tracking [35.96855931247585]
  UAV-ground visual tracking (UGVT) aims to simultaneously track the same object from both the UAV and the ground view.<n>Existing two-stream methods suffer from isolated feature extraction and rely heavily on implicit appearance matching.<n>We propose VL-UniTrack, a fully unified framework enhanced by visual-language prompts.
  arXiv  Detail & Related papers  (2026-05-06T07:23:53Z)
• VirPro: Visual-referred Probabilistic Prompt Learning for Weakly-Supervised Monocular 3D Detection [12.835071167163607]
  VirPro is an adaptive multi-modal pretraining paradigm that can be seamlessly integrated into diverse weakly supervised monocular 3D detection frameworks.<n>We generate a diverse set of learnable, instance-conditioned prompts across scenes and store them in an Adaptive Prompt Bank (APB)
  arXiv  Detail & Related papers  (2026-03-18T08:23:55Z)
• Constrained Prompt Enhancement for Improving Zero-Shot Generalization of Vision-Language Models [57.357091028792325]
  Vision-language models (VLMs) pre-trained on web-scale data exhibit promising zero-shot generalization but often suffer from semantic misalignment.<n>We propose a novel constrained prompt enhancement (CPE) method to improve visual-textual alignment.<n>Our approach consists of two key components: Topology-Guided Synonymous Semantic Generation (TGSSG) and Category-Agnostic Discriminative Region Selection (CADRS)
  arXiv  Detail & Related papers  (2025-08-24T15:45:22Z)
• Text-guided Visual Prompt DINO for Generic Segmentation [31.33676182634522]
  We propose Prompt-DINO, a text-guided visual Prompt DINO framework.<n>First, we introduce an early fusion mechanism that unifies text/visual prompts and backbone features.<n>Second, we design order-aligned query selection for DETR-based architectures.<n>Third, we develop a generative data engine powered by the Recognize Anything via Prompting (RAP) model.
  arXiv  Detail & Related papers  (2025-08-08T09:09:30Z)
• FOCUS: Unified Vision-Language Modeling for Interactive Editing Driven by Referential Segmentation [55.01077993490845]
  Recent Large Vision Language Models (LVLMs) demonstrate promising capabilities in unifying visual understanding and generative modeling.<n>We introduce FOCUS, a unified LVLM that integrates segmentation-aware perception and controllable object-centric generation within an end-to-end framework.
  arXiv  Detail & Related papers  (2025-06-20T07:46:40Z)
• ConText: Driving In-context Learning for Text Removal and Segmentation [59.6299939669307]
  This paper presents the first study on adapting the visual in-context learning paradigm to optical character recognition tasks.<n>We propose a task-chaining compositor in the form of image-removal-segmentation.<n>We also introduce context-aware aggregation, integrating the chained prompt pattern into the latent query representation.
  arXiv  Detail & Related papers  (2025-06-04T10:06:32Z)
• Sequential Visual and Semantic Consistency for Semi-supervised Text Recognition [56.968108142307976]
  Scene text recognition (STR) is a challenging task that requires large-scale annotated data for training. Most existing STR methods resort to synthetic data, which may introduce domain discrepancy and degrade the performance of STR models. This paper proposes a novel semi-supervised learning method for STR that incorporates word-level consistency regularization from both visual and semantic aspects.
  arXiv  Detail & Related papers  (2024-02-24T13:00:54Z)
• TextFormer: A Query-based End-to-End Text Spotter with Mixed Supervision [61.186488081379]
  We propose TextFormer, a query-based end-to-end text spotter with Transformer architecture. TextFormer builds upon an image encoder and a text decoder to learn a joint semantic understanding for multi-task modeling. It allows for mutual training and optimization of classification, segmentation, and recognition branches, resulting in deeper feature sharing.
  arXiv  Detail & Related papers  (2023-06-06T03:37:41Z)
• CLIP4STR: A Simple Baseline for Scene Text Recognition with Pre-trained Vision-Language Model [55.321010757641524]
  We introduce CLIP4STR, a simple yet effective STR method built upon image and text encoders of CLIP.<n>We scale CLIP4STR in terms of the model size, pre-training data, and training data, achieving state-of-the-art performance on 13 STR benchmarks.
  arXiv  Detail & Related papers  (2023-05-23T12:51:20Z)
• Unified Vision and Language Prompt Learning [86.1530128487077]
  We present a systematic study on two representative prompt tuning methods, namely text prompt tuning and visual prompt tuning. A major finding is that text prompt tuning fails on data with high intra-class visual variances while visual prompt tuning cannot handle low inter-class variances. To combine the best from both worlds, we propose a simple approach called Unified Prompt Tuning (UPT), which essentially learns a tiny neural network to jointly optimize prompts across different modalities.
  arXiv  Detail & Related papers  (2022-10-13T17:50:24Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.