Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detection

• URL: http://arxiv.org/abs/2303.05499v5
• Date: Fri, 19 Jul 2024 06:00:41 GMT
• Title: Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detection
• Authors: Shilong Liu, Zhaoyang Zeng, Tianhe Ren, Feng Li, Hao Zhang, Jie Yang, Qing Jiang, Chunyuan Li, Jianwei Yang, Hang Su, Jun Zhu, Lei Zhang,
• Abstract summary: We present an open-set object detector, called Grounding DINO, by marrying Transformer-based detector DINO with grounded pre-training. The key solution of open-set object detection is introducing language to a closed-set detector for open-set concept generalization. Grounding DINO performs remarkably well on all three settings, including benchmarks on COCO, LVIS, ODinW, and RefCOCO/+/g.
• Score: 76.5120397167247
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: In this paper, we present an open-set object detector, called Grounding DINO, by marrying Transformer-based detector DINO with grounded pre-training, which can detect arbitrary objects with human inputs such as category names or referring expressions. The key solution of open-set object detection is introducing language to a closed-set detector for open-set concept generalization. To effectively fuse language and vision modalities, we conceptually divide a closed-set detector into three phases and propose a tight fusion solution, which includes a feature enhancer, a language-guided query selection, and a cross-modality decoder for cross-modality fusion. While previous works mainly evaluate open-set object detection on novel categories, we propose to also perform evaluations on referring expression comprehension for objects specified with attributes. Grounding DINO performs remarkably well on all three settings, including benchmarks on COCO, LVIS, ODinW, and RefCOCO/+/g. Grounding DINO achieves a $52.5$ AP on the COCO detection zero-shot transfer benchmark, i.e., without any training data from COCO. It sets a new record on the ODinW zero-shot benchmark with a mean $26.1$ AP. Code will be available at \url{https://github.com/IDEA-Research/GroundingDINO}.

Related papers

• Weakly Supervised Open-Vocabulary Object Detection [31.605276665964787]
  We propose a novel weakly supervised open-vocabulary object detection framework, namely WSOVOD, to extend traditional WSOD. To achieve this, we explore three vital strategies, including dataset-level feature adaptation, image-level salient object localization, and region-level vision-language alignment.
  arXiv  Detail & Related papers  (2023-12-19T18:59:53Z)
• Enhancing Novel Object Detection via Cooperative Foundational Models [75.30243629533277]
  We present a novel approach to transform existing closed-set detectors into open-set detectors. We surpass the current state-of-the-art by a margin of 7.2 $ textAP_50 $ for novel classes.
  arXiv  Detail & Related papers  (2023-11-19T17:28:28Z)
• What Makes Good Open-Vocabulary Detector: A Disassembling Perspective [6.623703413255309]
  Open-vocabulary detection (OVD) is a new object detection paradigm, aiming to localize and recognize unseen objects defined by an unbounded vocabulary. Previous works mainly focus on the open vocabulary classification part, with less attention on the localization part. We show in this work that improving localization as well as cross-modal classification complement each other, and compose a good OVD detector jointly.
  arXiv  Detail & Related papers  (2023-09-01T03:03:50Z)
• Open-Vocabulary Point-Cloud Object Detection without 3D Annotation [62.18197846270103]
  The goal of open-vocabulary 3D point-cloud detection is to identify novel objects based on arbitrary textual descriptions. We develop a point-cloud detector that can learn a general representation for localizing various objects. We also propose a novel de-biased triplet cross-modal contrastive learning to connect the modalities of image, point-cloud and text.
  arXiv  Detail & Related papers  (2023-04-03T08:22:02Z)
• Open-Set Object Detection Using Classification-free Object Proposal and Instance-level Contrastive Learning [25.935629339091697]
  Open-set object detection (OSOD) is a promising direction to handle the problem consisting of two subtasks: objects and background separation, and open-set object classification. We present Openset RCNN to address the challenging OSOD. We show that our Openset RCNN can endow the robot with an open-set perception ability to support robotic rearrangement tasks in cluttered environments.
  arXiv  Detail & Related papers  (2022-11-21T15:00:04Z)
• GLENet: Boosting 3D Object Detectors with Generative Label Uncertainty Estimation [70.75100533512021]
  In this paper, we formulate the label uncertainty problem as the diversity of potentially plausible bounding boxes of objects. We propose GLENet, a generative framework adapted from conditional variational autoencoders, to model the one-to-many relationship between a typical 3D object and its potential ground-truth bounding boxes with latent variables. The label uncertainty generated by GLENet is a plug-and-play module and can be conveniently integrated into existing deep 3D detectors.
  arXiv  Detail & Related papers  (2022-07-06T06:26:17Z)
• Disentangle Your Dense Object Detector [82.22771433419727]
  Deep learning-based dense object detectors have achieved great success in the past few years and have been applied to numerous multimedia applications such as video understanding. However, the current training pipeline for dense detectors is compromised to lots of conjunctions that may not hold. We propose Disentangled Dense Object Detector (DDOD), in which simple and effective disentanglement mechanisms are designed and integrated into the current state-of-the-art detectors.
  arXiv  Detail & Related papers  (2021-07-07T00:52:16Z)
• EHSOD: CAM-Guided End-to-end Hybrid-Supervised Object Detection with Cascade Refinement [53.69674636044927]
  We present EHSOD, an end-to-end hybrid-supervised object detection system. It can be trained in one shot on both fully and weakly-annotated data. It achieves comparable results on multiple object detection benchmarks with only 30% fully-annotated data.
  arXiv  Detail & Related papers  (2020-02-18T08:04:58Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.