COCO-O: A Benchmark for Object Detectors under Natural Distribution Shifts

• URL: http://arxiv.org/abs/2307.12730v2
• Date: Wed, 2 Aug 2023 12:10:55 GMT
• Title: COCO-O: A Benchmark for Object Detectors under Natural Distribution Shifts
• Authors: Xiaofeng Mao, Yuefeng Chen, Yao Zhu, Da Chen, Hang Su, Rong Zhang, Hui Xue
• Abstract summary: COCO-O is a test dataset based on COCO with 6 types of natural distribution shifts. COCO-O has a large distribution gap with training data and results in a significant 55.7% relative performance drop on a Faster R-CNN detector. We study the robustness effect on recent breakthroughs of detector's architecture design, augmentation and pre-training techniques.
• Score: 27.406639379618003
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Practical object detection application can lose its effectiveness on image inputs with natural distribution shifts. This problem leads the research community to pay more attention on the robustness of detectors under Out-Of-Distribution (OOD) inputs. Existing works construct datasets to benchmark the detector's OOD robustness for a specific application scenario, e.g., Autonomous Driving. However, these datasets lack universality and are hard to benchmark general detectors built on common tasks such as COCO. To give a more comprehensive robustness assessment, we introduce COCO-O(ut-of-distribution), a test dataset based on COCO with 6 types of natural distribution shifts. COCO-O has a large distribution gap with training data and results in a significant 55.7% relative performance drop on a Faster R-CNN detector. We leverage COCO-O to conduct experiments on more than 100 modern object detectors to investigate if their improvements are credible or just over-fitting to the COCO test set. Unfortunately, most classic detectors in early years do not exhibit strong OOD generalization. We further study the robustness effect on recent breakthroughs of detector's architecture design, augmentation and pre-training techniques. Some empirical findings are revealed: 1) Compared with detection head or neck, backbone is the most important part for robustness; 2) An end-to-end detection transformer design brings no enhancement, and may even reduce robustness; 3) Large-scale foundation models have made a great leap on robust object detection. We hope our COCO-O could provide a rich testbed for robustness study of object detection. The dataset will be available at https://github.com/alibaba/easyrobust/tree/main/benchmarks/coco_o.

Related papers

• From COCO to COCO-FP: A Deep Dive into Background False Positives for COCO Detectors [8.3561487803637]
  False alarms in fire and smoke detection are critical in real-world applications. COCO-FP is a new evaluation dataset derived from the ImageNet-1K dataset. Our evaluation of both standard and advanced object detectors shows a significant number of false positives in both closed-set and open-set scenarios.
  arXiv  Detail & Related papers  (2024-09-12T10:22:12Z)
• Resultant: Incremental Effectiveness on Likelihood for Unsupervised Out-of-Distribution Detection [63.93728560200819]
  Unsupervised out-of-distribution (U-OOD) detection is to identify data samples with a detector trained solely on unlabeled in-distribution (ID) data. Recent studies have developed various detectors based on DGMs to move beyond likelihood. We apply two techniques for each direction, specifically post-hoc prior and dataset entropy-mutual calibration. Experimental results demonstrate that the Resultant could be a new state-of-the-art U-OOD detector.
  arXiv  Detail & Related papers  (2024-09-05T02:58:13Z)
• Learning Feature Inversion for Multi-class Anomaly Detection under General-purpose COCO-AD Benchmark [101.23684938489413]
  Anomaly detection (AD) is often focused on detecting anomalies for industrial quality inspection and medical lesion examination. This work first constructs a large-scale and general-purpose COCO-AD dataset by extending COCO to the AD field. Inspired by the metrics in the segmentation field, we propose several more practical threshold-dependent AD-specific metrics.
  arXiv  Detail & Related papers  (2024-04-16T17:38:26Z)
• Open-Vocabulary Object Detectors: Robustness Challenges under Distribution Shifts [6.486569431242123]
  Vision-Language Models (VLMs) have recently achieved groundbreaking results. Investigating OOD robustness in VLM object detection is essential to increase the trustworthiness of these models. This study presents a comprehensive robustness evaluation of the zero-shot capabilities of three recent open-vocabulary (OV) foundation object detection models.
  arXiv  Detail & Related papers  (2024-04-01T14:18:15Z)
• Evaluating Out-of-Distribution Detectors Through Adversarial Generation of Outliers [8.913669910615226]
  We propose Evaluation-via-Generation for OOD detectors (EvG) EvG is a new protocol for investigating the robustness of OOD detectors under more realistic modes of variation in outliers. We perform a comprehensive benchmark comparison of the performance of state-of-the-art OOD detectors using EvG, uncovering previously overlooked weaknesses.
  arXiv  Detail & Related papers  (2022-08-20T22:10:36Z)
• Anchor-free Oriented Proposal Generator for Object Detection [59.54125119453818]
  Oriented object detection is a practical and challenging task in remote sensing image interpretation. Nowadays, oriented detectors mostly use horizontal boxes as intermedium to derive oriented boxes from them. We propose a novel Anchor-free Oriented Proposal Generator (AOPG) that abandons the horizontal boxes-related operations from the network architecture.
  arXiv  Detail & Related papers  (2021-10-05T10:45:51Z)
• Disentangle Your Dense Object Detector [82.22771433419727]
  Deep learning-based dense object detectors have achieved great success in the past few years and have been applied to numerous multimedia applications such as video understanding. However, the current training pipeline for dense detectors is compromised to lots of conjunctions that may not hold. We propose Disentangled Dense Object Detector (DDOD), in which simple and effective disentanglement mechanisms are designed and integrated into the current state-of-the-art detectors.
  arXiv  Detail & Related papers  (2021-07-07T00:52:16Z)
• Robust and Accurate Object Detection via Adversarial Learning [111.36192453882195]
  This work augments the fine-tuning stage for object detectors by exploring adversarial examples. Our approach boosts the performance of state-of-the-art EfficientDets by +1.1 mAP on the object detection benchmark.
  arXiv  Detail & Related papers  (2021-03-23T19:45:26Z)
• Relevance Attack on Detectors [24.318876747711055]
  This paper focuses on high-transferable adversarial attacks on detectors, which are hard to attack in a black-box manner. We are the first to suggest that the relevance map from interpreters for detectors is such a property. Based on it, we design a Relevance Attack on Detectors (RAD), which achieves a state-of-the-art transferability.
  arXiv  Detail & Related papers  (2020-08-16T02:44:25Z)
• Scope Head for Accurate Localization in Object Detection [135.9979405835606]
  We propose a novel detector coined as ScopeNet, which models anchors of each location as a mutually dependent relationship. With our concise and effective design, the proposed ScopeNet achieves state-of-the-art results on COCO.
  arXiv  Detail & Related papers  (2020-05-11T04:00:09Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.