NovelCraft: A Dataset for Novelty Detection and Discovery in Open Worlds

• URL: http://arxiv.org/abs/2206.11736v3
• Date: Tue, 28 Mar 2023 18:27:24 GMT
• Title: NovelCraft: A Dataset for Novelty Detection and Discovery in Open Worlds
• Authors: Patrick Feeney, Sarah Schneider, Panagiotis Lymperopoulos, Li-Ping Liu, Matthias Scheutz, Michael C. Hughes
• Abstract summary: NovelCraft dataset contains episodic data of the images and symbolic world-states seen by an agent completing a pogo stick assembly task within a modified Minecraft environment. Our visual novelty detection benchmark finds that methods that rank best on popular area-under-the-curve metrics may be outperformed by simpler alternatives. Further multimodal novelty detection experiments suggest that methods that fuse both visual and symbolic information can improve time until detection as well as overall discrimination.
• Score: 14.265615838391703
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: In order for artificial agents to successfully perform tasks in changing environments, they must be able to both detect and adapt to novelty. However, visual novelty detection research often only evaluates on repurposed datasets such as CIFAR-10 originally intended for object classification, where images focus on one distinct, well-centered object. New benchmarks are needed to represent the challenges of navigating the complex scenes of an open world. Our new NovelCraft dataset contains multimodal episodic data of the images and symbolic world-states seen by an agent completing a pogo stick assembly task within a modified Minecraft environment. In some episodes, we insert novel objects of varying size within the complex 3D scene that may impact gameplay. Our visual novelty detection benchmark finds that methods that rank best on popular area-under-the-curve metrics may be outperformed by simpler alternatives when controlling false positives matters most. Further multimodal novelty detection experiments suggest that methods that fuse both visual and symbolic information can improve time until detection as well as overall discrimination. Finally, our evaluation of recent generalized category discovery methods suggests that adapting to new imbalanced categories in complex scenes remains an exciting open problem.

Related papers

• Exploiting Unlabeled Data with Multiple Expert Teachers for Open Vocabulary Aerial Object Detection and Its Orientation Adaptation [58.37525311718006]
  We put forth a novel formulation of the aerial object detection problem, namely open-vocabulary aerial object detection (OVAD) We propose CastDet, a CLIP-activated student-teacher detection framework that serves as the first OVAD detector specifically designed for the challenging aerial scenario. Our framework integrates a robust localization teacher along with several box selection strategies to generate high-quality proposals for novel objects.
  arXiv  Detail & Related papers  (2024-11-04T12:59:13Z)
• A Comprehensive Library for Benchmarking Multi-class Visual Anomaly Detection [52.228708947607636]
  This paper introduces a comprehensive visual anomaly detection benchmark, ADer, which is a modular framework for new methods. The benchmark includes multiple datasets from industrial and medical domains, implementing fifteen state-of-the-art methods and nine comprehensive metrics. We objectively reveal the strengths and weaknesses of different methods and provide insights into the challenges and future directions of multi-class visual anomaly detection.
  arXiv  Detail & Related papers  (2024-06-05T13:40:07Z)
• Zero-shot Degree of Ill-posedness Estimation for Active Small Object Change Detection [8.977792536037956]
  In everyday indoor navigation, robots often needto detect non-distinctive small-change objects. Existing techniques rely on high-quality class-specific object priors to regularize a change detector model. In this study, we explore the concept of degree-of-ill-posedness (DoI) to improve both passive and activevision.
  arXiv  Detail & Related papers  (2024-05-10T01:56:39Z)
• Find n' Propagate: Open-Vocabulary 3D Object Detection in Urban Environments [67.83787474506073]
  We tackle the limitations of current LiDAR-based 3D object detection systems. We introduce a universal textscFind n' Propagate approach for 3D OV tasks. We achieve up to a 3.97-fold increase in Average Precision (AP) for novel object classes.
  arXiv  Detail & Related papers  (2024-03-20T12:51:30Z)
• OpenPatch: a 3D patchwork for Out-Of-Distribution detection [16.262921993755892]
  We present an approach for the task of semantic novelty detection on real-world point cloud samples when the reference known data are synthetic. OpenPatch builds on a large pre-trained model and simply extracts from its intermediate features a set of patch representations that describe each known class. We demonstrate that OpenPatch excels in both the full and few-shot known sample scenarios.
  arXiv  Detail & Related papers  (2023-10-05T08:49:51Z)
• Look Around and Learn: Self-Training Object Detection by Exploration [23.620820805804616]
  An agent learns to explore the environment using a pre-trained off-the-shelf detector to locate objects and associate pseudo-labels. By assuming that pseudo-labels for the same object must be consistent across different views, we learn the exploration policy Look Around to mine hard samples. We implement a unified benchmark of the current state-of-the-art and compare our approach with pre-existing exploration policies and perception mechanisms.
  arXiv  Detail & Related papers  (2023-02-07T16:26:45Z)
• Spatial Reasoning for Few-Shot Object Detection [21.3564383157159]
  We propose a spatial reasoning framework that detects novel objects with only a few training examples in a context. We employ a graph convolutional network as the RoIs and their relatedness are defined as nodes and edges, respectively. We demonstrate that the proposed method significantly outperforms the state-of-the-art methods and verify its efficacy through extensive ablation studies.
  arXiv  Detail & Related papers  (2022-11-02T12:38:08Z)
• OW-DETR: Open-world Detection Transformer [90.56239673123804]
  We introduce a novel end-to-end transformer-based framework, OW-DETR, for open-world object detection. OW-DETR comprises three dedicated components namely, attention-driven pseudo-labeling, novelty classification and objectness scoring. Our model outperforms the recently introduced OWOD approach, ORE, with absolute gains ranging from 1.8% to 3.3% in terms of unknown recall.
  arXiv  Detail & Related papers  (2021-12-02T18:58:30Z)
• The Pursuit of Knowledge: Discovering and Localizing Novel Categories using Dual Memory [85.01439251151203]
  We tackle object category discovery, which is the problem of discovering and localizing novel objects in a large unlabeled dataset. We propose a method to use prior knowledge about certain object categories to discover new categories by leveraging two memory modules. We show the performance of our detector on the COCO minival dataset to demonstrate its in-the-wild capabilities.
  arXiv  Detail & Related papers  (2021-05-04T17:55:59Z)
• Any-Shot Object Detection [81.88153407655334]
  'Any-shot detection' is where totally unseen and few-shot categories can simultaneously co-occur during inference. We propose a unified any-shot detection model, that can concurrently learn to detect both zero-shot and few-shot object classes. Our framework can also be used solely for Zero-shot detection and Few-shot detection tasks.
  arXiv  Detail & Related papers  (2020-03-16T03:43:15Z)
• SVIRO: Synthetic Vehicle Interior Rear Seat Occupancy Dataset and Benchmark [11.101588888002045]
  We release SVIRO, a synthetic dataset for sceneries in the passenger compartment of ten different vehicles. We analyze machine learning-based approaches for their generalization capacities and reliability when trained on a limited number of variations.
  arXiv  Detail & Related papers  (2020-01-10T14:44:23Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.