Zero-Shot In-Distribution Detection in Multi-Object Settings Using Vision-Language Foundation Models

• URL: http://arxiv.org/abs/2304.04521v3
• Date: Wed, 23 Aug 2023 13:11:20 GMT
• Title: Zero-Shot In-Distribution Detection in Multi-Object Settings Using Vision-Language Foundation Models
• Authors: Atsuyuki Miyai, Qing Yu, Go Irie, Kiyoharu Aizawa
• Abstract summary: In this paper, we propose a novel problem setting called zero-shot in-distribution (ID) detection. We identify images containing ID objects as ID images (even if they contain OOD objects) and images lacking ID objects as OOD images without any training. We present a simple and effective approach, Global-Local Concept Matching, based on both global and local visual-text alignments of CLIP features.
• Score: 37.36999826208225
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Extracting in-distribution (ID) images from noisy images scraped from the Internet is an important preprocessing for constructing datasets, which has traditionally been done manually. Automating this preprocessing with deep learning techniques presents two key challenges. First, images should be collected using only the name of the ID class without training on the ID data. Second, as we can see why COCO was created, it is crucial to identify images containing not only ID objects but also both ID and out-of-distribution (OOD) objects as ID images to create robust recognizers. In this paper, we propose a novel problem setting called zero-shot in-distribution (ID) detection, where we identify images containing ID objects as ID images (even if they contain OOD objects), and images lacking ID objects as OOD images without any training. To solve this problem, we leverage the powerful zero-shot capability of CLIP and present a simple and effective approach, Global-Local Maximum Concept Matching (GL-MCM), based on both global and local visual-text alignments of CLIP features. Extensive experiments demonstrate that GL-MCM outperforms comparison methods on both multi-object datasets and single-object ImageNet benchmarks. The code will be available via https://github.com/AtsuMiyai/GL-MCM.

Related papers

• MMO-IG: Multi-Class and Multi-Scale Object Image Generation for Remote Sensing [12.491684385808902]
  MMO-IG is designed to generate RS images with supervised object labels from global and local aspects simultaneously. Considering the complex interdependencies among MMOs, we construct a spatial-cross dependency knowledge graph. Our MMO-IG exhibits superior generation capabilities for RS images with dense MMO-supervised labels.
  arXiv  Detail & Related papers  (2024-12-18T10:19:12Z)
• EasyRef: Omni-Generalized Group Image Reference for Diffusion Models via Multimodal LLM [38.8308841469793]
  This paper introduces EasyRef, a novel plug-and-play adaptation method that enables diffusion models to be conditioned on multiple reference images and the text prompt. We leverage the multi-image comprehension and instruction-following capabilities of the multimodal large language model (MLLM) to exploit consistent visual elements within multiple images. Experimental results demonstrate EasyRef surpasses both tuning-free methods like IP-Adapter and tuning-based methods like LoRA, achieving superior aesthetic quality and robust zero-shot generalization across diverse domains.
  arXiv  Detail & Related papers  (2024-12-12T18:59:48Z)
• OSMLoc: Single Image-Based Visual Localization in OpenStreetMap with Geometric and Semantic Guidances [11.085165252259042]
  OSMLoc is a brain-inspired single-image visual localization method with semantic and geometric guidance to improve accuracy, robustness, and generalization ability. To validate the proposed OSMLoc, we collect a worldwide cross-area and cross-condition (CC) benchmark for extensive evaluation.
  arXiv  Detail & Related papers  (2024-11-13T14:59:00Z)
• Large Language Models for Multimodal Deformable Image Registration [50.91473745610945]
  We propose a novel coarse-to-fine MDIR framework,LLM-Morph, for aligning the deep features from different modal medical images. Specifically, we first utilize a CNN encoder to extract deep visual features from cross-modal image pairs, then we use the first adapter to adjust these tokens, and use LoRA in pre-trained LLMs to fine-tune their weights. Third, for the alignment of tokens, we utilize other four adapters to transform the LLM-encoded tokens into multi-scale visual features, generating multi-scale deformation fields and facilitating the coarse-to-fine MDIR task
  arXiv  Detail & Related papers  (2024-08-20T09:58:30Z)
• INF-LLaVA: Dual-perspective Perception for High-Resolution Multimodal Large Language Model [71.50973774576431]
  We propose a novel MLLM, INF-LLaVA, designed for effective high-resolution image perception. We introduce a Dual-perspective Cropping Module (DCM), which ensures that each sub-image contains continuous details from a local perspective. Second, we introduce Dual-perspective Enhancement Module (DEM) to enable the mutual enhancement of global and local features.
  arXiv  Detail & Related papers  (2024-07-23T06:02:30Z)
• From Global to Local: Multi-scale Out-of-distribution Detection [129.37607313927458]
  Out-of-distribution (OOD) detection aims to detect "unknown" data whose labels have not been seen during the in-distribution (ID) training process. Recent progress in representation learning gives rise to distance-based OOD detection. We propose Multi-scale OOD DEtection (MODE), a first framework leveraging both global visual information and local region details.
  arXiv  Detail & Related papers  (2023-08-20T11:56:25Z)
• Coarse-to-Fine: Learning Compact Discriminative Representation for Single-Stage Image Retrieval [11.696941841000985]
  Two-stage methods following retrieve-and-rerank paradigm have achieved excellent performance, but their separate local and global modules are inefficient to real-world applications. We propose a mechanism which attentively selects prominent local descriptors and infuse fine-grained semantic relations into the global representation. Our method achieves state-of-the-art single-stage image retrieval performance on benchmarks such as Revisited Oxford and Revisited Paris.
  arXiv  Detail & Related papers  (2023-08-08T03:06:10Z)
• Adaptive Graph Convolution Module for Salient Object Detection [7.278033100480174]
  We propose an adaptive graph convolution module (AGCM) to deal with complex scenes. Prototype features are extracted from the input image using a learnable region generation layer. The proposed AGCM dramatically improves the SOD performance both quantitatively and quantitatively.
  arXiv  Detail & Related papers  (2023-03-17T07:07:17Z)
• Multi-Content Complementation Network for Salient Object Detection in Optical Remote Sensing Images [108.79667788962425]
  salient object detection in optical remote sensing images (RSI-SOD) remains to be a challenging emerging topic. We propose a novel Multi-Content Complementation Network (MCCNet) to explore the complementarity of multiple content for RSI-SOD. In MCCM, we consider multiple types of features that are critical to RSI-SOD, including foreground features, edge features, background features, and global image-level features.
  arXiv  Detail & Related papers  (2021-12-02T04:46:40Z)
• Boosting Few-shot Semantic Segmentation with Transformers [81.43459055197435]
  TRansformer-based Few-shot Semantic segmentation method (TRFS) Our model consists of two modules: Global Enhancement Module (GEM) and Local Enhancement Module (LEM)
  arXiv  Detail & Related papers  (2021-08-04T20:09:21Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.