Label-Free Event-based Object Recognition via Joint Learning with Image Reconstruction from Events

• URL: http://arxiv.org/abs/2308.09383v1
• Date: Fri, 18 Aug 2023 08:28:17 GMT
• Title: Label-Free Event-based Object Recognition via Joint Learning with Image Reconstruction from Events
• Authors: Hoonhee Cho, Hyeonseong Kim, Yujeong Chae, and Kuk-Jin Yoon
• Abstract summary: We study label-free event-based object recognition where category labels and paired images are not available. Our method first reconstructs images from events and performs object recognition through Contrastive Language-Image Pre-training (CLIP) Since the category information is essential in reconstructing images, we propose category-guided attraction loss and category-agnostic repulsion loss.
• Score: 42.71383489578851
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Recognizing objects from sparse and noisy events becomes extremely difficult when paired images and category labels do not exist. In this paper, we study label-free event-based object recognition where category labels and paired images are not available. To this end, we propose a joint formulation of object recognition and image reconstruction in a complementary manner. Our method first reconstructs images from events and performs object recognition through Contrastive Language-Image Pre-training (CLIP), enabling better recognition through a rich context of images. Since the category information is essential in reconstructing images, we propose category-guided attraction loss and category-agnostic repulsion loss to bridge the textual features of predicted categories and the visual features of reconstructed images using CLIP. Moreover, we introduce a reliable data sampling strategy and local-global reconstruction consistency to boost joint learning of two tasks. To enhance the accuracy of prediction and quality of reconstruction, we also propose a prototype-based approach using unpaired images. Extensive experiments demonstrate the superiority of our method and its extensibility for zero-shot object recognition. Our project code is available at \url{https://github.com/Chohoonhee/Ev-LaFOR}.

Related papers

• Knowledge Fused Recognition: Fusing Hierarchical Knowledge for Image Recognition through Quantitative Relativity Modeling and Deep Metric Learning [18.534970504136254]
  We propose a novel deep metric learning based method to fuse hierarchical prior knowledge about image classes. Existing deep metric learning incorporated image classification mainly exploits qualitative relativity between image classes. A new triplet loss function term that exploits quantitative relativity and aligns distances in model latent space with those in knowledge space is also proposed and incorporated in the proposed dual-modality fusion method.
  arXiv  Detail & Related papers  (2024-07-30T07:24:33Z)
• RAFIC: Retrieval-Augmented Few-shot Image Classification [0.0]
  Few-shot image classification is the task of classifying unseen images to one of N mutually exclusive classes. We have developed a method for augmenting the set of K with an addition set of A retrieved images. We demonstrate that RAFIC markedly improves performance of few-shot image classification across two challenging datasets.
  arXiv  Detail & Related papers  (2023-12-11T22:28:51Z)
• Re-Scoring Using Image-Language Similarity for Few-Shot Object Detection [4.0208298639821525]
  Few-shot object detection, which focuses on detecting novel objects with few labels, is an emerging challenge in the community. Recent studies show that adapting a pre-trained model or modified loss function can improve performance. We propose Re-scoring using Image-language Similarity for Few-shot object detection (RISF) which extends Faster R-CNN.
  arXiv  Detail & Related papers  (2023-11-01T04:04:34Z)
• No Token Left Behind: Explainability-Aided Image Classification and Generation [79.4957965474334]
  We present a novel explainability-based approach, which adds a loss term to ensure that CLIP focuses on all relevant semantic parts of the input. Our method yields an improvement in the recognition rate, without additional training or fine-tuning.
  arXiv  Detail & Related papers  (2022-04-11T07:16:39Z)
• Unpaired Image Captioning by Image-level Weakly-Supervised Visual Concept Recognition [83.93422034664184]
  Unpaired image captioning (UIC) is to describe images without using image-caption pairs in the training phase. Most existing studies use off-the-shelf algorithms to obtain the visual concepts. We propose a novel approach to achieve cost-effective UIC using image-level labels.
  arXiv  Detail & Related papers  (2022-03-07T08:02:23Z)
• Unsupervised Part Discovery from Contrastive Reconstruction [90.88501867321573]
  The goal of self-supervised visual representation learning is to learn strong, transferable image representations. We propose an unsupervised approach to object part discovery and segmentation. Our method yields semantic parts consistent across fine-grained but visually distinct categories.
  arXiv  Detail & Related papers  (2021-11-11T17:59:42Z)
• Region-level Active Learning for Cluttered Scenes [60.93811392293329]
  We introduce a new strategy that subsumes previous Image-level and Object-level approaches into a generalized, Region-level approach. We show that this approach significantly decreases labeling effort and improves rare object search on realistic data with inherent class-imbalance and cluttered scenes.
  arXiv  Detail & Related papers  (2021-08-20T14:02:38Z)
• Rectifying the Shortcut Learning of Background: Shared Object Concentration for Few-Shot Image Recognition [101.59989523028264]
  Few-Shot image classification aims to utilize pretrained knowledge learned from a large-scale dataset to tackle a series of downstream classification tasks. We propose COSOC, a novel Few-Shot Learning framework, to automatically figure out foreground objects at both pretraining and evaluation stage.
  arXiv  Detail & Related papers  (2021-07-16T07:46:41Z)
• A Self-supervised GAN for Unsupervised Few-shot Object Recognition [39.79912546252623]
  This paper addresses unsupervised few-shot object recognition. All training images are unlabeled, and test images are divided into queries and a few labeled support images per object class of interest. We extend the vanilla GAN with two loss functions, both aimed at self-supervised learning.
  arXiv  Detail & Related papers  (2020-08-16T19:47:26Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.