Matching Neuromorphic Events and Color Images via Adversarial Learning

• URL: http://arxiv.org/abs/2003.00636v1
• Date: Mon, 2 Mar 2020 02:48:56 GMT
• Title: Matching Neuromorphic Events and Color Images via Adversarial Learning
• Authors: Fang Xu, Shijie Lin, Wen Yang, Lei Yu, Dengxin Dai, Gui-song Xia
• Abstract summary: We propose the Event-Based Image Retrieval (EBIR) problem to exploit the cross-modal matching task. We address the EBIR problem by proposing neuromorphic Events-Color image Feature Learning (ECFL) We also contribute to the community N-UKbench and EC180 dataset to promote the development of EBIR problem.
• Score: 49.447580124957966
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: The event camera has appealing properties: high dynamic range, low latency, low power consumption and low memory usage, and thus provides complementariness to conventional frame-based cameras. It only captures the dynamics of a scene and is able to capture almost "continuous" motion. However, different from frame-based camera that reflects the whole appearance as scenes are, the event camera casts away the detailed characteristics of objects, such as texture and color. To take advantages of both modalities, the event camera and frame-based camera are combined together for various machine vision tasks. Then the cross-modal matching between neuromorphic events and color images plays a vital and essential role. In this paper, we propose the Event-Based Image Retrieval (EBIR) problem to exploit the cross-modal matching task. Given an event stream depicting a particular object as query, the aim is to retrieve color images containing the same object. This problem is challenging because there exists a large modality gap between neuromorphic events and color images. We address the EBIR problem by proposing neuromorphic Events-Color image Feature Learning (ECFL). Particularly, the adversarial learning is employed to jointly model neuromorphic events and color images into a common embedding space. We also contribute to the community N-UKbench and EC180 dataset to promote the development of EBIR problem. Extensive experiments on our datasets show that the proposed method is superior in learning effective modality-invariant representation to link two different modalities.

Related papers

• EvPlug: Learn a Plug-and-Play Module for Event and Image Fusion [55.367269556557645]
  EvPlug learns a plug-and-play event and image fusion module from the supervision of the existing RGB-based model. We demonstrate the superiority of EvPlug in several vision tasks such as object detection, semantic segmentation, and 3D hand pose estimation.
  arXiv  Detail & Related papers  (2023-12-28T10:05:13Z)
• Implicit Event-RGBD Neural SLAM [54.74363487009845]
  Implicit neural SLAM has achieved remarkable progress recently. Existing methods face significant challenges in non-ideal scenarios. We propose EN-SLAM, the first event-RGBD implicit neural SLAM framework.
  arXiv  Detail & Related papers  (2023-11-18T08:48:58Z)
• Neuromorphic Imaging and Classification with Graph Learning [11.882239213276392]
  Bio-inspired neuromorphic cameras asynchronously record pixel brightness changes and generate sparse event streams. Due to the multidimensional address-event structure, most existing vision algorithms cannot properly handle asynchronous event streams. We propose a new graph representation of the event data and couple it with a Graph Transformer to perform accurate neuromorphic classification.
  arXiv  Detail & Related papers  (2023-09-27T12:58:18Z)
• Deformable Neural Radiance Fields using RGB and Event Cameras [65.40527279809474]
  We develop a novel method to model the deformable neural radiance fields using RGB and event cameras. The proposed method uses the asynchronous stream of events and sparse RGB frames. Experiments conducted on both realistically rendered graphics and real-world datasets demonstrate a significant benefit of the proposed method.
  arXiv  Detail & Related papers  (2023-09-15T14:19:36Z)
• Event-based Image Deblurring with Dynamic Motion Awareness [10.81953574179206]
  We introduce the first dataset containing pairs of real RGB blur images and related events during the exposure time. Our results show better robustness overall when using events, with improvements in PSNR by up to 1.57dB on synthetic data and 1.08 dB on real event data.
  arXiv  Detail & Related papers  (2022-08-24T09:39:55Z)
• End-to-end Multi-modal Video Temporal Grounding [105.36814858748285]
  We propose a multi-modal framework to extract complementary information from videos. We adopt RGB images for appearance, optical flow for motion, and depth maps for image structure. We conduct experiments on the Charades-STA and ActivityNet Captions datasets, and show that the proposed method performs favorably against state-of-the-art approaches.
  arXiv  Detail & Related papers  (2021-07-12T17:58:10Z)
• Event-based Motion Segmentation with Spatio-Temporal Graph Cuts [51.17064599766138]
  We have developed a method to identify independently objects acquired with an event-based camera. The method performs on par or better than the state of the art without having to predetermine the number of expected moving objects.
  arXiv  Detail & Related papers  (2020-12-16T04:06:02Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.