Rethinking Efficient and Effective Point-based Networks for Event Camera Classification and Regression: EventMamba

• URL: http://arxiv.org/abs/2405.06116v3
• Date: Wed, 3 Jul 2024 03:17:06 GMT
• Title: Rethinking Efficient and Effective Point-based Networks for Event Camera Classification and Regression: EventMamba
• Authors: Hongwei Ren, Yue Zhou, Jiadong Zhu, Haotian Fu, Yulong Huang, Xiaopeng Lin, Yuetong Fang, Fei Ma, Hao Yu, Bojun Cheng,
• Abstract summary: Event cameras efficiently detect changes in ambient light with low latency and high dynamic range while consuming minimal power. Most current approach to processing event data often involves converting it into frame-based representations. Point Cloud is a popular representation for 3D processing and is better suited to match the sparse and asynchronous nature of the event camera. We propose EventMamba, an efficient and effective Point Cloud framework that achieves competitive results even compared to the state-of-the-art (SOTA) frame-based method.
• Score: 11.400397931501338
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Event cameras, drawing inspiration from biological systems, efficiently detect changes in ambient light with low latency and high dynamic range while consuming minimal power. The most current approach to processing event data often involves converting it into frame-based representations, which is well-established in traditional vision. However, this approach neglects the sparsity of event data, loses fine-grained temporal information during the transformation process, and increases the computational burden, making it ineffective for characterizing event camera properties. In contrast, Point Cloud is a popular representation for 3D processing and is better suited to match the sparse and asynchronous nature of the event camera. Nevertheless, despite the theoretical compatibility of point-based methods with event cameras, the results show a performance gap that is not yet satisfactory compared to frame-based methods. In order to bridge the performance gap, we propose EventMamba, an efficient and effective Point Cloud framework that achieves competitive results even compared to the state-of-the-art (SOTA) frame-based method in both classification and regression tasks. This notable accomplishment is facilitated by our rethinking of the distinction between Event Cloud and Point Cloud, emphasizing effective temporal information extraction through optimized network structures. Specifically, EventMamba leverages temporal aggregation and State Space Model (SSM) based Mamba boasting enhanced temporal information extraction capabilities. Through a hierarchical structure, EventMamba is adept at abstracting local and global spatial features and implicit and explicit temporal features. By adhering to the lightweight design principle, EventMamba delivers impressive results with minimal computational resource utilization, demonstrating its efficiency and effectiveness.

Related papers

• Fast Window-Based Event Denoising with Spatiotemporal Correlation Enhancement [85.66867277156089]
  We propose window-based event denoising, which simultaneously deals with a stack of events. In spatial domain, we choose maximum a posteriori (MAP) to discriminate real-world event and noise. Our algorithm can remove event noise effectively and efficiently and improve the performance of downstream tasks.
  arXiv  Detail & Related papers  (2024-02-14T15:56:42Z)
• Representation Learning on Event Stream via an Elastic Net-incorporated Tensor Network [1.9515859963221267]
  We present a novel representation method which can capture global correlations of all events in the event stream simultaneously. Our method can achieve effective results in applications like filtering noise compared with the state-of-the-art methods.
  arXiv  Detail & Related papers  (2024-01-16T02:51:47Z)
• Implicit Event-RGBD Neural SLAM [54.74363487009845]
  Implicit neural SLAM has achieved remarkable progress recently. Existing methods face significant challenges in non-ideal scenarios. We propose EN-SLAM, the first event-RGBD implicit neural SLAM framework.
  arXiv  Detail & Related papers  (2023-11-18T08:48:58Z)
• Generalizing Event-Based Motion Deblurring in Real-World Scenarios [62.995994797897424]
  Event-based motion deblurring has shown promising results by exploiting low-latency events. We propose a scale-aware network that allows flexible input spatial scales and enables learning from different temporal scales of motion blur. A two-stage self-supervised learning scheme is then developed to fit real-world data distribution.
  arXiv  Detail & Related papers  (2023-08-11T04:27:29Z)
• HALSIE: Hybrid Approach to Learning Segmentation by Simultaneously Exploiting Image and Event Modalities [6.543272301133159]
  Event cameras detect changes in per-pixel intensity to generate asynchronous event streams. They offer great potential for accurate semantic map retrieval in real-time autonomous systems. Existing implementations for event segmentation suffer from sub-based performance. We propose hybrid end-to-end learning framework HALSIE to reduce inference cost by up to $20times$ versus art.
  arXiv  Detail & Related papers  (2022-11-19T17:09:50Z)
• Asynchronous Optimisation for Event-based Visual Odometry [53.59879499700895]
  Event cameras open up new possibilities for robotic perception due to their low latency and high dynamic range. We focus on event-based visual odometry (VO) We propose an asynchronous structure-from-motion optimisation back-end.
  arXiv  Detail & Related papers  (2022-03-02T11:28:47Z)
• AET-EFN: A Versatile Design for Static and Dynamic Event-Based Vision [33.4444564715323]
  Event data are noisy, sparse, and nonuniform in the spatial-temporal domain with an extremely high temporal resolution. Existing methods encode events into point-cloud-based or voxel-based representations, but suffer from noise and/or information loss. This work proposes the Aligned Event Frame (AET) as a novel event data representation, and a neat framework called Event Frame Net (EFN) The proposed AET and EFN are evaluated on various datasets, and proved to surpass existing state-of-the-art methods by large margins.
  arXiv  Detail & Related papers  (2021-03-22T08:09:03Z)
• Unsupervised Feature Learning for Event Data: Direct vs Inverse Problem Formulation [53.850686395708905]
  Event-based cameras record an asynchronous stream of per-pixel brightness changes. In this paper, we focus on single-layer architectures for representation learning from event data. We show improvements of up to 9 % in the recognition accuracy compared to the state-of-the-art methods.
  arXiv  Detail & Related papers  (2020-09-23T10:40:03Z)
• Event-based visual place recognition with ensembles of temporal windows [29.6328152991222]
  Event cameras are bio-inspired sensors capable of providing a continuous stream of events with low latency and high range. We develop a novel ensemble-based scheme for combining temporal windows of varying lengths that are processed in parallel. We show that our proposed ensemble scheme significantly outperforms all single-window baselines and conventional model-based ensembles.
  arXiv  Detail & Related papers  (2020-05-22T05:33:35Z)
• Event-based Asynchronous Sparse Convolutional Networks [54.094244806123235]
  Event cameras are bio-inspired sensors that respond to per-pixel brightness changes in the form of asynchronous and sparse "events" We present a general framework for converting models trained on synchronous image-like event representations into asynchronous models with identical output. We show both theoretically and experimentally that this drastically reduces the computational complexity and latency of high-capacity, synchronous neural networks.
  arXiv  Detail & Related papers  (2020-03-20T08:39:49Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.