Event Stream based Human Action Recognition: A High-Definition Benchmark Dataset and Algorithms

• URL: http://arxiv.org/abs/2408.09764v1
• Date: Mon, 19 Aug 2024 07:52:20 GMT
• Title: Event Stream based Human Action Recognition: A High-Definition Benchmark Dataset and Algorithms
• Authors: Xiao Wang, Shiao Wang, Pengpeng Shao, Bo Jiang, Lin Zhu, Yonghong Tian,
• Abstract summary: We propose a large-scale, high-definition ($1280 times 800$) human action recognition dataset based on the CeleX-V event camera. To build a more comprehensive benchmark dataset, we report over 20 mainstream HAR models for future works to compare.
• Score: 29.577583619354314
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Human Action Recognition (HAR) stands as a pivotal research domain in both computer vision and artificial intelligence, with RGB cameras dominating as the preferred tool for investigation and innovation in this field. However, in real-world applications, RGB cameras encounter numerous challenges, including light conditions, fast motion, and privacy concerns. Consequently, bio-inspired event cameras have garnered increasing attention due to their advantages of low energy consumption, high dynamic range, etc. Nevertheless, most existing event-based HAR datasets are low resolution ($346 \times 260$). In this paper, we propose a large-scale, high-definition ($1280 \times 800$) human action recognition dataset based on the CeleX-V event camera, termed CeleX-HAR. It encompasses 150 commonly occurring action categories, comprising a total of 124,625 video sequences. Various factors such as multi-view, illumination, action speed, and occlusion are considered when recording these data. To build a more comprehensive benchmark dataset, we report over 20 mainstream HAR models for future works to compare. In addition, we also propose a novel Mamba vision backbone network for event stream based HAR, termed EVMamba, which equips the spatial plane multi-directional scanning and novel voxel temporal scanning mechanism. By encoding and mining the spatio-temporal information of event streams, our EVMamba has achieved favorable results across multiple datasets. Both the dataset and source code will be released on \url{https://github.com/Event-AHU/CeleX-HAR}

Related papers

• SpikMamba: When SNN meets Mamba in Event-based Human Action Recognition [13.426390494116776]
  Human action recognition (HAR) plays a key role in various applications such as video analysis, surveillance, autonomous driving, robotics, and healthcare. Most HAR algorithms are developed from RGB images, which capture detailed visual information. Event cameras offer a promising solution by capturing scene brightness changes sparsely at the pixel level, without capturing full images.
  arXiv  Detail & Related papers  (2024-10-22T07:00:43Z)
• DailyDVS-200: A Comprehensive Benchmark Dataset for Event-Based Action Recognition [51.96660522869841]
  DailyDVS-200 is a benchmark dataset tailored for the event-based action recognition community. It covers 200 action categories across real-world scenarios, recorded by 47 participants, and comprises more than 22,000 event sequences. DailyDVS-200 is annotated with 14 attributes, ensuring a detailed characterization of the recorded actions.
  arXiv  Detail & Related papers  (2024-07-06T15:25:10Z)
• Event Stream-based Visual Object Tracking: A High-Resolution Benchmark Dataset and A Novel Baseline [38.42400442371156]
  Existing works either utilize aligned RGB and event data for accurate tracking or directly learn an event-based tracker. We propose a novel hierarchical knowledge distillation framework that can fully utilize multi-modal / multi-view information during training to facilitate knowledge transfer. We propose the first large-scale high-resolution ($1280 times 720$) dataset named EventVOT. It contains 1141 videos and covers a wide range of categories such as pedestrians, vehicles, UAVs, ping pongs, etc.
  arXiv  Detail & Related papers  (2023-09-26T01:42:26Z)
• EventTransAct: A video transformer-based framework for Event-camera based action recognition [52.537021302246664]
  Event cameras offer new opportunities compared to standard action recognition in RGB videos. In this study, we employ a computationally efficient model, namely the video transformer network (VTN), which initially acquires spatial embeddings per event-frame. In order to better adopt the VTN for the sparse and fine-grained nature of event data, we design Event-Contrastive Loss ($mathcalL_EC$) and event-specific augmentations.
  arXiv  Detail & Related papers  (2023-08-25T23:51:07Z)
• Event-based Simultaneous Localization and Mapping: A Comprehensive Survey [52.73728442921428]
  Review of event-based vSLAM algorithms that exploit the benefits of asynchronous and irregular event streams for localization and mapping tasks. Paper categorizes event-based vSLAM methods into four main categories: feature-based, direct, motion-compensation, and deep learning methods.
  arXiv  Detail & Related papers  (2023-04-19T16:21:14Z)
• Dual Memory Aggregation Network for Event-Based Object Detection with Learnable Representation [79.02808071245634]
  Event-based cameras are bio-inspired sensors that capture brightness change of every pixel in an asynchronous manner. Event streams are divided into grids in the x-y-t coordinates for both positive and negative polarity, producing a set of pillars as 3D tensor representation. Long memory is encoded in the hidden state of adaptive convLSTMs while short memory is modeled by computing spatial-temporal correlation between event pillars.
  arXiv  Detail & Related papers  (2023-03-17T12:12:41Z)
• HARDVS: Revisiting Human Activity Recognition with Dynamic Vision Sensors [40.949347728083474]
  The main streams of human activity recognition (HAR) algorithms are developed based on RGB cameras which are suffered from illumination, fast motion, privacy-preserving, and large energy consumption. Meanwhile, the biologically inspired event cameras attracted great interest due to their unique features, such as high dynamic range, dense temporal but sparse spatial resolution, low latency, low power, etc. As it is a newly arising sensor, even there is no realistic large-scale dataset for HAR. We propose a large-scale benchmark dataset, termed HARDVS, which contains 300 categories and more than 100K event sequences.
  arXiv  Detail & Related papers  (2022-11-17T16:48:50Z)
• TUM-VIE: The TUM Stereo Visual-Inertial Event Dataset [50.8779574716494]
  Event cameras are bio-inspired vision sensors which measure per pixel brightness changes. They offer numerous benefits over traditional, frame-based cameras, including low latency, high dynamic range, high temporal resolution and low power consumption. To foster the development of 3D perception and navigation algorithms with event cameras, we present the TUM-VIE dataset.
  arXiv  Detail & Related papers  (2021-08-16T19:53:56Z)
• VisEvent: Reliable Object Tracking via Collaboration of Frame and Event Flows [93.54888104118822]
  We propose a large-scale Visible-Event benchmark (termed VisEvent) due to the lack of a realistic and scaled dataset for this task. Our dataset consists of 820 video pairs captured under low illumination, high speed, and background clutter scenarios. Based on VisEvent, we transform the event flows into event images and construct more than 30 baseline methods.
  arXiv  Detail & Related papers  (2021-08-11T03:55:12Z)
• Event-based Robotic Grasping Detection with Neuromorphic Vision Sensor and Event-Stream Dataset [8.030163836902299]
  Neuromorphic vision is a small and young community of research. Compared to traditional frame-based computer vision, neuromorphic vision is a small and young community of research. We construct a robotic grasping dataset named Event-Stream dataset with 91 objects. As leds blink at high frequency, the Event-Stream dataset is annotated in a high frequency of 1 kHz. We develop a deep neural network for grasping detection which consider the angle learning problem as classification instead of regression.
  arXiv  Detail & Related papers  (2020-04-28T16:55:19Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.