SpikeMOT: Event-based Multi-Object Tracking with Sparse Motion Features

• URL: http://arxiv.org/abs/2309.16987v1
• Date: Fri, 29 Sep 2023 05:13:43 GMT
• Title: SpikeMOT: Event-based Multi-Object Tracking with Sparse Motion Features
• Authors: Song Wang, Zhu Wang, Can Li, Xiaojuan Qi, Hayden Kwok-Hay So
• Abstract summary: SpikeMOT is an event-based multi-object tracker. SpikeMOT uses spiking neural networks to extract sparsetemporal features from event streams associated with objects.
• Score: 52.213656737672935
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: In comparison to conventional RGB cameras, the superior temporal resolution of event cameras allows them to capture rich information between frames, making them prime candidates for object tracking. Yet in practice, despite their theoretical advantages, the body of work on event-based multi-object tracking (MOT) remains in its infancy, especially in real-world settings where events from complex background and camera motion can easily obscure the true target motion. In this work, an event-based multi-object tracker, called SpikeMOT, is presented to address these challenges. SpikeMOT leverages spiking neural networks to extract sparse spatiotemporal features from event streams associated with objects. The resulting spike train representations are used to track the object movement at high frequency, while a simultaneous object detector provides updated spatial information of these objects at an equivalent frame rate. To evaluate the effectiveness of SpikeMOT, we introduce DSEC-MOT, the first large-scale event-based MOT benchmark incorporating fine-grained annotations for objects experiencing severe occlusions, frequent trajectory intersections, and long-term re-identification in real-world contexts. Extensive experiments employing DSEC-MOT and another event-based dataset, named FE240hz, demonstrate SpikeMOT's capability to achieve high tracking accuracy amidst challenging real-world scenarios, advancing the state-of-the-art in event-based multi-object tracking.

Related papers

• BlinkTrack: Feature Tracking over 100 FPS via Events and Images [50.98675227695814]
  We propose a novel framework, BlinkTrack, which integrates event data with RGB images for high-frequency feature tracking. Our method extends the traditional Kalman filter into a learning-based framework, utilizing differentiable Kalman filters in both event and image branches. Experimental results indicate that BlinkTrack significantly outperforms existing event-based methods.
  arXiv  Detail & Related papers  (2024-09-26T15:54:18Z)
• Distractor-aware Event-based Tracking [45.07711356111249]
  We propose a distractor-aware event-based tracker that introduces transformer modules into Siamese network architecture (named DANet) Our model is mainly composed of a motion-aware network and a target-aware network, which simultaneously exploits both motion cues and object contours from event data. Our DANet can be trained in an end-to-end manner without any post-processing and can run at over 80 FPS on a single V100.
  arXiv  Detail & Related papers  (2023-10-22T05:50:20Z)
• MotionTrack: Learning Motion Predictor for Multiple Object Tracking [68.68339102749358]
  We introduce a novel motion-based tracker, MotionTrack, centered around a learnable motion predictor. Our experimental results demonstrate that MotionTrack yields state-of-the-art performance on datasets such as Dancetrack and SportsMOT.
  arXiv  Detail & Related papers  (2023-06-05T04:24:11Z)
• Event-Free Moving Object Segmentation from Moving Ego Vehicle [88.33470650615162]
  Moving object segmentation (MOS) in dynamic scenes is an important, challenging, but under-explored research topic for autonomous driving. Most segmentation methods leverage motion cues obtained from optical flow maps. We propose to exploit event cameras for better video understanding, which provide rich motion cues without relying on optical flow.
  arXiv  Detail & Related papers  (2023-04-28T23:43:10Z)
• Dual Memory Aggregation Network for Event-Based Object Detection with Learnable Representation [79.02808071245634]
  Event-based cameras are bio-inspired sensors that capture brightness change of every pixel in an asynchronous manner. Event streams are divided into grids in the x-y-t coordinates for both positive and negative polarity, producing a set of pillars as 3D tensor representation. Long memory is encoded in the hidden state of adaptive convLSTMs while short memory is modeled by computing spatial-temporal correlation between event pillars.
  arXiv  Detail & Related papers  (2023-03-17T12:12:41Z)
• RGB-Event Fusion for Moving Object Detection in Autonomous Driving [3.5397758597664306]
  Moving Object Detection (MOD) is a critical vision task for successfully achieving safe autonomous driving. Recent advances in sensor technologies, especially the Event camera, can naturally complement the conventional camera approach to better model moving objects. We propose RENet, a novel RGB-Event fusion Network, that jointly exploits the two complementary modalities to achieve more robust MOD.
  arXiv  Detail & Related papers  (2022-09-17T12:59:08Z)
• IA-MOT: Instance-Aware Multi-Object Tracking with Motion Consistency [40.354708148590696]
  "instance-aware MOT" (IA-MOT) can track multiple objects in either static or moving cameras. Our proposed method won the first place in Track 3 of the BMTT Challenge in CVPR 2020 workshops.
  arXiv  Detail & Related papers  (2020-06-24T03:53:36Z)
• End-to-end Learning of Object Motion Estimation from Retinal Events for Event-based Object Tracking [35.95703377642108]
  We propose a novel deep neural network to learn and regress a parametric object-level motion/transform model for event-based object tracking. To achieve this goal, we propose a synchronous Time-Surface with Linear Time Decay representation. We feed the sequence of TSLTD frames to a novel Retinal Motion Regression Network (RMRNet) perform to an end-to-end 5-DoF object motion regression.
  arXiv  Detail & Related papers  (2020-02-14T08:19:50Z)
• Asynchronous Tracking-by-Detection on Adaptive Time Surfaces for Event-based Object Tracking [87.0297771292994]
  We propose an Event-based Tracking-by-Detection (ETD) method for generic bounding box-based object tracking. To achieve this goal, we present an Adaptive Time-Surface with Linear Time Decay (ATSLTD) event-to-frame conversion algorithm. We compare the proposed ETD method with seven popular object tracking methods, that are based on conventional cameras or event cameras, and two variants of ETD.
  arXiv  Detail & Related papers  (2020-02-13T15:58:31Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.