Data-driven Feature Tracking for Event Cameras

• URL: http://arxiv.org/abs/2211.12826v3
• Date: Tue, 25 Apr 2023 16:27:51 GMT
• Title: Data-driven Feature Tracking for Event Cameras
• Authors: Nico Messikommer, Carter Fang, Mathias Gehrig, Davide Scaramuzza
• Abstract summary: We introduce the first data-driven feature tracker for event cameras, which leverages low-latency events to track features detected in a grayscale frame. By directly transferring zero-shot from synthetic to real data, our data-driven tracker outperforms existing approaches in relative feature age by up to 120%. This performance gap is further increased to 130% by adapting our tracker to real data with a novel self-supervision strategy.
• Score: 48.04815194265117
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Because of their high temporal resolution, increased resilience to motion blur, and very sparse output, event cameras have been shown to be ideal for low-latency and low-bandwidth feature tracking, even in challenging scenarios. Existing feature tracking methods for event cameras are either handcrafted or derived from first principles but require extensive parameter tuning, are sensitive to noise, and do not generalize to different scenarios due to unmodeled effects. To tackle these deficiencies, we introduce the first data-driven feature tracker for event cameras, which leverages low-latency events to track features detected in a grayscale frame. We achieve robust performance via a novel frame attention module, which shares information across feature tracks. By directly transferring zero-shot from synthetic to real data, our data-driven tracker outperforms existing approaches in relative feature age by up to 120% while also achieving the lowest latency. This performance gap is further increased to 130% by adapting our tracker to real data with a novel self-supervision strategy.

Related papers

• BlinkTrack: Feature Tracking over 100 FPS via Events and Images [50.98675227695814]
  We propose a novel framework, BlinkTrack, which integrates event data with RGB images for high-frequency feature tracking. Our method extends the traditional Kalman filter into a learning-based framework, utilizing differentiable Kalman filters in both event and image branches. Experimental results indicate that BlinkTrack significantly outperforms existing event-based methods.
  arXiv  Detail & Related papers  (2024-09-26T15:54:18Z)
• Exploring Dynamic Transformer for Efficient Object Tracking [58.120191254379854]
  We propose DyTrack, a dynamic transformer framework for efficient tracking. DyTrack automatically learns to configure proper reasoning routes for various inputs, gaining better utilization of the available computational budget. Experiments on multiple benchmarks demonstrate that DyTrack achieves promising speed-precision trade-offs with only a single model.
  arXiv  Detail & Related papers  (2024-03-26T12:31:58Z)
• Distractor-aware Event-based Tracking [45.07711356111249]
  We propose a distractor-aware event-based tracker that introduces transformer modules into Siamese network architecture (named DANet) Our model is mainly composed of a motion-aware network and a target-aware network, which simultaneously exploits both motion cues and object contours from event data. Our DANet can be trained in an end-to-end manner without any post-processing and can run at over 80 FPS on a single V100.
  arXiv  Detail & Related papers  (2023-10-22T05:50:20Z)
• Asynchronous Optimisation for Event-based Visual Odometry [53.59879499700895]
  Event cameras open up new possibilities for robotic perception due to their low latency and high dynamic range. We focus on event-based visual odometry (VO) We propose an asynchronous structure-from-motion optimisation back-end.
  arXiv  Detail & Related papers  (2022-03-02T11:28:47Z)
• Moving Object Detection for Event-based vision using Graph Spectral Clustering [6.354824287948164]
  Moving object detection has been a central topic of discussion in computer vision for its wide range of applications. We present an unsupervised Graph Spectral Clustering technique for Moving Object Detection in Event-based data. We additionally show how the optimum number of moving objects can be automatically determined.
  arXiv  Detail & Related papers  (2021-09-30T10:19:22Z)
• Bridging the Gap between Events and Frames through Unsupervised Domain Adaptation [57.22705137545853]
  We propose a task transfer method that allows models to be trained directly with labeled images and unlabeled event data. We leverage the generative event model to split event features into content and motion features. Our approach unlocks the vast amount of existing image datasets for the training of event-based neural networks.
  arXiv  Detail & Related papers  (2021-09-06T17:31:37Z)
• VisEvent: Reliable Object Tracking via Collaboration of Frame and Event Flows [93.54888104118822]
  We propose a large-scale Visible-Event benchmark (termed VisEvent) due to the lack of a realistic and scaled dataset for this task. Our dataset consists of 820 video pairs captured under low illumination, high speed, and background clutter scenarios. Based on VisEvent, we transform the event flows into event images and construct more than 30 baseline methods.
  arXiv  Detail & Related papers  (2021-08-11T03:55:12Z)
• Learning to Detect Objects with a 1 Megapixel Event Camera [14.949946376335305]
  Event cameras encode visual information with high temporal precision, low data-rate, and high-dynamic range. Due to the novelty of the field, the performance of event-based systems on many vision tasks is still lower compared to conventional frame-based solutions.
  arXiv  Detail & Related papers  (2020-09-28T16:03:59Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.