BlinkVision: A Benchmark for Optical Flow, Scene Flow and Point Tracking Estimation using RGB Frames and Events

• URL: http://arxiv.org/abs/2410.20451v1
• Date: Sun, 27 Oct 2024 13:59:21 GMT
• Title: BlinkVision: A Benchmark for Optical Flow, Scene Flow and Point Tracking Estimation using RGB Frames and Events
• Authors: Yijin Li, Yichen Shen, Zhaoyang Huang, Shuo Chen, Weikang Bian, Xiaoyu Shi, Fu-Yun Wang, Keqiang Sun, Hujun Bao, Zhaopeng Cui, Guofeng Zhang, Hongsheng Li,
• Abstract summary: We propose BlinkVision, a large-scale and diverse benchmark with multiple modalities and dense correspondence annotations. BlinkVision delivers photorealistic data and covers various naturalistic factors, such as camera shake and deformation. It enables extensive benchmarks on three types of correspondence tasks (optical flow, point tracking, and scene flow estimation) for both image-based and event-based methods.
• Score: 72.25918104830252
• License:
• Abstract: Recent advances in event-based vision suggest that these systems complement traditional cameras by providing continuous observation without frame rate limitations and a high dynamic range, making them well-suited for correspondence tasks such as optical flow and point tracking. However, there is still a lack of comprehensive benchmarks for correspondence tasks that include both event data and images. To address this gap, we propose BlinkVision, a large-scale and diverse benchmark with multiple modalities and dense correspondence annotations. BlinkVision offers several valuable features: 1) Rich modalities: It includes both event data and RGB images. 2) Extensive annotations: It provides dense per-pixel annotations covering optical flow, scene flow, and point tracking. 3) Large vocabulary: It contains 410 everyday categories, sharing common classes with popular 2D and 3D datasets like LVIS and ShapeNet. 4) Naturalistic: It delivers photorealistic data and covers various naturalistic factors, such as camera shake and deformation. BlinkVision enables extensive benchmarks on three types of correspondence tasks (optical flow, point tracking, and scene flow estimation) for both image-based and event-based methods, offering new observations, practices, and insights for future research. The benchmark website is https://www.blinkvision.net/.

Related papers

• BlinkTrack: Feature Tracking over 100 FPS via Events and Images [50.98675227695814]
  We propose a novel framework, BlinkTrack, which integrates event data with RGB images for high-frequency feature tracking. Our method extends the traditional Kalman filter into a learning-based framework, utilizing differentiable Kalman filters in both event and image branches. Experimental results indicate that BlinkTrack significantly outperforms existing event-based methods.
  arXiv  Detail & Related papers  (2024-09-26T15:54:18Z)
• MambaPupil: Bidirectional Selective Recurrent model for Event-based Eye tracking [50.26836546224782]
  Event-based eye tracking has shown great promise with the high temporal resolution and low redundancy. The diversity and abruptness of eye movement patterns, including blinking, fixating, saccades, and smooth pursuit, pose significant challenges for eye localization. This paper proposes a bidirectional long-term sequence modeling and time-varying state selection mechanism to fully utilize contextual temporal information.
  arXiv  Detail & Related papers  (2024-04-18T11:09:25Z)
• RPEFlow: Multimodal Fusion of RGB-PointCloud-Event for Joint Optical Flow and Scene Flow Estimation [43.358140897849616]
  In this paper, we incorporate RGB images, Point clouds and Events for joint optical flow and scene flow estimation with our proposed multi-stage multimodal fusion model, RPEFlow. Experiments on both synthetic and real datasets show that our model outperforms the existing state-of-the-art by a wide margin.
  arXiv  Detail & Related papers  (2023-09-26T17:23:55Z)
• NPF-200: A Multi-Modal Eye Fixation Dataset and Method for Non-Photorealistic Videos [51.409547544747284]
  NPF-200 is the first large-scale multi-modal dataset of purely non-photorealistic videos with eye fixations. We conduct a series of analyses to gain deeper insights into this task. We propose a universal frequency-aware multi-modal non-photorealistic saliency detection model called NPSNet.
  arXiv  Detail & Related papers  (2023-08-23T14:25:22Z)
• Modeling Continuous Motion for 3D Point Cloud Object Tracking [54.48716096286417]
  This paper presents a novel approach that views each tracklet as a continuous stream. At each timestamp, only the current frame is fed into the network to interact with multi-frame historical features stored in a memory bank. To enhance the utilization of multi-frame features for robust tracking, a contrastive sequence enhancement strategy is proposed.
  arXiv  Detail & Related papers  (2023-03-14T02:58:27Z)
• DarkVision: A Benchmark for Low-light Image/Video Perception [44.94878263751042]
  We contribute the first multi-illuminance, multi-camera, and low-light dataset, named DarkVision, for both image enhancement and object detection. The dataset consists of bright-dark pairs of 900 static scenes with objects from 15 categories, and 32 dynamic scenes with 4-category objects. For each scene, images/videos were captured at 5 illuminance levels using three cameras of different grades, and average photons can be reliably estimated.
  arXiv  Detail & Related papers  (2023-01-16T05:55:59Z)
• Residual 3D Scene Flow Learning with Context-Aware Feature Extraction [11.394559627312743]
  We propose a novel context-aware set conv layer to exploit contextual structure information of Euclidean space. We also propose an explicit residual flow learning structure in the residual flow refinement layer to cope with long-distance movement. Our method achieves state-of-the-art performance, surpassing all other previous works to the best of our knowledge by at least 25%.
  arXiv  Detail & Related papers  (2021-09-10T06:15:18Z)
• VisEvent: Reliable Object Tracking via Collaboration of Frame and Event Flows [93.54888104118822]
  We propose a large-scale Visible-Event benchmark (termed VisEvent) due to the lack of a realistic and scaled dataset for this task. Our dataset consists of 820 video pairs captured under low illumination, high speed, and background clutter scenarios. Based on VisEvent, we transform the event flows into event images and construct more than 30 baseline methods.
  arXiv  Detail & Related papers  (2021-08-11T03:55:12Z)
• Spot What Matters: Learning Context Using Graph Convolutional Networks for Weakly-Supervised Action Detection [0.0]
  We introduce an architecture based on self-attention and Convolutional Networks to improve human action detection in video. Our model aids explainability by visualizing the learned context as an attention map, even for actions and objects unseen during training. Experimental results show that our contextualized approach outperforms a baseline action detection approach by more than 2 points in Video-mAP.
  arXiv  Detail & Related papers  (2021-07-28T21:37:18Z)
• Hierarchical Attention Learning of Scene Flow in 3D Point Clouds [28.59260783047209]
  This paper studies the problem of scene flow estimation from two consecutive 3D point clouds. A novel hierarchical neural network with double attention is proposed for learning the correlation of point features in adjacent frames. Experiments show that the proposed network outperforms the state-of-the-art performance of 3D scene flow estimation.
  arXiv  Detail & Related papers  (2020-10-12T14:56:08Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.