Learning Dense and Continuous Optical Flow from an Event Camera
- URL: http://arxiv.org/abs/2211.09078v1
- Date: Wed, 16 Nov 2022 17:53:18 GMT
- Title: Learning Dense and Continuous Optical Flow from an Event Camera
- Authors: Zhexiong Wan, Yuchao Dai, Yuxin Mao
- Abstract summary: Event cameras such as DAVIS can simultaneously output high temporal resolution events and low frame-rate intensity images.
Most of the existing optical flow estimation methods are based on two consecutive image frames and can only estimate discrete flow at a fixed time interval.
We propose a novel deep learning-based dense and continuous optical flow estimation framework from a single image with event streams.
- Score: 28.77846425802558
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: Event cameras such as DAVIS can simultaneously output high temporal
resolution events and low frame-rate intensity images, which own great
potential in capturing scene motion, such as optical flow estimation. Most of
the existing optical flow estimation methods are based on two consecutive image
frames and can only estimate discrete flow at a fixed time interval. Previous
work has shown that continuous flow estimation can be achieved by changing the
quantities or time intervals of events. However, they are difficult to estimate
reliable dense flow , especially in the regions without any triggered events.
In this paper, we propose a novel deep learning-based dense and continuous
optical flow estimation framework from a single image with event streams, which
facilitates the accurate perception of high-speed motion. Specifically, we
first propose an event-image fusion and correlation module to effectively
exploit the internal motion from two different modalities of data. Then we
propose an iterative update network structure with bidirectional training for
optical flow prediction. Therefore, our model can estimate reliable dense flow
as two-frame-based methods, as well as estimate temporal continuous flow as
event-based methods. Extensive experimental results on both synthetic and real
captured datasets demonstrate that our model outperforms existing event-based
state-of-the-art methods and our designed baselines for accurate dense and
continuous optical flow estimation.
Related papers
- Motion-prior Contrast Maximization for Dense Continuous-Time Motion Estimation [34.529280562470746]
We introduce a novel self-supervised loss combining the Contrast Maximization framework with a non-linear motion prior in the form of pixel-level trajectories.
Their effectiveness is demonstrated in two scenarios: In dense continuous-time motion estimation, our method improves the zero-shot performance of a synthetically trained model by 29%.
arXiv Detail & Related papers (2024-07-15T15:18:28Z) - MemFlow: Optical Flow Estimation and Prediction with Memory [54.22820729477756]
We present MemFlow, a real-time method for optical flow estimation and prediction with memory.
Our method enables memory read-out and update modules for aggregating historical motion information in real-time.
Our approach seamlessly extends to the future prediction of optical flow based on past observations.
arXiv Detail & Related papers (2024-04-07T04:56:58Z) - Towards Anytime Optical Flow Estimation with Event Cameras [35.685866753715416]
Event cameras are capable of responding to log-brightness changes in microseconds.
Existing datasets collected via event cameras provide limited frame rate optical flow ground truth.
We propose EVA-Flow, an EVent-based Anytime Flow estimation network to produce high-frame-rate event optical flow.
arXiv Detail & Related papers (2023-07-11T06:15:12Z) - SSTM: Spatiotemporal Recurrent Transformers for Multi-frame Optical Flow
Estimation [0.0]
In optical flow estimates in and near cluded regions, and out-of-boundary regions are two of the current significant limitations of optical flow estimation algorithms.
Recent state-of-the-art optical flow estimation algorithms are two-frame based methods where optical flow is estimated sequentially for each consecutive image pair in a sequence.
We propose a learning-based multi-frame optical flow estimation method that estimates two or more consecutive optical flows in parallel from multi-frame image sequences.
arXiv Detail & Related papers (2023-04-26T23:39:40Z) - SCFlow: Optical Flow Estimation for Spiking Camera [50.770803466875364]
Spiking camera has enormous potential in real applications, especially for motion estimation in high-speed scenes.
Optical flow estimation has achieved remarkable success in image-based and event-based vision, but % existing methods cannot be directly applied in spike stream from spiking camera.
This paper presents, SCFlow, a novel deep learning pipeline for optical flow estimation for spiking camera.
arXiv Detail & Related papers (2021-10-08T06:16:45Z) - Dense Optical Flow from Event Cameras [55.79329250951028]
We propose to incorporate feature correlation and sequential processing into dense optical flow estimation from event cameras.
Our proposed approach computes dense optical flow and reduces the end-point error by 23% on MVSEC.
arXiv Detail & Related papers (2021-08-24T07:39:08Z) - TimeLens: Event-based Video Frame Interpolation [54.28139783383213]
We introduce Time Lens, a novel indicates equal contribution method that leverages the advantages of both synthesis-based and flow-based approaches.
We show an up to 5.21 dB improvement in terms of PSNR over state-of-the-art frame-based and event-based methods.
arXiv Detail & Related papers (2021-06-14T10:33:47Z) - Unsupervised Motion Representation Enhanced Network for Action
Recognition [4.42249337449125]
Motion representation between consecutive frames has proven to have great promotion to video understanding.
TV-L1 method, an effective optical flow solver, is time-consuming and expensive in storage for caching the extracted optical flow.
We propose UF-TSN, a novel end-to-end action recognition approach enhanced with an embedded lightweight unsupervised optical flow estimator.
arXiv Detail & Related papers (2021-03-05T04:14:32Z) - Learning Monocular Dense Depth from Events [53.078665310545745]
Event cameras produce brightness changes in the form of a stream of asynchronous events instead of intensity frames.
Recent learning-based approaches have been applied to event-based data, such as monocular depth prediction.
We propose a recurrent architecture to solve this task and show significant improvement over standard feed-forward methods.
arXiv Detail & Related papers (2020-10-16T12:36:23Z) - STaRFlow: A SpatioTemporal Recurrent Cell for Lightweight Multi-Frame
Optical Flow Estimation [64.99259320624148]
We present a new lightweight CNN-based algorithm for multi-frame optical flow estimation.
The resulting STaRFlow algorithm gives state-of-the-art performances on MPI Sintel and Kitti2015.
arXiv Detail & Related papers (2020-07-10T17:01:34Z) - Single Image Optical Flow Estimation with an Event Camera [38.92408855196647]
Event cameras are bio-inspired sensors that report intensity changes in microsecond resolution.
We propose a single image (potentially blurred) and events based optical flow estimation approach.
arXiv Detail & Related papers (2020-04-01T11:28:30Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.