TMA: Temporal Motion Aggregation for Event-based Optical Flow

• URL: http://arxiv.org/abs/2303.11629v2
• Date: Mon, 21 Aug 2023 07:07:43 GMT
• Title: TMA: Temporal Motion Aggregation for Event-based Optical Flow
• Authors: Haotian Liu, Guang Chen, Sanqing Qu, Yanping Zhang, Zhijun Li, Alois Knoll and Changjun Jiang
• Abstract summary: Event cameras have the ability to record continuous and detailed trajectories of objects with high temporal resolution. Most existing learning-based approaches for event optical flow estimation ignore the inherent temporal continuity of event data. We propose a novel Temporal Motion Aggregation (TMA) approach to unlock its potential.
• Score: 27.49029251605363
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Event cameras have the ability to record continuous and detailed trajectories of objects with high temporal resolution, thereby providing intuitive motion cues for optical flow estimation. Nevertheless, most existing learning-based approaches for event optical flow estimation directly remould the paradigm of conventional images by representing the consecutive event stream as static frames, ignoring the inherent temporal continuity of event data. In this paper, we argue that temporal continuity is a vital element of event-based optical flow and propose a novel Temporal Motion Aggregation (TMA) approach to unlock its potential. Technically, TMA comprises three components: an event splitting strategy to incorporate intermediate motion information underlying the temporal context, a linear lookup strategy to align temporally fine-grained motion features and a novel motion pattern aggregation module to emphasize consistent patterns for motion feature enhancement. By incorporating temporally fine-grained motion information, TMA can derive better flow estimates than existing methods at early stages, which not only enables TMA to obtain more accurate final predictions, but also greatly reduces the demand for a number of refinements. Extensive experiments on DSEC-Flow and MVSEC datasets verify the effectiveness and superiority of our TMA. Remarkably, compared to E-RAFT, TMA achieves a 6\% improvement in accuracy and a 40\% reduction in inference time on DSEC-Flow. Code will be available at \url{https://github.com/ispc-lab/TMA}.

Related papers

• Motion-prior Contrast Maximization for Dense Continuous-Time Motion Estimation [34.529280562470746]
  We introduce a novel self-supervised loss combining the Contrast Maximization framework with a non-linear motion prior in the form of pixel-level trajectories. Their effectiveness is demonstrated in two scenarios: In dense continuous-time motion estimation, our method improves the zero-shot performance of a synthetically trained model by 29%.
  arXiv  Detail & Related papers  (2024-07-15T15:18:28Z)
• Temporal Residual Guided Diffusion Framework for Event-Driven Video Reconstruction [23.493813870675197]
  Event-based video reconstruction has garnered increasing attention due to its advantages, such as high dynamic range and rapid motion capture capabilities. Current methods often prioritize the extraction of temporal information from continuous event flow, leading to an overemphasis on low-frequency texture features in the scene. We introduce a novel approach, the Temporal Residual Guided Diffusion Framework, which effectively leverages both temporal and frequency-based event priors.
  arXiv  Detail & Related papers  (2024-07-15T11:48:57Z)
• Generalizable Implicit Motion Modeling for Video Frame Interpolation [51.966062283735596]
  Motion is critical in flow-based Video Frame Interpolation (VFI) We introduce General Implicit Motion Modeling (IMM), a novel and effective approach to motion modeling VFI. Our GIMM can be easily integrated with existing flow-based VFI works by supplying accurately modeled motion.
  arXiv  Detail & Related papers  (2024-07-11T17:13:15Z)
• Motion-Aware Video Frame Interpolation [49.49668436390514]
  We introduce a Motion-Aware Video Frame Interpolation (MA-VFI) network, which directly estimates intermediate optical flow from consecutive frames. It not only extracts global semantic relationships and spatial details from input frames with different receptive fields, but also effectively reduces the required computational cost and complexity.
  arXiv  Detail & Related papers  (2024-02-05T11:00:14Z)
• Spatio-Temporal Branching for Motion Prediction using Motion Increments [55.68088298632865]
  Human motion prediction (HMP) has emerged as a popular research topic due to its diverse applications. Traditional methods rely on hand-crafted features and machine learning techniques. We propose a noveltemporal-temporal branching network using incremental information for HMP.
  arXiv  Detail & Related papers  (2023-08-02T12:04:28Z)
• Modeling Continuous Motion for 3D Point Cloud Object Tracking [54.48716096286417]
  This paper presents a novel approach that views each tracklet as a continuous stream. At each timestamp, only the current frame is fed into the network to interact with multi-frame historical features stored in a memory bank. To enhance the utilization of multi-frame features for robust tracking, a contrastive sequence enhancement strategy is proposed.
  arXiv  Detail & Related papers  (2023-03-14T02:58:27Z)
• Learning Dense and Continuous Optical Flow from an Event Camera [28.77846425802558]
  Event cameras such as DAVIS can simultaneously output high temporal resolution events and low frame-rate intensity images. Most of the existing optical flow estimation methods are based on two consecutive image frames and can only estimate discrete flow at a fixed time interval. We propose a novel deep learning-based dense and continuous optical flow estimation framework from a single image with event streams.
  arXiv  Detail & Related papers  (2022-11-16T17:53:18Z)
• Motion-aware Memory Network for Fast Video Salient Object Detection [15.967509480432266]
  We design a space-time memory (STM)-based network, which extracts useful temporal information of the current frame from adjacent frames as the temporal branch of VSOD. In the encoding stage, we generate high-level temporal features by using high-level features from the current and its adjacent frames. In the decoding stage, we propose an effective fusion strategy for spatial and temporal branches. The proposed model does not require optical flow or other preprocessing, and can reach a speed of nearly 100 FPS during inference.
  arXiv  Detail & Related papers  (2022-08-01T15:56:19Z)
• ProgressiveMotionSeg: Mutually Reinforced Framework for Event-Based Motion Segmentation [101.19290845597918]
  This paper presents a Motion Estimation (ME) module and an Event Denoising (ED) module jointly optimized in a mutually reinforced manner. Taking temporal correlation as guidance, ED module calculates the confidence that each event belongs to real activity events, and transmits it to ME module to update energy function of motion segmentation for noise suppression.
  arXiv  Detail & Related papers  (2022-03-22T13:40:26Z)
• Unsupervised Motion Representation Enhanced Network for Action Recognition [4.42249337449125]
  Motion representation between consecutive frames has proven to have great promotion to video understanding. TV-L1 method, an effective optical flow solver, is time-consuming and expensive in storage for caching the extracted optical flow. We propose UF-TSN, a novel end-to-end action recognition approach enhanced with an embedded lightweight unsupervised optical flow estimator.
  arXiv  Detail & Related papers  (2021-03-05T04:14:32Z)
• PAN: Towards Fast Action Recognition via Learning Persistence of Appearance [60.75488333935592]
  Most state-of-the-art methods heavily rely on dense optical flow as motion representation. In this paper, we shed light on fast action recognition by lifting the reliance on optical flow. We design a novel motion cue called Persistence of Appearance (PA) In contrast to optical flow, our PA focuses more on distilling the motion information at boundaries.
  arXiv  Detail & Related papers  (2020-08-08T07:09:54Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.