Unsupervised Motion Representation Enhanced Network for Action Recognition

• URL: http://arxiv.org/abs/2103.03465v1
• Date: Fri, 5 Mar 2021 04:14:32 GMT
• Title: Unsupervised Motion Representation Enhanced Network for Action Recognition
• Authors: Xiaohang Yang, Lingtong Kong, Jie Yang
• Abstract summary: Motion representation between consecutive frames has proven to have great promotion to video understanding. TV-L1 method, an effective optical flow solver, is time-consuming and expensive in storage for caching the extracted optical flow. We propose UF-TSN, a novel end-to-end action recognition approach enhanced with an embedded lightweight unsupervised optical flow estimator.
• Score: 4.42249337449125
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Learning reliable motion representation between consecutive frames, such as optical flow, has proven to have great promotion to video understanding. However, the TV-L1 method, an effective optical flow solver, is time-consuming and expensive in storage for caching the extracted optical flow. To fill the gap, we propose UF-TSN, a novel end-to-end action recognition approach enhanced with an embedded lightweight unsupervised optical flow estimator. UF-TSN estimates motion cues from adjacent frames in a coarse-to-fine manner and focuses on small displacement for each level by extracting pyramid of feature and warping one to the other according to the estimated flow of the last level. Due to the lack of labeled motion for action datasets, we constrain the flow prediction with multi-scale photometric consistency and edge-aware smoothness. Compared with state-of-the-art unsupervised motion representation learning methods, our model achieves better accuracy while maintaining efficiency, which is competitive with some supervised or more complicated approaches.

Related papers

• Motion-prior Contrast Maximization for Dense Continuous-Time Motion Estimation [34.529280562470746]
  We introduce a novel self-supervised loss combining the Contrast Maximization framework with a non-linear motion prior in the form of pixel-level trajectories. Their effectiveness is demonstrated in two scenarios: In dense continuous-time motion estimation, our method improves the zero-shot performance of a synthetically trained model by 29%.
  arXiv  Detail & Related papers  (2024-07-15T15:18:28Z)
• OCAI: Improving Optical Flow Estimation by Occlusion and Consistency Aware Interpolation [55.676358801492114]
  We propose OCAI, a method that supports robust frame ambiguities by generating intermediate video frames alongside optical flows in between. Our evaluations demonstrate superior quality and enhanced optical flow accuracy on established benchmarks such as Sintel and KITTI.
  arXiv  Detail & Related papers  (2024-03-26T20:23:48Z)
• Motion-Aware Video Frame Interpolation [49.49668436390514]
  We introduce a Motion-Aware Video Frame Interpolation (MA-VFI) network, which directly estimates intermediate optical flow from consecutive frames. It not only extracts global semantic relationships and spatial details from input frames with different receptive fields, but also effectively reduces the required computational cost and complexity.
  arXiv  Detail & Related papers  (2024-02-05T11:00:14Z)
• Flow Dynamics Correction for Action Recognition [43.95003560364798]
  We show that existing action recognition models which rely on optical flow are able to get performance boosted with our corrected optical flow. We integrate our corrected flow dynamics into popular models through a simple step by selecting only the best performing optical flow features.
  arXiv  Detail & Related papers  (2023-10-16T04:49:06Z)
• TransFlow: Transformer as Flow Learner [22.727953339383344]
  We propose TransFlow, a pure transformer architecture for optical flow estimation. It provides more accurate correlation and trustworthy matching in flow estimation. It recovers more compromised information in flow estimation through long-range temporal association in dynamic scenes.
  arXiv  Detail & Related papers  (2023-04-23T03:11:23Z)
• EM-driven unsupervised learning for efficient motion segmentation [3.5232234532568376]
  This paper presents a CNN-based fully unsupervised method for motion segmentation from optical flow. We use the Expectation-Maximization (EM) framework to leverage the loss function and the training procedure of our motion segmentation neural network. Our method outperforms comparable unsupervised methods and is very efficient.
  arXiv  Detail & Related papers  (2022-01-06T14:35:45Z)
• Optical Flow Estimation from a Single Motion-blurred Image [66.2061278123057]
  Motion blur in an image may have practical interests in fundamental computer vision problems. We propose a novel framework to estimate optical flow from a single motion-blurred image in an end-to-end manner.
  arXiv  Detail & Related papers  (2021-03-04T12:45:18Z)
• FLAVR: Flow-Agnostic Video Representations for Fast Frame Interpolation [97.99012124785177]
  FLAVR is a flexible and efficient architecture that uses 3D space-time convolutions to enable end-to-end learning and inference for video framesupervised. We demonstrate that FLAVR can serve as a useful self- pretext task for action recognition, optical flow estimation, and motion magnification.
  arXiv  Detail & Related papers  (2020-12-15T18:59:30Z)
• PAN: Towards Fast Action Recognition via Learning Persistence of Appearance [60.75488333935592]
  Most state-of-the-art methods heavily rely on dense optical flow as motion representation. In this paper, we shed light on fast action recognition by lifting the reliance on optical flow. We design a novel motion cue called Persistence of Appearance (PA) In contrast to optical flow, our PA focuses more on distilling the motion information at boundaries.
  arXiv  Detail & Related papers  (2020-08-08T07:09:54Z)
• Joint Unsupervised Learning of Optical Flow and Egomotion with Bi-Level Optimization [59.9673626329892]
  We exploit the global relationship between optical flow and camera motion using epipolar geometry. We use implicit differentiation to enable back-propagation through the lower-level geometric optimization layer independent of its implementation.
  arXiv  Detail & Related papers  (2020-02-26T22:28:00Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.