MoLo: Motion-augmented Long-short Contrastive Learning for Few-shot Action Recognition

• URL: http://arxiv.org/abs/2304.00946v1
• Date: Mon, 3 Apr 2023 13:09:39 GMT
• Title: MoLo: Motion-augmented Long-short Contrastive Learning for Few-shot Action Recognition
• Authors: Xiang Wang, Shiwei Zhang, Zhiwu Qing, Changxin Gao, Yingya Zhang, Deli Zhao, Nong Sang
• Abstract summary: We develop a Motion-augmented Long-short Contrastive Learning (MoLo) method that contains two crucial components, including a long-short contrastive objective and a motion autodecoder. MoLo can simultaneously learn long-range temporal context and motion cues for comprehensive few-shot matching.
• Score: 50.345327516891615
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Current state-of-the-art approaches for few-shot action recognition achieve promising performance by conducting frame-level matching on learned visual features. However, they generally suffer from two limitations: i) the matching procedure between local frames tends to be inaccurate due to the lack of guidance to force long-range temporal perception; ii) explicit motion learning is usually ignored, leading to partial information loss. To address these issues, we develop a Motion-augmented Long-short Contrastive Learning (MoLo) method that contains two crucial components, including a long-short contrastive objective and a motion autodecoder. Specifically, the long-short contrastive objective is to endow local frame features with long-form temporal awareness by maximizing their agreement with the global token of videos belonging to the same class. The motion autodecoder is a lightweight architecture to reconstruct pixel motions from the differential features, which explicitly embeds the network with motion dynamics. By this means, MoLo can simultaneously learn long-range temporal context and motion cues for comprehensive few-shot matching. To demonstrate the effectiveness, we evaluate MoLo on five standard benchmarks, and the results show that MoLo favorably outperforms recent advanced methods. The source code is available at https://github.com/alibaba-mmai-research/MoLo.

Related papers

• Fine-Grained Spatiotemporal Motion Alignment for Contrastive Video Representation Learning [16.094271750354835]
  Motion information is critical to a robust and generalized video representation. Recent works have adopted frame difference as the source of motion information in video contrastive learning. We present a framework capable of introducing well-aligned and significant motion information.
  arXiv  Detail & Related papers  (2023-09-01T07:03:27Z)
• MotionTrack: Learning Robust Short-term and Long-term Motions for Multi-Object Tracking [56.92165669843006]
  We propose MotionTrack, which learns robust short-term and long-term motions in a unified framework to associate trajectories from a short to long range. For dense crowds, we design a novel Interaction Module to learn interaction-aware motions from short-term trajectories, which can estimate the complex movement of each target. For extreme occlusions, we build a novel Refind Module to learn reliable long-term motions from the target's history trajectory, which can link the interrupted trajectory with its corresponding detection.
  arXiv  Detail & Related papers  (2023-03-18T12:38:33Z)
• Slow-Fast Visual Tempo Learning for Video-based Action Recognition [78.3820439082979]
  Action visual tempo characterizes the dynamics and the temporal scale of an action. Previous methods capture the visual tempo either by sampling raw videos with multiple rates, or by hierarchically sampling backbone features. We propose a Temporal Correlation Module (TCM) to extract action visual tempo from low-level backbone features at single-layer remarkably.
  arXiv  Detail & Related papers  (2022-02-24T14:20:04Z)
• Exploring Motion and Appearance Information for Temporal Sentence Grounding [52.01687915910648]
  We propose a Motion-Appearance Reasoning Network (MARN) to solve temporal sentence grounding. We develop separate motion and appearance branches to learn motion-guided and appearance-guided object relations. Our proposed MARN significantly outperforms previous state-of-the-art methods by a large margin.
  arXiv  Detail & Related papers  (2022-01-03T02:44:18Z)
• Long-Short Temporal Modeling for Efficient Action Recognition [32.159784061961886]
  We propose a new two-stream action recognition network, termed as MENet, consisting of a Motion Enhancement (ME) module and a Video-level Aggregation (VLA) module. For short-term motions, we design an efficient ME module to enhance the short-term motions by mingling the motion saliency among neighboring segments. As for long-term aggregations, VLA is adopted at the top of the appearance branch to integrate the long-term dependencies across all segments.
  arXiv  Detail & Related papers  (2021-06-30T02:54:13Z)
• Self-supervised Motion Learning from Static Images [36.85209332144106]
  Motion from Static Images (MoSI) learns to encode motion information. MoSI can discover regions with large motion even without fine-tuning on the downstream datasets. We demonstrate that MoSI can discover regions with large motion even without fine-tuning on the downstream datasets.
  arXiv  Detail & Related papers  (2021-04-01T03:55:50Z)
• Learning Comprehensive Motion Representation for Action Recognition [124.65403098534266]
  2D CNN-based methods are efficient but may yield redundant features due to applying the same 2D convolution kernel to each frame. Recent efforts attempt to capture motion information by establishing inter-frame connections while still suffering the limited temporal receptive field or high latency. We propose a Channel-wise Motion Enhancement (CME) module to adaptively emphasize the channels related to dynamic information with a channel-wise gate vector. We also propose a Spatial-wise Motion Enhancement (SME) module to focus on the regions with the critical target in motion, according to the point-to-point similarity between adjacent feature maps.
  arXiv  Detail & Related papers  (2021-03-23T03:06:26Z)
• PAN: Towards Fast Action Recognition via Learning Persistence of Appearance [60.75488333935592]
  Most state-of-the-art methods heavily rely on dense optical flow as motion representation. In this paper, we shed light on fast action recognition by lifting the reliance on optical flow. We design a novel motion cue called Persistence of Appearance (PA) In contrast to optical flow, our PA focuses more on distilling the motion information at boundaries.
  arXiv  Detail & Related papers  (2020-08-08T07:09:54Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.