SOAP: Enhancing Spatio-Temporal Relation and Motion Information Capturing for Few-Shot Action Recognition

• URL: http://arxiv.org/abs/2407.16344v3
• Date: Wed, 21 Aug 2024 16:07:08 GMT
• Title: SOAP: Enhancing Spatio-Temporal Relation and Motion Information Capturing for Few-Shot Action Recognition
• Authors: Wenbo Huang, Jinghui Zhang, Xuwei Qian, Zhen Wu, Meng Wang, Lei Zhang,
• Abstract summary: Large amounts of video samples are continuously required for traditional data-driven research. We propose a novel plug-and-play architecture for action recognition called Stemp-Oral frAme tuwenle (SOAP) in this paper. SOAP-Net achieves new state-of-the-art performance across well-known benchmarks such as SthSthV2, Kinetics, UCF101, and SOAP51.
• Score: 18.542942459854867
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: High frame-rate (HFR) videos of action recognition improve fine-grained expression while reducing the spatio-temporal relation and motion information density. Thus, large amounts of video samples are continuously required for traditional data-driven training. However, samples are not always sufficient in real-world scenarios, promoting few-shot action recognition (FSAR) research. We observe that most recent FSAR works build spatio-temporal relation of video samples via temporal alignment after spatial feature extraction, cutting apart spatial and temporal features within samples. They also capture motion information via narrow perspectives between adjacent frames without considering density, leading to insufficient motion information capturing. Therefore, we propose a novel plug-and-play architecture for FSAR called Spatio-tempOral frAme tuPle enhancer (SOAP) in this paper. The model we designed with such architecture refers to SOAP-Net. Temporal connections between different feature channels and spatio-temporal relation of features are considered instead of simple feature extraction. Comprehensive motion information is also captured, using frame tuples with multiple frames containing more motion information than adjacent frames. Combining frame tuples of diverse frame counts further provides a broader perspective. SOAP-Net achieves new state-of-the-art performance across well-known benchmarks such as SthSthV2, Kinetics, UCF101, and HMDB51. Extensive empirical evaluations underscore the competitiveness, pluggability, generalization, and robustness of SOAP. The code is released at https://github.com/wenbohuang1002/SOAP.

Related papers

• Video Frame Interpolation with Densely Queried Bilateral Correlation [52.823751291070906]
  Video Frame Interpolation (VFI) aims to synthesize non-existent intermediate frames between existent frames. Flow-based VFI algorithms estimate intermediate motion fields to warp the existent frames. We propose Densely Queried Bilateral Correlation (DQBC) that gets rid of the receptive field dependency problem.
  arXiv  Detail & Related papers  (2023-04-26T14:45:09Z)
• Implicit Temporal Modeling with Learnable Alignment for Video Recognition [95.82093301212964]
  We propose a novel Implicit Learnable Alignment (ILA) method, which minimizes the temporal modeling effort while achieving incredibly high performance. ILA achieves a top-1 accuracy of 88.7% on Kinetics-400 with much fewer FLOPs compared with Swin-L and ViViT-H.
  arXiv  Detail & Related papers  (2023-04-20T17:11:01Z)
• You Can Ground Earlier than See: An Effective and Efficient Pipeline for Temporal Sentence Grounding in Compressed Videos [56.676761067861236]
  Given an untrimmed video, temporal sentence grounding aims to locate a target moment semantically according to a sentence query. Previous respectable works have made decent success, but they only focus on high-level visual features extracted from decoded frames. We propose a new setting, compressed-domain TSG, which directly utilizes compressed videos rather than fully-decompressed frames as the visual input.
  arXiv  Detail & Related papers  (2023-03-14T12:53:27Z)
• SWTF: Sparse Weighted Temporal Fusion for Drone-Based Activity Recognition [2.7677069267434873]
  Drone-camera based human activity recognition (HAR) has received significant attention from the computer vision research community. We propose a novel Sparse Weighted Temporal Fusion (SWTF) module to utilize sparsely sampled video frames. The proposed model has received an accuracy of 72.76%, 92.56%, and 78.86% on the respective datasets.
  arXiv  Detail & Related papers  (2022-11-10T12:45:43Z)
• FuTH-Net: Fusing Temporal Relations and Holistic Features for Aerial Video Classification [49.06447472006251]
  We propose a novel deep neural network, termed FuTH-Net, to model not only holistic features, but also temporal relations for aerial video classification. Our model is evaluated on two aerial video classification datasets, ERA and Drone-Action, and achieves the state-of-the-art results.
  arXiv  Detail & Related papers  (2022-09-22T21:15:58Z)
• Flow-Guided Sparse Transformer for Video Deblurring [124.11022871999423]
  FlowGuided Sparse Transformer (F GST) is a framework for video deblurring. FGSW-MSA enjoys the guidance of the estimated optical flow to globally sample spatially sparse elements corresponding to the same scene patch in neighboring frames. Our proposed F GST outperforms state-of-the-art patches on both DVD and GOPRO datasets and even yields more visually pleasing results in real video deblurring.
  arXiv  Detail & Related papers  (2022-01-06T02:05:32Z)
• PAN: Towards Fast Action Recognition via Learning Persistence of Appearance [60.75488333935592]
  Most state-of-the-art methods heavily rely on dense optical flow as motion representation. In this paper, we shed light on fast action recognition by lifting the reliance on optical flow. We design a novel motion cue called Persistence of Appearance (PA) In contrast to optical flow, our PA focuses more on distilling the motion information at boundaries.
  arXiv  Detail & Related papers  (2020-08-08T07:09:54Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.