MVP-Shot: Multi-Velocity Progressive-Alignment Framework for Few-Shot Action Recognition

• URL: http://arxiv.org/abs/2405.02077v3
• Date: Thu, 23 May 2024 10:17:23 GMT
• Title: MVP-Shot: Multi-Velocity Progressive-Alignment Framework for Few-Shot Action Recognition
• Authors: Hongyu Qu, Rui Yan, Xiangbo Shu, Hailiang Gao, Peng Huang, Guo-Sen Xie,
• Abstract summary: MVP-Shot is a framework to learn and align semantic-related action features at multi-velocity levels. MVFA module measures similarity between features from support and query videos with different velocity scales. PST module injects velocity-tailored text information into the video feature via feature interaction on channel and temporal domains.
• Score: 36.426688592783975
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Recent few-shot action recognition (FSAR) methods typically perform semantic matching on learned discriminative features to achieve promising performance. However, most FSAR methods focus on single-scale (e.g., frame-level, segment-level, etc) feature alignment, which ignores that human actions with the same semantic may appear at different velocities. To this end, we develop a novel Multi-Velocity Progressive-alignment (MVP-Shot) framework to progressively learn and align semantic-related action features at multi-velocity levels. Concretely, a Multi-Velocity Feature Alignment (MVFA) module is designed to measure the similarity between features from support and query videos with different velocity scales and then merge all similarity scores in a residual fashion. To avoid the multiple velocity features deviating from the underlying motion semantic, our proposed Progressive Semantic-Tailored Interaction (PSTI) module injects velocity-tailored text information into the video feature via feature interaction on channel and temporal domains at different velocities. The above two modules compensate for each other to make more accurate query sample predictions under the few-shot settings. Experimental results show our method outperforms current state-of-the-art methods on multiple standard few-shot benchmarks (i.e., HMDB51, UCF101, Kinetics, and SSv2-small).

Related papers

• Task-Specific Alignment and Multiple Level Transformer for Few-Shot Action Recognition [11.700737340560796]
  In recent years, some works have used the Transformer to deal with frames, then get the attention feature and the enhanced prototype, and the results are competitive. We address these problems through an end-to-end method named "Task-Specific Alignment and Multiple-level Transformer Network (TSA-MLT)" Our method achieves state-of-the-art results on the HMDB51 and UCF101 datasets and a competitive result on the benchmark of Kinetics and something 2-something V2 datasets.
  arXiv  Detail & Related papers  (2023-07-05T02:13:25Z)
• Video Semantic Segmentation with Inter-Frame Feature Fusion and Inner-Frame Feature Refinement [39.06589186472675]
  We propose a spatial-temporal fusion (STF) module to model dense pairwise relationships among multi-frame features. Besides, we propose a novel memory-augmented refinement (MAR) module to tackle difficult predictions among semantic boundaries.
  arXiv  Detail & Related papers  (2023-01-10T07:57:05Z)
• HyRSM++: Hybrid Relation Guided Temporal Set Matching for Few-shot Action Recognition [51.2715005161475]
  We propose a novel Hybrid Relation guided temporal Set Matching approach for few-shot action recognition. The core idea of HyRSM++ is to integrate all videos within the task to learn discriminative representations. We show that our method achieves state-of-the-art performance under various few-shot settings.
  arXiv  Detail & Related papers  (2023-01-09T13:32:50Z)
• Hybrid Relation Guided Set Matching for Few-shot Action Recognition [51.3308583226322]
  We propose a novel Hybrid Relation guided Set Matching (HyRSM) approach that incorporates two key components. The purpose of the hybrid relation module is to learn task-specific embeddings by fully exploiting associated relations within and cross videos in an episode. We evaluate HyRSM on six challenging benchmarks, and the experimental results show its superiority over the state-of-the-art methods by a convincing margin.
  arXiv  Detail & Related papers  (2022-04-28T11:43:41Z)
• Modeling Motion with Multi-Modal Features for Text-Based Video Segmentation [56.41614987789537]
  Text-based video segmentation aims to segment the target object in a video based on a describing sentence. We propose a method to fuse and align appearance, motion, and linguistic features to achieve accurate segmentation.
  arXiv  Detail & Related papers  (2022-04-06T02:42:33Z)
• Fine-grained Temporal Contrastive Learning for Weakly-supervised Temporal Action Localization [87.47977407022492]
  This paper argues that learning by contextually comparing sequence-to-sequence distinctions offers an essential inductive bias in weakly-supervised action localization. Under a differentiable dynamic programming formulation, two complementary contrastive objectives are designed, including Fine-grained Sequence Distance (FSD) contrasting and Longest Common Subsequence (LCS) contrasting. Our method achieves state-of-the-art performance on two popular benchmarks.
  arXiv  Detail & Related papers  (2022-03-31T05:13:50Z)
• Slow-Fast Visual Tempo Learning for Video-based Action Recognition [78.3820439082979]
  Action visual tempo characterizes the dynamics and the temporal scale of an action. Previous methods capture the visual tempo either by sampling raw videos with multiple rates, or by hierarchically sampling backbone features. We propose a Temporal Correlation Module (TCM) to extract action visual tempo from low-level backbone features at single-layer remarkably.
  arXiv  Detail & Related papers  (2022-02-24T14:20:04Z)
• TEA: Temporal Excitation and Aggregation for Action Recognition [31.076707274791957]
  We propose a Temporal Excitation and Aggregation block, including a motion excitation module and a multiple temporal aggregation module. For short-range motion modeling, the ME module calculates the feature-level temporal differences fromtemporal features. The MTA module proposes to deform the local convolution to a group of sub-convolutions, forming a hierarchical residual architecture.
  arXiv  Detail & Related papers  (2020-04-03T06:53:30Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.