Dynamic Appearance: A Video Representation for Action Recognition with Joint Training

• URL: http://arxiv.org/abs/2211.12748v2
• Date: Thu, 24 Nov 2022 03:54:38 GMT
• Title: Dynamic Appearance: A Video Representation for Action Recognition with Joint Training
• Authors: Guoxi Huang, Adrian G. Bors
• Abstract summary: We introduce a new concept, Dynamic Appearance (DA), summarizing the appearance information relating to movement in a video. We consider distilling the dynamic appearance from raw video data as a means of efficient video understanding. We provide extensive experimental results on four action recognition benchmarks.
• Score: 11.746833714322154
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Static appearance of video may impede the ability of a deep neural network to learn motion-relevant features in video action recognition. In this paper, we introduce a new concept, Dynamic Appearance (DA), summarizing the appearance information relating to movement in a video while filtering out the static information considered unrelated to motion. We consider distilling the dynamic appearance from raw video data as a means of efficient video understanding. To this end, we propose the Pixel-Wise Temporal Projection (PWTP), which projects the static appearance of a video into a subspace within its original vector space, while the dynamic appearance is encoded in the projection residual describing a special motion pattern. Moreover, we integrate the PWTP module with a CNN or Transformer into an end-to-end training framework, which is optimized by utilizing multi-objective optimization algorithms. We provide extensive experimental results on four action recognition benchmarks: Kinetics400, Something-Something V1, UCF101 and HMDB51.

Related papers

• Enhancing Bandwidth Efficiency for Video Motion Transfer Applications using Deep Learning Based Keypoint Prediction [4.60378493357739]
  We propose a deep learning based novel prediction framework for enhanced bandwidth reduction in motion transfer enabled video applications. For real-time applications, our results show the effectiveness of our proposed architecture by enabling up to 2x additional bandwidth reduction.
  arXiv  Detail & Related papers  (2024-03-17T20:36:43Z)
• Self-Supervised Video Representation Learning via Latent Time Navigation [12.721647696921865]
  Self-supervised video representation learning aims at maximizing similarity between different temporal segments of one video. We propose Latent Time Navigation (LTN) to capture fine-grained motions. Our experimental analysis suggests that learning video representations by LTN consistently improves performance of action classification.
  arXiv  Detail & Related papers  (2023-05-10T20:06:17Z)
• Bidirectional Cross-Modal Knowledge Exploration for Video Recognition with Pre-trained Vision-Language Models [149.1331903899298]
  We propose a novel framework called BIKE, which utilizes the cross-modal bridge to explore bidirectional knowledge. We present a Temporal Concept Spotting mechanism that uses the Text-to-Video expertise to capture temporal saliency in a parameter-free manner. Our best model achieves a state-of-the-art accuracy of 88.6% on the challenging Kinetics-400 using the released CLIP model.
  arXiv  Detail & Related papers  (2022-12-31T11:36:53Z)
• Differentiable Frequency-based Disentanglement for Aerial Video Action Recognition [56.91538445510214]
  We present a learning algorithm for human activity recognition in videos. Our approach is designed for UAV videos, which are mainly acquired from obliquely placed dynamic cameras. We conduct extensive experiments on the UAV Human dataset and the NEC Drone dataset.
  arXiv  Detail & Related papers  (2022-09-15T22:16:52Z)
• Self-Supervised Video Representation Learning with Motion-Contrastive Perception [13.860736711747284]
  Motion-Contrastive Perception Network (MCPNet) MCPNet consists of two branches, namely, Motion Information Perception (MIP) and Contrastive Instance Perception (CIP) Our method outperforms current state-of-the-art visual-only self-supervised approaches.
  arXiv  Detail & Related papers  (2022-04-10T05:34:46Z)
• Motion-aware Dynamic Graph Neural Network for Video Compressive Sensing [14.67994875448175]
  Video snapshot imaging (SCI) utilizes a 2D detector to capture sequential video frames and compress them into a single measurement. Most existing reconstruction methods are incapable of efficiently capturing long-range spatial and temporal dependencies. We propose a flexible and robust approach based on the graph neural network (GNN) to efficiently model non-local interactions between pixels in space and time regardless of the distance.
  arXiv  Detail & Related papers  (2022-03-01T12:13:46Z)
• Wide and Narrow: Video Prediction from Context and Motion [54.21624227408727]
  We propose a new framework to integrate these complementary attributes to predict complex pixel dynamics through deep networks. We present global context propagation networks that aggregate the non-local neighboring representations to preserve the contextual information over the past frames. We also devise local filter memory networks that generate adaptive filter kernels by storing the motion of moving objects in the memory.
  arXiv  Detail & Related papers  (2021-10-22T04:35:58Z)
• EAN: Event Adaptive Network for Enhanced Action Recognition [66.81780707955852]
  We propose a unified action recognition framework to investigate the dynamic nature of video content. First, when extracting local cues, we generate the spatial-temporal kernels of dynamic-scale to adaptively fit the diverse events. Second, to accurately aggregate these cues into a global video representation, we propose to mine the interactions only among a few selected foreground objects by a Transformer.
  arXiv  Detail & Related papers  (2021-07-22T15:57:18Z)
• Hybrid Dynamic-static Context-aware Attention Network for Action Assessment in Long Videos [96.45804577283563]
  We present a novel hybrid dynAmic-static Context-aware attenTION NETwork (ACTION-NET) for action assessment in long videos. We learn the video dynamic information but also focus on the static postures of the detected athletes in specific frames. We combine the features of the two streams to regress the final video score, supervised by ground-truth scores given by experts.
  arXiv  Detail & Related papers  (2020-08-13T15:51:42Z)
• MotionSqueeze: Neural Motion Feature Learning for Video Understanding [46.82376603090792]
  Motion plays a crucial role in understanding videos and most state-of-the-art neural models for video classification incorporate motion information. In this work, we replace external and heavy computation of optical flows with internal and light-weight learning of motion features. We demonstrate that the proposed method provides a significant gain on four standard benchmarks for action recognition with only a small amount of additional cost.
  arXiv  Detail & Related papers  (2020-07-20T08:30:14Z)
• Motion-Attentive Transition for Zero-Shot Video Object Segmentation [99.44383412488703]
  We present a Motion-Attentive Transition Network (MATNet) for zero-shot object segmentation. An asymmetric attention block, called Motion-Attentive Transition (MAT), is designed within a two-stream encoder. In this way, the encoder becomes deeply internative, allowing for closely hierarchical interactions between object motion and appearance.
  arXiv  Detail & Related papers  (2020-03-09T16:58:42Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.