3D CNNs with Adaptive Temporal Feature Resolutions

• URL: http://arxiv.org/abs/2011.08652v4
• Date: Wed, 11 Aug 2021 09:14:20 GMT
• Title: 3D CNNs with Adaptive Temporal Feature Resolutions
• Authors: Mohsen Fayyaz, Emad Bahrami, Ali Diba, Mehdi Noroozi, Ehsan Adeli, Luc Van Gool, Juergen Gall
• Abstract summary: Similarity Guided Sampling (SGS) module can be plugged into any existing 3D CNN architecture. SGS empowers 3D CNNs by learning the similarity of temporal features and grouping similar features together. Our evaluations show that the proposed module improves the state-of-the-art by reducing the computational cost (GFLOPs) by half while preserving or even improving the accuracy.
• Score: 83.43776851586351
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: While state-of-the-art 3D Convolutional Neural Networks (CNN) achieve very good results on action recognition datasets, they are computationally very expensive and require many GFLOPs. While the GFLOPs of a 3D CNN can be decreased by reducing the temporal feature resolution within the network, there is no setting that is optimal for all input clips. In this work, we therefore introduce a differentiable Similarity Guided Sampling (SGS) module, which can be plugged into any existing 3D CNN architecture. SGS empowers 3D CNNs by learning the similarity of temporal features and grouping similar features together. As a result, the temporal feature resolution is not anymore static but it varies for each input video clip. By integrating SGS as an additional layer within current 3D CNNs, we can convert them into much more efficient 3D CNNs with adaptive temporal feature resolutions (ATFR). Our evaluations show that the proposed module improves the state-of-the-art by reducing the computational cost (GFLOPs) by half while preserving or even improving the accuracy. We evaluate our module by adding it to multiple state-of-the-art 3D CNNs on various datasets such as Kinetics-600, Kinetics-400, mini-Kinetics, Something-Something V2, UCF101, and HMDB51.

Related papers

• OA-CNNs: Omni-Adaptive Sparse CNNs for 3D Semantic Segmentation [70.17681136234202]
  We reexamine the design distinctions and test the limits of what a sparse CNN can achieve. We propose two key components, i.e., adaptive receptive fields (spatially) and adaptive relation, to bridge the gap. This exploration led to the creation of Omni-Adaptive 3D CNNs (OA-CNNs), a family of networks that integrates a lightweight module.
  arXiv  Detail & Related papers  (2024-03-21T14:06:38Z)
• Maximizing Spatio-Temporal Entropy of Deep 3D CNNs for Efficient Video Recognition [25.364148451584356]
  3D convolution neural networks (CNNs) have been the prevailing option for video recognition. We propose to automatically design efficient 3D CNN architectures via a novel training-free neural architecture search approach. Experiments on Something-Something V1&V2 and Kinetics400 demonstrate that the E3D family achieves state-of-the-art performance.
  arXiv  Detail & Related papers  (2023-03-05T15:11:53Z)
• In Defense of Image Pre-Training for Spatiotemporal Recognition [32.56468478601864]
  Key to effectively leveraging image pre-training lies in the decomposition of learning spatial and temporal features. New pipeline consistently achieves better results on video recognition with significant speedup.
  arXiv  Detail & Related papers  (2022-05-03T18:45:44Z)
• Gate-Shift-Fuse for Video Action Recognition [43.8525418821458]
  Gate-Fuse (GSF) is a novel-temporal feature extraction module which controls interactions in-temporal decomposition and learns to adaptively route features through time and combine them in a data dependent manner. GSF can be inserted into existing 2D CNNs to convert them into efficient and high performing, with negligible parameter and compute overhead. We perform an extensive analysis of GSF using two popular 2D CNN families and achieve state-of-the-art or competitive performance on five standard action recognition benchmarks.
  arXiv  Detail & Related papers  (2022-03-16T19:19:04Z)
• Continual 3D Convolutional Neural Networks for Real-time Processing of Videos [93.73198973454944]
  We introduce Continual 3D Contemporalal Neural Networks (Co3D CNNs) Co3D CNNs process videos frame-by-frame rather than by clip by clip. We show that Co3D CNNs initialised on the weights from preexisting state-of-the-art video recognition models reduce floating point operations for frame-wise computations by 10.0-12.4x while improving accuracy on Kinetics-400 by 2.3-3.8x.
  arXiv  Detail & Related papers  (2021-05-31T18:30:52Z)
• MoViNets: Mobile Video Networks for Efficient Video Recognition [52.49314494202433]
  3D convolutional neural networks (CNNs) are accurate at video recognition but require large computation and memory budgets. We propose a three-step approach to improve computational efficiency while substantially reducing the peak memory usage of 3D CNNs.
  arXiv  Detail & Related papers  (2021-03-21T23:06:38Z)
• RT3D: Achieving Real-Time Execution of 3D Convolutional Neural Networks on Mobile Devices [57.877112704841366]
  This paper proposes RT3D, a model compression and mobile acceleration framework for 3D CNNs. For the first time, real-time execution of 3D CNNs is achieved on off-the-shelf mobiles.
  arXiv  Detail & Related papers  (2020-07-20T02:05:32Z)
• A Real-time Action Representation with Temporal Encoding and Deep Compression [115.3739774920845]
  We propose a new real-time convolutional architecture, called Temporal Convolutional 3D Network (T-C3D), for action representation. T-C3D learns video action representations in a hierarchical multi-granularity manner while obtaining a high process speed. Our method achieves clear improvements on UCF101 action recognition benchmark against state-of-the-art real-time methods by 5.4% in terms of accuracy and 2 times faster in terms of inference speed with a less than 5MB storage model.
  arXiv  Detail & Related papers  (2020-06-17T06:30:43Z)
• Would Mega-scale Datasets Further Enhance Spatiotemporal 3D CNNs? [18.95620388632382]
  In the early era of deep neural networks, 2D CNNs have been better than 3D CNNs in the context of video recognition. Recent studies revealed that 3D CNNs can outperform 2D CNNs trained on a large-scale video dataset.
  arXiv  Detail & Related papers  (2020-04-10T09:44:19Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.