A Framework Combining 3D CNN and Transformer for Video-Based Behavior Recognition

• URL: http://arxiv.org/abs/2508.06528v1
• Date: Sat, 02 Aug 2025 07:33:29 GMT
• Title: A Framework Combining 3D CNN and Transformer for Video-Based Behavior Recognition
• Authors: Xiuliang Zhang, Tadiwa Elisha Nyamasvisva, Chuntao Liu,
• Abstract summary: We propose a hybrid framework combining 3D CNN and Transformer architectures.<n>The 3D CNN module extracts low-leveltemporal features, while the Transformer module captures long-range temporal dependencies.<n>The proposed model outperforms traditional 3D CNN and standalone Transformers, achieving higher recognition accuracy with manageable complexity.
• Score: 0.0
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Video-based behavior recognition is essential in fields such as public safety, intelligent surveillance, and human-computer interaction. Traditional 3D Convolutional Neural Network (3D CNN) effectively capture local spatiotemporal features but struggle with modeling long-range dependencies. Conversely, Transformers excel at learning global contextual information but face challenges with high computational costs. To address these limitations, we propose a hybrid framework combining 3D CNN and Transformer architectures. The 3D CNN module extracts low-level spatiotemporal features, while the Transformer module captures long-range temporal dependencies, with a fusion mechanism integrating both representations. Evaluated on benchmark datasets, the proposed model outperforms traditional 3D CNN and standalone Transformers, achieving higher recognition accuracy with manageable complexity. Ablation studies further validate the complementary strengths of the two modules. This hybrid framework offers an effective and scalable solution for video-based behavior recognition.

Related papers

• Lemon: A Unified and Scalable 3D Multimodal Model for Universal Spatial Understanding [80.66591664266744]
  Lemon is a unified transformer architecture that processes 3D point cloud patches and language tokens as a single sequence.<n>To handle the complexity of 3D data, we develop a structured patchification and tokenization scheme that preserves spatial context.<n>Lemon establishes new state-of-the-art performance across comprehensive 3D understanding and reasoning tasks.
  arXiv  Detail & Related papers  (2025-12-14T20:02:43Z)
• Puppeteer: Rig and Animate Your 3D Models [105.11046762553121]
  Puppeteer is a comprehensive framework that addresses both automatic rigging and animation for diverse 3D objects.<n>Our system first predicts plausible skeletal structures via an auto-regressive transformer.<n>It then infers skinning weights via an attention-based architecture.
  arXiv  Detail & Related papers  (2025-08-14T17:59:31Z)
• SETransformer: A Hybrid Attention-Based Architecture for Robust Human Activity Recognition [7.291558599547268]
  Human Activity Recognition (HAR) using wearable sensor data has become a central task in mobile computing, healthcare, and human-computer interaction.<n>We propose SETransformer, a hybrid deep neural architecture that combines Transformer-based temporal modeling with channel-wise squeeze-and-excitation (SE) attention and a learnable temporal attention pooling mechanism.<n>We evaluate SETransformer on the WISDM dataset and demonstrate that it significantly outperforms conventional models including LSTM, GRU, BiLSTM, and CNN baselines.
  arXiv  Detail & Related papers  (2025-05-25T23:39:34Z)
• VidFormer: A novel end-to-end framework fused by 3DCNN and Transformer for Video-based Remote Physiological Measurement [9.605944796068046]
  We introduce VidFormer, a novel framework that integrates convolutional networks (CNNs) and models for r tasks.<n>Our evaluation on five publicly available datasets demonstrates that VidFormer outperforms current state-of-the-art (SOTA) methods.
  arXiv  Detail & Related papers  (2025-01-03T08:18:08Z)
• RAVEN: Rethinking Adversarial Video Generation with Efficient Tri-plane Networks [93.18404922542702]
  We present a novel video generative model designed to address long-term spatial and temporal dependencies. Our approach incorporates a hybrid explicit-implicit tri-plane representation inspired by 3D-aware generative frameworks. Our model synthesizes high-fidelity video clips at a resolution of $256times256$ pixels, with durations extending to more than $5$ seconds at a frame rate of 30 fps.
  arXiv  Detail & Related papers  (2024-01-11T16:48:44Z)
• A Two-stream Hybrid CNN-Transformer Network for Skeleton-based Human Interaction Recognition [6.490564374810672]
  We propose a Two-stream Hybrid CNN-Transformer Network (THCT-Net) It exploits the local specificity of CNN and models global dependencies through the Transformer. We show that the proposed method can better comprehend and infer the meaning and context of various actions, outperforming state-of-the-art methods.
  arXiv  Detail & Related papers  (2023-12-31T06:46:46Z)
• Video-FocalNets: Spatio-Temporal Focal Modulation for Video Action Recognition [112.66832145320434]
  Video-FocalNet is an effective and efficient architecture for video recognition that models both local global contexts. Video-FocalNet is based on a-temporal focal modulation architecture that reverses the interaction and aggregation steps of self-attention. We show that Video-FocalNets perform favorably well against state-of-the-art transformer-based models for video recognition on five large-scale datasets.
  arXiv  Detail & Related papers  (2023-07-13T17:59:33Z)
• Video Mobile-Former: Video Recognition with Efficient Global Spatial-temporal Modeling [125.95527079960725]
  Transformer-based models have achieved top performance on major video recognition benchmarks. Video Mobile-Former is the first Transformer-based video model which constrains the computational budget within 1G FLOPs.
  arXiv  Detail & Related papers  (2022-08-25T17:59:00Z)
• Gate-Shift-Fuse for Video Action Recognition [43.8525418821458]
  Gate-Fuse (GSF) is a novel-temporal feature extraction module which controls interactions in-temporal decomposition and learns to adaptively route features through time and combine them in a data dependent manner. GSF can be inserted into existing 2D CNNs to convert them into efficient and high performing, with negligible parameter and compute overhead. We perform an extensive analysis of GSF using two popular 2D CNN families and achieve state-of-the-art or competitive performance on five standard action recognition benchmarks.
  arXiv  Detail & Related papers  (2022-03-16T19:19:04Z)
• Video Frame Interpolation Transformer [86.20646863821908]
  We propose a Transformer-based video framework that allows content-aware aggregation weights and considers long-range dependencies with the self-attention operations. To avoid the high computational cost of global self-attention, we introduce the concept of local attention into video. In addition, we develop a multi-scale frame scheme to fully realize the potential of Transformers.
  arXiv  Detail & Related papers  (2021-11-27T05:35:10Z)
• Dense Interaction Learning for Video-based Person Re-identification [75.03200492219003]
  We propose a hybrid framework, Dense Interaction Learning (DenseIL), to tackle video-based person re-ID difficulties. DenseIL contains a CNN encoder and a Dense Interaction (DI) decoder. Our experiments consistently and significantly outperform all the state-of-the-art methods on multiple standard video-based re-ID datasets.
  arXiv  Detail & Related papers  (2021-03-16T12:22:08Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.