Pose Matters: Evaluating Vision Transformers and CNNs for Human Action Recognition on Small COCO Subsets

• URL: http://arxiv.org/abs/2506.11678v1
• Date: Fri, 13 Jun 2025 11:16:50 GMT
• Title: Pose Matters: Evaluating Vision Transformers and CNNs for Human Action Recognition on Small COCO Subsets
• Authors: MingZe Tang, Madiha Kazi,
• Abstract summary: This study explores human recognition using a three-class subset of the COCO image corpus.<n>The binary Vision Transformer (ViT) achieved 90% mean test accuracy.
• Score: 0.0
• License: http://creativecommons.org/licenses/by-sa/4.0/
• Abstract: This study explores human action recognition using a three-class subset of the COCO image corpus, benchmarking models from simple fully connected networks to transformer architectures. The binary Vision Transformer (ViT) achieved 90% mean test accuracy, significantly exceeding multiclass classifiers such as convolutional networks (approximately 35%) and CLIP-based models (approximately 62-64%). A one-way ANOVA (F = 61.37, p < 0.001) confirmed these differences are statistically significant. Qualitative analysis with SHAP explainer and LeGrad heatmaps indicated that the ViT localizes pose-specific regions (e.g., lower limbs for walking or running), while simpler feed-forward models often focus on background textures, explaining their errors. These findings emphasize the data efficiency of transformer representations and the importance of explainability techniques in diagnosing class-specific failures.

Related papers

• Heuristical Comparison of Vision Transformers Against Convolutional Neural Networks for Semantic Segmentation on Remote Sensing Imagery [0.0]
  Vision Transformers (ViT) have brought a new wave of research in the field of computer vision.<n>This paper focuses on the comparison of three key factors of using (or not using) ViT for semantic segmentation of aerial images.<n>We show that the novel combined weighted loss function significantly boosts the CNN model's performance compared to transfer learning with ViT.
  arXiv  Detail & Related papers  (2024-11-14T00:18:04Z)
• Swin Transformer for Robust Differentiation of Real and Synthetic Images: Intra- and Inter-Dataset Analysis [0.0]
  This study proposes a Swin Transformer-based model for accurate differentiation between natural and synthetic images. The model's performance was evaluated through intra-dataset and inter-dataset testing across three distinct datasets.
  arXiv  Detail & Related papers  (2024-09-07T06:43:17Z)
• Unveiling Backbone Effects in CLIP: Exploring Representational Synergies and Variances [49.631908848868505]
  Contrastive Language-Image Pretraining (CLIP) stands out as a prominent method for image representation learning. We investigate the differences in CLIP performance among various neural architectures. We propose a simple, yet effective approach to combine predictions from multiple backbones, leading to a notable performance boost of up to 6.34%.
  arXiv  Detail & Related papers  (2023-12-22T03:01:41Z)
• Affine-Consistent Transformer for Multi-Class Cell Nuclei Detection [76.11864242047074]
  We propose a novel Affine-Consistent Transformer (AC-Former), which directly yields a sequence of nucleus positions. We introduce an Adaptive Affine Transformer (AAT) module, which can automatically learn the key spatial transformations to warp original images for local network training. Experimental results demonstrate that the proposed method significantly outperforms existing state-of-the-art algorithms on various benchmarks.
  arXiv  Detail & Related papers  (2023-10-22T02:27:02Z)
• Human Action Recognition in Still Images Using ConViT [0.11510009152620665]
  This paper proposes a new module that functions like a convolutional layer that uses Vision Transformer (ViT) It is shown that the proposed model, compared to a simple CNN, can extract meaningful parts of an image and suppress the misleading parts.
  arXiv  Detail & Related papers  (2023-07-18T06:15:23Z)
• ViTAEv2: Vision Transformer Advanced by Exploring Inductive Bias for Image Recognition and Beyond [76.35955924137986]
  We propose a Vision Transformer Advanced by Exploring intrinsic IB from convolutions, i.e., ViTAE. ViTAE has several spatial pyramid reduction modules to downsample and embed the input image into tokens with rich multi-scale context. We obtain the state-of-the-art classification performance, i.e., 88.5% Top-1 classification accuracy on ImageNet validation set and the best 91.2% Top-1 accuracy on ImageNet real validation set.
  arXiv  Detail & Related papers  (2022-02-21T10:40:05Z)
• Class-Aware Generative Adversarial Transformers for Medical Image Segmentation [39.14169989603906]
  We present CA-GANformer, a novel type of generative adversarial transformers, for medical image segmentation. First, we take advantage of the pyramid structure to construct multi-scale representations and handle multi-scale variations. We then design a novel class-aware transformer module to better learn the discriminative regions of objects with semantic structures.
  arXiv  Detail & Related papers  (2022-01-26T03:50:02Z)
• Vision Transformers for femur fracture classification [59.99241204074268]
  The Vision Transformer (ViT) was able to correctly predict 83% of the test images. Good results were obtained in sub-fractures with the largest and richest dataset ever.
  arXiv  Detail & Related papers  (2021-08-07T10:12:42Z)
• Vision Transformers are Robust Learners [65.91359312429147]
  We study the robustness of the Vision Transformer (ViT) against common corruptions and perturbations, distribution shifts, and natural adversarial examples. We present analyses that provide both quantitative and qualitative indications to explain why ViTs are indeed more robust learners.
  arXiv  Detail & Related papers  (2021-05-17T02:39:22Z)
• From Sound Representation to Model Robustness [82.21746840893658]
  We investigate the impact of different standard environmental sound representations (spectrograms) on the recognition performance and adversarial attack robustness of a victim residual convolutional neural network. Averaged over various experiments on three environmental sound datasets, we found the ResNet-18 model outperforms other deep learning architectures.
  arXiv  Detail & Related papers  (2020-07-27T17:30:49Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.