Exploring the Use of Contrastive Language-Image Pre-Training for Human Posture Classification: Insights from Yoga Pose Analysis

• URL: http://arxiv.org/abs/2501.07221v1
• Date: Mon, 13 Jan 2025 11:20:44 GMT
• Title: Exploring the Use of Contrastive Language-Image Pre-Training for Human Posture Classification: Insights from Yoga Pose Analysis
• Authors: Andrzej D. Dobrzycki, Ana M. Bernardos, Luca Bergesio, Andrzej Pomirski, Daniel Sáez-Trigueros,
• Abstract summary: This study aims to assess the effectiveness of Contrastive Language-Image Pretraining (CLIP) in classifying human postures. Applying transfer learning on 15,301 images (real and synthetic) with 82 classes has shown promising results. The fine-tuned CLIP model, tested on 3826 images, achieves an accuracy of over 85%.
• Score: 0.6524460254566905
• License:
• Abstract: Accurate human posture classification in images and videos is crucial for automated applications across various fields, including work safety, physical rehabilitation, sports training, or daily assisted living. Recently, multimodal learning methods, such as Contrastive Language-Image Pretraining (CLIP), have advanced significantly in jointly understanding images and text. This study aims to assess the effectiveness of CLIP in classifying human postures, focusing on its application in yoga. Despite the initial limitations of the zero-shot approach, applying transfer learning on 15,301 images (real and synthetic) with 82 classes has shown promising results. The article describes the full procedure for fine-tuning, including the choice for image description syntax, models and hyperparameters adjustment. The fine-tuned CLIP model, tested on 3826 images, achieves an accuracy of over 85%, surpassing the current state-of-the-art of previous works on the same dataset by approximately 6%, its training time being 3.5 times lower than what is needed to fine-tune a YOLOv8-based model. For more application-oriented scenarios, with smaller datasets of six postures each, containing 1301 and 401 training images, the fine-tuned models attain an accuracy of 98.8% and 99.1%, respectively. Furthermore, our experiments indicate that training with as few as 20 images per pose can yield around 90% accuracy in a six-class dataset. This study demonstrates that this multimodal technique can be effectively used for yoga pose classification, and possibly for human posture classification, in general. Additionally, CLIP inference time (around 7 ms) supports that the model can be integrated into automated systems for posture evaluation, e.g., for developing a real-time personal yoga assistant for performance assessment.

Related papers

• L-WISE: Boosting Human Image Category Learning Through Model-Based Image Selection And Enhancement [12.524893323311108]
  We propose to augment visual learning in humans in a way that improves human categorization accuracy at test time. Our learning augmentation approach consists of selecting images based on their model-estimated recognition difficulty, and (ii) using image perturbations that aid recognition for novice learners. To the best of our knowledge, this is the first application of ANNs to increase visual learning performance in humans by enhancing category-specific features.
  arXiv  Detail & Related papers  (2024-12-12T23:57:01Z)
• Human Pose Descriptions and Subject-Focused Attention for Improved Zero-Shot Transfer in Human-Centric Classification Tasks [89.1896982106731]
  We present a novel pipeline for creating contextual descriptions of human body poses in images using only auxiliary attributes. We demonstrate the effectiveness of our pose descriptions in enabling zero-shot human-centric classification using CLIP. Our models were pretrained on the MPII Pose Descriptions dataset and their zero-shot performance was evaluated on five unseen datasets.
  arXiv  Detail & Related papers  (2024-03-11T16:56:37Z)
• HAP: Structure-Aware Masked Image Modeling for Human-Centric Perception [97.55089867970874]
  We introduce masked image modeling (MIM) as a pre-training approach for this task. Motivated by this insight, we incorporate an intuitive human structure prior - human parts - into pre-training. This encourages the model to concentrate more on body structure information during pre-training, yielding substantial benefits across a range of human-centric perception tasks.
  arXiv  Detail & Related papers  (2023-10-31T17:56:11Z)
• No Data Augmentation? Alternative Regularizations for Effective Training on Small Datasets [0.0]
  We study alternative regularization strategies to push the limits of supervised learning on small image classification datasets. In particular, we employ a agnostic to select (semi) optimal learning rate and weight decay couples via the norm of model parameters. We reach a test accuracy of 66.5%, on par with the best state-of-the-art methods.
  arXiv  Detail & Related papers  (2023-09-04T16:13:59Z)
• UniBoost: Unsupervised Unimodal Pre-training for Boosting Zero-shot Vision-Language Tasks [60.46473247205654]
  Using large-scale unsupervised unimodal models as pre-training can enhance the zero-shot performance of image-text pair models. Our experiments show that unimodal pre-training outperforms state-of-the-art CLIP-based models.
  arXiv  Detail & Related papers  (2023-06-07T18:26:22Z)
• The effectiveness of MAE pre-pretraining for billion-scale pretraining [65.98338857597935]
  We introduce an additional pre-pretraining stage that is simple and uses the self-supervised MAE technique to initialize the model. We measure the effectiveness of pre-pretraining on 10 different visual recognition tasks spanning image classification, video recognition, object detection, low-shot classification and zero-shot recognition.
  arXiv  Detail & Related papers  (2023-03-23T17:56:12Z)
• LiT Tuned Models for Efficient Species Detection [22.3395465641384]
  Our paper introduces a simple methodology for adapting any fine-grained image classification dataset for distributed vision-language pretraining. We implement this methodology on the challenging iNaturalist-2021 dataset, comprised of approximately 2.7 million images of macro-organisms across 10,000 classes. Our model (trained using a new method called locked-image text tuning) uses a pre-trained, frozen vision representation, proving that language alignment alone can attain strong transfer learning performance.
  arXiv  Detail & Related papers  (2023-02-12T20:36:55Z)
• Learning Customized Visual Models with Retrieval-Augmented Knowledge [104.05456849611895]
  We propose REACT, a framework to acquire the relevant web knowledge to build customized visual models for target domains. We retrieve the most relevant image-text pairs from the web-scale database as external knowledge, and propose to customize the model by only training new modualized blocks while freezing all the original weights. The effectiveness of REACT is demonstrated via extensive experiments on classification, retrieval, detection and segmentation tasks, including zero, few, and full-shot settings.
  arXiv  Detail & Related papers  (2023-01-17T18:59:06Z)
• How many images do I need? Understanding how sample size per class affects deep learning model performance metrics for balanced designs in autonomous wildlife monitoring [0.0]
  We explore in depth the issues of deep learning model performance for progressively increasing per class (species) sample sizes. We provide ecologists with an approximation formula to estimate how many images per animal species they need for certain accuracy level a priori.
  arXiv  Detail & Related papers  (2020-10-16T06:28:35Z)
• CPS++: Improving Class-level 6D Pose and Shape Estimation From Monocular Images With Self-Supervised Learning [74.53664270194643]
  Modern monocular 6D pose estimation methods can only cope with a handful of object instances. We propose a novel method for class-level monocular 6D pose estimation, coupled with metric shape retrieval. We experimentally demonstrate that we can retrieve precise 6D poses and metric shapes from a single RGB image.
  arXiv  Detail & Related papers  (2020-03-12T15:28:13Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.