How Effective are Self-Supervised Models for Contact Identification in Videos

• URL: http://arxiv.org/abs/2408.00498v2
• Date: Wed, 25 Sep 2024 05:32:25 GMT
• Title: How Effective are Self-Supervised Models for Contact Identification in Videos
• Authors: Malitha Gunawardhana, Limalka Sadith, Liel David, Daniel Harari, Muhammad Haris Khan,
• Abstract summary: This work aims to employ eight different CNNs based video SSL models to identify instances of physical contact within video sequences specifically. The Something-Something v2 (SSv2) and Epic-Kitchen (EK-100) datasets were chosen for evaluating these approaches.
• Score: 6.527178779672975
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: The exploration of video content via Self-Supervised Learning (SSL) models has unveiled a dynamic field of study, emphasizing both the complex challenges and unique opportunities inherent in this area. Despite the growing body of research, the ability of SSL models to detect physical contacts in videos remains largely unexplored, particularly the effectiveness of methods such as downstream supervision with linear probing or full fine-tuning. This work aims to bridge this gap by employing eight different convolutional neural networks (CNNs) based video SSL models to identify instances of physical contact within video sequences specifically. The Something-Something v2 (SSv2) and Epic-Kitchen (EK-100) datasets were chosen for evaluating these approaches due to the promising results on UCF101 and HMDB51, coupled with their limited prior assessment on SSv2 and EK-100. Additionally, these datasets feature diverse environments and scenarios, essential for testing the robustness and accuracy of video-based models. This approach not only examines the effectiveness of each model in recognizing physical contacts but also explores the performance in the action recognition downstream task. By doing so, valuable insights into the adaptability of SSL models in interpreting complex, dynamic visual information are contributed.

Related papers

• Deep Learning Approaches for Human Action Recognition in Video Data [0.8080830346931087]
  This study conducts an in-depth analysis of various deep learning models to address this challenge. We focus on Convolutional Neural Networks (CNNs), Recurrent Neural Networks (RNNs), and Two-Stream ConvNets. The results of this study underscore the potential of composite models in achieving robust human action recognition.
  arXiv  Detail & Related papers  (2024-03-11T15:31:25Z)
• Diffusion-Based Neural Network Weights Generation [80.89706112736353]
  D2NWG is a diffusion-based neural network weights generation technique that efficiently produces high-performing weights for transfer learning. Our method extends generative hyper-representation learning to recast the latent diffusion paradigm for neural network weights generation. Our approach is scalable to large architectures such as large language models (LLMs), overcoming the limitations of current parameter generation techniques.
  arXiv  Detail & Related papers  (2024-02-28T08:34:23Z)
• CLearViD: Curriculum Learning for Video Description [3.5293199207536627]
  Video description entails automatically generating coherent natural language sentences that narrate the content of a given video. We introduce CLearViD, a transformer-based model for video description generation that leverages curriculum learning to accomplish this task. The results on two datasets, namely ActivityNet Captions and YouCook2, show that CLearViD significantly outperforms existing state-of-the-art models in terms of both accuracy and diversity metrics.
  arXiv  Detail & Related papers  (2023-11-08T06:20:32Z)
• Physics Inspired Hybrid Attention for SAR Target Recognition [61.01086031364307]
  We propose a physics inspired hybrid attention (PIHA) mechanism and the once-for-all (OFA) evaluation protocol to address the issues. PIHA leverages the high-level semantics of physical information to activate and guide the feature group aware of local semantics of target. Our method outperforms other state-of-the-art approaches in 12 test scenarios with same ASC parameters.
  arXiv  Detail & Related papers  (2023-09-27T14:39:41Z)
• Improving Video Violence Recognition with Human Interaction Learning on 3D Skeleton Point Clouds [88.87985219999764]
  We develop a method for video violence recognition from a new perspective of skeleton points. We first formulate 3D skeleton point clouds from human sequences extracted from videos. We then perform interaction learning on these 3D skeleton point clouds.
  arXiv  Detail & Related papers  (2023-08-26T12:55:18Z)
• PIGNet2: A Versatile Deep Learning-based Protein-Ligand Interaction Prediction Model for Binding Affinity Scoring and Virtual Screening [0.0]
  Prediction of protein-ligand interactions (PLI) plays a crucial role in drug discovery. The development of a versatile model capable of accurately scoring binding affinity and conducting efficient virtual screening remains a challenge. Here, we propose a viable solution by introducing a novel data augmentation strategy combined with a physics-informed graph neural network.
  arXiv  Detail & Related papers  (2023-07-03T14:46:49Z)
• Pre-training Contextualized World Models with In-the-wild Videos for Reinforcement Learning [54.67880602409801]
  In this paper, we study the problem of pre-training world models with abundant in-the-wild videos for efficient learning of visual control tasks. We introduce Contextualized World Models (ContextWM) that explicitly separate context and dynamics modeling. Our experiments show that in-the-wild video pre-training equipped with ContextWM can significantly improve the sample efficiency of model-based reinforcement learning.
  arXiv  Detail & Related papers  (2023-05-29T14:29:12Z)
• Human activity recognition using deep learning approaches and single frame cnn and convolutional lstm [0.0]
  We explore two deep learning-based approaches, namely single frame Convolutional Neural Networks (CNNs) and convolutional Long Short-Term Memory to recognise human actions from videos. The two models were trained and evaluated on a benchmark action recognition dataset, UCF50, and another dataset that was created for the experimentation. Though both models exhibit good accuracies, the single frame CNN model outperforms the Convolutional LSTM model by having an accuracy of 99.8% with the UCF50 dataset.
  arXiv  Detail & Related papers  (2023-04-18T01:33:29Z)
• Large-scale Robustness Analysis of Video Action Recognition Models [10.017292176162302]
  We study robustness of six state-of-the-art action recognition models against 90 different perturbations. The study reveals some interesting findings, 1) transformer based models are consistently more robust compared to CNN based models, 2) Pretraining improves robustness for Transformer based models more than CNN based models, and 3) All of the studied models are robust to temporal perturbations for all datasets but SSv2.
  arXiv  Detail & Related papers  (2022-07-04T13:29:34Z)
• CONVIQT: Contrastive Video Quality Estimator [63.749184706461826]
  Perceptual video quality assessment (VQA) is an integral component of many streaming and video sharing platforms. Here we consider the problem of learning perceptually relevant video quality representations in a self-supervised manner. Our results indicate that compelling representations with perceptual bearing can be obtained using self-supervised learning.
  arXiv  Detail & Related papers  (2022-06-29T15:22:01Z)
• Unsupervised Learning of Video Representations via Dense Trajectory Clustering [86.45054867170795]
  This paper addresses the task of unsupervised learning of representations for action recognition in videos. We first propose to adapt two top performing objectives in this class - instance recognition and local aggregation. We observe promising performance, but qualitative analysis shows that the learned representations fail to capture motion patterns.
  arXiv  Detail & Related papers  (2020-06-28T22:23:03Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.