Language-Assisted Deep Learning for Autistic Behaviors Recognition

• URL: http://arxiv.org/abs/2211.09310v3
• Date: Thu, 4 Jan 2024 22:05:15 GMT
• Title: Language-Assisted Deep Learning for Autistic Behaviors Recognition
• Authors: Andong Deng and Taojiannan Yang and Chen Chen and Qian Chen and Leslie Neely and Sakiko Oyama
• Abstract summary: We show that a vision-based problem behaviors recognition system can achieve high accuracy and outperform the previous methods by a large margin. We propose a two-branch multimodal deep learning framework by incorporating the "freely available" language description for each type of problem behavior. Experimental results demonstrate that incorporating additional language supervision can bring an obvious performance boost for the autism problem behaviors recognition task.
• Score: 13.200025637384897
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Correctly recognizing the behaviors of children with Autism Spectrum Disorder (ASD) is of vital importance for the diagnosis of Autism and timely early intervention. However, the observation and recording during the treatment from the parents of autistic children may not be accurate and objective. In such cases, automatic recognition systems based on computer vision and machine learning (in particular deep learning) technology can alleviate this issue to a large extent. Existing human action recognition models can now achieve persuasive performance on challenging activity datasets, e.g. daily activity, and sports activity. However, problem behaviors in children with ASD are very different from these general activities, and recognizing these problem behaviors via computer vision is less studied. In this paper, we first evaluate a strong baseline for action recognition, i.e. Video Swin Transformer, on two autism behaviors datasets (SSBD and ESBD) and show that it can achieve high accuracy and outperform the previous methods by a large margin, demonstrating the feasibility of vision-based problem behaviors recognition. Moreover, we propose language-assisted training to further enhance the action recognition performance. Specifically, we develop a two-branch multimodal deep learning framework by incorporating the "freely available" language description for each type of problem behavior. Experimental results demonstrate that incorporating additional language supervision can bring an obvious performance boost for the autism problem behaviors recognition task as compared to using the video information only (i.e. 3.49% improvement on ESBD and 1.46% on SSBD).

Related papers

• Localizing Moments of Actions in Untrimmed Videos of Infants with Autism Spectrum Disorder [5.2289135066938375]
  We introduce a self-attention based TAL model designed to identify ASD-related behaviors in infant videos. This study is the first to conduct end-to-end temporal action localization in untrimmed videos of infants with ASD. We achieve 70% accuracy for look face, 79% accuracy for look object, 72% for smile and 65% for vocalization.
  arXiv  Detail & Related papers  (2024-04-08T20:31:27Z)
• Hear Me, See Me, Understand Me: Audio-Visual Autism Behavior Recognition [47.550391816383794]
  We introduce a novel problem of audio-visual autism behavior recognition. Social behavior recognition is an essential aspect previously omitted in AI-assisted autism screening research. We will release our dataset, code, and pre-trained models.
  arXiv  Detail & Related papers  (2024-03-22T22:52:35Z)
• Video-Based Autism Detection with Deep Learning [0.0]
  We develop a deep learning model that analyzes video clips of children reacting to sensory stimuli. Results show that our model effectively generalizes and understands key differences in the distinct movements of the children.
  arXiv  Detail & Related papers  (2024-02-26T17:45:00Z)
• Leveraging Pretrained Representations with Task-related Keywords for Alzheimer's Disease Detection [69.53626024091076]
  Alzheimer's disease (AD) is particularly prominent in older adults. Recent advances in pre-trained models motivate AD detection modeling to shift from low-level features to high-level representations. This paper presents several efficient methods to extract better AD-related cues from high-level acoustic and linguistic features.
  arXiv  Detail & Related papers  (2023-03-14T16:03:28Z)
• Comparison of Probabilistic Deep Learning Methods for Autism Detection [0.0]
  Autism Spectrum Disorder (ASD) is one neuro developmental disorder that is now widespread in the world. Early detection of the disorder helps in the onset treatment and helps one to lead a normal life.
  arXiv  Detail & Related papers  (2023-03-09T17:49:37Z)
• Vision-Based Activity Recognition in Children with Autism-Related Behaviors [15.915410623440874]
  We demonstrate the effect of a region-based computer vision system to help clinicians and parents analyze a child's behavior. The data is pre-processed by detecting the target child in the video to reduce the impact of background noise. Motivated by the effectiveness of temporal convolutional models, we propose both light-weight and conventional models capable of extracting action features from video frames.
  arXiv  Detail & Related papers  (2022-08-08T15:12:27Z)
• Towards Unbiased Visual Emotion Recognition via Causal Intervention [63.74095927462]
  We propose a novel Emotion Recognition Network (IERN) to alleviate the negative effects brought by the dataset bias. A series of designed tests validate the effectiveness of IERN, and experiments on three emotion benchmarks demonstrate that IERN outperforms other state-of-the-art approaches.
  arXiv  Detail & Related papers  (2021-07-26T10:40:59Z)
• A Two-stage Multi-modal Affect Analysis Framework for Children with Autism Spectrum Disorder [3.029434408969759]
  We present an open-source two-stage multi-modal approach leveraging acoustic and visual cues to predict three main affect states of children with ASD's affect states in real-world play therapy scenarios. This work presents a novel way to combine human expertise and machine intelligence for ASD affect recognition by proposing a two-stage schema.
  arXiv  Detail & Related papers  (2021-06-17T01:28:53Z)
• Affect Analysis in-the-wild: Valence-Arousal, Expressions, Action Units and a Unified Framework [83.21732533130846]
  The paper focuses on large in-the-wild databases, i.e., Aff-Wild and Aff-Wild2. It presents the design of two classes of deep neural networks trained with these databases. A novel multi-task and holistic framework is presented which is able to jointly learn and effectively generalize and perform affect recognition.
  arXiv  Detail & Related papers  (2021-03-29T17:36:20Z)
• Early Autism Spectrum Disorders Diagnosis Using Eye-Tracking Technology [62.997667081978825]
  Lack of money, absence of qualified specialists, and low level of trust to the correction methods are the main issues that affect the in-time diagnoses of ASD. Our team developed the algorithm that will be able to predict the chances of ASD according to the information from the gaze activity of the child.
  arXiv  Detail & Related papers  (2020-08-21T20:22:55Z)
• Continuous Emotion Recognition via Deep Convolutional Autoencoder and Support Vector Regressor [70.2226417364135]
  It is crucial that the machine should be able to recognize the emotional state of the user with high accuracy. Deep neural networks have been used with great success in recognizing emotions. We present a new model for continuous emotion recognition based on facial expression recognition.
  arXiv  Detail & Related papers  (2020-01-31T17:47:16Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.