Weakly Supervised Online Action Detection for Infant General Movements
- URL: http://arxiv.org/abs/2208.03648v1
- Date: Sun, 7 Aug 2022 05:49:55 GMT
- Title: Weakly Supervised Online Action Detection for Infant General Movements
- Authors: Tongyi Luo, Jia Xiao, Chuncao Zhang, Siheng Chen, Yuan Tian, Guangjun
Yu, Kang Dang, Xiaowei Ding
- Abstract summary: We propose a novel approach named WO-GMA to perform fidgety movements localization in the weakly supervised online setting.
Experimental results on dataset with 757 videos of different infants show that WO-GMA can get state-of-the-art video-level classification and cliplevel detection results.
- Score: 20.80510092848336
- License: http://creativecommons.org/licenses/by-nc-sa/4.0/
- Abstract: To make the earlier medical intervention of infants' cerebral palsy (CP),
early diagnosis of brain damage is critical. Although general movements
assessment(GMA) has shown promising results in early CP detection, it is
laborious. Most existing works take videos as input to make fidgety
movements(FMs) classification for the GMA automation. Those methods require a
complete observation of videos and can not localize video frames containing
normal FMs. Therefore we propose a novel approach named WO-GMA to perform FMs
localization in the weakly supervised online setting. Infant body keypoints are
first extracted as the inputs to WO-GMA. Then WO-GMA performs local
spatio-temporal extraction followed by two network branches to generate pseudo
clip labels and model online actions. With the clip-level pseudo labels, the
action modeling branch learns to detect FMs in an online fashion. Experimental
results on a dataset with 757 videos of different infants show that WO-GMA can
get state-of-the-art video-level classification and cliplevel detection
results. Moreover, only the first 20% duration of the video is needed to get
classification results as good as fully observed, implying a significantly
shortened FMs diagnosis time. Code is available at:
https://github.com/scofiedluo/WO-GMA.
Related papers
- Automatic Classification of General Movements in Newborns [9.308383767711367]
General movements (GMs) are reliable predictors for neurodevelopmental disorders.
To scale up newborn screening, there is a need for an algorithm that can automatically classify GMs from infant video recordings.
In this work, we introduce a tool for extracting features from these recordings and explore various machine learning techniques for automated GM classification.
arXiv Detail & Related papers (2024-11-14T21:53:46Z) - VideoGLUE: Video General Understanding Evaluation of Foundation Models [89.07145427268948]
We evaluate video understanding capabilities of foundation models (FMs) using a carefully designed experiment protocol.
We jointly profile FMs' hallmark and efficacy efficiency when adapting to general video understanding tasks.
arXiv Detail & Related papers (2023-07-06T17:47:52Z) - Semi-supervised Body Parsing and Pose Estimation for Enhancing Infant
General Movement Assessment [11.33138866472943]
General movement assessment (GMA) of infant movement videos (IMVs) is an effective method for early detection of cerebral palsy (CP) in infants.
We demonstrate in this paper that end-to-end trainable neural networks for image sequence recognition can be applied to achieve good results in GMA.
We propose a semi-supervised model, termed SiamParseNet (SPN), which consists of two branches, one for intra-frame body parts segmentation and another for inter-frame label propagation.
arXiv Detail & Related papers (2022-10-14T18:46:30Z) - Self-Supervised Masked Convolutional Transformer Block for Anomaly
Detection [122.4894940892536]
We present a novel self-supervised masked convolutional transformer block (SSMCTB) that comprises the reconstruction-based functionality at a core architectural level.
In this work, we extend our previous self-supervised predictive convolutional attentive block (SSPCAB) with a 3D masked convolutional layer, a transformer for channel-wise attention, as well as a novel self-supervised objective based on Huber loss.
arXiv Detail & Related papers (2022-09-25T04:56:10Z) - Automated Classification of General Movements in Infants Using a
Two-stream Spatiotemporal Fusion Network [5.541644538483947]
The assessment of general movements (GMs) in infants is a useful tool in the early diagnosis of neurodevelopmental disorders.
Recent video-based GMs classification has attracted attention, but this approach would be strongly affected by irrelevant information.
We propose an automated GMs classification method, which consists of preprocessing networks that remove unnecessary background information.
arXiv Detail & Related papers (2022-07-04T05:21:09Z) - A Spatio-temporal Attention-based Model for Infant Movement Assessment
from Videos [44.71923220732036]
We develop a new method for fidgety movement assessment using human poses extracted from short clips.
Human poses capture only relevant motion profiles of joints and limbs and are free from irrelevant appearance artifacts.
Our experiments show that the proposed method achieves the ROC-AUC score of 81.87%, significantly outperforming existing competing methods with better interpretability.
arXiv Detail & Related papers (2021-05-20T14:31:54Z) - One to Many: Adaptive Instrument Segmentation via Meta Learning and
Dynamic Online Adaptation in Robotic Surgical Video [71.43912903508765]
MDAL is a dynamic online adaptive learning scheme for instrument segmentation in robot-assisted surgery.
It learns the general knowledge of instruments and the fast adaptation ability through the video-specific meta-learning paradigm.
It outperforms other state-of-the-art methods on two datasets.
arXiv Detail & Related papers (2021-03-24T05:02:18Z) - Learning from Weakly-labeled Web Videos via Exploring Sub-Concepts [89.06560404218028]
We introduce a new method for pre-training video action recognition models using queried web videos.
Instead of trying to filter out, we propose to convert the potential noises in these queried videos to useful supervision signals.
We show that SPL outperforms several existing pre-training strategies using pseudo-labels.
arXiv Detail & Related papers (2021-01-11T05:50:16Z) - SiamParseNet: Joint Body Parsing and Label Propagation in Infant
Movement Videos [12.99371655893686]
General movement assessment (GMA) of infant movement videos (IMVs) is an effective method for the early detection of cerebral palsy (CP) in infants.
We propose a semi-supervised body parsing model, termed SiamParseNet (SPN), to jointly learn single frame body parsing and label propagation between frames in a semi-supervised fashion.
arXiv Detail & Related papers (2020-07-16T21:14:25Z) - Gabriella: An Online System for Real-Time Activity Detection in
Untrimmed Security Videos [72.50607929306058]
We propose a real-time online system to perform activity detection on untrimmed security videos.
The proposed method consists of three stages: tubelet extraction, activity classification and online tubelet merging.
We demonstrate the effectiveness of the proposed approach in terms of speed (100 fps) and performance with state-of-the-art results.
arXiv Detail & Related papers (2020-04-23T22:20:10Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.