Multimodal Laryngoscopic Video Analysis for Assisted Diagnosis of Vocal Fold Paralysis

• URL: http://arxiv.org/abs/2409.03597v3
• Date: Tue, 22 Apr 2025 15:32:41 GMT
• Title: Multimodal Laryngoscopic Video Analysis for Assisted Diagnosis of Vocal Fold Paralysis
• Authors: Yucong Zhang, Xin Zou, Jinshan Yang, Wenjun Chen, Juan Liu, Faya Liang, Ming Li,
• Abstract summary: The system integrates video-based glottis detection with an audio keyword spotting method to analyze both video and audio data.<n>Pre-trained audio encoders are utilized to encode the patient voice to get the audio features.<n>Visual features are generated by measuring the angle deviation of both the left and right vocal folds to the estimated glottal midline on the segmented glottis masks.
• Score: 9.530028450239394
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: This paper presents the Multimodal Laryngoscopic Video Analyzing System (MLVAS), a novel system that leverages both audio and video data to automatically extract key video segments and metrics from raw laryngeal videostroboscopic videos for assisted clinical assessment. The system integrates video-based glottis detection with an audio keyword spotting method to analyze both video and audio data, identifying patient vocalizations and refining video highlights to ensure optimal inspection of vocal fold movements. Beyond key video segment extraction from the raw laryngeal videos, MLVAS is able to generate effective audio and visual features for Vocal Fold Paralysis (VFP) detection. Pre-trained audio encoders are utilized to encode the patient voice to get the audio features. Visual features are generated by measuring the angle deviation of both the left and right vocal folds to the estimated glottal midline on the segmented glottis masks. To get better masks, we introduce a diffusion-based refinement that follows traditional U-Net segmentation to reduce false positives. We conducted several ablation studies to demonstrate the effectiveness of each module and modalities in the proposed MLVAS. The experimental results on a public segmentation dataset show the effectiveness of our proposed segmentation module. In addition, unilateral VFP classification results on a real-world clinic dataset demonstrate MLVAS's ability of providing reliable and objective metrics as well as visualization for assisted clinical diagnosis.

Related papers

• MCAT: Visual Query-Based Localization of Standard Anatomical Clips in Fetal Ultrasound Videos Using Multi-Tier Class-Aware Token Transformer [6.520396145278936]
  We introduce a visual query-based video clip localization (VQ) method to assist sonographers by enabling them to capture a quick US sweep. MCAT returns the video clip containing the standard frames for that anatomy, facilitating thorough screening for potential anomalies. Our model outperforms state-of-the-art methods by 10% and 13% mIoU on the ultrasound datasets and by 5.35% mIoU on the Ego4D dataset, using 96% fewer tokens.
  arXiv  Detail & Related papers  (2025-04-08T14:29:15Z)
• $C^2$AV-TSE: Context and Confidence-aware Audio Visual Target Speaker Extraction [80.57232374640911]
  We propose a model-agnostic strategy called the Mask-And-Recover (MAR) MAR integrates both inter- and intra-modality contextual correlations to enable global inference within extraction modules. To better target challenging parts within each sample, we introduce a Fine-grained Confidence Score (FCS) model.
  arXiv  Detail & Related papers  (2025-04-01T13:01:30Z)
• MMSummary: Multimodal Summary Generation for Fetal Ultrasound Video [13.231546105751015]
  We present the first automated multimodal generation, MMSummary, for medical imaging video, particularly with a focus on fetal ultrasound analysis. MMSummary is designed as a three-stage pipeline, progressing from anatomy detection to captioning and finally segmentation and measurement. Based on reported experiments is estimated to reduce scanning time by approximately 31.5%, thereby suggesting the potential to enhance workflow efficiency.
  arXiv  Detail & Related papers  (2024-08-07T13:30:58Z)
• Dr-LLaVA: Visual Instruction Tuning with Symbolic Clinical Grounding [53.629132242389716]
  Vision-Language Models (VLM) can support clinicians by analyzing medical images and engaging in natural language interactions. VLMs often exhibit "hallucinogenic" behavior, generating textual outputs not grounded in contextual multimodal information. We propose a new alignment algorithm that uses symbolic representations of clinical reasoning to ground VLMs in medical knowledge.
  arXiv  Detail & Related papers  (2024-05-29T23:19:28Z)
• Multi-View Spectrogram Transformer for Respiratory Sound Classification [32.346046623638394]
  A Multi-View Spectrogram Transformer (MVST) is proposed to embed different views of time-frequency characteristics into the vision transformer. Experimental results on the ICBHI dataset demonstrate that the proposed MVST significantly outperforms state-of-the-art methods for classifying respiratory sounds.
  arXiv  Detail & Related papers  (2023-11-16T08:17:02Z)
• A Unified Approach for Comprehensive Analysis of Various Spectral and Tissue Doppler Echocardiography [3.7775754350457746]
  We introduce a novel unified framework using a convolutional neural network for comprehensive analysis of spectral and tissue Doppler echocardiography images. The network automatically recognizes key features across various Doppler views, with novel Doppler shape embedding and anti-aliasing modules. Empirical results indicate a consistent outperformance in performance metrics, including dice similarity coefficients (DSC) and intersection over union (IoU)
  arXiv  Detail & Related papers  (2023-11-14T15:10:05Z)
• Show from Tell: Audio-Visual Modelling in Clinical Settings [58.88175583465277]
  We consider audio-visual modelling in a clinical setting, providing a solution to learn medical representations without human expert annotation. A simple yet effective multi-modal self-supervised learning framework is proposed for this purpose. The proposed approach is able to localise anatomical regions of interest during ultrasound imaging, with only speech audio as a reference.
  arXiv  Detail & Related papers  (2023-10-25T08:55:48Z)
• AVTENet: Audio-Visual Transformer-based Ensemble Network Exploiting Multiple Experts for Video Deepfake Detection [53.448283629898214]
  The recent proliferation of hyper-realistic deepfake videos has drawn attention to the threat of audio and visual forgeries. Most previous work on detecting AI-generated fake videos only utilize visual modality or audio modality. We propose an Audio-Visual Transformer-based Ensemble Network (AVTENet) framework that considers both acoustic manipulation and visual manipulation.
  arXiv  Detail & Related papers  (2023-10-19T19:01:26Z)
• Analysis and Detection of Pathological Voice using Glottal Source Features [18.80191660913831]
  Glottal source features are extracted using glottal flows estimated with the quasi-closed phase (QCP) glottal inverse filtering method. We derive mel-frequency cepstral coefficients (MFCCs) from the glottal source waveforms computed by QCP and ZFF. Analysis of features revealed that the glottal source contains information that discriminates normal and pathological voice.
  arXiv  Detail & Related papers  (2023-09-25T12:14:25Z)
• Unified Frequency-Assisted Transformer Framework for Detecting and Grounding Multi-Modal Manipulation [109.1912721224697]
  We present the Unified Frequency-Assisted transFormer framework, named UFAFormer, to address the DGM4 problem. By leveraging the discrete wavelet transform, we decompose images into several frequency sub-bands, capturing rich face forgery artifacts. Our proposed frequency encoder, incorporating intra-band and inter-band self-attentions, explicitly aggregates forgery features within and across diverse sub-bands.
  arXiv  Detail & Related papers  (2023-09-18T11:06:42Z)
• GEMTrans: A General, Echocardiography-based, Multi-Level Transformer Framework for Cardiovascular Diagnosis [14.737295160286939]
  Vision-based machine learning (ML) methods have gained popularity to act as secondary layers of verification. We propose a General, Echo-based, Multi-Level Transformer (GEMTrans) framework that provides explainability. We show the flexibility of our framework by considering two critical tasks including ejection fraction (EF) and aortic stenosis (AS) severity detection.
  arXiv  Detail & Related papers  (2023-08-25T07:30:18Z)
• DopUS-Net: Quality-Aware Robotic Ultrasound Imaging based on Doppler Signal [48.97719097435527]
  DopUS-Net combines the Doppler images with B-mode images to increase the segmentation accuracy and robustness of small blood vessels. An artery re-identification module qualitatively evaluate the real-time segmentation results and automatically optimize the probe pose for enhanced Doppler images.
  arXiv  Detail & Related papers  (2023-05-15T18:19:29Z)
• Audio-visual multi-channel speech separation, dereverberation and recognition [70.34433820322323]
  This paper proposes an audio-visual multi-channel speech separation, dereverberation and recognition approach. The advantage of the additional visual modality over using audio only is demonstrated on two neural dereverberation approaches. Experiments conducted on the LRS2 dataset suggest that the proposed audio-visual multi-channel speech separation, dereverberation and recognition system outperforms the baseline.
  arXiv  Detail & Related papers  (2022-04-05T04:16:03Z)
• Acoustic To Articulatory Speech Inversion Using Multi-Resolution Spectro-Temporal Representations Of Speech Signals [5.743287315640403]
  We train a feed-forward deep neural network to estimate articulatory trajectories of six tract variables. Experiments achieved a correlation of 0.675 with ground-truth tract variables.
  arXiv  Detail & Related papers  (2022-03-11T07:27:42Z)
• Visualizing Classifier Adjacency Relations: A Case Study in Speaker Verification and Voice Anti-Spoofing [72.4445825335561]
  We propose a simple method to derive 2D representation from detection scores produced by an arbitrary set of binary classifiers. Based upon rank correlations, our method facilitates a visual comparison of classifiers with arbitrary scores. While the approach is fully versatile and can be applied to any detection task, we demonstrate the method using scores produced by automatic speaker verification and voice anti-spoofing systems.
  arXiv  Detail & Related papers  (2021-06-11T13:03:33Z)
• RespVAD: Voice Activity Detection via Video-Extracted Respiration Patterns [5.716047866174048]
  Voice Activity Detection (VAD) refers to the task of identification of regions of human speech in digital signals such as audio and video. respiration forms the primary source of energy for speech production. An audio-independent VAD technique using the respiration pattern extracted from the speakers' video is developed.
  arXiv  Detail & Related papers  (2020-08-21T13:26:24Z)
• Multi-Modal Video Forensic Platform for Investigating Post-Terrorist Attack Scenarios [55.82693757287532]
  Large scale Video Analytic Platforms (VAP) assist law enforcement agencies (LEA) in identifying suspects and securing evidence. We present a video analytic platform that integrates visual and audio analytic modules and fuses information from surveillance cameras and video uploads from eyewitnesses.
  arXiv  Detail & Related papers  (2020-04-02T14:29:27Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.