Exploring ASR-Based Wav2Vec2 for Automated Speech Disorder Assessment: Insights and Analysis

• URL: http://arxiv.org/abs/2410.08250v1
• Date: Thu, 10 Oct 2024 13:12:17 GMT
• Title: Exploring ASR-Based Wav2Vec2 for Automated Speech Disorder Assessment: Insights and Analysis
• Authors: Tuan Nguyen, Corinne Fredouille, Alain Ghio, Mathieu Balaguer, Virginie Woisard,
• Abstract summary: The Wav2Vec2 ASR-based model has been fine-tuned for automated speech disorder quality assessment tasks. This paper presents the first analysis of this baseline model for speech quality assessment, focusing on intelligibility and severity tasks.
• Score: 7.567181073057191
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: With the rise of SSL and ASR technologies, the Wav2Vec2 ASR-based model has been fine-tuned for automated speech disorder quality assessment tasks, yielding impressive results and setting a new baseline for Head and Neck Cancer speech contexts. This demonstrates that the ASR dimension from Wav2Vec2 closely aligns with assessment dimensions. Despite its effectiveness, this system remains a black box with no clear interpretation of the connection between the model ASR dimension and clinical assessments. This paper presents the first analysis of this baseline model for speech quality assessment, focusing on intelligibility and severity tasks. We conduct a layer-wise analysis to identify key layers and compare different SSL and ASR Wav2Vec2 models based on pre-trained data. Additionally, post-hoc XAI methods, including Canonical Correlation Analysis (CCA) and visualization techniques, are used to track model evolution and visualize embeddings for enhanced interpretability.

Related papers

• Transferable Adversarial Attacks against ASR [43.766547483367795]
  We study the vulnerability of practical black-box attacks in cutting-edge automatic speech recognition models. We propose a speech-aware gradient optimization approach (SAGO) for ASR, which forces mistranscription with minimal impact on human imperceptibility. Our comprehensive experimental results reveal performance enhancements compared to baseline approaches across five models on two databases.
  arXiv  Detail & Related papers  (2024-11-14T06:32:31Z)
• Towards objective and interpretable speech disorder assessment: a comparative analysis of CNN and transformer-based models [7.774205081900019]
  Head and Neck Cancers (HNC) significantly impact patients' ability to speak, affecting their quality of life. This study proposes a self-supervised Wav2Vec2-based model for phone classification with HNC patients, to enhance accuracy and improve the discrimination of phonetic features for subsequent interpretability purpose.
  arXiv  Detail & Related papers  (2024-06-07T08:51:52Z)
• Multi-Modal Prompt Learning on Blind Image Quality Assessment [65.0676908930946]
  Image Quality Assessment (IQA) models benefit significantly from semantic information, which allows them to treat different types of objects distinctly. Traditional methods, hindered by a lack of sufficiently annotated data, have employed the CLIP image-text pretraining model as their backbone to gain semantic awareness. Recent approaches have attempted to address this mismatch using prompt technology, but these solutions have shortcomings. This paper introduces an innovative multi-modal prompt-based methodology for IQA.
  arXiv  Detail & Related papers  (2024-04-23T11:45:32Z)
• Exploring Pathological Speech Quality Assessment with ASR-Powered Wav2Vec2 in Data-Scarce Context [7.567181073057191]
  This paper introduces a novel approach where the system learns at the audio level instead of segments despite data scarcity. It shows that the ASR based Wav2Vec2 model brings the best results and may indicate a strong correlation between ASR and speech quality assessment.
  arXiv  Detail & Related papers  (2024-03-29T13:59:34Z)
• Diffusion Model Based Visual Compensation Guidance and Visual Difference Analysis for No-Reference Image Quality Assessment [82.13830107682232]
  We propose a novel class of state-of-the-art (SOTA) generative model, which exhibits the capability to model intricate relationships. We devise a new diffusion restoration network that leverages the produced enhanced image and noise-containing images. Two visual evaluation branches are designed to comprehensively analyze the obtained high-level feature information.
  arXiv  Detail & Related papers  (2024-02-22T09:39:46Z)
• A Quantitative Approach to Understand Self-Supervised Models as Cross-lingual Feature Extractors [9.279391026742658]
  We analyze the effect of model size, training objectives, and model architecture on the models' performance as a feature extractor. We develop a novel metric, the Phonetic-Syntax Ratio (PSR), to measure the phonetic and synthetic information in the extracted representations.
  arXiv  Detail & Related papers  (2023-11-27T15:58:28Z)
• QualEval: Qualitative Evaluation for Model Improvement [82.73561470966658]
  We propose QualEval, which augments quantitative scalar metrics with automated qualitative evaluation as a vehicle for model improvement. QualEval uses a powerful LLM reasoner and our novel flexible linear programming solver to generate human-readable insights. We demonstrate that leveraging its insights, for example, improves the absolute performance of the Llama 2 model by up to 15% points relative.
  arXiv  Detail & Related papers  (2023-11-06T00:21:44Z)
• End-to-End Speech Recognition: A Survey [68.35707678386949]
  The goal of this survey is to provide a taxonomy of E2E ASR models and corresponding improvements. All relevant aspects of E2E ASR are covered in this work, accompanied by discussions of performance and deployment opportunities.
  arXiv  Detail & Related papers  (2023-03-03T01:46:41Z)
• ASR-Aware End-to-end Neural Diarization [15.172086811068962]
  We present a Conformer-based end-to-end neural diarization (EEND) model that uses both acoustic input and features derived from an automatic speech recognition (ASR) model. Three modifications to the Conformer-based EEND architecture are proposed to incorporate the features. Experiments on the two-speaker English conversations of Switchboard+SRE data sets show that multi-task learning with position-in-word information is the most effective way of utilizing ASR features.
  arXiv  Detail & Related papers  (2022-02-02T21:17:14Z)
• A multi-stage machine learning model on diagnosis of esophageal manometry [50.591267188664666]
  The framework includes deep-learning models at the swallow-level stage and feature-based machine learning models at the study-level stage. This is the first artificial-intelligence-style model to automatically predict CC diagnosis of HRM study from raw multi-swallow data.
  arXiv  Detail & Related papers  (2021-06-25T20:09:23Z)
• Joint Contextual Modeling for ASR Correction and Language Understanding [60.230013453699975]
  We propose multi-task neural approaches to perform contextual language correction on ASR outputs jointly with language understanding (LU) We show that the error rates of off the shelf ASR and following LU systems can be reduced significantly by 14% relative with joint models trained using small amounts of in-domain data.
  arXiv  Detail & Related papers  (2020-01-28T22:09:25Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.