Performance evaluation of SLAM-ASR: The Good, the Bad, the Ugly, and the Way Forward

• URL: http://arxiv.org/abs/2411.03866v1
• Date: Wed, 06 Nov 2024 12:22:04 GMT
• Title: Performance evaluation of SLAM-ASR: The Good, the Bad, the Ugly, and the Way Forward
• Authors: Shashi Kumar, Iuliia Thorbecke, Sergio Burdisso, Esaú Villatoro-Tello, Manjunath K E, Kadri Hacioğlu, Pradeep Rangappa, Petr Motlicek, Aravind Ganapathiraju, Andreas Stolcke,
• Abstract summary: Recent research has demonstrated that training a linear connector between speech foundation encoders and large language models (LLMs) enables this architecture to achieve strong ASR capabilities. Despite the impressive results, it remains unclear whether these simple approaches are robust enough across different scenarios and speech conditions. We present novel empirical findings that offer insights on how to effectively utilize the SLAM-ASR architecture across a wide range of settings.
• Score: 10.914414815406275
• License:
• Abstract: Recent research has demonstrated that training a linear connector between speech foundation encoders and large language models (LLMs) enables this architecture to achieve strong ASR capabilities. Despite the impressive results, it remains unclear whether these simple approaches are robust enough across different scenarios and speech conditions, such as domain shifts and different speech perturbations. In this paper, we address these questions by conducting various ablation experiments using a recent and widely adopted approach called SLAM-ASR. We present novel empirical findings that offer insights on how to effectively utilize the SLAM-ASR architecture across a wide range of settings. Our main findings indicate that the SLAM-ASR exhibits poor performance in cross-domain evaluation settings. Additionally, speech perturbations within in-domain data, such as changes in speed or the presence of additive noise, can significantly impact performance. Our findings offer critical insights for fine-tuning and configuring robust LLM-based ASR models, tailored to different data characteristics and computational resources.

Related papers

• Reinforcement Learning for Aligning Large Language Models Agents with Interactive Environments: Quantifying and Mitigating Prompt Overfitting [40.78026627009521]
  Reinforcement learning (RL) is a promising approach for aligning large language models (LLMs) knowledge with sequential decision-making tasks. We propose a novel framework to analyze the sensitivity of LLMs to prompt formulations following RL training in a textual environment.
  arXiv  Detail & Related papers  (2024-10-25T18:25:35Z)
• Disentangled Noisy Correspondence Learning [56.06801962154915]
  Cross-modal retrieval is crucial in understanding latent correspondences across modalities. DisNCL is a novel information-theoretic framework for feature Disentanglement in Noisy Correspondence Learning.
  arXiv  Detail & Related papers  (2024-08-10T09:49:55Z)
• Fine-Tuning or Fine-Failing? Debunking Performance Myths in Large Language Models [0.8399688944263842]
  Large Language Models (LLMs) have the capability to understand and generate human-like text from input queries. This study extends this concept to the integration of LLMs within Retrieval-Augmented Generation (RAG) pipelines. We evaluate the impact of fine-tuning on the LLMs' capacity for data extraction and contextual understanding.
  arXiv  Detail & Related papers  (2024-06-17T04:35:17Z)
• Enhancing Noise Robustness of Retrieval-Augmented Language Models with Adaptive Adversarial Training [39.21885486667879]
  Large Language Models (LLMs) exhibit substantial capabilities yet encounter challenges, including hallucination, outdated knowledge, and untraceable reasoning processes. Retrieval-augmented generation (RAG) has emerged as a promising solution, integrating knowledge from external databases to mitigate these challenges. We propose a novel RAG approach known as Retrieval-augmented Adaptive Adrial Training (RAAT)
  arXiv  Detail & Related papers  (2024-05-31T16:24:53Z)
• Comprehensive Reassessment of Large-Scale Evaluation Outcomes in LLMs: A Multifaceted Statistical Approach [64.42462708687921]
  Evaluations have revealed that factors such as scaling, training types, architectures and other factors profoundly impact the performance of LLMs. Our study embarks on a thorough re-examination of these LLMs, targeting the inadequacies in current evaluation methods. This includes the application of ANOVA, Tukey HSD tests, GAMM, and clustering technique.
  arXiv  Detail & Related papers  (2024-03-22T14:47:35Z)
• RAGGED: Towards Informed Design of Retrieval Augmented Generation Systems [51.171355532527365]
  Retrieval-augmented generation (RAG) can significantly improve the performance of language models (LMs) RAGGED is a framework for analyzing RAG configurations across various document-based question answering tasks.
  arXiv  Detail & Related papers  (2024-03-14T02:26:31Z)
• A Neuromorphic Architecture for Reinforcement Learning from Real-Valued Observations [0.34410212782758043]
  Reinforcement Learning (RL) provides a powerful framework for decision-making in complex environments. This paper presents a novel Spiking Neural Network (SNN) architecture for solving RL problems with real-valued observations.
  arXiv  Detail & Related papers  (2023-07-06T12:33:34Z)
• Robustness and Generalization Performance of Deep Learning Models on Cyber-Physical Systems: A Comparative Study [71.84852429039881]
  Investigation focuses on the models' ability to handle a range of perturbations, such as sensor faults and noise. We test the generalization and transfer learning capabilities of these models by exposing them to out-of-distribution (OOD) samples.
  arXiv  Detail & Related papers  (2023-06-13T12:43:59Z)
• ASR: Attention-alike Structural Re-parameterization [53.019657810468026]
  We propose a simple-yet-effective attention-alike structural re- parameterization (ASR) that allows us to achieve SRP for a given network while enjoying the effectiveness of the attention mechanism. In this paper, we conduct extensive experiments from a statistical perspective and discover an interesting phenomenon Stripe Observation, which reveals that channel attention values quickly approach some constant vectors during training.
  arXiv  Detail & Related papers  (2023-04-13T08:52:34Z)
• Enhancing the Generalization for Intent Classification and Out-of-Domain Detection in SLU [70.44344060176952]
  Intent classification is a major task in spoken language understanding (SLU) Recent works have shown that using extra data and labels can improve the OOD detection performance. This paper proposes to train a model with only IND data while supporting both IND intent classification and OOD detection.
  arXiv  Detail & Related papers  (2021-06-28T08:27:38Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.