Benchmarking Automatic Speech Recognition coupled LLM Modules for Medical Diagnostics

• URL: http://arxiv.org/abs/2502.13982v1
• Date: Tue, 18 Feb 2025 14:05:13 GMT
• Title: Benchmarking Automatic Speech Recognition coupled LLM Modules for Medical Diagnostics
• Authors: Kabir Kumar,
• Abstract summary: This report serves as my self project wherein models finetuned on medical call recordings are analysed. Automatic Speech Recognition (ASR) for speech transcription and a Large Language Model (LLM) for context-aware, professional responses are analysed.
• Score: 0.0
• License:
• Abstract: Natural Language Processing (NLP) and Voice Recognition agents are rapidly evolving healthcare by enabling efficient, accessible, and professional patient support while automating grunt work. This report serves as my self project wherein models finetuned on medical call recordings are analysed through a two-stage system: Automatic Speech Recognition (ASR) for speech transcription and a Large Language Model (LLM) for context-aware, professional responses. ASR, finetuned on phone call recordings provides generalised transcription of diverse patient speech over call, while the LLM matches transcribed text to medical diagnosis. A novel audio preprocessing strategy, is deployed to provide invariance to incoming recording/call data, laden with sufficient augmentation with noise/clipping to make the pipeline robust to the type of microphone and ambient conditions the patient might have while calling/recording.

Related papers

• Large Language Models Are Strong Audio-Visual Speech Recognition Learners [53.142635674428874]
  Multimodal large language models (MLLMs) have recently become a focal point of research due to their formidable multimodal understanding capabilities. We propose Llama-AVSR, a new MLLM with strong audio-visual speech recognition capabilities. We evaluate our proposed approach on LRS3, the largest public AVSR benchmark, and we achieve new state-of-the-art results for the tasks of ASR and AVSR with a WER of 0.81% and 0.77%, respectively.
  arXiv  Detail & Related papers  (2024-09-18T21:17:27Z)
• MEDSAGE: Enhancing Robustness of Medical Dialogue Summarization to ASR Errors with LLM-generated Synthetic Dialogues [41.23757609484281]
  Speech recognition errors can significantly degrade the performance of downstream tasks like summarization. We propose MEDSAGE, an approach for generating synthetic samples for data augmentation using Large Language Models. LLMs can effectively model ASR noise, and incorporating this noisy data into the training process significantly improves the robustness and accuracy of medical dialogue summarization systems.
  arXiv  Detail & Related papers  (2024-08-26T17:04:00Z)
• DiscreteSLU: A Large Language Model with Self-Supervised Discrete Speech Units for Spoken Language Understanding [51.32965203977845]
  We propose the use of discrete speech units (DSU) instead of continuous-valued speech encoder outputs. The proposed model shows robust performance on speech inputs from seen/unseen domains and instruction-following capability in spoken question answering. Our findings suggest that the ASR task and datasets are not crucial in instruction-tuning for spoken question answering tasks.
  arXiv  Detail & Related papers  (2024-06-13T17:28:13Z)
• Learning Speech Representation From Contrastive Token-Acoustic Pretraining [57.08426714676043]
  We propose "Contrastive Token-Acoustic Pretraining (CTAP)", which uses two encoders to bring phoneme and speech into a joint multimodal space. The proposed CTAP model is trained on 210k speech and phoneme pairs, achieving minimally-supervised TTS, VC, and ASR.
  arXiv  Detail & Related papers  (2023-09-01T12:35:43Z)
• Careful Whisper -- leveraging advances in automatic speech recognition for robust and interpretable aphasia subtype classification [0.0]
  This paper presents a fully automated approach for identifying speech anomalies from voice recordings to aid in the assessment of speech impairments. By combining Connectionist Temporal Classification (CTC) and encoder-decoder-based automatic speech recognition models, we generate rich acoustic and clean transcripts. We then apply several natural language processing methods to extract features from these transcripts to produce prototypes of healthy speech.
  arXiv  Detail & Related papers  (2023-08-02T15:53:59Z)
• Introducing Semantics into Speech Encoders [91.37001512418111]
  We propose an unsupervised way of incorporating semantic information from large language models into self-supervised speech encoders without labeled audio transcriptions. Our approach achieves similar performance as supervised methods trained on over 100 hours of labeled audio transcripts.
  arXiv  Detail & Related papers  (2022-11-15T18:44:28Z)
• Clinical Dialogue Transcription Error Correction using Seq2Seq Models [1.663938381339885]
  We present a seq2seq learning approach for ASR transcription error correction of clinical dialogues. We fine-tune a seq2seq model on a mask-filling task using a domain-specific dataset which we have shared publicly for future research.
  arXiv  Detail & Related papers  (2022-05-26T18:27:17Z)
• Wav2Seq: Pre-training Speech-to-Text Encoder-Decoder Models Using Pseudo Languages [58.43299730989809]
  We introduce Wav2Seq, the first self-supervised approach to pre-train both parts of encoder-decoder models for speech data. We induce a pseudo language as a compact discrete representation, and formulate a self-supervised pseudo speech recognition task. This process stands on its own, or can be applied as low-cost second-stage pre-training.
  arXiv  Detail & Related papers  (2022-05-02T17:59:02Z)
• PriMock57: A Dataset Of Primary Care Mock Consultations [66.29154510369372]
  We detail the development of a public access, high quality dataset comprising of57 mocked primary care consultations. Our work illustrates how the dataset can be used as a benchmark for conversational medical ASR as well as consultation note generation from transcripts.
  arXiv  Detail & Related papers  (2022-04-01T10:18:28Z)
• Robust Prediction of Punctuation and Truecasing for Medical ASR [18.08508027663331]
  This paper proposes a conditional joint modeling framework for prediction of punctuation and truecasing. We also present techniques for domain and task specific adaptation by fine-tuning masked language models with medical domain data.
  arXiv  Detail & Related papers  (2020-07-04T07:15:13Z)
• MultiQT: Multimodal Learning for Real-Time Question Tracking in Speech [4.384576489684272]
  We propose a novel approach to real-time sequence labeling in speech. Our model treats speech and its own textual representation as two separate modalities or views. We show significant gains of jointly learning from the two modalities when compared to text or audio only.
  arXiv  Detail & Related papers  (2020-05-02T12:16:14Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.