Joint vs Sequential Speaker-Role Detection and Automatic Speech Recognition for Air-traffic Control

• URL: http://arxiv.org/abs/2406.13842v1
• Date: Wed, 19 Jun 2024 21:11:01 GMT
• Title: Joint vs Sequential Speaker-Role Detection and Automatic Speech Recognition for Air-traffic Control
• Authors: Alexander Blatt, Aravind Krishnan, Dietrich Klakow,
• Abstract summary: We propose a transformer-based joint ASR-SRD system that solves both tasks jointly while relying on a standard ASR architecture. We compare this joint system against two cascaded approaches for ASR and SRD on multiple ATC datasets.
• Score: 60.35553925189286
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Utilizing air-traffic control (ATC) data for downstream natural-language processing tasks requires preprocessing steps. Key steps are the transcription of the data via automatic speech recognition (ASR) and speaker diarization, respectively speaker role detection (SRD) to divide the transcripts into pilot and air-traffic controller (ATCO) transcripts. While traditional approaches take on these tasks separately, we propose a transformer-based joint ASR-SRD system that solves both tasks jointly while relying on a standard ASR architecture. We compare this joint system against two cascaded approaches for ASR and SRD on multiple ATC datasets. Our study shows in which cases our joint system can outperform the two traditional approaches and in which cases the other architectures are preferable. We additionally evaluate how acoustic and lexical differences influence all architectures and show how to overcome them for our joint architecture.

Related papers

• Mutual Learning for Acoustic Matching and Dereverberation via Visual Scene-driven Diffusion [93.32354378820648]
  We introduce MVSD, a mutual learning framework based on diffusion models. MVSD considers the two tasks symmetrically, exploiting the reciprocal relationship to facilitate learning from inverse tasks. Our framework can improve the performance of the reverberator and dereverberator.
  arXiv  Detail & Related papers  (2024-07-15T00:47:56Z)
• MLCA-AVSR: Multi-Layer Cross Attention Fusion based Audio-Visual Speech Recognition [62.89464258519723]
  We propose a multi-layer cross-attention fusion based AVSR approach that promotes representation of each modality by fusing them at different levels of audio/visual encoders. Our proposed approach surpasses the first-place system, establishing a new SOTA cpCER of 29.13% on this dataset.
  arXiv  Detail & Related papers  (2024-01-07T08:59:32Z)
• Unified Modeling of Multi-Talker Overlapped Speech Recognition and Diarization with a Sidecar Separator [42.8787280791491]
  Multi-talker overlapped speech poses a significant challenge for speech recognition and diarization. We propose a cost-effective method to convert a single-talker automatic speech recognition system into a multi-talker one. We incorporate a diarization branch into the Sidecar, allowing for unified modeling of both ASR and diarization with a negligible overhead of only 768 parameters.
  arXiv  Detail & Related papers  (2023-05-25T17:18:37Z)
• A Virtual Simulation-Pilot Agent for Training of Air Traffic Controllers [0.797970449705065]
  We propose a novel virtual simulation-pilot engine for speeding up air traffic controller (ATCo) training. The engine receives spoken communications from ATCo trainees, and it performs automatic speech recognition and understanding. To the best of our knowledge, this is the first work fully based on open-source ATC resources and AI tools.
  arXiv  Detail & Related papers  (2023-04-16T17:45:21Z)
• Attention-based Multi-hypothesis Fusion for Speech Summarization [83.04957603852571]
  Speech summarization can be achieved by combining automatic speech recognition (ASR) and text summarization (TS) ASR errors directly affect the quality of the output summary in the cascade approach. We propose a cascade speech summarization model that is robust to ASR errors and that exploits multiple hypotheses generated by ASR to attenuate the effect of ASR errors on the summary.
  arXiv  Detail & Related papers  (2021-11-16T03:00:29Z)
• A Comparative Study of Speaker Role Identification in Air Traffic Communication Using Deep Learning Approaches [9.565067058593316]
  We formulate the speaker role identification (SRI) task of controller-pilot communication as a binary classification problem. To ablate the impacts of the comparative approaches, various advanced neural network architectures are applied. The proposed MMSRINet shows the competitive performance and robustness than the other methods on both seen and unseen data.
  arXiv  Detail & Related papers  (2021-11-03T07:00:20Z)
• BERTraffic: A Robust BERT-Based Approach for Speaker Change Detection and Role Identification of Air-Traffic Communications [2.270534915073284]
  Speech Activity Detection (SAD) or diarization system fails and then two or more single speaker segments are in the same recording. We developed a system that combines the segmentation of a SAD module with a BERT-based model that performs Speaker Change Detection (SCD) and Speaker Role Identification (SRI) based on ASR transcripts (i.e., diarization + SRI) The proposed model reaches up to 0.90/0.95 F1-score on ATCO/pilot for SRI on several test sets.
  arXiv  Detail & Related papers  (2021-10-12T07:25:12Z)
• Grammar Based Identification Of Speaker Role For Improving ATCO And Pilot ASR [1.1391158217994781]
  Assistant Based Speech Recognition (ABSR) for air traffic control is generally trained by pooling both Air Traffic Controller (ATCO) and pilot data. Due to data imbalance of ATCO and pilot and varying acoustic conditions, the ASR performance is usually significantly better for ATCOs than pilots.
  arXiv  Detail & Related papers  (2021-08-27T08:40:08Z)
• Directional ASR: A New Paradigm for E2E Multi-Speaker Speech Recognition with Source Localization [73.62550438861942]
  This paper proposes a new paradigm for handling far-field multi-speaker data in an end-to-end neural network manner, called directional automatic speech recognition (D-ASR) In D-ASR, the azimuth angle of the sources with respect to the microphone array is defined as a latent variable. This angle controls the quality of separation, which in turn determines the ASR performance.
  arXiv  Detail & Related papers  (2020-10-30T20:26:28Z)
• Audio-visual Multi-channel Recognition of Overlapped Speech [79.21950701506732]
  This paper presents an audio-visual multi-channel overlapped speech recognition system featuring tightly integrated separation front-end and recognition back-end. Experiments suggest that the proposed multi-channel AVSR system outperforms the baseline audio-only ASR system by up to 6.81% (26.83% relative) and 22.22% (56.87% relative) absolute word error rate (WER) reduction on overlapped speech constructed using either simulation or replaying of the lipreading sentence 2 dataset respectively.
  arXiv  Detail & Related papers  (2020-05-18T10:31:19Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.