On the Use of Semantically-Aligned Speech Representations for Spoken Language Understanding

• URL: http://arxiv.org/abs/2210.05291v1
• Date: Tue, 11 Oct 2022 09:40:34 GMT
• Title: On the Use of Semantically-Aligned Speech Representations for Spoken Language Understanding
• Abstract summary: We employ the recently-introduced SAMU-XLSR model, which is designed to generate a single embedding that captures the semantics at the utterance level. We show that the use of the SAMU-XLSR model instead of the initial XLS-R model improves significantly the performance in the framework of end-to-end spoken language understanding.
• Score: 9.319029899398673
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: In this paper we examine the use of semantically-aligned speech representations for end-to-end spoken language understanding (SLU). We employ the recently-introduced SAMU-XLSR model, which is designed to generate a single embedding that captures the semantics at the utterance level, semantically aligned across different languages. This model combines the acoustic frame-level speech representation learning model (XLS-R) with the Language Agnostic BERT Sentence Embedding (LaBSE) model. We show that the use of the SAMU-XLSR model instead of the initial XLS-R model improves significantly the performance in the framework of end-to-end SLU. Finally, we present the benefits of using this model towards language portability in SLU.

Related papers

• SENSE models: an open source solution for multilingual and multimodal semantic-based tasks [11.951746684966205]
  SENSE (Shared Embedding for N-lingual Speech and tExt) is an open-source solution inspired by the SAMU-XLSR framework.<n>We describe how the original SAMU-XLSR method has been updated by selecting a stronger teacher text model and a better initial speech encoder.<n>We report experimental results on multilingual and multimodal semantic tasks, where our SENSE model achieves highly competitive performance.
  arXiv  Detail & Related papers  (2025-09-15T16:18:51Z)
• Improving Transducer-Based Spoken Language Understanding with Self-Conditioned CTC and Knowledge Transfer [11.362681035467121]
  We propose to improve end-to-end (E2E) spoken language understand (SLU) in an RNN transducer model (RNN-T) Our proposed model is akin to an E2E differentiable cascaded model which performs ASR and SLU sequentially.
  arXiv  Detail & Related papers  (2025-01-03T18:19:12Z)
• Align-SLM: Textless Spoken Language Models with Reinforcement Learning from AI Feedback [50.84142264245052]
  This work introduces the Align-SLM framework to enhance the semantic understanding of textless Spoken Language Models (SLMs) Our approach generates multiple speech continuations from a given prompt and uses semantic metrics to create preference data for Direct Preference Optimization (DPO) We evaluate the framework using ZeroSpeech 2021 benchmarks for lexical and syntactic modeling, the spoken version of the StoryCloze dataset for semantic coherence, and other speech generation metrics, including the GPT4-o score and human evaluation.
  arXiv  Detail & Related papers  (2024-11-04T06:07:53Z)
• ML-SUPERB 2.0: Benchmarking Multilingual Speech Models Across Modeling Constraints, Languages, and Datasets [106.7760874400261]
  This paper presents ML-SUPERB2.0, which is a new benchmark for evaluating pre-trained SSL and supervised speech models. We find performance improvements over the setup of ML-SUPERB, but performance depends on the downstream model design. Also, we find large performance differences between languages and datasets, suggesting the need for more targeted approaches.
  arXiv  Detail & Related papers  (2024-06-12T21:01:26Z)
• Scaling Properties of Speech Language Models [4.0142527158949415]
  Speech Language Models (SLMs) aim to learn language from raw audio, without textual resources. We estimate the scale at which our current methods will yield a SLM with the English proficiency of text-based Large Language Models (LLMs)
  arXiv  Detail & Related papers  (2024-03-31T13:30:12Z)
• Towards ASR Robust Spoken Language Understanding Through In-Context Learning With Word Confusion Networks [68.79880423713597]
  We introduce a method that utilizes the ASR system's lattice output instead of relying solely on the top hypothesis. Our in-context learning experiments, covering spoken question answering and intent classification, underline the LLM's resilience to noisy speech transcripts.
  arXiv  Detail & Related papers  (2024-01-05T17:58:10Z)
• Improving Aspect-Based Sentiment with End-to-End Semantic Role Labeling Model [6.85316573653194]
  This paper presents a series of approaches aimed at enhancing the performance of Aspect-Based Sentiment Analysis (ABSA) We propose a novel end-to-end Semantic Role Labeling model that effectively captures most of the structured semantic information within the Transformer hidden state. We evaluate the proposed models in two languages, English and Czech, employing ELECTRA-small models.
  arXiv  Detail & Related papers  (2023-07-27T11:28:16Z)
• Semantic enrichment towards efficient speech representations [9.30840529284715]
  This study investigates a specific in-domain semantic enrichment of the SAMU-XLSR model. We show the benefits of the use of same-domain French and Italian benchmarks for low-resource language portability.
  arXiv  Detail & Related papers  (2023-07-03T19:52:56Z)
• Prompting Language-Informed Distribution for Compositional Zero-Shot Learning [73.49852821602057]
  Compositional zero-shot learning (CZSL) task aims to recognize unseen compositional visual concepts. We propose a model by prompting the language-informed distribution, aka., PLID, for the task. Experimental results on MIT-States, UT-Zappos, and C-GQA datasets show the superior performance of the PLID to the prior arts.
  arXiv  Detail & Related papers  (2023-05-23T18:00:22Z)
• End-to-end model for named entity recognition from speech without paired training data [12.66131972249388]
  We propose an approach to build an end-to-end neural model to extract semantic information. Our approach is based on the use of an external model trained to generate a sequence of vectorial representations from text. Experiments on named entity recognition, carried out on the QUAERO corpus, show that this approach is very promising.
  arXiv  Detail & Related papers  (2022-04-02T08:14:27Z)
• Multi-Modal Zero-Shot Sign Language Recognition [51.07720650677784]
  We propose a multi-modal Zero-Shot Sign Language Recognition model. A Transformer-based model along with a C3D model is used for hand detection and deep features extraction. A semantic space is used to map the visual features to the lingual embedding of the class labels.
  arXiv  Detail & Related papers  (2021-09-02T09:10:39Z)
• SPLAT: Speech-Language Joint Pre-Training for Spoken Language Understanding [61.02342238771685]
  Spoken language understanding requires a model to analyze input acoustic signal to understand its linguistic content and make predictions. Various pre-training methods have been proposed to learn rich representations from large-scale unannotated speech and text. We propose a novel semi-supervised learning framework, SPLAT, to jointly pre-train the speech and language modules.
  arXiv  Detail & Related papers  (2020-10-05T19:29:49Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.