Sentence-wise Speech Summarization: Task, Datasets, and End-to-End Modeling with LM Knowledge Distillation

• URL: http://arxiv.org/abs/2408.00205v1
• Date: Thu, 1 Aug 2024 00:18:21 GMT
• Title: Sentence-wise Speech Summarization: Task, Datasets, and End-to-End Modeling with LM Knowledge Distillation
• Authors: Kohei Matsuura, Takanori Ashihara, Takafumi Moriya, Masato Mimura, Takatomo Kano, Atsunori Ogawa, Marc Delcroix,
• Abstract summary: Sen-SSum generates text summaries from a spoken document in a sentence-by-sentence manner. We present two datasets for Sen-SSum: Mega-SSum and CSJ-SSum.
• Score: 44.332577357986324
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: This paper introduces a novel approach called sentence-wise speech summarization (Sen-SSum), which generates text summaries from a spoken document in a sentence-by-sentence manner. Sen-SSum combines the real-time processing of automatic speech recognition (ASR) with the conciseness of speech summarization. To explore this approach, we present two datasets for Sen-SSum: Mega-SSum and CSJ-SSum. Using these datasets, our study evaluates two types of Transformer-based models: 1) cascade models that combine ASR and strong text summarization models, and 2) end-to-end (E2E) models that directly convert speech into a text summary. While E2E models are appealing to develop compute-efficient models, they perform worse than cascade models. Therefore, we propose knowledge distillation for E2E models using pseudo-summaries generated by the cascade models. Our experiments show that this proposed knowledge distillation effectively improves the performance of the E2E model on both datasets.

Related papers

• SimpleSpeech 2: Towards Simple and Efficient Text-to-Speech with Flow-based Scalar Latent Transformer Diffusion Models [64.40250409933752]
  We build upon our previous publication by implementing a simple and efficient non-autoregressive (NAR) TTS framework, termed SimpleSpeech 2. SimpleSpeech 2 effectively combines the strengths of both autoregressive (AR) and non-autoregressive (NAR) methods. We show a significant improvement in generation performance and generation speed compared to our previous work and other state-of-the-art (SOTA) large-scale TTS models.
  arXiv  Detail & Related papers  (2024-08-25T17:07:39Z)
• Transfer Learning from Pre-trained Language Models Improves End-to-End Speech Summarization [48.35495352015281]
  End-to-end speech summarization (E2E SSum) directly summarizes input speech into easy-to-read short sentences with a single model. Due to the high cost of collecting speech-summary pairs, an E2E SSum model tends to suffer from training data scarcity and output unnatural sentences. We propose for the first time to integrate a pre-trained language model (LM) into the E2E SSum decoder via transfer learning.
  arXiv  Detail & Related papers  (2023-06-07T08:23:58Z)
• Towards End-to-end Speech-to-text Summarization [0.0]
  Speech-to-text (S2T) summarization is a time-saving technique for filtering and keeping up with the broadcast news uploaded online on a daily basis. End-to-end (E2E) modelling of S2T abstractive summarization is a promising approach that offers the possibility of generating rich latent representations. We model S2T summarization both with a cascade and an E2E system for a corpus of broadcast news in French.
  arXiv  Detail & Related papers  (2023-06-06T15:22:16Z)
• Leveraging Large Text Corpora for End-to-End Speech Summarization [58.673480990374635]
  End-to-end speech summarization (E2E SSum) is a technique to directly generate summary sentences from speech. We present two novel methods that leverage a large amount of external text summarization data for E2E SSum training.
  arXiv  Detail & Related papers  (2023-03-02T05:19:49Z)
• Attention-based Multi-hypothesis Fusion for Speech Summarization [83.04957603852571]
  Speech summarization can be achieved by combining automatic speech recognition (ASR) and text summarization (TS) ASR errors directly affect the quality of the output summary in the cascade approach. We propose a cascade speech summarization model that is robust to ASR errors and that exploits multiple hypotheses generated by ASR to attenuate the effect of ASR errors on the summary.
  arXiv  Detail & Related papers  (2021-11-16T03:00:29Z)
• Speech Summarization using Restricted Self-Attention [79.89680891246827]
  We introduce a single model optimized end-to-end for speech summarization. We demonstrate that the proposed model learns to directly summarize speech for the How-2 corpus of instructional videos.
  arXiv  Detail & Related papers  (2021-10-12T18:21:23Z)
• MeetSum: Transforming Meeting Transcript Summarization using Transformers! [2.1915057426589746]
  We utilize a Transformer-based Pointer Generator Network to generate abstract summaries for meeting transcripts. This model uses 2 LSTMs as an encoder and a decoder, a Pointer network which copies words from the inputted text, and a Generator network to produce out-of-vocabulary words. We show that training the model on a news summary dataset and using zero-shot learning to test it on the meeting dataset proves to produce better results than training it on the AMI meeting dataset.
  arXiv  Detail & Related papers  (2021-08-13T16:34:09Z)
• Abstractive Text Summarization based on Language Model Conditioning and Locality Modeling [4.525267347429154]
  We train a Transformer-based neural model on the BERT language model. In addition, we propose a new method of BERT-windowing, which allows chunk-wise processing of texts longer than the BERT window size. The results of our models are compared to a baseline and the state-of-the-art models on the CNN/Daily Mail dataset.
  arXiv  Detail & Related papers  (2020-03-29T14:00:17Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.