Unsupervised Speech Segmentation: A General Approach Using Speech Language Models

• URL: http://arxiv.org/abs/2501.03711v1
• Date: Tue, 07 Jan 2025 11:32:13 GMT
• Title: Unsupervised Speech Segmentation: A General Approach Using Speech Language Models
• Authors: Avishai Elmakies, Omri Abend, Yossi Adi,
• Abstract summary: We introduce an unsupervised approach for Speech, which builds on previously researched approaches, e.g., Speaker Diarization. We empirically demonstrate the effectiveness of the proposed approach by considering several setups. Results suggest that the proposed method is superior to the evaluated baselines on boundary detection, segment purity, and over-segmentation.
• Score: 35.497690338358055
• License:
• Abstract: In this paper, we introduce an unsupervised approach for Speech Segmentation, which builds on previously researched approaches, e.g., Speaker Diarization, while being applicable to an inclusive set of acoustic-semantic distinctions, paving a path towards a general Unsupervised Speech Segmentation approach. Unlike traditional speech and audio segmentation, which mainly focuses on spectral changes in the input signal, e.g., phone segmentation, our approach tries to segment the spoken utterance into chunks with differing acoustic-semantic styles, focusing on acoustic-semantic information that does not translate well into text, e.g., emotion or speaker. While most Speech Segmentation tasks only handle one style change, e.g., emotion diarization, our approach tries to handle multiple acoustic-semantic style changes. Leveraging recent advances in Speech Language Models (SLMs), we propose a simple unsupervised method to segment a given speech utterance. We empirically demonstrate the effectiveness of the proposed approach by considering several setups. Results suggest that the proposed method is superior to the evaluated baselines on boundary detection, segment purity, and over-segmentation. Code is available at https://github.com/avishaiElmakies/unsupervised_speech_segmentation_using_slm.

Related papers

• Label-anticipated Event Disentanglement for Audio-Visual Video Parsing [61.08434062821899]
  We introduce a new decoding paradigm, underlinelabel sunderlineemunderlineantic-based underlineprojection (LEAP) LEAP works by iteratively projecting encoded latent features of audio/visual segments onto semantically independent label embeddings. To facilitate the LEAP paradigm, we propose a semantic-aware optimization strategy, which includes a novel audio-visual semantic similarity loss function.
  arXiv  Detail & Related papers  (2024-07-11T01:57:08Z)
• Audio-Visual Neural Syntax Acquisition [91.14892278795892]
  We study phrase structure induction from visually-grounded speech. We present the Audio-Visual Neural Syntax Learner (AV-NSL) that learns phrase structure by listening to audio and looking at images, without ever being exposed to text.
  arXiv  Detail & Related papers  (2023-10-11T16:54:57Z)
• Learning Speech Representation From Contrastive Token-Acoustic Pretraining [57.08426714676043]
  We propose "Contrastive Token-Acoustic Pretraining (CTAP)", which uses two encoders to bring phoneme and speech into a joint multimodal space. The proposed CTAP model is trained on 210k speech and phoneme pairs, achieving minimally-supervised TTS, VC, and ASR.
  arXiv  Detail & Related papers  (2023-09-01T12:35:43Z)
• Smart Speech Segmentation using Acousto-Linguistic Features with look-ahead [3.579111205766969]
  We present a hybrid approach that leverages both acoustic and language information to improve segmentation. On average, our models improve segmentation-F0.5 score by 9.8% over baseline. For the downstream task of machine translation, it improves the translation BLEU score by an average of 1.05 points.
  arXiv  Detail & Related papers  (2022-10-26T03:36:31Z)
• Self-Supervised Speech Representation Learning: A Review [105.1545308184483]
  Self-supervised representation learning methods promise a single universal model that would benefit a wide variety of tasks and domains. Speech representation learning is experiencing similar progress in three main categories: generative, contrastive, and predictive methods. This review presents approaches for self-supervised speech representation learning and their connection to other research areas.
  arXiv  Detail & Related papers  (2022-05-21T16:52:57Z)
• Speech Segmentation Optimization using Segmented Bilingual Speech Corpus for End-to-end Speech Translation [16.630616128169372]
  We propose a speech segmentation method using a binary classification model trained using a segmented bilingual speech corpus. Experimental results revealed that the proposed method is more suitable for cascade and end-to-end ST systems than conventional segmentation methods.
  arXiv  Detail & Related papers  (2022-03-29T12:26:56Z)
• On the Difficulty of Segmenting Words with Attention [32.97060026226872]
  We show, however, that even on monolingual data this approach is brittle. In experiments with different input types, data sizes, and segmentation algorithms, only models trained to predict phones from words succeed in the task.
  arXiv  Detail & Related papers  (2021-09-21T11:37:08Z)
• Segmental Contrastive Predictive Coding for Unsupervised Word Segmentation [33.35220574193796]
  We propose a segmental contrastive predictive coding (SCPC) framework that can model the signal structure at a higher level e.g. at the phoneme level. A differentiable boundary detector finds variable-length segments, which are then used to optimize a segment encoder via NCE. We show that our single model outperforms existing phoneme and word segmentation methods on TIMIT and Buckeye datasets.
  arXiv  Detail & Related papers  (2021-06-03T23:12:05Z)
• Unsupervised Pattern Discovery from Thematic Speech Archives Based on Multilingual Bottleneck Features [41.951988293049205]
  We propose a two-stage approach, which comprises unsupervised acoustic modeling and decoding, followed by pattern mining in acoustic unit sequences. The proposed system is able to effectively extract topic-related words and phrases from the lecture recordings on MIT OpenCourseWare.
  arXiv  Detail & Related papers  (2020-11-03T20:06:48Z)
• Bridging the Modality Gap for Speech-to-Text Translation [57.47099674461832]
  End-to-end speech translation aims to translate speech in one language into text in another language via an end-to-end way. Most existing methods employ an encoder-decoder structure with a single encoder to learn acoustic representation and semantic information simultaneously. We propose a Speech-to-Text Adaptation for Speech Translation model which aims to improve the end-to-end model performance by bridging the modality gap between speech and text.
  arXiv  Detail & Related papers  (2020-10-28T12:33:04Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.