More than words: Advancements and challenges in speech recognition for singing

• URL: http://arxiv.org/abs/2403.09298v1
• Date: Thu, 14 Mar 2024 11:37:02 GMT
• Title: More than words: Advancements and challenges in speech recognition for singing
• Authors: Anna Kruspe,
• Abstract summary: I will describe some of my own experiences when performing research on these tasks just as they were starting to gain traction. My goal is to illuminate the complexities of applying speech recognition to singing, evaluate current capabilities, and outline future research directions.
• Score: 1.4589372436314496
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: This paper addresses the challenges and advancements in speech recognition for singing, a domain distinctly different from standard speech recognition. Singing encompasses unique challenges, including extensive pitch variations, diverse vocal styles, and background music interference. We explore key areas such as phoneme recognition, language identification in songs, keyword spotting, and full lyrics transcription. I will describe some of my own experiences when performing research on these tasks just as they were starting to gain traction, but will also show how recent developments in deep learning and large-scale datasets have propelled progress in this field. My goal is to illuminate the complexities of applying speech recognition to singing, evaluate current capabilities, and outline future research directions.

Related papers

• GTSinger: A Global Multi-Technique Singing Corpus with Realistic Music Scores for All Singing Tasks [52.30565320125514]
  GTSinger is a large global, multi-technique, free-to-use, high-quality singing corpus with realistic music scores. We collect 80.59 hours of high-quality singing voices, forming the largest recorded singing dataset. We conduct four benchmark experiments: technique-controllable singing voice synthesis, technique recognition, style transfer, and speech-to-singing conversion.
  arXiv  Detail & Related papers  (2024-09-20T18:18:14Z)
• DASB -- Discrete Audio and Speech Benchmark [12.02056212008393]
  We release the Discrete Audio and Speech Benchmark (DASB), a leaderboard for benchmarking discrete audio tokens across a range of tasks. Our results show that, on average, semantic tokens outperform compression tokens across most discriminative and generative tasks. However, the performance gap between semantic tokens and standard continuous representations remains substantial.
  arXiv  Detail & Related papers  (2024-06-20T13:23:27Z)
• Singer Identity Representation Learning using Self-Supervised Techniques [0.0]
  We propose a framework for training singer identity encoders to extract representations suitable for various singing-related tasks. We explore different self-supervised learning techniques on a large collection of isolated vocal tracks. We evaluate the quality of the resulting representations on singer similarity and identification tasks.
  arXiv  Detail & Related papers  (2024-01-10T10:41:38Z)
• Qwen-Audio: Advancing Universal Audio Understanding via Unified Large-Scale Audio-Language Models [98.34889301515412]
  We develop the Qwen-Audio model and address the limitation by scaling up audio-language pre-training to cover over 30 tasks and various audio types. Qwen-Audio achieves impressive performance across diverse benchmark tasks without requiring any task-specific fine-tuning. We further develop Qwen-Audio-Chat, which allows for input from various audios and text inputs, enabling multi-turn dialogues and supporting various audio-central scenarios.
  arXiv  Detail & Related papers  (2023-11-14T05:34:50Z)
• A Phoneme-Informed Neural Network Model for Note-Level Singing Transcription [11.951441023641975]
  We propose a method of finding note onsets of singing voice more accurately by leveraging the linguistic characteristics of singing. Our approach substantially improves the performance of singing transcription and emphasizes the importance of linguistic features in singing analysis.
  arXiv  Detail & Related papers  (2023-04-12T15:36:01Z)
• Deep Learning for Visual Speech Analysis: A Survey [54.53032361204449]
  This paper presents a review of recent progress in deep learning methods on visual speech analysis. We cover different aspects of visual speech, including fundamental problems, challenges, benchmark datasets, a taxonomy of existing methods, and state-of-the-art performance.
  arXiv  Detail & Related papers  (2022-05-22T14:44:53Z)
• Self-Supervised Speech Representation Learning: A Review [105.1545308184483]
  Self-supervised representation learning methods promise a single universal model that would benefit a wide variety of tasks and domains. Speech representation learning is experiencing similar progress in three main categories: generative, contrastive, and predictive methods. This review presents approaches for self-supervised speech representation learning and their connection to other research areas.
  arXiv  Detail & Related papers  (2022-05-21T16:52:57Z)
• Automated Audio Captioning: an Overview of Recent Progress and New Challenges [56.98522404673527]
  Automated audio captioning is a cross-modal translation task that aims to generate natural language descriptions for given audio clips. We present a comprehensive review of the published contributions in automated audio captioning, from a variety of existing approaches to evaluation metrics and datasets.
  arXiv  Detail & Related papers  (2022-05-12T08:36:35Z)
• Audio-Visual Speech Inpainting with Deep Learning [30.59696039318939]
  We inpaint speech signals with gaps ranging from 100 ms to 1600 ms to investigate the contribution that vision can provide for gaps of different duration. Results show that the performance of audio-only speech inpainting approaches degrades rapidly when gaps get large. We show that multi-task learning is effective, although the largest contribution to performance comes from vision.
  arXiv  Detail & Related papers  (2020-10-09T13:23:01Z)
• Deep Audio-Visual Learning: A Survey [53.487938108404244]
  We divide the current audio-visual learning tasks into four different subfields. We discuss state-of-the-art methods as well as the remaining challenges of each subfield. We summarize the commonly used datasets and performance metrics.
  arXiv  Detail & Related papers  (2020-01-14T13:11:21Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.