Singing Voice Conversion with Accompaniment Using Self-Supervised Representation-Based Melody Features

• URL: http://arxiv.org/abs/2502.04722v1
• Date: Fri, 07 Feb 2025 07:46:19 GMT
• Title: Singing Voice Conversion with Accompaniment Using Self-Supervised Representation-Based Melody Features
• Authors: Wei Chen, Binzhu Sha, Jing Yang, Zhuo Wang, Fan Fan, Zhiyong Wu,
• Abstract summary: Melody preservation is crucial in singing voice conversion (SVC) In many scenarios, audio is often accompanied with background music (BGM), which can cause audio distortion and interfere with the extraction of melody and other key features. We introduce a novel SVC method that uses self-supervised representation-based melody features to improve melody modeling accuracy in the presence of BGM.
• Score: 15.77234185655295
• License:
• Abstract: Melody preservation is crucial in singing voice conversion (SVC). However, in many scenarios, audio is often accompanied with background music (BGM), which can cause audio distortion and interfere with the extraction of melody and other key features, significantly degrading SVC performance. Previous methods have attempted to address this by using more robust neural network-based melody extractors, but their performance drops sharply in the presence of complex accompaniment. Other approaches involve performing source separation before conversion, but this often introduces noticeable artifacts, leading to a significant drop in conversion quality and increasing the user's operational costs. To address these issues, we introduce a novel SVC method that uses self-supervised representation-based melody features to improve melody modeling accuracy in the presence of BGM. In our experiments, we compare the effectiveness of different self-supervised learning (SSL) models for melody extraction and explore for the first time how SSL benefits the task of melody extraction. The experimental results demonstrate that our proposed SVC model significantly outperforms existing baseline methods in terms of melody accuracy and shows higher similarity and naturalness in both subjective and objective evaluations across noisy and clean audio environments.

Related papers

• SPA-SVC: Self-supervised Pitch Augmentation for Singing Voice Conversion [12.454955437047573]
  We propose a Self-supervised Pitch Augmentation method for Singing Voice Conversion (SPA-SVC) We introduce a cycle pitch shifting training strategy and Structural Similarity Index (SSIM) loss into our SVC model, effectively enhancing its performance. Experimental results on the public singing datasets M4Singer indicate that our proposed method significantly improves model performance.
  arXiv  Detail & Related papers  (2024-06-09T08:34:01Z)
• Resource-constrained stereo singing voice cancellation [1.0962868591006976]
  We study the problem of stereo singing voice cancellation. Our approach is evaluated using objective offline metrics and a large-scale MUSHRA trial.
  arXiv  Detail & Related papers  (2024-01-22T16:05:30Z)
• Enhancing the vocal range of single-speaker singing voice synthesis with melody-unsupervised pre-training [82.94349771571642]
  This work proposes a melody-unsupervised multi-speaker pre-training method to enhance the vocal range of the single-speaker. It is the first to introduce a differentiable duration regulator to improve the rhythm naturalness of the synthesized voice. Experimental results verify that the proposed SVS system outperforms the baseline on both sound quality and naturalness.
  arXiv  Detail & Related papers  (2023-09-01T06:40:41Z)
• Towards Improving Harmonic Sensitivity and Prediction Stability for Singing Melody Extraction [36.45127093978295]
  We propose an input feature modification and a training objective modification based on two assumptions. To enhance the model's sensitivity on the trailing harmonics, we modify the Combined Frequency and Periodicity representation using discrete z-transform. We apply these modifications to several models, including MSNet, FTANet, and a newly introduced model, PianoNet, modified from a piano transcription network.
  arXiv  Detail & Related papers  (2023-08-04T21:59:40Z)
• RMSSinger: Realistic-Music-Score based Singing Voice Synthesis [56.51475521778443]
  RMS-SVS aims to generate high-quality singing voices given realistic music scores with different note types. We propose RMSSinger, the first RMS-SVS method, which takes realistic music scores as input. In RMSSinger, we introduce word-level modeling to avoid the time-consuming phoneme duration annotation and the complicated phoneme-level mel-note alignment.
  arXiv  Detail & Related papers  (2023-05-18T03:57:51Z)
• A Comparative Analysis Of Latent Regressor Losses For Singing Voice Conversion [15.691936529849539]
  Singer identity embedding (SIE) network on mel-spectrograms of singer recordings to produce singer-specific variance encodings. We propose a pitch-matching mechanism between source and target singers to ensure these evaluations are not influenced by differences in pitch register.
  arXiv  Detail & Related papers  (2023-02-27T11:26:57Z)
• DiffSVC: A Diffusion Probabilistic Model for Singing Voice Conversion [51.83469048737548]
  We propose DiffSVC, an SVC system based on denoising diffusion probabilistic model. A denoising module is trained in DiffSVC, which takes destroyed mel spectrogram and its corresponding step information as input to predict the added Gaussian noise. Experiments show that DiffSVC can achieve superior conversion performance in terms of naturalness and voice similarity to current state-of-the-art SVC approaches.
  arXiv  Detail & Related papers  (2021-05-28T14:26:40Z)
• DiffSinger: Diffusion Acoustic Model for Singing Voice Synthesis [53.19363127760314]
  DiffSinger is a parameterized Markov chain which iteratively converts the noise into mel-spectrogram conditioned on the music score. The evaluations conducted on the Chinese singing dataset demonstrate that DiffSinger outperforms state-of-the-art SVS work with a notable margin.
  arXiv  Detail & Related papers  (2021-05-06T05:21:42Z)
• PPG-based singing voice conversion with adversarial representation learning [18.937609682084034]
  Singing voice conversion aims to convert the voice of one singer to that of other singers while keeping the singing content and melody. We build an end-to-end architecture, taking posteriorgrams as inputs and generating mel spectrograms. Our methods can significantly improve the conversion performance in terms of naturalness, melody, and voice similarity.
  arXiv  Detail & Related papers  (2020-10-28T08:03:27Z)
• Unsupervised Cross-Domain Singing Voice Conversion [105.1021715879586]
  We present a wav-to-wav generative model for the task of singing voice conversion from any identity. Our method utilizes both an acoustic model, trained for the task of automatic speech recognition, together with melody extracted features to drive a waveform-based generator.
  arXiv  Detail & Related papers  (2020-08-06T18:29:11Z)
• Audio Impairment Recognition Using a Correlation-Based Feature Representation [85.08880949780894]
  We propose a new representation of hand-crafted features that is based on the correlation of feature pairs. We show superior performance in terms of compact feature dimensionality and improved computational speed in the test stage.
  arXiv  Detail & Related papers  (2020-03-22T13:34:37Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.