DAFMSVC: One-Shot Singing Voice Conversion with Dual Attention Mechanism and Flow Matching

• URL: http://arxiv.org/abs/2508.05978v1
• Date: Fri, 08 Aug 2025 03:24:19 GMT
• Title: DAFMSVC: One-Shot Singing Voice Conversion with Dual Attention Mechanism and Flow Matching
• Authors: Wei Chen, Binzhu Sha, Dan Luo, Jing Yang, Zhuo Wang, Fan Fan, Zhiyong Wu,
• Abstract summary: Key challenge in any-to-any Singing Voice Conversion is adapting unseen speaker timbres to source audio without quality degradation.<n>We propose DAFMSVC, where the self-supervised learning features from the source audio are replaced with the most similar SSL features from the target audio.<n>It also incorporates a dual cross-attention mechanism for the adaptive fusion of speaker embeddings, melody, and linguistic content.
• Score: 17.823734573531
• License: http://creativecommons.org/licenses/by-sa/4.0/
• Abstract: Singing Voice Conversion (SVC) transfers a source singer's timbre to a target while keeping melody and lyrics. The key challenge in any-to-any SVC is adapting unseen speaker timbres to source audio without quality degradation. Existing methods either face timbre leakage or fail to achieve satisfactory timbre similarity and quality in the generated audio. To address these challenges, we propose DAFMSVC, where the self-supervised learning (SSL) features from the source audio are replaced with the most similar SSL features from the target audio to prevent timbre leakage. It also incorporates a dual cross-attention mechanism for the adaptive fusion of speaker embeddings, melody, and linguistic content. Additionally, we introduce a flow matching module for high quality audio generation from the fused features. Experimental results show that DAFMSVC significantly enhances timbre similarity and naturalness, outperforming state-of-the-art methods in both subjective and objective evaluations.

Related papers

• HQ-SVC: Towards High-Quality Zero-Shot Singing Voice Conversion in Low-Resource Scenarios [18.036712630643205]
  HQ-SVC is an efficient framework for high-quality zero-shot singing voice conversion.<n> HQ-SVC first extracts jointly content and speaker features using a decoupled model.<n>It then enhances fidelity through pitch and volume modeling, preserving critical acoustic information.
  arXiv  Detail & Related papers  (2025-11-11T17:33:30Z)
• MuteSwap: Visual-informed Silent Video Identity Conversion [18.395223784732806]
  We introduce Silent Face-based Voice Conversion (SFVC)<n>SFVC generates intelligible speech and converting identity using only visual cues.<n>MuteSwap is a novel framework that employs contrastive learning to align cross-modality identities.
  arXiv  Detail & Related papers  (2025-07-01T07:13:34Z)
• LeVo: High-Quality Song Generation with Multi-Preference Alignment [49.94713419553945]
  We introduce LeVo, an LM-based framework consisting of LeLM and a music accompaniment.<n>LeVo is capable of parallelly modeling two types of tokens: mixed tokens, which represent the combined audio of vocals and to achieve vocal-instrument harmony, and dual-track tokens, which separately encode vocals and accompaniment.<n> Experimental results demonstrate that LeVo consistently outperforms existing methods on both objective and subjective metrics.
  arXiv  Detail & Related papers  (2025-06-09T07:57:24Z)
• FlowDubber: Movie Dubbing with LLM-based Semantic-aware Learning and Flow Matching based Voice Enhancing [78.83988199306901]
  Movie Dubbing aims to convert scripts into speeches that align with the given movie clip in both temporal and emotional aspects.<n>Existing methods focus primarily on reducing the word error rate while ignoring the importance of lip-sync and acoustic quality.<n>We propose FlowDubber, which achieves high-quality audio-visual sync and pronunciation by incorporating a large speech language model and dual contrastive aligning.
  arXiv  Detail & Related papers  (2025-05-02T13:30:19Z)
• Singing Voice Conversion with Accompaniment Using Self-Supervised Representation-Based Melody Features [15.77234185655295]
  Melody preservation is crucial in singing voice conversion (SVC)<n>In many scenarios, audio is often accompanied with background music (BGM), which can cause audio distortion and interfere with the extraction of melody and other key features.<n>We introduce a novel SVC method that uses self-supervised representation-based melody features to improve melody modeling accuracy in the presence of BGM.
  arXiv  Detail & Related papers  (2025-02-07T07:46:19Z)
• Seeing Your Speech Style: A Novel Zero-Shot Identity-Disentanglement Face-based Voice Conversion [5.483488375189695]
  Face-based Voice Conversion (FVC) is a novel task that leverages facial images to generate the target speaker's voice style. Previous work has two shortcomings: (1) suffering from obtaining facial embeddings that are well-aligned with the speaker's voice identity information, and (2) inadequacy in decoupling content and speaker identity information from the audio input. We present a novel FVC method, Identity-Disentanglement Face-based Voice Conversion (ID-FaceVC), which overcomes the above two limitations.
  arXiv  Detail & Related papers  (2024-09-01T11:51:18Z)
• Combining audio control and style transfer using latent diffusion [1.705371629600151]
  In this paper, we aim to unify explicit control and style transfer within a single model. Our model can generate audio matching a timbre target, while specifying structure either with explicit controls or through another audio example. We show that our method can generate cover versions of complete musical pieces by transferring rhythmic and melodic content to the style of a target audio in a different genre.
  arXiv  Detail & Related papers  (2024-07-31T23:27:27Z)
• StyleSinger: Style Transfer for Out-of-Domain Singing Voice Synthesis [63.18764165357298]
  Style transfer for out-of-domain singing voice synthesis (SVS) focuses on generating high-quality singing voices with unseen styles.<n>StyleSinger is the first singing voice synthesis model for zero-shot style transfer of out-of-domain reference singing voice samples.<n>Our evaluations in zero-shot style transfer undeniably establish that StyleSinger outperforms baseline models in both audio quality and similarity to the reference singing voice samples.
  arXiv  Detail & Related papers  (2023-12-17T15:26:16Z)
• Enhancing the vocal range of single-speaker singing voice synthesis with melody-unsupervised pre-training [82.94349771571642]
  This work proposes a melody-unsupervised multi-speaker pre-training method to enhance the vocal range of the single-speaker. It is the first to introduce a differentiable duration regulator to improve the rhythm naturalness of the synthesized voice. Experimental results verify that the proposed SVS system outperforms the baseline on both sound quality and naturalness.
  arXiv  Detail & Related papers  (2023-09-01T06:40:41Z)
• DiffSVC: A Diffusion Probabilistic Model for Singing Voice Conversion [51.83469048737548]
  We propose DiffSVC, an SVC system based on denoising diffusion probabilistic model. A denoising module is trained in DiffSVC, which takes destroyed mel spectrogram and its corresponding step information as input to predict the added Gaussian noise. Experiments show that DiffSVC can achieve superior conversion performance in terms of naturalness and voice similarity to current state-of-the-art SVC approaches.
  arXiv  Detail & Related papers  (2021-05-28T14:26:40Z)
• PPG-based singing voice conversion with adversarial representation learning [18.937609682084034]
  Singing voice conversion aims to convert the voice of one singer to that of other singers while keeping the singing content and melody. We build an end-to-end architecture, taking posteriorgrams as inputs and generating mel spectrograms. Our methods can significantly improve the conversion performance in terms of naturalness, melody, and voice similarity.
  arXiv  Detail & Related papers  (2020-10-28T08:03:27Z)
• Unsupervised Cross-Domain Singing Voice Conversion [105.1021715879586]
  We present a wav-to-wav generative model for the task of singing voice conversion from any identity. Our method utilizes both an acoustic model, trained for the task of automatic speech recognition, together with melody extracted features to drive a waveform-based generator.
  arXiv  Detail & Related papers  (2020-08-06T18:29:11Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.