ViolinDiff: Enhancing Expressive Violin Synthesis with Pitch Bend Conditioning

• URL: http://arxiv.org/abs/2409.12477v2
• Date: Tue, 04 Feb 2025 11:43:34 GMT
• Title: ViolinDiff: Enhancing Expressive Violin Synthesis with Pitch Bend Conditioning
• Abstract summary: We present ViolinDiff, a two-stage diffusion-based synthesis framework.<n>The proposed model generates more realistic violin sounds than the model without explicit pitch bend modeling.
• Score: 2.9153096940947796
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Modeling the natural contour of fundamental frequency (F0) plays a critical role in music audio synthesis. However, transcribing and managing multiple F0 contours in polyphonic music is challenging, and explicit F0 contour modeling has not yet been explored for polyphonic instrumental synthesis. In this paper, we present ViolinDiff, a two-stage diffusion-based synthesis framework. For a given violin MIDI file, the first stage estimates the F0 contour as pitch bend information, and the second stage generates mel spectrogram incorporating these expressive details. The quantitative metrics and listening test results show that the proposed model generates more realistic violin sounds than the model without explicit pitch bend modeling. Audio samples are available online: daewoung.github.io/ViolinDiff-Demo.

Related papers

• Annotation-Free MIDI-to-Audio Synthesis via Concatenative Synthesis and Generative Refinement [0.0]
  CoSaRef is a MIDI-to-audio synthesis method that does not require MIDI-audio paired datasets.<n>It generates a synthetic audio track using concatenative synthesis based on MIDI input, then refines it with a diffusion-based deep generative model trained on datasets without MIDI annotations.<n>It allows detailed control over timbres and expression through audio sample selection and extra MIDI design, similar to traditional functions in digital audio workstations.
  arXiv  Detail & Related papers  (2024-10-22T08:01:40Z)
• Accompanied Singing Voice Synthesis with Fully Text-controlled Melody [61.147446955297625]
  Text-to-song (TTSong) is a music generation task that synthesizes accompanied singing voices. We present MelodyLM, the first TTSong model that generates high-quality song pieces with fully text-controlled melodies.
  arXiv  Detail & Related papers  (2024-07-02T08:23:38Z)
• Expressive Acoustic Guitar Sound Synthesis with an Instrument-Specific Input Representation and Diffusion Outpainting [9.812666469580872]
  We propose an expressive acoustic guitar sound synthesis model with a customized input representation to the instrument. We implement the proposed approach using diffusion-based outpainting which can generate audio with long-term consistency. Our proposed model has higher audio quality than the baseline model and generates more realistic timbre sounds.
  arXiv  Detail & Related papers  (2024-01-24T14:44:01Z)
• Make-A-Voice: Unified Voice Synthesis With Discrete Representation [77.3998611565557]
  Make-A-Voice is a unified framework for synthesizing and manipulating voice signals from discrete representations. We show that Make-A-Voice exhibits superior audio quality and style similarity compared with competitive baseline models.
  arXiv  Detail & Related papers  (2023-05-30T17:59:26Z)
• Noise2Music: Text-conditioned Music Generation with Diffusion Models [73.74580231353684]
  We introduce Noise2Music, where a series of diffusion models is trained to generate high-quality 30-second music clips from text prompts. We find that the generated audio is not only able to faithfully reflect key elements of the text prompt such as genre, tempo, instruments, mood, and era. Pretrained large language models play a key role in this story -- they are used to generate paired text for the audio of the training set and to extract embeddings of the text prompts ingested by the diffusion models.
  arXiv  Detail & Related papers  (2023-02-08T07:27:27Z)
• Multi-instrument Music Synthesis with Spectrogram Diffusion [19.81982315173444]
  We focus on a middle ground of neural synthesizers that can generate audio from MIDI sequences with arbitrary combinations of instruments in realtime. We use a simple two-stage process: MIDI to spectrograms with an encoder-decoder Transformer, then spectrograms to audio with a generative adversarial network (GAN) spectrogram inverter. We find this to be a promising first step towards interactive and expressive neural synthesis for arbitrary combinations of instruments and notes.
  arXiv  Detail & Related papers  (2022-06-11T03:26:15Z)
• Deep Performer: Score-to-Audio Music Performance Synthesis [30.95307878579825]
  Deep Performer is a novel system for score-to-audio music performance synthesis. Unlike speech, music often contains polyphony and long notes. We show that our proposed model can synthesize music with clear polyphony and harmonic structures.
  arXiv  Detail & Related papers  (2022-02-12T10:36:52Z)
• WaveGrad 2: Iterative Refinement for Text-to-Speech Synthesis [80.60577805727624]
  WaveGrad 2 is a non-autoregressive generative model for text-to-speech synthesis. It can generate high fidelity audio, approaching the performance of a state-of-the-art neural TTS system.
  arXiv  Detail & Related papers  (2021-06-17T17:09:21Z)
• DiffSinger: Diffusion Acoustic Model for Singing Voice Synthesis [53.19363127760314]
  DiffSinger is a parameterized Markov chain which iteratively converts the noise into mel-spectrogram conditioned on the music score. The evaluations conducted on the Chinese singing dataset demonstrate that DiffSinger outperforms state-of-the-art SVS work with a notable margin.
  arXiv  Detail & Related papers  (2021-05-06T05:21:42Z)
• HpRNet : Incorporating Residual Noise Modeling for Violin in a Variational Parametric Synthesizer [11.4219428942199]
  We introduce a dataset of Carnatic Violin Recordings where bow noise is an integral part of the playing style of higher pitched notes. We obtain insights about each of the harmonic and residual components of the signal, as well as their interdependence.
  arXiv  Detail & Related papers  (2020-08-19T12:48:32Z)
• Generative Modelling for Controllable Audio Synthesis of Expressive Piano Performance [6.531546527140474]
  controllable neural audio synthesizer based on Gaussian Mixture Variational Autoencoders (GM-VAE) We demonstrate how the model is able to apply fine-grained style morphing over the course of the audio.
  arXiv  Detail & Related papers  (2020-06-16T12:54:41Z)
• VaPar Synth -- A Variational Parametric Model for Audio Synthesis [78.3405844354125]
  We present VaPar Synth - a Variational Parametric Synthesizer which utilizes a conditional variational autoencoder (CVAE) trained on a suitable parametric representation. We demonstrate our proposed model's capabilities via the reconstruction and generation of instrumental tones with flexible control over their pitch.
  arXiv  Detail & Related papers  (2020-03-30T16:05:47Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.