Annotation-Free MIDI-to-Audio Synthesis via Concatenative Synthesis and Generative Refinement

• URL: http://arxiv.org/abs/2410.16785v1
• Date: Tue, 22 Oct 2024 08:01:40 GMT
• Title: Annotation-Free MIDI-to-Audio Synthesis via Concatenative Synthesis and Generative Refinement
• Authors: Osamu Take, Taketo Akama,
• Abstract summary: CoSaRef is a MIDI-to-audio synthesis method that can be developed without MIDI-audio paired datasets. It first performs concatenative synthesis based on MIDI inputs and then refines the resulting audio into realistic tracks using a diffusion-based deep generative model trained on audio-only datasets.
• Score: 0.0
• License:
• Abstract: Recent MIDI-to-audio synthesis methods have employed deep neural networks to successfully generate high-quality and expressive instrumental tracks. However, these methods require MIDI annotations for supervised training, limiting the diversity of the output audio in terms of instrument timbres, and expression styles. We propose CoSaRef, a MIDI-to-audio synthesis method that can be developed without MIDI-audio paired datasets. CoSaRef first performs concatenative synthesis based on MIDI inputs and then refines the resulting audio into realistic tracks using a diffusion-based deep generative model trained on audio-only datasets. This approach enhances the diversity of audio timbres and expression styles. It also allows for control over the output timbre based on audio sample selection, similar to traditional functions in digital audio workstations. Experiments show that while inherently capable of generating general tracks with high control over timbre, CoSaRef can also perform comparably to conventional methods in generating realistic audio.

Related papers

• Combining audio control and style transfer using latent diffusion [1.705371629600151]
  In this paper, we aim to unify explicit control and style transfer within a single model. Our model can generate audio matching a timbre target, while specifying structure either with explicit controls or through another audio example. We show that our method can generate cover versions of complete musical pieces by transferring rhythmic and melodic content to the style of a target audio in a different genre.
  arXiv  Detail & Related papers  (2024-07-31T23:27:27Z)
• Accompanied Singing Voice Synthesis with Fully Text-controlled Melody [61.147446955297625]
  Text-to-song (TTSong) is a music generation task that synthesizes accompanied singing voices. We present MelodyLM, the first TTSong model that generates high-quality song pieces with fully text-controlled melodies.
  arXiv  Detail & Related papers  (2024-07-02T08:23:38Z)
• Creative Text-to-Audio Generation via Synthesizer Programming [1.1203110769488043]
  We propose a text-to-audio generation method that leverages a virtual modular sound synthesizer with only 78 parameters. Our method, CTAG, iteratively updates a synthesizer's parameters to produce high-quality audio renderings of text prompts.
  arXiv  Detail & Related papers  (2024-06-01T04:08:31Z)
• Prompt-Singer: Controllable Singing-Voice-Synthesis with Natural Language Prompt [50.25271407721519]
  We propose Prompt-Singer, the first SVS method that enables attribute controlling on singer gender, vocal range and volume with natural language. We adopt a model architecture based on a decoder-only transformer with a multi-scale hierarchy, and design a range-melody decoupled pitch representation. Experiments show that our model achieves favorable controlling ability and audio quality.
  arXiv  Detail & Related papers  (2024-03-18T13:39:05Z)
• DiffMoog: a Differentiable Modular Synthesizer for Sound Matching [48.33168531500444]
  DiffMoog is a differentiable modular synthesizer with a comprehensive set of modules typically found in commercial instruments. Being differentiable, it allows integration into neural networks, enabling automated sound matching. We introduce an open-source platform that comprises DiffMoog and an end-to-end sound matching framework.
  arXiv  Detail & Related papers  (2024-01-23T08:59:21Z)
• Make-A-Voice: Unified Voice Synthesis With Discrete Representation [77.3998611565557]
  Make-A-Voice is a unified framework for synthesizing and manipulating voice signals from discrete representations. We show that Make-A-Voice exhibits superior audio quality and style similarity compared with competitive baseline models.
  arXiv  Detail & Related papers  (2023-05-30T17:59:26Z)
• Multi-instrument Music Synthesis with Spectrogram Diffusion [19.81982315173444]
  We focus on a middle ground of neural synthesizers that can generate audio from MIDI sequences with arbitrary combinations of instruments in realtime. We use a simple two-stage process: MIDI to spectrograms with an encoder-decoder Transformer, then spectrograms to audio with a generative adversarial network (GAN) spectrogram inverter. We find this to be a promising first step towards interactive and expressive neural synthesis for arbitrary combinations of instruments and notes.
  arXiv  Detail & Related papers  (2022-06-11T03:26:15Z)
• MIDI-DDSP: Detailed Control of Musical Performance via Hierarchical Modeling [6.256118777336895]
  Musical expression requires control of both what notes are played, and how they are performed. We introduce MIDI-DDSP, a hierarchical model of musical instruments that enables both realistic neural audio synthesis and detailed user control. We demonstrate that this hierarchy can reconstruct high-fidelity audio, accurately predict performance attributes for a note sequence, independently manipulate the attributes of a given performance, and as a complete system, generate realistic audio from a novel note sequence.
  arXiv  Detail & Related papers  (2021-12-17T04:15:42Z)
• Rapping-Singing Voice Synthesis based on Phoneme-level Prosody Control [47.33830090185952]
  A text-to-rapping/singing system is introduced, which can be adapted to any speaker's voice. It utilizes a Tacotron-based multispeaker acoustic model trained on read-only speech data. Results show that the proposed approach can produce high quality rapping/singing voice with increased naturalness.
  arXiv  Detail & Related papers  (2021-11-17T14:31:55Z)
• Unsupervised Cross-Domain Singing Voice Conversion [105.1021715879586]
  We present a wav-to-wav generative model for the task of singing voice conversion from any identity. Our method utilizes both an acoustic model, trained for the task of automatic speech recognition, together with melody extracted features to drive a waveform-based generator.
  arXiv  Detail & Related papers  (2020-08-06T18:29:11Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.