WavFlow: Audio Generation in Waveform Space

• URL: http://arxiv.org/abs/2605.18749v1
• Date: Mon, 18 May 2026 17:59:10 GMT
• Title: WavFlow: Audio Generation in Waveform Space
• Abstract summary: WavFlow is a framework that generates high-fidelity audio directly in raw waveform space without intermediate representations.<n>We leverage an automated data pipeline to curate 5 million high-quality video-text-audio triplets.<n>WavFlow achieves competitive performance on the video-to-audio benchmark VGGSound (FD_PaSST: 59.98, IS_PANNs: 17.40, DeSync: 0.44) and the text-to-audio benchmark AudioCaps (FD_PANNs: 10.63, IS_PANNs: 12.62)
• Score: 25.62432990858879
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Modern audio generation predominantly relies on latent-space compression, introducing additional complexity and potential information loss. In this work, we challenge this paradigm with WavFlow, a framework that generates high-fidelity audio directly in raw waveform space without intermediate representations. To overcome the inherent difficulties of modeling high-dimensional and low-energy signals, we reshape audio into 2D token grids through waveform patchify and introduce amplitude lifting to align signal scales, enabling stable optimization via direct x-prediction in flow matching. To capture complex semantic alignment and temporal synchronization, we leverage an automated data pipeline to curate 5 million high-quality video-text-audio triplets, allowing the model to learn fine-grained acoustic patterns from scratch. Experimental results show that WavFlow achieves competitive performance on the video-to-audio benchmark VGGSound (FD_PaSST: 59.98, IS_PANNs: 17.40, DeSync: 0.44) and the text-to-audio benchmark AudioCaps (FD_PANNs: 10.63, IS_PANNs: 12.62), matching or exceeding the performance of established latent-based methods. Our work demonstrates that intermediate compression is not a prerequisite for high-quality synthesis, offering a simpler and more scalable alternative for multimodal audio generation.

Related papers

• Foley-Flow: Coordinated Video-to-Audio Generation with Masked Audio-Visual Alignment and Dynamic Conditional Flows [75.44753202066171]
  Coordinated audio generation based on video inputs typically requires a strict audio-visual (AV) alignment.<n>We propose FoleyFlow to first align unimodal AV encoders via masked modeling training.<n>After training, the AV encoders which are separately pretrained using only unimodal data are aligned with semantic and rhythmic consistency.
  arXiv  Detail & Related papers  (2026-03-09T09:06:25Z)
• ViSAudio: End-to-End Video-Driven Binaural Spatial Audio Generation [55.76423101183408]
  ViSAudio is an end-to-end framework that employs conditional flow matching with a dual-branch audio generation architecture.<n>It generates high-quality audio with spatial immersion that adapts to viewpoint changes, sound-source motion, and diverse acoustic environments.
  arXiv  Detail & Related papers  (2025-12-02T18:56:12Z)
• UniverSR: Unified and Versatile Audio Super-Resolution via Vocoder-Free Flow Matching [20.92242470770289]
  We present a framework for audio super-resolution that employs a flow matching generative model to capture the conditional distribution of complex-valued spectral coefficients.<n> Experiments show that our model consistently produces high-fidelity 48 kHz audio across diverse upsampling factors.
  arXiv  Detail & Related papers  (2025-10-01T11:04:53Z)
• Kling-Foley: Multimodal Diffusion Transformer for High-Quality Video-to-Audio Generation [27.20097004987987]
  We propose a large-scale multimodal Video-to-Audio generation model that synthesizes high-quality audio synchronized with video content.<n>Our experiments show that Kling-Foley trained with the flow matching objective achieves new audio-visual SOTA performance.
  arXiv  Detail & Related papers  (2025-06-24T16:39:39Z)
• MMAudio: Taming Multimodal Joint Training for High-Quality Video-to-Audio Synthesis [56.01110988816489]
  We propose to synthesize high-quality and synchronized audio, given video and optional text conditions, using a novel multimodal joint training framework MMAudio.<n> MMAudio is jointly trained with larger-scale, readily available text-audio data to learn to generate semantically aligned high-quality audio samples.<n> MMAudio achieves surprisingly competitive performance in text-to-audio generation, showing that joint training does not hinder single-modality performance.
  arXiv  Detail & Related papers  (2024-12-19T18:59:55Z)
• Frieren: Efficient Video-to-Audio Generation Network with Rectified Flow Matching [51.70360630470263]
  Video-to-audio (V2A) generation aims to synthesize content-matching audio from silent video.<n>We propose Frieren, a V2A model based on rectified flow matching.<n>Experiments indicate that Frieren achieves state-of-the-art performance in both generation quality and temporal alignment.
  arXiv  Detail & Related papers  (2024-06-01T06:40:22Z)
• From Discrete Tokens to High-Fidelity Audio Using Multi-Band Diffusion [84.138804145918]
  Deep generative models can generate high-fidelity audio conditioned on various types of representations. These models are prone to generate audible artifacts when the conditioning is flawed or imperfect. We propose a high-fidelity multi-band diffusion-based framework that generates any type of audio modality from low-bitrate discrete representations.
  arXiv  Detail & Related papers  (2023-08-02T22:14:29Z)
• Make-An-Audio: Text-To-Audio Generation with Prompt-Enhanced Diffusion Models [65.18102159618631]
  multimodal generative modeling has created milestones in text-to-image and text-to-video generation. Its application to audio still lags behind for two main reasons: the lack of large-scale datasets with high-quality text-audio pairs, and the complexity of modeling long continuous audio data. We propose Make-An-Audio with a prompt-enhanced diffusion model that addresses these gaps.
  arXiv  Detail & Related papers  (2023-01-30T04:44:34Z)
• RAVE: A variational autoencoder for fast and high-quality neural audio synthesis [2.28438857884398]
  We introduce a Realtime Audio Variational autoEncoder (RAVE) allowing both fast and high-quality audio waveform synthesis. We show that our model is the first able to generate 48kHz audio signals, while simultaneously running 20 times faster than real-time on a standard laptop CPU.
  arXiv  Detail & Related papers  (2021-11-09T09:07:30Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.