PhysWave: Physics-Guided Latent Diffusion Models for Controllable Spatial Audio Generation

• URL: http://arxiv.org/abs/2608.29549v1
• Date: Sun, 30 Aug 2026 04:37:50 GMT
• Title: PhysWave: Physics-Guided Latent Diffusion Models for Controllable Spatial Audio Generation
• Abstract summary: We present PhysWave, a physics-guided latent diffusion model for controllable text-to-FOA generation.<n> PhysWave unifies natural-language and trajectory control through a shared waypoint-caption representation.<n>Extensive results show that the proposed priors help PhysWave generate spatially consistent FOA audio while maintaining competitive audio quality.
• Score: 11.945977059126008
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Text-to-spatial audio generation, such as text-to-First-Order Ambisonics (FOA), provides a convenient way to create spatial audio for billion-dollar gaming and film industries. However, existing text-to-FOA methods are largely data-driven and may produce audio that violates acoustic relations between source direction and distance. They also separate descriptive and parametric control, forcing users to trade usability for precision. In this paper, we present PhysWave, a physics-guided latent diffusion model for controllable text-to-FOA generation. PhysWave unifies natural-language and trajectory control through a shared waypoint-caption representation, and augments diffusion training with two differentiable acoustic priors: spherical-harmonic direction consistency and inverse-square distance consistency. To support dynamic spatial generation, we further construct a 300K-clip FOA dataset with diverse sound categories and source trajectories. Extensive results show that the proposed priors help PhysWave generate spatially consistent FOA audio while maintaining competitive audio quality. Further analyses show that these physics priors improve spatial consistency during training and can also be used as inference-time guidance for training-free spatial refinement.

Related papers

• Spatio-Temporal Audio Language Modeling for Dynamic Sound Sources [57.68713138159972]
  We present a time-resolved audio encoder that learns event semantics together with source trajectories.<n>Experiments show that this representation improves the semanticlocalization and yields stronger reasoning than spatial and localization-oriented tradeoffs.
  arXiv  Detail & Related papers  (2026-06-12T05:58:31Z)
• BareWave: Waveform-Native Flow-Matching Text-to-Speech [76.5390412686083]
  We present BareWave, a fully waveform-native framework for direct text-to-wave generation in flow-matching TTS.<n>We develop a direct text-to-wave training framework that combines training-time representation alignment, staged noise scheduling, and velocity-aware perceptual alignment.<n> Experiments on zero-shot voice cloning show that strong intelligibility, speaker similarity, and naturalness can be achieved under a fully waveform-native inference path.
  arXiv  Detail & Related papers  (2026-06-08T05:36:42Z)
• UniSonate: A Unified Model for Speech, Music, and Sound Effect Generation with Text Instructions [55.622295453533475]
  We introduce UniSonate, a unified flow-matching framework capable of synthesizing speech, music, and sound effects.<n>We propose a novel dynamic token injection mechanism that projects unstructured environmental sounds into a structured temporal latent space.<n>Experiments demonstrate that UniSonate achieves state-of-the-art performance in instruction-based TTS and TTM.
  arXiv  Detail & Related papers  (2026-04-24T04:26:04Z)
• PhyAVBench: A Challenging Audio Physics-Sensitivity Benchmark for Physically Grounded Text-to-Audio-Video Generation [63.3417467957431]
  Text-to-audio-video (T2AV) generation underpins a wide range of applications demanding realistic audio-visual content.<n>We present PhyAVBench, a challenging audio physics-sensitivity benchmark designed to evaluate the audio physics grounding capabilities of existing T2AV models.<n>Unlike prior benchmarks that primarily focus on audio-video synchronization, PhyAVBench explicitly evaluates models' understanding of the physical mechanisms underlying sound generation.
  arXiv  Detail & Related papers  (2025-12-30T05:22:31Z)
• Generating Moving 3D Soundscapes with Latent Diffusion Models [4.792255437561157]
  We introduce SonicMotion, a framework capable of generating Ambisonics audio with explicit control over moving sound sources.<n>SonicMotion achieves state-of-the-art semantic alignment and perceptual quality comparable to leading text-to-audio systems.
  arXiv  Detail & Related papers  (2025-07-09T22:31:06Z)
• Adaptive Control Attention Network for Underwater Acoustic Localization and Domain Adaptation [8.017203108408973]
  Localizing acoustic sound sources in the ocean is a challenging task due to the complex and dynamic nature of the environment.<n>We propose a multi-branch network architecture designed to accurately predict the distance between a moving acoustic source and a receiver.<n>Our proposed method outperforms state-of-the-art (SOTA) approaches in similar settings.
  arXiv  Detail & Related papers  (2025-06-20T18:13:30Z)
• DualSpec: Text-to-spatial-audio Generation via Dual-Spectrogram Guided Diffusion Model [46.60765174200236]
  We propose a text-to-spatial-audio (TTSA) generation framework named DualSpec.<n>It first trains variational autoencoders (VAEs) for extracting the latent acoustic representations from sound event audio.<n>Finally, it trains a diffusion model from the latent acoustic representations and text features for the spatial audio generation.
  arXiv  Detail & Related papers  (2025-02-26T09:01:59Z)
• ImmerseDiffusion: A Generative Spatial Audio Latent Diffusion Model [2.2927722373373247]
  We introduce ImmerseDiffusion, an end-to-end generative audio model that produces 3D immersive soundscapes conditioned on the spatial, temporal, and environmental conditions of sound objects.
  arXiv  Detail & Related papers  (2024-10-19T02:28:53Z)
• DiffAR: Denoising Diffusion Autoregressive Model for Raw Speech Waveform Generation [25.968115316199246]
  This work proposes a diffusion probabilistic end-to-end model for generating a raw speech waveform. Our model is autoregressive, generating overlapping frames sequentially, where each frame is conditioned on a portion of the previously generated one. Experiments show that the proposed model generates speech with superior quality compared with other state-of-the-art neural speech generation systems.
  arXiv  Detail & Related papers  (2023-10-02T17:42:22Z)
• Make-An-Audio 2: Temporal-Enhanced Text-to-Audio Generation [72.7915031238824]
  Large diffusion models have been successful in text-to-audio (T2A) synthesis tasks. They often suffer from common issues such as semantic misalignment and poor temporal consistency. We propose Make-an-Audio 2, a latent diffusion-based T2A method that builds on the success of Make-an-Audio.
  arXiv  Detail & Related papers  (2023-05-29T10:41:28Z)
• Data Fusion for Audiovisual Speaker Localization: Extending Dynamic Stream Weights to the Spatial Domain [103.3388198420822]
  Esting the positions of multiple speakers can be helpful for tasks like automatic speech recognition or speaker diarization. This paper proposes a novel audiovisual data fusion framework for speaker localization by assigning individual dynamic stream weights to specific regions. A performance evaluation using audiovisual recordings yields promising results, with the proposed fusion approach outperforming all baseline models.
  arXiv  Detail & Related papers  (2021-02-23T09:59:31Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.