AI-Based Sound Effect Generation: A Narrative Review of Generative Models Across Input Modalities

• URL: http://arxiv.org/abs/2608.03742v1
• Date: Tue, 04 Aug 2026 14:36:34 GMT
• Title: AI-Based Sound Effect Generation: A Narrative Review of Generative Models Across Input Modalities
• Abstract summary: This chapter reviews and analyzes recent AI-based generative models for sound effect synthesis.<n>It examines 30 peer-reviewed articles sourced from Google Scholar, IEEE Xplore, and the ACM Digital Library.<n>The results show that multiple models achieved state-of-the-art performance, producing high-fidelity, semantically aligned, and increasingly temporally coherent sound effects across tasks.
• Score: 0.0
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: Sound effects play a crucial role in conveying actions, events, and environmental cues across digital applications, often requiring a high degree of variation and contextual adaptability. Artificial intelligence (AI)-driven audio generative models are rapidly growing in popularity and have the potential to transform the way sound is synthesized and used across various applications. In response to this growing momentum, this chapter reviews and analyzes recent AI-based generative models for sound effect synthesis, with a focus on how different input modalities (text, visual, audio, and multimodal) affect the quality, controllability, and contextual relevance of the generated audio. It examines 30 peer-reviewed articles sourced from Google Scholar, IEEE Xplore, and the ACM Digital Library, exploring the evolution of AI generative models over the past five years. The results show that multiple models achieved state-of-the-art performance, producing high-fidelity, semantically aligned, and increasingly temporally coherent sound effects across tasks. However, despite these advances, the review identifies persistent challenges, including limitations in temporal synchronization for complex multi-event scenarios, gaps between objective metrics and human perception, and trade-offs between controllability and generative diversity. Overall, the chapter highlights that AI-driven sound effect generation is progressing toward more adaptive, scalable, and context-aware systems, offering significant implications for future sound design workflows and interactive media applications.

Related papers

• ImmersiveTTS: Environment-Aware Text-to-Speech with Multimodal Diffusion Transformer and Domain-Specific Representation Alignment [52.31881685769569]
  We propose ImmersiveTTS, an environment-aware text-to-speech (TTS) model that generates natural speech seamlessly integrated within environmental contexts.<n>Our model builds on a multimodal diffusion transformer and fuses transcript-aligned speech latent with text-conditioned environmental context.<n> Experimental results show that ImmersiveTTS achieves higher naturalness, intelligibility, and audio fidelity than existing approaches.
  arXiv  Detail & Related papers  (2026-05-29T07:58:54Z)
• Audio-Visual Intelligence in Large Foundation Models [123.78167820053545]
  Audio-Visual Intelligence (AVI) has emerged as a central frontier in artificial intelligence.<n>Despite rapid progress, the literature remains fragmented, spanning diverse tasks, inconsistent, and heterogeneous evaluation practices.<n>This survey provides the first comprehensive review of AVI through the lens of large foundation models.
  arXiv  Detail & Related papers  (2026-05-05T17:59:16Z)
• FusAD: Time-Frequency Fusion with Adaptive Denoising for General Time Series Analysis [92.23551599659186]
  Time series analysis plays a vital role in fields such as finance, healthcare, industry, and meteorology.<n>FusAD is a unified analysis framework designed for diverse time series tasks.
  arXiv  Detail & Related papers  (2025-12-16T04:34:27Z)
• Auto-Regressive vs Flow-Matching: a Comparative Study of Modeling Paradigms for Text-to-Music Generation [36.11407262818182]
  State-of-the-art (SOTA) systems differ significantly in many dimensions, such as training datasets, modeling paradigms, and architectural choices.<n>This study focuses exclusively on the modeling paradigm.<n>We compare the two arguably most common modeling paradigms: auto-regressive decoding and conditional flow-matching.
  arXiv  Detail & Related papers  (2025-06-10T08:37:45Z)
• AVadCLIP: Audio-Visual Collaboration for Robust Video Anomaly Detection [57.649223695021114]
  We present a novel weakly supervised framework that leverages audio-visual collaboration for robust video anomaly detection.<n>Our framework demonstrates superior performance across multiple benchmarks, with audio integration significantly boosting anomaly detection accuracy.
  arXiv  Detail & Related papers  (2025-04-06T13:59:16Z)
• Comparative Analysis of Audio Feature Extraction for Real-Time Talking Portrait Synthesis [3.210706100833053]
  We propose and implement a fully integrated system that replaces conventional AFE models with Open AI's Whisper. We show that Whisper not only accelerates processing but also improves specific aspects of rendering quality, resulting in more realistic and responsive talking-head interactions.
  arXiv  Detail & Related papers  (2024-11-20T11:18:05Z)
• Can Large Audio-Language Models Truly Hear? Tackling Hallucinations with Multi-Task Assessment and Stepwise Audio Reasoning [55.2480439325792]
  Large audio-language models (LALMs) have shown impressive capabilities in understanding and reasoning about audio and speech information.<n>These models still face challenges, including hallucinating non-existent sound events, misidentifying the order of sound events, and incorrectly attributing sound sources.
  arXiv  Detail & Related papers  (2024-10-21T15:55:27Z)
• Where are we in audio deepfake detection? A systematic analysis over generative and detection models [59.09338266364506]
  SONAR is a synthetic AI-Audio Detection Framework and Benchmark.<n>It provides a comprehensive evaluation for distinguishing cutting-edge AI-synthesized auditory content.<n>It is the first framework to uniformly benchmark AI-audio detection across both traditional and foundation model-based detection systems.
  arXiv  Detail & Related papers  (2024-10-06T01:03:42Z)
• Unveiling and Mitigating Bias in Audio Visual Segmentation [9.427676046134374]
  Community researchers have developed a range of advanced audio-visual segmentation models to improve the quality of sounding objects' masks. While masks created by these models may initially appear plausible, they occasionally exhibit anomalies with incorrect grounding logic. We attribute this to real-world inherent preferences and distributions as a simpler signal for learning than the complex audio-visual grounding.
  arXiv  Detail & Related papers  (2024-07-23T16:55:04Z)
• CMMD: Contrastive Multi-Modal Diffusion for Video-Audio Conditional Modeling [21.380988939240844]
  We introduce a multi-modal diffusion model tailored for the bi-directional conditional generation of video and audio. We propose a joint contrastive training loss to improve the synchronization between visual and auditory occurrences.
  arXiv  Detail & Related papers  (2023-12-08T23:55:19Z)
• A Comprehensive Multi-scale Approach for Speech and Dynamics Synchrony in Talking Head Generation [16.033455552126348]
  We propose a multi-scale audio-visual synchrony loss and a multi-scale autoregressive GAN.<n>We train a stack of syncer models on multimodal input pyramids and use these models as guidance in a multi-scale generator network.<n>Experiments show significant improvements over the state-of-the-art in head motion dynamics quality.
  arXiv  Detail & Related papers  (2023-07-04T08:29:59Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.