MMAudio-LABEL: Audio Event Labeling via Audio Generation for Silent Video

• URL: http://arxiv.org/abs/2605.00495v1
• Date: Fri, 01 May 2026 08:09:06 GMT
• Title: MMAudio-LABEL: Audio Event Labeling via Audio Generation for Silent Video
• Abstract summary: MMAudio-LABEL is an event-aware audio generation framework built on a foundational audio generation model.<n>We evaluate our method on the Greatest Hits dataset for onset detection and 17-class material classification.<n>Results suggest that jointly learning audio generation and event prediction enables a more interpretable and practical video-to-audio synthesis.
• Score: 28.779831823146313
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Recent advances in multimodal generation have enabled high-quality audio generation from silent videos. Practical applications, such as sound production, demand not only the generated audio but also explicit sound event labels detailing the type and timing of sounds. One straightforward approach involves applying a standard sound event detection to the generated audio. However, this post-hoc pipeline is inherently limited, as it is prone to error accumulation. To address this limitation, we propose MMAudio-LABEL (LAtent-Based Event Labeling), an event-aware audio generation framework built on a foundational audio generation model as its backbone that jointly generates audio and frame-aligned sound event predictions from silent videos. We evaluate our method on the Greatest Hits dataset for onset detection and 17-class material classification. Our approach improves onset-detection accuracy from 46.7% to 75.0% and material-classification accuracy from 40.6% to 61.0% over baselines. These results suggest that jointly learning audio generation and event prediction enables a more interpretable and practical video-to-audio synthesis.

Related papers

• PRISM-Bench: An Audio-Centric Diagnostic Benchmark for Text-to-Audio-Video Generation [23.127789678422698]
  Text-to-audio-video (T2AV) generation has advanced rapidly, but its evaluation still underestimates the audio modality.<n>We present PRISM-Bench, the first audio-centric diagnostic benchmark for T2AV generation.
  arXiv  Detail & Related papers  (2026-09-04T08:27:10Z)
• DreamAudio: Customized Text-to-Audio Generation with Diffusion Models [38.963121219471354]
  We present DreamAudio for customized text-to-audio generation (CTTA)<n>We introduce a new framework that is designed to enable the model to identify auditory information from user-provided reference concepts for audio generation.<n>Given a few reference audio samples containing personalized audio events, our system can generate new audio samples that include these specific events.
  arXiv  Detail & Related papers  (2025-09-07T12:06:21Z)
• Kimi-Audio Technical Report [67.69331679172303]
  Kimi-Audio is an open-source audio foundation model that excels in audio understanding, generation, and conversation.<n>We detail the practices in building Kimi-Audio, including model architecture, data curation, training recipe, inference deployment, and evaluation.
  arXiv  Detail & Related papers  (2025-04-25T15:31:46Z)
• MMAudio: Taming Multimodal Joint Training for High-Quality Video-to-Audio Synthesis [56.01110988816489]
  We propose to synthesize high-quality and synchronized audio, given video and optional text conditions, using a novel multimodal joint training framework MMAudio.<n> MMAudio is jointly trained with larger-scale, readily available text-audio data to learn to generate semantically aligned high-quality audio samples.<n> MMAudio achieves surprisingly competitive performance in text-to-audio generation, showing that joint training does not hinder single-modality performance.
  arXiv  Detail & Related papers  (2024-12-19T18:59:55Z)
• AdVerb: Visually Guided Audio Dereverberation [49.958724234969445]
  We present AdVerb, a novel audio-visual dereverberation framework. It uses visual cues in addition to the reverberant sound to estimate clean audio.
  arXiv  Detail & Related papers  (2023-08-23T18:20:59Z)
• Large-scale unsupervised audio pre-training for video-to-speech synthesis [64.86087257004883]
  Video-to-speech synthesis is the task of reconstructing the speech signal from a silent video of a speaker. In this paper we propose to train encoder-decoder models on more than 3,500 hours of audio data at 24kHz. We then use the pre-trained decoders to initialize the audio decoders for the video-to-speech synthesis task.
  arXiv  Detail & Related papers  (2023-06-27T13:31:33Z)
• A dataset for Audio-Visual Sound Event Detection in Movies [33.59510253345295]
  We present a dataset of audio events called Subtitle-Aligned Movie Sounds (SAM-S) We use publicly-available closed-caption transcripts to automatically mine over 110K audio events from 430 movies. We identify three dimensions to categorize audio events: sound, source, quality, and present the steps involved to produce a final taxonomy of 245 sounds.
  arXiv  Detail & Related papers  (2023-02-14T19:55:39Z)
• AudioGen: Textually Guided Audio Generation [116.57006301417306]
  We tackle the problem of generating audio samples conditioned on descriptive text captions. In this work, we propose AaudioGen, an auto-regressive model that generates audio samples conditioned on text inputs.
  arXiv  Detail & Related papers  (2022-09-30T10:17:05Z)
• Generating Visually Aligned Sound from Videos [83.89485254543888]
  We focus on the task of generating sound from natural videos. The sound should be both temporally and content-wise aligned with visual signals. Some sounds generated outside of a camera can not be inferred from video content.
  arXiv  Detail & Related papers  (2020-07-14T07:51:06Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.