Temporally Aligned Audio for Video with Autoregression

• URL: http://arxiv.org/abs/2409.13689v1
• Date: Fri, 20 Sep 2024 17:59:01 GMT
• Title: Temporally Aligned Audio for Video with Autoregression
• Authors: Ilpo Viertola, Vladimir Iashin, Esa Rahtu,
• Abstract summary: V-AURA is the first autoregressive model to achieve high temporal alignment and relevance in video-to-audio generation. VisualSound is a benchmark dataset with high audio-visual relevance.
• Score: 17.019400481122872
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: We introduce V-AURA, the first autoregressive model to achieve high temporal alignment and relevance in video-to-audio generation. V-AURA uses a high-framerate visual feature extractor and a cross-modal audio-visual feature fusion strategy to capture fine-grained visual motion events and ensure precise temporal alignment. Additionally, we propose VisualSound, a benchmark dataset with high audio-visual relevance. VisualSound is based on VGGSound, a video dataset consisting of in-the-wild samples extracted from YouTube. During the curation, we remove samples where auditory events are not aligned with the visual ones. V-AURA outperforms current state-of-the-art models in temporal alignment and semantic relevance while maintaining comparable audio quality. Code, samples, VisualSound and models are available at https://v-aura.notion.site

Related papers

• From Vision to Audio and Beyond: A Unified Model for Audio-Visual Representation and Generation [17.95017332858846]
  We introduce a novel framework called Vision to Audio and Beyond (VAB) to bridge the gap between audio-visual representation learning and vision-to-audio generation. VAB uses a pre-trained audio tokenizer and an image encoder to obtain audio tokens and visual features, respectively. Our experiments showcase the efficiency of VAB in producing high-quality audio from video, and its capability to acquire semantic audio-visual features.
  arXiv  Detail & Related papers  (2024-09-27T20:26:34Z)
• Unveiling Visual Biases in Audio-Visual Localization Benchmarks [52.76903182540441]
  We identify a significant issue in existing benchmarks. The sounding objects are often easily recognized based solely on visual cues, which we refer to as visual bias. Our findings suggest that existing AVSL benchmarks need further refinement to facilitate audio-visual learning.
  arXiv  Detail & Related papers  (2024-08-25T04:56:08Z)
• Video-Foley: Two-Stage Video-To-Sound Generation via Temporal Event Condition For Foley Sound [6.638504164134713]
  Foley sound synthesis is crucial for multimedia production, enhancing user experience by synchronizing audio and video both temporally and semantically. Recent studies on automating this labor-intensive process through video-to-sound generation face significant challenges. We propose Video-Foley, a video-to-sound system using Root Mean Square (RMS) as a temporal event condition with semantic timbre prompts.
  arXiv  Detail & Related papers  (2024-08-21T18:06:15Z)
• Text-to-feature diffusion for audio-visual few-shot learning [59.45164042078649]
  Few-shot learning from video data is a challenging and underexplored, yet much cheaper, setup. We introduce a unified audio-visual few-shot video classification benchmark on three datasets. We show that AV-DIFF obtains state-of-the-art performance on our proposed benchmark for audio-visual few-shot learning.
  arXiv  Detail & Related papers  (2023-09-07T17:30:36Z)
• Diff-Foley: Synchronized Video-to-Audio Synthesis with Latent Diffusion Models [12.898486592791604]
  We present Diff-Foley, a synchronized Video-to-Audio synthesis method with a latent diffusion model (LDM) We show Diff-Foley achieves state-of-the-art V2A performance on current large scale V2A dataset.
  arXiv  Detail & Related papers  (2023-06-29T12:39:58Z)
• V2Meow: Meowing to the Visual Beat via Video-to-Music Generation [47.076283429992664]
  V2Meow is a video-to-music generation system capable of producing high-quality music audio for a diverse range of video input types. It synthesizes high-fidelity music audio waveforms solely by conditioning on pre-trained general-purpose visual features extracted from video frames.
  arXiv  Detail & Related papers  (2023-05-11T06:26:41Z)
• Audio-Visual Contrastive Learning with Temporal Self-Supervision [84.11385346896412]
  We propose a self-supervised learning approach for videos that learns representations of both the RGB frames and the accompanying audio without human supervision. To leverage the temporal and aural dimension inherent to videos, our method extends temporal self-supervision to the audio-visual setting.
  arXiv  Detail & Related papers  (2023-02-15T15:00:55Z)
• Dual-path Attention is All You Need for Audio-Visual Speech Extraction [34.7260610874298]
  We propose a new way to fuse audio-visual features. The proposed algorithm incorporates the visual features as an additional feature stream. Results show we achieve superior results compared with other time-domain based audio-visual fusion models.
  arXiv  Detail & Related papers  (2022-07-09T07:27:46Z)
• AVATAR: Unconstrained Audiovisual Speech Recognition [75.17253531162608]
  We propose a new sequence-to-sequence AudioVisual ASR TrAnsformeR (AVATAR) trained end-to-end from spectrograms and full-frame RGB. We demonstrate the contribution of the visual modality on the How2 AV-ASR benchmark, especially in the presence of simulated noise. We also create a new, real-world test bed for AV-ASR called VisSpeech, which demonstrates the contribution of the visual modality under challenging audio conditions.
  arXiv  Detail & Related papers  (2022-06-15T17:33:19Z)
• Localizing Visual Sounds the Hard Way [149.84890978170174]
  We train the network to explicitly discriminate challenging image fragments, even for images that do contain the object emitting the sound. We show that our algorithm achieves state-of-the-art performance on the popular Flickr SoundNet dataset. We introduce the VGG-Sound Source (VGG-SS) benchmark, a new set of annotations for the recently-introduced VGG-Sound dataset.
  arXiv  Detail & Related papers  (2021-04-06T17:38:18Z)
• Generating Visually Aligned Sound from Videos [83.89485254543888]
  We focus on the task of generating sound from natural videos. The sound should be both temporally and content-wise aligned with visual signals. Some sounds generated outside of a camera can not be inferred from video content.
  arXiv  Detail & Related papers  (2020-07-14T07:51:06Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.