Soundify: Matching Sound Effects to Video

• URL: http://arxiv.org/abs/2112.09726v4
• Date: Tue, 25 Jun 2024 13:28:04 GMT
• Title: Soundify: Matching Sound Effects to Video
• Authors: David Chuan-En Lin, Anastasis Germanidis, Cristóbal Valenzuela, Yining Shi, Nikolas Martelaro,
• Abstract summary: This paper presents Soundify, a system that assists editors in matching sounds to video. Given a video, Soundify identifies matching sounds, synchronizes the sounds to the video, and dynamically adjusts panning and volume to create spatial audio.
• Score: 4.225919537333002
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: In the art of video editing, sound helps add character to an object and immerse the viewer within a space. Through formative interviews with professional editors (N=10), we found that the task of adding sounds to video can be challenging. This paper presents Soundify, a system that assists editors in matching sounds to video. Given a video, Soundify identifies matching sounds, synchronizes the sounds to the video, and dynamically adjusts panning and volume to create spatial audio. In a human evaluation study (N=889), we show that Soundify is capable of matching sounds to video out-of-the-box for a diverse range of audio categories. In a within-subjects expert study (N=12), we demonstrate the usefulness of Soundify in helping video editors match sounds to video with lighter workload, reduced task completion time, and improved usability.

Related papers

• Self-Supervised Audio-Visual Soundscape Stylization [22.734359700809126]
  We manipulate input speech to sound as though it was recorded within a different scene, given an audio-visual conditional example recorded from that scene. Our model learns through self-supervision, taking advantage of the fact that natural video contains recurring sound events and textures. We show that our model can be successfully trained using unlabeled, in-the-wild videos, and that an additional visual signal can improve its sound prediction abilities.
  arXiv  Detail & Related papers  (2024-09-22T06:57:33Z)
• Action2Sound: Ambient-Aware Generation of Action Sounds from Egocentric Videos [87.32349247938136]
  Existing approaches implicitly assume total correspondence between the video and audio during training. We propose a novel ambient-aware audio generation model, AV-LDM. Our approach is the first to focus video-to-audio generation faithfully on the observed visual content.
  arXiv  Detail & Related papers  (2024-06-13T16:10:19Z)
• AudioScenic: Audio-Driven Video Scene Editing [55.098754835213995]
  We introduce AudioScenic, an audio-driven framework designed for video scene editing. AudioScenic integrates audio semantics into the visual scene through a temporal-aware audio semantic injection process. We present an audio Magnitude Modulator module that adjusts the temporal dynamics of the scene in response to changes in audio magnitude. Second, the audio Frequency Fuser module is designed to ensure temporal consistency by aligning the frequency of the audio with the dynamics of the video scenes.
  arXiv  Detail & Related papers  (2024-04-25T12:55:58Z)
• SyncFusion: Multimodal Onset-synchronized Video-to-Audio Foley Synthesis [9.118448725265669]
  One of the most time-consuming steps when designing sound is synchronizing audio with video. In video games and animations, no reference audio exists, requiring manual annotation of event timings from the video. We propose a system to extract repetitive actions onsets from a video, which are then used to condition a diffusion model trained to generate a new synchronized sound effects audio track.
  arXiv  Detail & Related papers  (2023-10-23T18:01:36Z)
• AdVerb: Visually Guided Audio Dereverberation [49.958724234969445]
  We present AdVerb, a novel audio-visual dereverberation framework. It uses visual cues in addition to the reverberant sound to estimate clean audio.
  arXiv  Detail & Related papers  (2023-08-23T18:20:59Z)
• WavJourney: Compositional Audio Creation with Large Language Models [38.39551216587242]
  We present WavJourney, a novel framework that leverages Large Language Models to connect various audio models for audio creation. WavJourney allows users to create storytelling audio content with diverse audio elements simply from textual descriptions. We show that WavJourney is capable of synthesizing realistic audio aligned with textually-described semantic, spatial and temporal conditions.
  arXiv  Detail & Related papers  (2023-07-26T17:54:04Z)
• Exploring the Role of Audio in Video Captioning [59.679122191706426]
  We present an audio-visual framework, which aims to fully exploit the potential of the audio modality for captioning. We propose new local-global fusion mechanisms to improve information exchange across audio and video.
  arXiv  Detail & Related papers  (2023-06-21T20:54:52Z)
• Conditional Generation of Audio from Video via Foley Analogies [19.681437827280757]
  Sound effects that designers add to videos are designed to convey a particular artistic effect and may be quite different from a scene's true sound. Inspired by the challenges of creating a soundtrack for a video that differs from its true sound, we propose the problem of conditional Foley. We show through human studies and automated evaluation metrics that our model successfully generates sound from video.
  arXiv  Detail & Related papers  (2023-04-17T17:59:45Z)
• Visually-Aware Audio Captioning With Adaptive Audio-Visual Attention [54.4258176885084]
  How to accurately recognize ambiguous sounds is a major challenge for audio captioning. We propose visually-aware audio captioning, which makes use of visual information to help the description of ambiguous sounding objects. Our proposed method achieves state-of-the-art results on machine translation metrics.
  arXiv  Detail & Related papers  (2022-10-28T22:45:41Z)
• Generating Visually Aligned Sound from Videos [83.89485254543888]
  We focus on the task of generating sound from natural videos. The sound should be both temporally and content-wise aligned with visual signals. Some sounds generated outside of a camera can not be inferred from video content.
  arXiv  Detail & Related papers  (2020-07-14T07:51:06Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.