Video Soundtrack Generation by Aligning Emotions and Temporal Boundaries

• URL: http://arxiv.org/abs/2502.10154v1
• Date: Fri, 14 Feb 2025 13:32:59 GMT
• Title: Video Soundtrack Generation by Aligning Emotions and Temporal Boundaries
• Authors: Serkan Sulun, Paula Viana, Matthew E. P. Davies,
• Abstract summary: EMSYNC is a video-based symbolic music generation model that aligns music with a video's emotional content and temporal boundaries.<n>We introduce boundary offsets, a novel temporal conditioning mechanism that enables the model to anticipate and align musical chords with scene cuts.<n>In subjective listening tests, EMSYNC outperforms state-of-the-art models across all subjective metrics, for music theory-aware participants as well as the general listeners.
• Score: 1.1743167854433303
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: We introduce EMSYNC, a video-based symbolic music generation model that aligns music with a video's emotional content and temporal boundaries. It follows a two-stage framework, where a pretrained video emotion classifier extracts emotional features, and a conditional music generator produces MIDI sequences guided by both emotional and temporal cues. We introduce boundary offsets, a novel temporal conditioning mechanism that enables the model to anticipate and align musical chords with scene cuts. Unlike existing models, our approach retains event-based encoding, ensuring fine-grained timing control and expressive musical nuances. We also propose a mapping scheme to bridge the video emotion classifier, which produces discrete emotion categories, with the emotion-conditioned MIDI generator, which operates on continuous-valued valence-arousal inputs. In subjective listening tests, EMSYNC outperforms state-of-the-art models across all subjective metrics, for music theory-aware participants as well as the general listeners.

Related papers

• Extending Visual Dynamics for Video-to-Music Generation [51.274561293909926]
  DyViM is a novel framework to enhance dynamics modeling for video-to-music generation. High-level semantics are conveyed through a cross-attention mechanism. Experiments demonstrate DyViM's superiority over state-of-the-art (SOTA) methods.
  arXiv  Detail & Related papers  (2025-04-10T09:47:26Z)
• Emotion-Guided Image to Music Generation [0.5461938536945723]
  This paper presents an emotion-guided image-to-music generation framework. It produces music that aligns with the emotional tone of a given image. The model employs a CNN-Transformer architecture, featuring pre-trained CNN image feature extractors and three Transformer encoders.
  arXiv  Detail & Related papers  (2024-10-29T17:47:51Z)
• MuVi: Video-to-Music Generation with Semantic Alignment and Rhythmic Synchronization [52.498942604622165]
  This paper presents MuVi, a framework to generate music that aligns with video content. MuVi analyzes video content through a specially designed visual adaptor to extract contextually and temporally relevant features. We show that MuVi demonstrates superior performance in both audio quality and temporal synchronization.
  arXiv  Detail & Related papers  (2024-10-16T18:44:56Z)
• Audio-Driven Emotional 3D Talking-Head Generation [47.6666060652434]
  We present a novel system for synthesizing high-fidelity, audio-driven video portraits with accurate emotional expressions. We propose a pose sampling method that generates natural idle-state (non-speaking) videos in response to silent audio inputs.
  arXiv  Detail & Related papers  (2024-10-07T08:23:05Z)
• Are We There Yet? A Brief Survey of Music Emotion Prediction Datasets, Models and Outstanding Challenges [9.62904012066486]
  We provide a comprehensive overview of the available music-emotion datasets and discuss evaluation standards as well as competitions in the field. We highlight the challenges that persist in accurately capturing emotion in music, including issues related to dataset quality, annotation consistency, and model generalization. We have complemented our findings with an accompanying GitHub repository.
  arXiv  Detail & Related papers  (2024-06-13T05:00:27Z)
• Emotion Manipulation Through Music -- A Deep Learning Interactive Visual Approach [0.0]
  We introduce a novel way to manipulate the emotional content of a song using AI tools. Our goal is to achieve the desired emotion while leaving the original melody as intact as possible. This research may contribute to on-demand custom music generation, the automated remixing of existing work, and music playlists tuned for emotional progression.
  arXiv  Detail & Related papers  (2024-06-12T20:12:29Z)
• GMTalker: Gaussian Mixture-based Audio-Driven Emotional Talking Video Portraits [60.05683966405544]
  We present GMTalker, a Gaussian mixture-based emotional talking portraits generation framework. Specifically, we propose a continuous and disentangled latent space, achieving more flexible emotion manipulation. We also introduce a normalizing flow-based motion generator pretrained on a large dataset to generate diverse head poses, blinks, and eyeball movements.
  arXiv  Detail & Related papers  (2023-12-12T19:03:04Z)
• Simple and Controllable Music Generation [94.61958781346176]
  MusicGen is a single Language Model (LM) that operates over several streams of compressed discrete music representation, i.e., tokens. Unlike prior work, MusicGen is comprised of a single-stage transformer LM together with efficient token interleaving patterns.
  arXiv  Detail & Related papers  (2023-06-08T15:31:05Z)
• REMAST: Real-time Emotion-based Music Arrangement with Soft Transition [29.34094293561448]
  Music as an emotional intervention medium has important applications in scenarios such as music therapy, games, and movies. We propose REMAST to achieve emotion real-time fit and smooth transition simultaneously. According to the evaluation results, REMAST surpasses the state-of-the-art methods in objective and subjective metrics.
  arXiv  Detail & Related papers  (2023-05-14T00:09:48Z)
• V2Meow: Meowing to the Visual Beat via Video-to-Music Generation [47.076283429992664]
  V2Meow is a video-to-music generation system capable of producing high-quality music audio for a diverse range of video input types. It synthesizes high-fidelity music audio waveforms solely by conditioning on pre-trained general-purpose visual features extracted from video frames.
  arXiv  Detail & Related papers  (2023-05-11T06:26:41Z)
• Dilated Context Integrated Network with Cross-Modal Consensus for Temporal Emotion Localization in Videos [128.70585652795637]
  TEL presents three unique challenges compared to temporal action localization. The emotions have extremely varied temporal dynamics. The fine-grained temporal annotations are complicated and labor-intensive.
  arXiv  Detail & Related papers  (2022-08-03T10:00:49Z)
• Music Gesture for Visual Sound Separation [121.36275456396075]
  "Music Gesture" is a keypoint-based structured representation to explicitly model the body and finger movements of musicians when they perform music. We first adopt a context-aware graph network to integrate visual semantic context with body dynamics, and then apply an audio-visual fusion model to associate body movements with the corresponding audio signals.
  arXiv  Detail & Related papers  (2020-04-20T17:53:46Z)
• Emotional Video to Audio Transformation Using Deep Recurrent Neural Networks and a Neuro-Fuzzy System [8.900866276512364]
  Current approaches overlook the video's emotional characteristics in the music generation step. We propose a novel hybrid deep neural network that uses an Adaptive Neuro-Fuzzy Inference System to predict a video's emotion. Our model can effectively generate audio that matches the scene eliciting a similar emotion from the viewer in both datasets.
  arXiv  Detail & Related papers  (2020-04-05T07:18:28Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.