Leveraging Audio Gestalt to Predict Media Memorability

• URL: http://arxiv.org/abs/2012.15635v1
• Date: Thu, 31 Dec 2020 14:50:42 GMT
• Title: Leveraging Audio Gestalt to Predict Media Memorability
• Authors: Lorin Sweeney, Graham Healy, Alan F. Smeaton
• Abstract summary: Memorability determines what evanesces into emptiness, and what worms its way into the deepest furrows of our minds. The Predicting Media Memorability task in MediaEval 2020 aims to address the question of media memorability by setting the task of automatically predicting video memorability. Our approach is a multimodal deep learning-based late fusion that combines visual, semantic, and auditory features.
• Score: 1.8506048493564673
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Memorability determines what evanesces into emptiness, and what worms its way into the deepest furrows of our minds. It is the key to curating more meaningful media content as we wade through daily digital torrents. The Predicting Media Memorability task in MediaEval 2020 aims to address the question of media memorability by setting the task of automatically predicting video memorability. Our approach is a multimodal deep learning-based late fusion that combines visual, semantic, and auditory features. We used audio gestalt to estimate the influence of the audio modality on overall video memorability, and accordingly inform which combination of features would best predict a given video's memorability scores.

Related papers

• Audio Does Matter: Importance-Aware Multi-Granularity Fusion for Video Moment Retrieval [33.114796739109075]
  Video Moment Retrieval (VMR) aims to retrieve a specific moment semantically related to a given query.<n>Most existing VMR methods solely focus on the visual and textual modalities while neglecting the complementary but important audio modality.<n>We propose a novel Importance-aware Multi-Granularity fusion model (IMG), which learns to dynamically and selectively aggregate the audio-vision-text contexts for VMR.
  arXiv  Detail & Related papers  (2025-08-06T09:58:43Z)
• ThinkSound: Chain-of-Thought Reasoning in Multimodal Large Language Models for Audio Generation and Editing [52.33281620699459]
  ThinkSound is a novel framework that leverages Chain-of-Thought (CoT) reasoning to enable stepwise, interactive audio generation and editing for videos.<n>Our approach decomposes the process into three complementary stages: semantically coherent, interactive object-centric refinement through precise user interactions, and targeted editing guided by natural language instructions.<n> Experiments demonstrate that ThinkSound achieves state-of-the-art performance in video-to-audio generation across both audio metrics and CoT metrics.
  arXiv  Detail & Related papers  (2025-06-26T16:32:06Z)
• Learning to Highlight Audio by Watching Movies [37.9846964966927]
  We introduce visually-guided acoustic highlighting, which aims to transform audio to deliver appropriate highlighting effects guided by the accompanying video.<n>To train our model, we also introduce a new dataset -- the muddy mix dataset, leveraging the meticulous audio and video crafting found in movies.<n>Our approach consistently outperforms several baselines in both quantitative and subjective evaluation.
  arXiv  Detail & Related papers  (2025-05-17T22:03:57Z)
• Query-centric Audio-Visual Cognition Network for Moment Retrieval, Segmentation and Step-Captioning [56.873534081386]
  A new topic HIREST is presented, including video retrieval, moment retrieval, moment segmentation, and step-captioning. We propose a query-centric audio-visual cognition network to construct a reliable multi-modal representation for three tasks. This can cognize user-preferred content and thus attain a query-centric audio-visual representation for three tasks.
  arXiv  Detail & Related papers  (2024-12-18T06:43:06Z)
• MMTrail: A Multimodal Trailer Video Dataset with Language and Music Descriptions [69.9122231800796]
  We present MMTrail, a large-scale multi-modality video-language dataset incorporating more than 20M trailer clips with visual captions. We propose a systemic captioning framework, achieving various modality annotations with more than 27.1k hours of trailer videos. Our dataset potentially paves the path for fine-grained large multimodal-language model training.
  arXiv  Detail & Related papers  (2024-07-30T16:43:24Z)
• Missingness-resilient Video-enhanced Multimodal Disfluency Detection [3.3281516035025285]
  We present a practical multimodal disfluency detection approach that leverages available video data together with audio. Our resilient design accommodates real-world scenarios where the video modality may sometimes be missing during inference. In experiments across five disfluency-detection tasks, our unified multimodal approach significantly outperforms Audio-only unimodal methods.
  arXiv  Detail & Related papers  (2024-06-11T05:47:16Z)
• Tango 2: Aligning Diffusion-based Text-to-Audio Generations through Direct Preference Optimization [70.13218512896032]
  Generation of audio from text prompts is an important aspect of such processes in the music and film industry. Our hypothesis is focusing on how these aspects of audio generation could improve audio generation performance in the presence of limited data. We synthetically create a preference dataset where each prompt has a winner audio output and some loser audio outputs for the diffusion model to learn from.
  arXiv  Detail & Related papers  (2024-04-15T17:31:22Z)
• SoundingActions: Learning How Actions Sound from Narrated Egocentric Videos [77.55518265996312]
  We propose a novel self-supervised embedding to learn how actions sound from narrated in-the-wild egocentric videos. Our multimodal contrastive-consensus coding (MC3) embedding reinforces the associations between audio, language, and vision when all modality pairs agree.
  arXiv  Detail & Related papers  (2024-04-08T05:19:28Z)
• A Video Is Worth 4096 Tokens: Verbalize Videos To Understand Them In Zero Shot [67.00455874279383]
  We propose verbalizing long videos to generate descriptions in natural language, then performing video-understanding tasks on the generated story as opposed to the original video. Our method, despite being zero-shot, achieves significantly better results than supervised baselines for video understanding. To alleviate a lack of story understanding benchmarks, we publicly release the first dataset on a crucial task in computational social science on persuasion strategy identification.
  arXiv  Detail & Related papers  (2023-05-16T19:13:11Z)
• Predicting emotion from music videos: exploring the relative contribution of visual and auditory information to affective responses [0.0]
  We present MusicVideos (MuVi), a novel dataset for affective multimedia content analysis. The data were collected by presenting music videos to participants in three conditions: music, visual, and audiovisual.
  arXiv  Detail & Related papers  (2022-02-19T07:36:43Z)
• MERLOT Reserve: Neural Script Knowledge through Vision and Language and Sound [90.1857707251566]
  We introduce MERLOT Reserve, a model that represents videos jointly over time. We replace snippets of text and audio with a MASK token; the model learns by choosing the correct masked-out snippet. Our objective learns faster than alternatives, and performs well at scale.
  arXiv  Detail & Related papers  (2022-01-07T19:00:21Z)
• The Influence of Audio on Video Memorability with an Audio Gestalt Regulated Video Memorability System [1.8506048493564673]
  We find evidence to suggest that audio can facilitate overall video recognition memorability rich in high-level (gestalt) audio features. We introduce a novel multimodal deep learning-based late-fusion system that uses audio gestalt to estimate the influence of a given video's audio on its overall short-term recognition memorability. We benchmark our audio gestalt based system on the Memento10k short-term video memorability dataset, achieving top-2 state-of-the-art results.
  arXiv  Detail & Related papers  (2021-04-23T12:53:33Z)
• Multi-modal Ensemble Models for Predicting Video Memorability [3.8367329188121824]
  This work introduces and demonstrates the efficacy and high generalizability of extracted audio embeddings as a feature for the task of predicting media memorability.
  arXiv  Detail & Related papers  (2021-02-01T21:16:52Z)
• Learning Speech Representations from Raw Audio by Joint Audiovisual Self-Supervision [63.564385139097624]
  We propose a method to learn self-supervised speech representations from the raw audio waveform. We train a raw audio encoder by combining audio-only self-supervision (by predicting informative audio attributes) with visual self-supervision (by generating talking faces from audio) Our results demonstrate the potential of multimodal self-supervision in audiovisual speech for learning good audio representations.
  arXiv  Detail & Related papers  (2020-07-08T14:07:06Z)
• Audio Summarization with Audio Features and Probability Distribution Divergence [1.0587107940165885]
  We focus on audio summarization based on audio features and the probability of distribution divergence. Our method, based on an extractive summarization approach, aims to select the most relevant segments until a time threshold is reached.
  arXiv  Detail & Related papers  (2020-01-20T13:10:01Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.