MM-AU:Towards Multimodal Understanding of Advertisement Videos

• URL: http://arxiv.org/abs/2308.14052v1
• Date: Sun, 27 Aug 2023 09:11:46 GMT
• Title: MM-AU:Towards Multimodal Understanding of Advertisement Videos
• Authors: Digbalay Bose, Rajat Hebbar, Tiantian Feng, Krishna Somandepalli, Anfeng Xu, Shrikanth Narayanan
• Abstract summary: We introduce a multimodal multilingual benchmark called MM-AU composed of over 8.4K videos (147 hours) curated from multiple web sources. We explore multiple zero-shot reasoning baselines through the application of large language models on the ads transcripts.
• Score: 38.117243603403175
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Advertisement videos (ads) play an integral part in the domain of Internet e-commerce as they amplify the reach of particular products to a broad audience or can serve as a medium to raise awareness about specific issues through concise narrative structures. The narrative structures of advertisements involve several elements like reasoning about the broad content (topic and the underlying message) and examining fine-grained details involving the transition of perceived tone due to the specific sequence of events and interaction among characters. In this work, to facilitate the understanding of advertisements along the three important dimensions of topic categorization, perceived tone transition, and social message detection, we introduce a multimodal multilingual benchmark called MM-AU composed of over 8.4K videos (147 hours) curated from multiple web sources. We explore multiple zero-shot reasoning baselines through the application of large language models on the ads transcripts. Further, we demonstrate that leveraging signals from multiple modalities, including audio, video, and text, in multimodal transformer-based supervised models leads to improved performance compared to unimodal approaches.

Related papers

• Prompting Video-Language Foundation Models with Domain-specific Fine-grained Heuristics for Video Question Answering [71.62961521518731]
  HeurVidQA is a framework that leverages domain-specific entity-actions to refine pre-trained video-language foundation models. Our approach treats these models as implicit knowledge engines, employing domain-specific entity-action prompters to direct the model's focus toward precise cues that enhance reasoning.
  arXiv  Detail & Related papers  (2024-10-12T06:22:23Z)
• MMTrail: A Multimodal Trailer Video Dataset with Language and Music Descriptions [69.9122231800796]
  We present MMTrail, a large-scale multi-modality video-language dataset incorporating more than 20M trailer clips with visual captions. We propose a systemic captioning framework, achieving various modality annotations with more than 27.1k hours of trailer videos. Our dataset potentially paves the path for fine-grained large multimodal-language model training.
  arXiv  Detail & Related papers  (2024-07-30T16:43:24Z)
• DM$^2$S$^2$: Deep Multi-Modal Sequence Sets with Hierarchical Modality Attention [8.382710169577447]
  Methods for extracting important information from multimodal data rely on a mid-fusion architecture. We propose a new concept that considers multimodal inputs as a set of sequences, namely, deep multimodal sequence sets. Our concept exhibits performance that is comparable to or better than the previous set-level models.
  arXiv  Detail & Related papers  (2022-09-07T13:25:09Z)
• Modeling Motion with Multi-Modal Features for Text-Based Video Segmentation [56.41614987789537]
  Text-based video segmentation aims to segment the target object in a video based on a describing sentence. We propose a method to fuse and align appearance, motion, and linguistic features to achieve accurate segmentation.
  arXiv  Detail & Related papers  (2022-04-06T02:42:33Z)
• Multi-modal Representation Learning for Video Advertisement Content Structuring [10.45050088240847]
  Video advertisement content structuring aims to segment a given video advertisement and label each segment on various dimensions. Video advertisements contain sufficient and useful multi-modal content like caption and speech. We propose a multi-modal encoder to learn multi-modal representation from video advertisements by interacting between video-audio and text.
  arXiv  Detail & Related papers  (2021-09-04T09:08:29Z)
• MONAH: Multi-Modal Narratives for Humans to analyze conversations [9.178828168133206]
  We introduce a system that automatically expands the verbatim transcripts of video-recorded conversations using multimodal data streams. This system uses a set of preprocessing rules to weave multimodal annotations into the verbatim transcripts and promote interpretability.
  arXiv  Detail & Related papers  (2021-01-18T21:55:58Z)
• Cross-Media Keyphrase Prediction: A Unified Framework with Multi-Modality Multi-Head Attention and Image Wordings [63.79979145520512]
  We explore the joint effects of texts and images in predicting the keyphrases for a multimedia post. We propose a novel Multi-Modality Multi-Head Attention (M3H-Att) to capture the intricate cross-media interactions. Our model significantly outperforms the previous state of the art based on traditional attention networks.
  arXiv  Detail & Related papers  (2020-11-03T08:44:18Z)
• VMSMO: Learning to Generate Multimodal Summary for Video-based News Articles [63.32111010686954]
  We propose the task of Video-based Multimodal Summarization with Multimodal Output (VMSMO) The main challenge in this task is to jointly model the temporal dependency of video with semantic meaning of article. We propose a Dual-Interaction-based Multimodal Summarizer (DIMS), consisting of a dual interaction module and multimodal generator.
  arXiv  Detail & Related papers  (2020-10-12T02:19:16Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.