ContextIQ: A Multimodal Expert-Based Video Retrieval System for Contextual Advertising

• URL: http://arxiv.org/abs/2410.22233v2
• Date: Wed, 06 Nov 2024 19:52:58 GMT
• Title: ContextIQ: A Multimodal Expert-Based Video Retrieval System for Contextual Advertising
• Authors: Ashutosh Chaubey, Anoubhav Agarwaal, Sartaki Sinha Roy, Aayush Agrawal, Susmita Ghose,
• Abstract summary: Contextual advertising serves ads that are aligned to the content that the user is viewing. Current text-to-video retrieval models based on joint multimodal training demand large datasets and computational resources. We introduce ContextIQ, a multimodal expert-based video retrieval system designed specifically for contextual advertising.
• Score: 2.330164376631038
• License:
• Abstract: Contextual advertising serves ads that are aligned to the content that the user is viewing. The rapid growth of video content on social platforms and streaming services, along with privacy concerns, has increased the need for contextual advertising. Placing the right ad in the right context creates a seamless and pleasant ad viewing experience, resulting in higher audience engagement and, ultimately, better ad monetization. From a technology standpoint, effective contextual advertising requires a video retrieval system capable of understanding complex video content at a very granular level. Current text-to-video retrieval models based on joint multimodal training demand large datasets and computational resources, limiting their practicality and lacking the key functionalities required for ad ecosystem integration. We introduce ContextIQ, a multimodal expert-based video retrieval system designed specifically for contextual advertising. ContextIQ utilizes modality-specific experts-video, audio, transcript (captions), and metadata such as objects, actions, emotion, etc.-to create semantically rich video representations. We show that our system, without joint training, achieves better or comparable results to state-of-the-art models and commercial solutions on multiple text-to-video retrieval benchmarks. Our ablation studies highlight the benefits of leveraging multiple modalities for enhanced video retrieval accuracy instead of using a vision-language model alone. Furthermore, we show how video retrieval systems such as ContextIQ can be used for contextual advertising in an ad ecosystem while also addressing concerns related to brand safety and filtering inappropriate content.

Related papers

• Prompting Video-Language Foundation Models with Domain-specific Fine-grained Heuristics for Video Question Answering [71.62961521518731]
  HeurVidQA is a framework that leverages domain-specific entity-actions to refine pre-trained video-language foundation models. Our approach treats these models as implicit knowledge engines, employing domain-specific entity-action prompters to direct the model's focus toward precise cues that enhance reasoning.
  arXiv  Detail & Related papers  (2024-10-12T06:22:23Z)
• GQE: Generalized Query Expansion for Enhanced Text-Video Retrieval [56.610806615527885]
  This paper introduces a novel data-centric approach, Generalized Query Expansion (GQE), to address the inherent information imbalance between text and video. By adaptively segmenting videos into short clips and employing zero-shot captioning, GQE enriches the training dataset with comprehensive scene descriptions. GQE achieves state-of-the-art performance on several benchmarks, including MSR-VTT, MSVD, LSMDC, and VATEX.
  arXiv  Detail & Related papers  (2024-08-14T01:24:09Z)
• How Good is my Video LMM? Complex Video Reasoning and Robustness Evaluation Suite for Video-LMMs [98.37571997794072]
  We present the Complex Video Reasoning and Robustness Evaluation Suite (CVRR-ES) CVRR-ES comprehensively assesses the performance of Video-LMMs across 11 diverse real-world video dimensions. Our findings provide valuable insights for building the next generation of human-centric AI systems.
  arXiv  Detail & Related papers  (2024-05-06T17:59:45Z)
• Improving Video Corpus Moment Retrieval with Partial Relevance Enhancement [72.7576395034068]
  Video Corpus Moment Retrieval (VCMR) is a new video retrieval task aimed at retrieving a relevant moment from a large corpus of untrimmed videos using a text query. We argue that effectively capturing the partial relevance between the query and video is essential for the VCMR task. For video retrieval, we introduce a multi-modal collaborative video retriever, generating different query representations for the two modalities. For moment localization, we propose the focus-then-fuse moment localizer, utilizing modality-specific gates to capture essential content.
  arXiv  Detail & Related papers  (2024-02-21T07:16:06Z)
• MM-AU:Towards Multimodal Understanding of Advertisement Videos [38.117243603403175]
  We introduce a multimodal multilingual benchmark called MM-AU composed of over 8.4K videos (147 hours) curated from multiple web sources. We explore multiple zero-shot reasoning baselines through the application of large language models on the ads transcripts.
  arXiv  Detail & Related papers  (2023-08-27T09:11:46Z)
• A Video Is Worth 4096 Tokens: Verbalize Videos To Understand Them In Zero Shot [67.00455874279383]
  We propose verbalizing long videos to generate descriptions in natural language, then performing video-understanding tasks on the generated story as opposed to the original video. Our method, despite being zero-shot, achieves significantly better results than supervised baselines for video understanding. To alleviate a lack of story understanding benchmarks, we publicly release the first dataset on a crucial task in computational social science on persuasion strategy identification.
  arXiv  Detail & Related papers  (2023-05-16T19:13:11Z)
• Multi-modal Representation Learning for Video Advertisement Content Structuring [10.45050088240847]
  Video advertisement content structuring aims to segment a given video advertisement and label each segment on various dimensions. Video advertisements contain sufficient and useful multi-modal content like caption and speech. We propose a multi-modal encoder to learn multi-modal representation from video advertisements by interacting between video-audio and text.
  arXiv  Detail & Related papers  (2021-09-04T09:08:29Z)
• A Multimodal Framework for Video Ads Understanding [64.70769354696019]
  We develop a multimodal system to improve the ability of structured analysis of advertising video content. Our solution achieved a score of 0.2470 measured in consideration of localization and prediction accuracy, ranking fourth in the 2021 TAAC final leaderboard.
  arXiv  Detail & Related papers  (2021-08-29T16:06:00Z)
• Fashion Focus: Multi-modal Retrieval System for Video Commodity Localization in E-commerce [18.651201334846352]
  We present an innovative demonstration of multi-modal retrieval system called "Fashion Focus" It enables to exactly localize the product images in the online video as the focuses. Our system employs two procedures for analysis, including video content structuring and multi-modal retrieval, to automatically achieve accurate video-to-shop matching.
  arXiv  Detail & Related papers  (2021-02-09T09:45:04Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.