SUMMA: A Multimodal Large Language Model for Advertisement Summarization

• URL: http://arxiv.org/abs/2508.20582v2
• Date: Fri, 10 Oct 2025 09:22:47 GMT
• Title: SUMMA: A Multimodal Large Language Model for Advertisement Summarization
• Authors: Weitao Jia, Shuo Yin, Zhoufutu Wen, Han Wang, Zehui Dai, Kun Zhang, Zhenyu Li, Tao Zeng, Xiaohui Lv,
• Abstract summary: We propose SUMMA, a model that processes video ads into summaries highlighting the content of highest commercial value.<n> SUMMA is developed via a two-stage training strategy-multimodal supervised fine-tuning followed by reinforcement learning.<n>Online experiments show a statistically significant 1.5% increase in advertising revenue.
• Score: 15.514886325064792
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Understanding multimodal video ads is crucial for improving query-ad matching and relevance ranking on short video platforms, enhancing advertising effectiveness and user experience. However, the effective utilization of multimodal information with high commercial value still largely constrained by reliance on highly compressed video embeddings-has long been inadequate. To address this, we propose SUMMA (the abbreviation of Summarizing MultiModal Ads), a multimodal model that automatically processes video ads into summaries highlighting the content of highest commercial value, thus improving their comprehension and ranking in Douyin search-advertising systems. SUMMA is developed via a two-stage training strategy-multimodal supervised fine-tuning followed by reinforcement learning with a mixed reward mechanism-on domain-specific data containing video frames and ASR/OCR transcripts, generating commercially valuable and explainable summaries. We integrate SUMMA-generated summaries into our production pipeline, directly enhancing the candidate retrieval and relevance ranking stages in real search-advertising systems. Both offline and online experiments show substantial improvements over baselines, with online results indicating a statistically significant 1.5% increase in advertising revenue. Our work establishes a novel paradigm for condensing multimodal information into representative texts, effectively aligning visual ad content with user query intent in retrieval and recommendation scenarios.

Related papers

• TeamCMU at Touché: Adversarial Co-Evolution for Advertisement Integration and Detection in Conversational Search [1.187456026346823]
  integration of advertisements into generated responses presents both commercial opportunities and challenges for user experience.<n>We propose a modular pipeline for advertisement management in RAG-based conversational systems, consisting of an ad-rewriter for seamless ad integration and a robust ad-classifier for detection.
  arXiv  Detail & Related papers  (2025-07-01T07:24:29Z)
• Learning Item Representations Directly from Multimodal Features for Effective Recommendation [51.49251689107541]
  multimodal recommender systems predominantly leverage Bayesian Personalized Ranking (BPR) optimization to learn item representations.<n>We propose a novel model (i.e., LIRDRec) that learns item representations directly from multimodal features to augment recommendation performance.
  arXiv  Detail & Related papers  (2025-05-08T05:42:22Z)
• HCMRM: A High-Consistency Multimodal Relevance Model for Search Ads [10.61722566941537]
  This paper focuses on improving query-to-video relevance matching to enhance the effectiveness of ranking in ad systems.<n>It utilizes a simple yet effective method to enhance the consistency between pre-training and relevance tasks.<n>The proposed method has been deployed in the Kuaishou search advertising system for over a year, contributing to a 6.1% reduction in the proportion of irrelevant ads and a 1.4% increase in ad revenue.
  arXiv  Detail & Related papers  (2025-02-09T09:07:11Z)
• CTR-Driven Advertising Image Generation with Multimodal Large Language Models [53.40005544344148]
  We explore the use of Multimodal Large Language Models (MLLMs) for generating advertising images by optimizing for Click-Through Rate (CTR) as the primary objective.<n>To further improve the CTR of generated images, we propose a novel reward model to fine-tune pre-trained MLLMs through Reinforcement Learning (RL)<n>Our method achieves state-of-the-art performance in both online and offline metrics.
  arXiv  Detail & Related papers  (2025-02-05T09:06:02Z)
• ContextIQ: A Multimodal Expert-Based Video Retrieval System for Contextual Advertising [2.330164376631038]
  Contextual advertising serves ads that are aligned to the content that the user is viewing.<n>Current text-to-video retrieval models based on joint multimodal training demand large datasets and computational resources.<n>We introduce ContextIQ, a multimodal expert-based video retrieval system designed specifically for contextual advertising.
  arXiv  Detail & Related papers  (2024-10-29T17:01:05Z)
• DialCLIP: Empowering CLIP as Multi-Modal Dialog Retriever [83.33209603041013]
  We propose a parameter-efficient prompt-tuning method named DialCLIP for multi-modal dialog retrieval. Our approach introduces a multi-modal context generator to learn context features which are distilled into prompts within the pre-trained vision-language model CLIP. To facilitate various types of retrieval, we also design multiple experts to learn mappings from CLIP outputs to multi-modal representation space.
  arXiv  Detail & Related papers  (2024-01-02T07:40:12Z)
• MM-AU:Towards Multimodal Understanding of Advertisement Videos [38.117243603403175]
  We introduce a multimodal multilingual benchmark called MM-AU composed of over 8.4K videos (147 hours) curated from multiple web sources. We explore multiple zero-shot reasoning baselines through the application of large language models on the ads transcripts.
  arXiv  Detail & Related papers  (2023-08-27T09:11:46Z)
• MMAPS: End-to-End Multi-Grained Multi-Modal Attribute-Aware Product Summarization [93.5217515566437]
  Multi-modal Product Summarization (MPS) aims to increase customers' desire to purchase by highlighting product characteristics. Existing MPS methods can produce promising results, but they still lack end-to-end product summarization. We propose an end-to-end multi-modal attribute-aware product summarization method (MMAPS) for generating high-quality product summaries in e-commerce.
  arXiv  Detail & Related papers  (2023-08-22T11:00:09Z)
• MHMS: Multimodal Hierarchical Multimedia Summarization [80.18786847090522]
  We propose a multimodal hierarchical multimedia summarization (MHMS) framework by interacting visual and language domains. Our method contains video and textual segmentation and summarization module, respectively. It formulates a cross-domain alignment objective with optimal transport distance to generate the representative and textual summary.
  arXiv  Detail & Related papers  (2022-04-07T21:00:40Z)
• Multi-modal Representation Learning for Video Advertisement Content Structuring [10.45050088240847]
  Video advertisement content structuring aims to segment a given video advertisement and label each segment on various dimensions. Video advertisements contain sufficient and useful multi-modal content like caption and speech. We propose a multi-modal encoder to learn multi-modal representation from video advertisements by interacting between video-audio and text.
  arXiv  Detail & Related papers  (2021-09-04T09:08:29Z)
• A Multimodal Framework for Video Ads Understanding [64.70769354696019]
  We develop a multimodal system to improve the ability of structured analysis of advertising video content. Our solution achieved a score of 0.2470 measured in consideration of localization and prediction accuracy, ranking fourth in the 2021 TAAC final leaderboard.
  arXiv  Detail & Related papers  (2021-08-29T16:06:00Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.