ViCo: Engaging Video Comment Generation with Human Preference Rewards

• URL: http://arxiv.org/abs/2308.11171v1
• Date: Tue, 22 Aug 2023 04:01:01 GMT
• Title: ViCo: Engaging Video Comment Generation with Human Preference Rewards
• Authors: Yuchong Sun, Bei Liu, Xu Chen, Ruihua Song, Jianlong Fu
• Abstract summary: We propose ViCo with three novel designs to tackle the challenges for generating engaging Video Comments. To quantify the engagement of comments, we utilize the number of "likes" each comment receives as a proxy of human preference. To automatically evaluate the engagement of comments, we train a reward model to align its judgment to the above proxy.
• Score: 68.50351391812723
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Engaging video comments play an important role in video social media, as they are the carrier of feelings, thoughts, or humor of the audience. Preliminary works have made initial exploration for video comment generation by adopting caption-style encoder-decoder models. However, comment generation presents some unique challenges distinct from caption generation, which makes these methods somewhat less effective at generating engaging comments. In contrast to the objective and descriptive nature of captions, comments tend to be inherently subjective, making it hard to quantify and evaluate the engagement of comments. Furthermore, the scarcity of truly engaging comments brings difficulty to collecting enough high-quality training examples. In this paper, we propose ViCo with three novel designs to tackle the above challenges for generating engaging Video Comments. Firstly, to quantify the engagement of comments, we utilize the number of "likes" each comment receives as a proxy of human preference after an appropriate debiasing procedure. Secondly, to automatically evaluate the engagement of comments, we train a reward model to align its judgment to the above proxy. Our user studies indicate that this reward model effectively aligns with human judgments. Lastly, to alleviate the scarcity of high-quality comments, an initial generator is trained on readily available but noisy data to generate comments. Then the reward model is employed to offer feedback on the generated comments, thus optimizing the initial generator. To facilitate the research of video commenting, we collect a large video comment-dataset (ViCo-20k) with rich metadata from a popular video website. Experiments on ViCo-20k show that the comments generated by our ViCo model exhibit the best performance in terms of both quantitative and qualitative results, particularly when engagement is considered.

Related papers

• HOTVCOM: Generating Buzzworthy Comments for Videos [49.39846630199698]
  This study introduces textscHotVCom, the largest Chinese video hot-comment dataset, comprising 94k diverse videos and 137 million comments. We also present the textttComHeat framework, which synergistically integrates visual, auditory, and textual data to generate influential hot-comments on the Chinese video dataset.
  arXiv  Detail & Related papers  (2024-09-23T16:45:13Z)
• Beyond Thumbs Up/Down: Untangling Challenges of Fine-Grained Feedback for Text-to-Image Generation [67.88747330066049]
  Fine-grained feedback captures nuanced distinctions in image quality and prompt-alignment. We show that demonstrating its superiority to coarse-grained feedback is not automatic. We identify key challenges in eliciting and utilizing fine-grained feedback.
  arXiv  Detail & Related papers  (2024-06-24T17:19:34Z)
• Knowledge Enhanced Model for Live Video Comment Generation [40.762720398152766]
  We propose a knowledge enhanced generation model inspired by the divergent and informative nature of live video comments. Our model adopts a pre-training encoder-decoder framework and incorporates external knowledge. The MovieLC dataset and our code will be released.
  arXiv  Detail & Related papers  (2023-04-28T07:03:50Z)
• Classifying YouTube Comments Based on Sentiment and Type of Sentence [0.0]
  We address the challenge of text extraction and classification from YouTube comments using well-known statistical measures and machine learning models. The results show that our approach that incorporates conventional methods performs well on the classification task, validating its potential in assisting content creators increase viewer engagement on their channel.
  arXiv  Detail & Related papers  (2021-10-31T18:08:10Z)
• Can We Automate Scientific Reviewing? [89.50052670307434]
  We discuss the possibility of using state-of-the-art natural language processing (NLP) models to generate first-pass peer reviews for scientific papers. We collect a dataset of papers in the machine learning domain, annotate them with different aspects of content covered in each review, and train targeted summarization models that take in papers to generate reviews. Comprehensive experimental results show that system-generated reviews tend to touch upon more aspects of the paper than human-written reviews.
  arXiv  Detail & Related papers  (2021-01-30T07:16:53Z)
• A Multi-modal Approach to Fine-grained Opinion Mining on Video Reviews [47.726065950436585]
  We propose a multi-modal approach for mining fine-grained opinions from video reviews. Our approach works at the sentence level without the need for time annotations.
  arXiv  Detail & Related papers  (2020-05-27T13:46:11Z)
• What comprises a good talking-head video generation?: A Survey and Benchmark [40.26689818789428]
  We present a benchmark for evaluating talking-head video generation with standardized dataset pre-processing strategies. We propose new metrics or select the most appropriate ones to evaluate results in what we consider as desired properties for a good talking-head video.
  arXiv  Detail & Related papers  (2020-05-07T01:58:05Z)
• Multimodal Matching Transformer for Live Commenting [97.06576354830736]
  Automatic live commenting aims to provide real-time comments on videos for viewers. Recent work on this task adopts encoder-decoder models to generate comments. We propose a multimodal matching transformer to capture the relationships among comments, vision, and audio.
  arXiv  Detail & Related papers  (2020-02-07T07:19:15Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.