Video2Reaction: Training Foundation Video Models to Predict Audience Reaction

• URL: http://arxiv.org/abs/2609.01816v1
• Date: Tue, 01 Sep 2026 19:44:23 GMT
• Title: Video2Reaction: Training Foundation Video Models to Predict Audience Reaction
• Abstract summary: Video2Reaction is a dataset that maps short movie segments to the induced emotional reactions of viewers in the wild.<n>We benchmark two vision-language models (VLMs) finetuned with LoRA.<n>We show that VLMs learn effectively from Video2Reaction and outperform specialized baselines on dominant reaction prediction.
• Score: 12.752010716495205
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: We introduce Video2Reaction, a multimodal dataset that maps short movie segments to the induced emotional reactions of viewers in the wild, as expressed through social media comments. Video2Reaction captures the natural diversity of emotional responses by aggregating reactions from online comments at scale, modeling labels as distributions over categorical emotions to better reflect the subjective and ambiguous nature of emotional perception. We benchmark two vision-language models (VLMs) finetuned with LoRA, showing that VLMs learn effectively from Video2Reaction and outperform specialized baselines on dominant reaction prediction. We further demonstrate that VLMs pre-finetuned on Video2Reaction transfer effectively to VCE, another induced emotion dataset with a different taxonomy and video domain. Notably, LLaVA-NeXT-Video-7B pre-finetuned on Video2Reaction and adapted on only 1% of VCE training data achieves a top-3 accuracy of 0.682, on par with the best reported VCE performance trained on the full dataset. The dataset is available at https://huggingface.co/datasets/infofusionlab/Video2Reaction

Related papers

• Benchmarking Dynamic Affective Reasoning: A Viewer-Centric Video Emotion Dataset [51.92925649424503]
  We introduce Dynamic Affective Reasoning, the first large-scale benchmark for viewer-centric affect transitions and causal reasoning over video events.<n>Dar contains 15,087 videos and 36,908 event-aligned affective segments annotated with 27 emotion categories.<n>Dar-R1 sets a new state-of-the-art for dynamic affective reasoning, in terms of both emotional localization and affective reasoning.
  arXiv  Detail & Related papers  (2026-07-11T09:58:29Z)
• Video2Reaction: Mapping Video to Audience Reaction Distribution in the Wild [12.752010716495205]
  $textbfVideo2Reaction$ is a dataset that maps short movie segments to a distribution of $textitinduced emotions$ of viewers in the wild.<n>$textbfVideo2Reaction$ spans more than 10,000 videos and serves as a reliable benchmark.<n>We develop a two-stage multi-agent pipeline using only open-source LLMs, achieving 86% correctness under blind human verification.
  arXiv  Detail & Related papers  (2026-07-08T00:17:20Z)
• FACE-net: Factual Calibration and Emotion Augmentation for Retrieval-enhanced Emotional Video Captioning [81.33341786837974]
  Emotional Video Captioning (EVC) is an emerging task, which aims to describe factual content with the intrinsic emotions expressed in videos.<n>We propose a retrieval-enhanced framework with FActual and Emotion augmentation (FACE-net)<n>FACE-net collaboratively mines factual-emotional semantics and provides adaptive and accurate guidance for generation.
  arXiv  Detail & Related papers  (2026-03-18T07:53:15Z)
• Saliency-guided Emotion Modeling: Predicting Viewer Reactions from Video Stimuli [0.0]
  We introduce a novel saliency-based approach to emotion prediction by extracting two key features: saliency area and number of salient regions.<n>Using the HD2S saliency model and OpenFace facial action unit analysis, we examine the relationship between video saliency and viewer emotions.
  arXiv  Detail & Related papers  (2025-05-25T14:52:36Z)
• SS-VAERR: Self-Supervised Apparent Emotional Reaction Recognition from Video [61.21388780334379]
  This work focuses on the apparent emotional reaction recognition from the video-only input, conducted in a self-supervised fashion. The network is first pre-trained on different self-supervised pretext tasks and later fine-tuned on the downstream target task.
  arXiv  Detail & Related papers  (2022-10-20T15:21:51Z)
• How Would The Viewer Feel? Estimating Wellbeing From Video Scenarios [73.24092762346095]
  We introduce two large-scale datasets with over 60,000 videos annotated for emotional response and subjective wellbeing. The Video Cognitive Empathy dataset contains annotations for distributions of fine-grained emotional responses, allowing models to gain a detailed understanding of affective states. The Video to Valence dataset contains annotations of relative pleasantness between videos, which enables predicting a continuous spectrum of wellbeing.
  arXiv  Detail & Related papers  (2022-10-18T17:58:25Z)
• Affect2MM: Affective Analysis of Multimedia Content Using Emotion Causality [84.69595956853908]
  We present Affect2MM, a learning method for time-series emotion prediction for multimedia content. Our goal is to automatically capture the varying emotions depicted by characters in real-life human-centric situations and behaviors.
  arXiv  Detail & Related papers  (2021-03-11T09:07:25Z)
• Enhancing Unsupervised Video Representation Learning by Decoupling the Scene and the Motion [86.56202610716504]
  Action categories are highly related with the scene where the action happens, making the model tend to degrade to a solution where only the scene information is encoded. We propose to decouple the scene and the motion (DSM) with two simple operations, so that the model attention towards the motion information is better paid.
  arXiv  Detail & Related papers  (2020-09-12T09:54:11Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.