FACE-net: Factual Calibration and Emotion Augmentation for Retrieval-enhanced Emotional Video Captioning

• URL: http://arxiv.org/abs/2603.17455v1
• Date: Wed, 18 Mar 2026 07:53:15 GMT
• Title: FACE-net: Factual Calibration and Emotion Augmentation for Retrieval-enhanced Emotional Video Captioning
• Abstract summary: Emotional Video Captioning (EVC) is an emerging task, which aims to describe factual content with the intrinsic emotions expressed in videos.<n>We propose a retrieval-enhanced framework with FActual and Emotion augmentation (FACE-net)<n>FACE-net collaboratively mines factual-emotional semantics and provides adaptive and accurate guidance for generation.
• Score: 81.33341786837974
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Emotional Video Captioning (EVC) is an emerging task, which aims to describe factual content with the intrinsic emotions expressed in videos. Existing works perceive global emotional cues and then combine with video content to generate descriptions. However, insufficient factual and emotional cues mining and coordination during generation make their methods difficult to deal with the factual-emotional bias, which refers to the factual and emotional requirements being different in different samples on generation. To this end, we propose a retrieval-enhanced framework with FActual Calibration and Emotion augmentation (FACE-net), which through a unified architecture collaboratively mines factual-emotional semantics and provides adaptive and accurate guidance for generation, breaking through the compromising tendency of factual-emotional descriptions in all sample learning. Technically, we firstly introduces an external repository and retrieves the most relevant sentences with the video content to augment the semantic information. Subsequently, our factual calibration via uncertainty estimation module splits the retrieved information into subject-predicate-object triplets, and self-refines and cross-refines different components through video content to effectively mine the factual semantics; while our progressive visual emotion augmentation module leverages the calibrated factual semantics as experts, interacts with the video content and emotion dictionary to generate visual queries and candidate emotions, and then aggregates them to adaptively augment emotions to each factual semantics. Moreover, to alleviate the factual-emotional bias, we design a dynamic bias adjustment routing module to predict and adjust the degree of bias of a sample.

Related papers

• Towards Accurate Emotion-Attributed Video Captioning via Fine-grained Emotion-Cause Pair Extraction [47.762139272076126]
  Emotional Video Captioning (EVC) is a challenging task that aims to generate factually accurate and emotionally rich descriptions for videos.<n>Existing EVC methods leverage holistic visual features to mine global emotional cues, and then aggregate multimodal features to guide the emotional caption generation.<n>We propose a fine-grained emotion-cause pair extraction framework for emotion-attributed video captioning.
  arXiv  Detail & Related papers  (2026-06-07T10:43:41Z)
• Incorporating Scene Context and Semantic Labels for Enhanced Group-level Emotion Recognition [39.138182195807424]
  Group-level emotion recognition (GER) aims to identify holistic emotions within a scene involving multiple individuals.<n>Current existed methods underestimate the importance of visual scene contextual information in modeling individual relationships.<n>We propose a novel framework that incorporates visual scene context and label-guided semantic information to improve GER performance.
  arXiv  Detail & Related papers  (2025-09-26T01:25:39Z)
• EmoCAST: Emotional Talking Portrait via Emotive Text Description [56.42674612728354]
  EmoCAST is a diffusion-based framework for precise text-driven emotional synthesis.<n>In appearance modeling, emotional prompts are integrated through a text-guided decoupled emotive module.<n>EmoCAST achieves state-of-the-art performance in generating realistic, emotionally expressive, and audio-synchronized talking-head videos.
  arXiv  Detail & Related papers  (2025-08-28T10:02:06Z)
• Taming Transformer for Emotion-Controllable Talking Face Generation [61.835295250047196]
  We propose a novel method to tackle the emotion-controllable talking face generation task discretely.<n>Specifically, we employ two pre-training strategies to disentangle audio into independent components and quantize videos into combinations of visual tokens.<n>We conduct experiments on the MEAD dataset that controls the emotion of videos conditioned on multiple emotional audios.
  arXiv  Detail & Related papers  (2025-08-20T02:16:52Z)
• Emotion-Qwen: A Unified Framework for Emotion and Vision Understanding [26.36195886824082]
  Emotion-Qwen is a unified multimodal framework designed to simultaneously enable robust emotion understanding and preserve general reasoning capabilities.<n>We develop the Video Emotion Reasoning dataset, a large-scale bilingual resource containing over 40K video clips annotated with detailed context-aware emotional descriptions.
  arXiv  Detail & Related papers  (2025-05-10T16:15:26Z)
• Contrastive Decoupled Representation Learning and Regularization for Speech-Preserving Facial Expression Manipulation [58.189703277322224]
  Speech-preserving facial expression manipulation (SPFEM) aims to modify a talking head to display a specific reference emotion.<n>Emotion and content information existing in reference and source inputs can provide direct and accurate supervision signals for SPFEM models.<n>We propose to learn content and emotion priors as guidance augmented with contrastive learning to learn decoupled content and emotion representation.
  arXiv  Detail & Related papers  (2025-04-08T04:34:38Z)
• Enriching Multimodal Sentiment Analysis through Textual Emotional Descriptions of Visual-Audio Content [56.62027582702816]
  Multimodal Sentiment Analysis seeks to unravel human emotions by amalgamating text, audio, and visual data.<n>Yet, discerning subtle emotional nuances within audio and video expressions poses a formidable challenge.<n>We introduce DEVA, a progressive fusion framework founded on textual sentiment descriptions.
  arXiv  Detail & Related papers  (2024-12-12T11:30:41Z)
• SPECTRUM: Semantic Processing and Emotion-informed video-Captioning Through Retrieval and Understanding Modalities [0.7510165488300369]
  This paper proposes a novel Semantic Processing and Emotion-informed video-Captioning Through Retrieval and Understanding Modalities (SPECTRUM) framework. SPECTRUM discerns multimodal semantics and emotional themes using Visual Text Attribute Investigation (VTAI) and determines the orientation of descriptive captions. They exploit video-to-text retrieval capabilities and the multifaceted nature of video content to estimate the emotional probabilities of candidate captions.
  arXiv  Detail & Related papers  (2024-11-04T10:51:47Z)
• Dual-path Collaborative Generation Network for Emotional Video Captioning [33.230028098522254]
  Emotional Video Captioning is an emerging task that aims to describe factual content with the intrinsic emotions expressed in videos. Existing emotional video captioning methods perceive global visual emotional cues at first, and then combine them with the video features to guide the emotional caption generation. We propose a dual-path collaborative generation network, which dynamically perceives visual emotional cues evolutions while generating emotional captions.
  arXiv  Detail & Related papers  (2024-08-06T07:30:53Z)
• Emotion Rendering for Conversational Speech Synthesis with Heterogeneous Graph-Based Context Modeling [50.99252242917458]
  Conversational Speech Synthesis (CSS) aims to accurately express an utterance with the appropriate prosody and emotional inflection within a conversational setting. To address the issue of data scarcity, we meticulously create emotional labels in terms of category and intensity. Our model outperforms the baseline models in understanding and rendering emotions.
  arXiv  Detail & Related papers  (2023-12-19T08:47:50Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.