Advancing Reference-free Evaluation of Video Captions with Factual Analysis

• URL: http://arxiv.org/abs/2509.16538v1
• Date: Sat, 20 Sep 2025 05:04:41 GMT
• Title: Advancing Reference-free Evaluation of Video Captions with Factual Analysis
• Authors: Shubhashis Roy Dipta, Tz-Ying Wu, Subarna Tripathi,
• Abstract summary: We introduce VC-Inspector, a novel caption quality evaluator that is both reference-free and factually grounded.<n>Our approach demonstrates superior alignment with human judgments on the VATEX-Eval dataset, outperforming existing methods.
• Score: 11.012178413572066
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Video captions offer concise snapshots of actors, objects, and actions within a video, serving as valuable assets for applications such as question answering and event localization. However, acquiring human annotations for video captions is costly or even impractical, especially when dealing with diverse video domains. Existing models trained on supervised datasets face challenges in evaluating performance across different domains due to the reliance on reference-based evaluation protocols, which necessitate ground truth captions. This assumption is unrealistic for evaluating videos in the wild. To address these limitations, we propose a reference-free evaluation framework that does not require ground truth captions, focusing on factual grounding to ensure accurate assessment of caption quality. We introduce VC-Inspector, a novel caption quality evaluator that is both reference-free and factually grounded. Utilizing large language models, we generate pseudo captions of varying quality based on supervised data, which are subsequently used to train a multimodal model (i.e., Qwen2.5-VL) as the evaluator. Our approach demonstrates superior alignment with human judgments on the VATEX-Eval dataset, outperforming existing methods. The performance also generalizes to image caption datasets, Flickr8K-Expert and Flickr8K-CF, when viewing images as 1-frame videos. Overall, VC-Inspector offers a scalable and generalizable solution for evaluating the factual accuracy of video captions, paving the way for more effective and objective assessment methodologies in diverse video domains.

Related papers

• CAMP-VQA: Caption-Embedded Multimodal Perception for No-Reference Quality Assessment of Compressed Video [9.172799792564009]
  We propose CAMP-VQA, a novel NR-VQA framework that exploits the semantic understanding capabilities of large models.<n>Our approach introduces a quality-aware video metadata mechanism that integrates key fragments extracted from inter-frame variations.<n>Our model consistently outperforms existing NR-VQA methods, achieving improved accuracy without the need for costly manual fine-grained annotations.
  arXiv  Detail & Related papers  (2025-11-10T16:37:47Z)
• No-Reference Rendered Video Quality Assessment: Dataset and Metrics [13.445406215772449]
  We present a large rendering-oriented video dataset with subjective quality annotations.<n>We calibrate our NR-VQA metric to assess rendered video quality by looking at both image quality and temporal stability.
  arXiv  Detail & Related papers  (2025-10-15T09:36:52Z)
• VideoScore2: Think before You Score in Generative Video Evaluation [69.43069741467603]
  VideoScore2 is a multi-dimensional, interpretable, and human-aligned framework that explicitly evaluates visual quality, text-to-video alignment, and physical/common-sense consistency.<n>Our model is trained on a large-scale dataset VideoFeedback2 containing 27,168 human-annotated videos.
  arXiv  Detail & Related papers  (2025-09-26T18:09:03Z)
• AVC-DPO: Aligned Video Captioning via Direct Preference Optimization [50.08618093204503]
  Video multimodal large language models (video MLLMs) have achieved substantial progress in video captioning tasks.<n>We propose Aligned Video Captioning via Direct Preference Optimization (AVC-DPO), a post-training framework designed to enhance captioning capabilities in video MLLMs through preference alignment.<n>We have achieved exceptional performance in the LOVE@PRCV'25 Workshop Track 1A: Video Detailed Captioning Challenge, achieving first place on the Video Detailed Captioning benchmark.
  arXiv  Detail & Related papers  (2025-07-02T08:51:45Z)
• VCapsBench: A Large-scale Fine-grained Benchmark for Video Caption Quality Evaluation [23.701884816475403]
  Video captions play a crucial role in text-to-video generation tasks.<n>Existing benchmarks inadequately address fine-grained evaluation.<n>We introduce the Fine-grained Video Caption Evaluation Benchmark (VCapsBench)
  arXiv  Detail & Related papers  (2025-05-29T14:34:25Z)
• Painting with Words: Elevating Detailed Image Captioning with Benchmark and Alignment Learning [56.31096024472269]
  We introduce DeCapBench along with a novel metric, DCScore, specifically designed for detailed captioning tasks.<n>DCScore evaluates hallucinations and fine-grained comprehensiveness by deconstructing responses into the smallest self-sufficient units.<n>DeCapBench exhibits a high correlation with VLM arena results on descriptive tasks, surpassing existing benchmarks for vision-language models.
  arXiv  Detail & Related papers  (2025-03-10T22:53:56Z)
• CAPability: A Comprehensive Visual Caption Benchmark for Evaluating Both Correctness and Thoroughness [30.44039177018447]
  CAPability is a comprehensive benchmark for evaluating visual captioning across 12 dimensions spanning six critical views.<n>We curate nearly 11K human-annotated images and videos with visual element annotations to evaluate the generated captions.
  arXiv  Detail & Related papers  (2025-02-19T07:55:51Z)
• CLIPVQA:Video Quality Assessment via CLIP [56.94085651315878]
  We propose an efficient CLIP-based Transformer method for the VQA problem ( CLIPVQA) The proposed CLIPVQA achieves new state-of-the-art VQA performance and up to 37% better generalizability than existing benchmark VQA methods.
  arXiv  Detail & Related papers  (2024-07-06T02:32:28Z)
• Live Video Captioning [0.6291443816903801]
  We introduce a groundbreaking paradigm: Live Video Captioning (LVC), where captions must be generated for video streams in an online manner.<n>We formally define the novel problem of LVC and propose innovative evaluation metrics specifically designed for this online scenario.<n>We present a new model that combines deformable transformers with temporal filtering, enabling effective captioning over video streams.
  arXiv  Detail & Related papers  (2024-06-20T11:25:16Z)
• Learning text-to-video retrieval from image captioning [59.81537951811595]
  We describe a protocol to study text-to-video retrieval training with unlabeled videos. We assume (i) no access to labels for any videos, and (ii) access to labeled images in the form of text. We show that automatically labeling video frames with image captioning allows text-to-video retrieval training.
  arXiv  Detail & Related papers  (2024-04-26T15:56:08Z)
• VLM-Eval: A General Evaluation on Video Large Language Models [16.92780012093112]
  We introduce a unified evaluation that encompasses multiple video tasks, including captioning, question and answering, retrieval, and action recognition. We propose a simple baseline: Video-LLaVA, which uses a single linear projection and outperforms existing video LLMs. We evaluate video LLMs beyond academic datasets, which show encouraging recognition and reasoning capabilities in driving scenarios with only hundreds of video-instruction pairs for fine-tuning.
  arXiv  Detail & Related papers  (2023-11-20T16:02:10Z)
• Helping Hands: An Object-Aware Ego-Centric Video Recognition Model [60.350851196619296]
  We introduce an object-aware decoder for improving the performance of ego-centric representations on ego-centric videos. We show that the model can act as a drop-in replacement for an ego-awareness video model to improve performance through visual-text grounding.
  arXiv  Detail & Related papers  (2023-08-15T17:58:11Z)
• Perceptual Quality Assessment of Virtual Reality Videos in the Wild [53.94620993606658]
  Existing panoramic video databases only consider synthetic distortions, assume fixed viewing conditions, and are limited in size. We construct the VR Video Quality in the Wild (VRVQW) database, containing $502$ user-generated videos with diverse content and distortion characteristics. We conduct a formal psychophysical experiment to record the scanpaths and perceived quality scores from $139$ participants under two different viewing conditions.
  arXiv  Detail & Related papers  (2022-06-13T02:22:57Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.