Putting Captions to the Test: Evaluating Video Caption Quality through Multiple-Choice Question Answering

• URL: http://arxiv.org/abs/2609.09973v1
• Date: Wed, 09 Sep 2026 10:00:43 GMT
• Title: Putting Captions to the Test: Evaluating Video Caption Quality through Multiple-Choice Question Answering
• Abstract summary: CapQuiz is a novel reference-free benchmark that assesses captions based on their utility in answering human-verified, fine-grained, multiple-choice questions.<n>We show that CapQuiz correlates significantly better with human judgments than existing metrics and offers interpretable insights into model performance.
• Score: 18.82625951944924
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: Evaluating video captioning remains a critical challenge for Visual Large Language Models (VLLMs). Existing metrics primarily rely on matching generated text against ground-truth references. This paradigm suffers from the ``one-to-many'' nature of video description, where high-quality captions are often penalized for lexical mismatches or valid shifts in visual focus. Furthermore, such assessments are typically one-dimensional, failing to provide a fine-grained analysis of caption quality. To address this, we redefine caption quality through the lens of information fidelity: A caption must maximize the coverage of salient visual information while ensuring strict factuality. We introduce CapQuiz, a novel reference-free benchmark that assesses captions based on their utility in answering human-verified, fine-grained, multiple-choice questions derived from the video. CapQuiz features a hierarchical taxonomy of 10 question types (spanning Descriptive and Inferential categories) across 24 diverse video domains. Extensive experiments demonstrate that CapQuiz correlates significantly better with human judgments than existing metrics and offers interpretable insights into model performance.

Related papers

• CAPEval: A Decoupled Caption Evaluation across Understanding and Generation [29.355541032573285]
  We design a decoupled caption evaluation benchmark, CAPEval, with human-written ground-truth captions and human-verified atomic checklist items.<n>We find a consistent task-dependent dissociation: Coverage serves as the stronger correlate for understanding performance, whereas Precision acts as the dominant predictor for generation performance.
  arXiv  Detail & Related papers  (2026-08-03T17:57:03Z)
• CapRiCorn-1K: A Comprehensive Benchmark for Video Captioning and Subject Referential Consistency Across Temporal Scales [41.187897807081875]
  CapRiCorn-1K is a benchmark designed to evaluate both video captioning quality and subject referential consistency across long temporal horizons and diverse video domains.<n>Our benchmark supports both audiovisual and visual-only settings.<n>Our evaluation metrics exhibit strong correlations with the performance of downstream understanding and generation tasks conditioned on the generated captions.
  arXiv  Detail & Related papers  (2026-06-20T08:37:32Z)
• CapRL++: Unified Reinforcement Learning with Verifiable Rewards for Dense Image and Video Captioning [86.9494763617273]
  We introduce Captioning Reinforcement Learning++ (CapRL++), a reference-free training framework that redefines caption quality through its utility.<n>We show that CapRL++ improves dense caption quality and strengthens caption-based pretraining across tasks such as spatial and temporal understanding.
  arXiv  Detail & Related papers  (2026-06-08T12:09:20Z)
• Advancing Reference-free Evaluation of Video Captions with Factual Analysis [11.012178413572066]
  We introduce VC-Inspector, a novel caption quality evaluator that is both reference-free and factually grounded.<n>Our approach demonstrates superior alignment with human judgments on the VATEX-Eval dataset, outperforming existing methods.
  arXiv  Detail & Related papers  (2025-09-20T05:04:41Z)
• ScaleCap: Inference-Time Scalable Image Captioning via Dual-Modality Debiasing [128.8346376825612]
  Key challenges of high-quality image captioning lie in the inherent biases of LVLMs.<n>We propose a scalable debiased captioning strategy, which continuously enriches and calibrates the caption with increased inference budget.<n>Annotating 450K images with ScaleCap and using them for LVLM pretraining leads to consistent performance gains across 11 widely used benchmarks.
  arXiv  Detail & Related papers  (2025-06-24T17:59:55Z)
• The Devil is in the Distributions: Explicit Modeling of Scene Content is Key in Zero-Shot Video Captioning [89.64905703368255]
  We propose a novel progressive multi-granularity textual prompting strategy for zero-shot video captioning.<n>Our approach constructs three distinct memory banks, encompassing noun phrases, scene graphs of noun phrases, and entire sentences.
  arXiv  Detail & Related papers  (2025-03-31T03:00:19Z)
• CAPability: A Comprehensive Visual Caption Benchmark for Evaluating Both Correctness and Thoroughness [30.44039177018447]
  CAPability is a comprehensive benchmark for evaluating visual captioning across 12 dimensions spanning six critical views.<n>We curate nearly 11K human-annotated images and videos with visual element annotations to evaluate the generated captions.
  arXiv  Detail & Related papers  (2025-02-19T07:55:51Z)
• AuroraCap: Efficient, Performant Video Detailed Captioning and a New Benchmark [89.73538448786405]
  We propose AuroraCap, a video captioner based on a large multimodal model.<n>We implement the token merging strategy, reducing the number of input visual tokens.<n>AuroraCap shows superior performance on various video and image captioning benchmarks.
  arXiv  Detail & Related papers  (2024-10-04T00:13:54Z)
• Transparent Human Evaluation for Image Captioning [70.03979566548823]
  We develop a rubric-based human evaluation protocol for image captioning models. We show that human-generated captions show substantially higher quality than machine-generated ones. We hope that this work will promote a more transparent evaluation protocol for image captioning.
  arXiv  Detail & Related papers  (2021-11-17T07:09:59Z)
• Intrinsic Image Captioning Evaluation [53.51379676690971]
  We propose a learning based metrics for image captioning, which we call Intrinsic Image Captioning Evaluation(I2CE) Experiment results show that our proposed method can keep robust performance and give more flexible scores to candidate captions when encountered with semantic similar expression or less aligned semantics.
  arXiv  Detail & Related papers  (2020-12-14T08:36:05Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.