CAPEval: A Decoupled Caption Evaluation across Understanding and Generation

• URL: http://arxiv.org/abs/2608.02589v1
• Date: Mon, 03 Aug 2026 17:57:03 GMT
• Title: CAPEval: A Decoupled Caption Evaluation across Understanding and Generation
• Abstract summary: We design a decoupled caption evaluation benchmark, CAPEval, with human-written ground-truth captions and human-verified atomic checklist items.<n>We find a consistent task-dependent dissociation: Coverage serves as the stronger correlate for understanding performance, whereas Precision acts as the dominant predictor for generation performance.
• Score: 29.355541032573285
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Captions serve as a primary supervision signal for both multimodal understanding and text-to-image generation. However, previous evaluations treat the caption quality as a single scalar objective, which conflates two distinct properties: (1) how much visual information a caption covers and (2) how reliably the image supports its stated claims. To this end, we design a decoupled caption evaluation benchmark, CAPEval (Coverage And Precision Evaluation), with human-written ground-truth captions and human-verified atomic checklist items. Specifically, CAPEval decomposes caption quality into Coverage and Precision. The former quantifies how thoroughly a caption covers ground-truth factual content, while the latter reflects the factual correctness rate of all claims expressed in the caption. We select 10 captioners and further conduct controlled downstream end-to-end experiments with them from four model families, where the caption source is the only variable. Empirically, we find a consistent task-dependent dissociation: Coverage serves as the stronger correlate for understanding performance, whereas Precision acts as the dominant predictor for generation performance. This decoupled evaluation paradigm not only delivers a more fine-grained diagnosis of caption quality, but also offers actionable guidance for selecting and optimizing captioners tailored to different downstream tasks.

Related papers

• Putting Captions to the Test: Evaluating Video Caption Quality through Multiple-Choice Question Answering [18.82625951944924]
  CapQuiz is a novel reference-free benchmark that assesses captions based on their utility in answering human-verified, fine-grained, multiple-choice questions.<n>We show that CapQuiz correlates significantly better with human judgments than existing metrics and offers interpretable insights into model performance.
  arXiv  Detail & Related papers  (2026-09-09T10:00:43Z)
• Controllable Image Captioning with Prompt-Conditioned Scene Rewards [13.154350707762179]
  Fine-grained Captioning Control Using Scene Rewards (FoCUS) is a controllable image captioning method that lets users steer captions toward specific semantic emphases.<n>FoCUS consistently improves controllability and fine-grained caption quality without degrading general caption performance.
  arXiv  Detail & Related papers  (2026-09-01T04:36:23Z)
• VCap: Hypergeometric Rewards for Weak-to-Strong Visual Captioning [57.588999592609646]
  We propose VCap, a Witness-Adjudicator reward that pairs the reference caption (a witness) with the visual signal (an adjudicator)<n>VCap delivers a reward signal with hypergeometric-distribution-level precision for caption quality verification.<n>In our experiments, an 8B model trained with VCap outperforms open- and closed-source SOTA models on multiple image and video captioning benchmarks.
  arXiv  Detail & Related papers  (2026-05-27T06:27:04Z)
• CCCaption: Dual-Reward Reinforcement Learning for Complete and Correct Image Captioning [23.289413412387223]
  We introduce CCCaption: a dual-reward reinforcement learning framework with a dedicated fine-tuning corpus.<n>For completeness, we use diverse LVLMs to disentangle the image into a set of visual queries, and reward captions that answer more of these queries.<n>For correctness, we penalize captions that contain hallucinations by validating the authenticity of sub-caption queries.
  arXiv  Detail & Related papers  (2026-02-25T07:34:26Z)
• ScaleCap: Inference-Time Scalable Image Captioning via Dual-Modality Debiasing [128.8346376825612]
  Key challenges of high-quality image captioning lie in the inherent biases of LVLMs.<n>We propose a scalable debiased captioning strategy, which continuously enriches and calibrates the caption with increased inference budget.<n>Annotating 450K images with ScaleCap and using them for LVLM pretraining leads to consistent performance gains across 11 widely used benchmarks.
  arXiv  Detail & Related papers  (2025-06-24T17:59:55Z)
• CAPability: A Comprehensive Visual Caption Benchmark for Evaluating Both Correctness and Thoroughness [30.44039177018447]
  CAPability is a comprehensive benchmark for evaluating visual captioning across 12 dimensions spanning six critical views.<n>We curate nearly 11K human-annotated images and videos with visual element annotations to evaluate the generated captions.
  arXiv  Detail & Related papers  (2025-02-19T07:55:51Z)
• What Makes for Good Image Captions? [50.48589893443939]
  Our framework posits that good image captions should balance three key aspects: informationally sufficient, minimally redundant, and readily comprehensible by humans.<n>We introduce the Pyramid of Captions (PoCa) method, which generates enriched captions by integrating local and global visual information.
  arXiv  Detail & Related papers  (2024-05-01T12:49:57Z)
• Simple Token-Level Confidence Improves Caption Correctness [117.33497608933169]
  Token-Level Confidence, or TLC, is a simple yet surprisingly effective method to assess caption correctness. We fine-tune a vision-language model on image captioning, input an image and proposed caption to the model, and aggregate token confidences over words or sequences to estimate image-caption consistency.
  arXiv  Detail & Related papers  (2023-05-11T17:58:17Z)
• Transparent Human Evaluation for Image Captioning [70.03979566548823]
  We develop a rubric-based human evaluation protocol for image captioning models. We show that human-generated captions show substantially higher quality than machine-generated ones. We hope that this work will promote a more transparent evaluation protocol for image captioning.
  arXiv  Detail & Related papers  (2021-11-17T07:09:59Z)
• Intrinsic Image Captioning Evaluation [53.51379676690971]
  We propose a learning based metrics for image captioning, which we call Intrinsic Image Captioning Evaluation(I2CE) Experiment results show that our proposed method can keep robust performance and give more flexible scores to candidate captions when encountered with semantic similar expression or less aligned semantics.
  arXiv  Detail & Related papers  (2020-12-14T08:36:05Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.