CapProbe: Evaluating Detailed Image Captions via Full-Scene Dense Question Answering

• URL: http://arxiv.org/abs/2608.11074v1
• Date: Tue, 11 Aug 2026 15:36:10 GMT
• Title: CapProbe: Evaluating Detailed Image Captions via Full-Scene Dense Question Answering
• Abstract summary: CapProbe is a full-scene dense QA benchmark that turns detailed caption evaluation into region-aligned factual checking.<n>It comprises 346 images, 1,868 regions, and 25,650 questions, averaging 74 QA pairs per image.<n>A language judge answers from the caption alone; an Uncertain option and Effective Accuracy provide a judge-dependent proxy for distinguishing unanswered probes from incorrectly resolved ones.
• Score: 6.875316321208349
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Evaluating detailed image captions from Vision-Language Models (VLMs) requires going beyond surface-level semantic similarity. Reference-based metrics (e.g., CIDEr and SPICE) and LLM-as-scorer protocols struggle to verify dense factual claims, while existing QA-based alternatives generally offer lower probe density, narrower domain coverage, or no explicit alignment between individual questions and segmented image regions. We introduce CapProbe, a full-scene dense QA benchmark that turns detailed caption evaluation into region-aligned factual checking. Each image is decomposed into coarse semantic regions covering both foreground and background elements; for every retained region, we generate multiple-choice questions spanning 10 semantic categories, forming a dense checklist of probed visual facts. Guided by a two-tier taxonomy of 37 L1 domains and 219 L2 sub-domains, CapProbe comprises 346 images, 1,868 regions, and 25,650 questions, averaging 74 QA pairs per image. A language judge answers from the caption alone; an Uncertain option and Effective Accuracy provide a judge-dependent proxy for distinguishing unanswered probes from incorrectly resolved ones, while density-based metrics penalize verbose yet uninformative captions. The protocol is cost-effective: by converting unconstrained scalar scoring into structured MCQ reading, it reduces open-ended scoring bias while remaining judge-conditioned and yields relatively stable model rankings under a fixed reader. Experiments on 13 VLMs show large Coverage gaps across models, a clear competency-efficiency trade-off, and failure modes that sparse or overlap-based evaluation often misses. The benchmark data, annotations, and evaluation code will be released soon.

Related papers

• Putting Captions to the Test: Evaluating Video Caption Quality through Multiple-Choice Question Answering [18.82625951944924]
  CapQuiz is a novel reference-free benchmark that assesses captions based on their utility in answering human-verified, fine-grained, multiple-choice questions.<n>We show that CapQuiz correlates significantly better with human judgments than existing metrics and offers interpretable insights into model performance.
  arXiv  Detail & Related papers  (2026-09-09T10:00:43Z)
• CAPEval: A Decoupled Caption Evaluation across Understanding and Generation [29.355541032573285]
  We design a decoupled caption evaluation benchmark, CAPEval, with human-written ground-truth captions and human-verified atomic checklist items.<n>We find a consistent task-dependent dissociation: Coverage serves as the stronger correlate for understanding performance, whereas Precision acts as the dominant predictor for generation performance.
  arXiv  Detail & Related papers  (2026-08-03T17:57:03Z)
• Beyond Static Anchors: Bounded Prototype Conditioning for Language-Free Medical Anomaly Detection [52.986874008247554]
  We propose ReCAP, a language-free framework that replaces static anchors with input-conditioned visual prototypes.<n>ReCAP re-centers separated normal and abnormal prototypes for each image through a bounded gated modulation.<n>It achieves the best image-level AUROC on all zero-shot and 23 of 24 few-shot settings, and the best zero-shot pixel-level AUROC on all three segmentation datasets.
  arXiv  Detail & Related papers  (2026-08-01T04:46:55Z)
• What to Test Next: Interpretable Coverage Gap Discovery in Driving VLMs [52.50210189669399]
  Driving vision-language models (VLMs) must accurately understand scenes across diverse conditions defined by Operational Design Domains (ODDs)<n>We propose SliceScorer, a deterministic scoring rule for missing-slice recommendation.<n>We show that SliceNav surfaces high-risk coverage gaps more effectively than prior slice-discovery methods.
  arXiv  Detail & Related papers  (2026-06-01T03:18:01Z)
• CaptionQA: Is Your Caption as Useful as the Image Itself? [39.852352842429376]
  Image captions serve as efficient surrogates for visual content in systems such as retrieval, recommendation, and multi-step agentic inference pipelines.<n>We propose a utility-based benchmark, CaptionQA, to evaluate model-generated captions.<n>We release CaptionQA along with an open-source pipeline for extension to new domains.
  arXiv  Detail & Related papers  (2025-11-26T03:43:32Z)
• BRIDGE: Bridging Gaps in Image Captioning Evaluation with Stronger Visual Cues [47.213906345208315]
  We propose BRIDGE, a new learnable and reference-free image captioning metric. Our proposal achieves state-of-the-art results compared to existing reference-free evaluation scores.
  arXiv  Detail & Related papers  (2024-07-29T18:00:17Z)
• CLAIR: Evaluating Image Captions with Large Language Models [69.46906537973518]
  We propose CLAIR, a novel method to evaluate machine-generated image captions. In our evaluations, CLAIR demonstrates a stronger correlation with human judgments of caption quality compared to existing measures. Clair provides noisily interpretable results by allowing the language model to identify the underlying reasoning behind its assigned score.
  arXiv  Detail & Related papers  (2023-10-19T17:59:01Z)
• CapDet: Unifying Dense Captioning and Open-World Detection Pretraining [68.8382821890089]
  We propose a novel open-world detector named CapDet to either predict under a given category list or directly generate the category of predicted bounding boxes. Specifically, we unify the open-world detection and dense caption tasks into a single yet effective framework by introducing an additional dense captioning head.
  arXiv  Detail & Related papers  (2023-03-04T19:53:00Z)
• Intrinsic Image Captioning Evaluation [53.51379676690971]
  We propose a learning based metrics for image captioning, which we call Intrinsic Image Captioning Evaluation(I2CE) Experiment results show that our proposed method can keep robust performance and give more flexible scores to candidate captions when encountered with semantic similar expression or less aligned semantics.
  arXiv  Detail & Related papers  (2020-12-14T08:36:05Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.