An Examination of the Robustness of Reference-Free Image Captioning Evaluation Metrics

• URL: http://arxiv.org/abs/2305.14998v2
• Date: Tue, 6 Feb 2024 04:20:15 GMT
• Title: An Examination of the Robustness of Reference-Free Image Captioning Evaluation Metrics
• Abstract summary: We evaluate CLIPScore, UMIC, and PAC-S in scenarios that require distinguishing between two captions with high lexical overlap but very different meanings. Our findings reveal that despite their high correlation with human judgments, CLIPScore, UMIC, and PAC-S struggle to identify fine-grained errors.
• Score: 6.137805050050414
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Recently, reference-free metrics such as CLIPScore (Hessel et al., 2021), UMIC (Lee et al., 2021), and PAC-S (Sarto et al., 2023) have been proposed for automatic reference-free evaluation of image captions. Our focus lies in evaluating the robustness of these metrics in scenarios that require distinguishing between two captions with high lexical overlap but very different meanings. Our findings reveal that despite their high correlation with human judgments, CLIPScore, UMIC, and PAC-S struggle to identify fine-grained errors. While all metrics exhibit strong sensitivity to visual grounding errors, their sensitivity to caption implausibility errors is limited. Furthermore, we found that all metrics are sensitive to variations in the size of image-relevant objects mentioned in the caption, while CLIPScore and PAC-S are also sensitive to the number of mentions of image-relevant objects in the caption. Regarding linguistic aspects of a caption, all metrics show weak comprehension of negation, and CLIPScore and PAC-S are insensitive to the structure of the caption to a great extent. We hope our findings will guide further improvements in reference-free evaluation of image captioning.

Related papers

• ITIScore: An Image-to-Text-to-Image Rating Framework for the Image Captioning Ability of MLLMs [84.09282931360089]
  ICBench is a large-scale image captioning benchmark that covers 12 content categories and consists of both short and long captions on 2K images.<n>We conduct extensive human subjective studies to obtain mean opinion scores (MOSs) across fine-grained evaluation dimensions.<n>We propose an automated evaluation metric, textbfITIScore, based on an image-to-text-to-image framework, which measures caption quality through reconstruction consistency.
  arXiv  Detail & Related papers  (2026-04-04T15:33:07Z)
• CAPability: A Comprehensive Visual Caption Benchmark for Evaluating Both Correctness and Thoroughness [30.44039177018447]
  CAPability is a comprehensive benchmark for evaluating visual captioning across 12 dimensions spanning six critical views.<n>We curate nearly 11K human-annotated images and videos with visual element annotations to evaluate the generated captions.
  arXiv  Detail & Related papers  (2025-02-19T07:55:51Z)
• BRIDGE: Bridging Gaps in Image Captioning Evaluation with Stronger Visual Cues [47.213906345208315]
  We propose BRIDGE, a new learnable and reference-free image captioning metric. Our proposal achieves state-of-the-art results compared to existing reference-free evaluation scores.
  arXiv  Detail & Related papers  (2024-07-29T18:00:17Z)
• HICEScore: A Hierarchical Metric for Image Captioning Evaluation [10.88292081473071]
  We propose a novel reference-free metric for image captioning evaluation, dubbed Hierarchical Image Captioning Evaluation Score (HICE-S) By detecting local visual regions and textual phrases, HICE-S builds an interpretable hierarchical scoring mechanism. Our proposed metric achieves the SOTA performance on several benchmarks, outperforming existing reference-free metrics.
  arXiv  Detail & Related papers  (2024-07-26T08:24:30Z)
• CLAIR: Evaluating Image Captions with Large Language Models [69.46906537973518]
  We propose CLAIR, a novel method to evaluate machine-generated image captions. In our evaluations, CLAIR demonstrates a stronger correlation with human judgments of caption quality compared to existing measures. Clair provides noisily interpretable results by allowing the language model to identify the underlying reasoning behind its assigned score.
  arXiv  Detail & Related papers  (2023-10-19T17:59:01Z)
• InfoMetIC: An Informative Metric for Reference-free Image Caption Evaluation [69.1642316502563]
  We propose an Informative Metric for Reference-free Image Caption evaluation (InfoMetIC) Given an image and a caption, InfoMetIC is able to report incorrect words and unmentioned image regions at fine-grained level. We also construct a token-level evaluation dataset and demonstrate the effectiveness of InfoMetIC in fine-grained evaluation.
  arXiv  Detail & Related papers  (2023-05-10T09:22:44Z)
• CapDet: Unifying Dense Captioning and Open-World Detection Pretraining [68.8382821890089]
  We propose a novel open-world detector named CapDet to either predict under a given category list or directly generate the category of predicted bounding boxes. Specifically, we unify the open-world detection and dense caption tasks into a single yet effective framework by introducing an additional dense captioning head.
  arXiv  Detail & Related papers  (2023-03-04T19:53:00Z)
• Transparent Human Evaluation for Image Captioning [70.03979566548823]
  We develop a rubric-based human evaluation protocol for image captioning models. We show that human-generated captions show substantially higher quality than machine-generated ones. We hope that this work will promote a more transparent evaluation protocol for image captioning.
  arXiv  Detail & Related papers  (2021-11-17T07:09:59Z)
• Contrastive Semantic Similarity Learning for Image Captioning Evaluation with Intrinsic Auto-encoder [52.42057181754076]
  Motivated by the auto-encoder mechanism and contrastive representation learning advances, we propose a learning-based metric for image captioning. We develop three progressive model structures to learn the sentence level representations. Experiment results show that our proposed method can align well with the scores generated from other contemporary metrics.
  arXiv  Detail & Related papers  (2021-06-29T12:27:05Z)
• CLIPScore: A Reference-free Evaluation Metric for Image Captioning [44.14502257230038]
  We show that CLIP, a cross-modal model pretrained on 400M image+caption pairs from the web, can be used for robust automatic evaluation of image captioning without the need for references. Experiments spanning several corpora demonstrate that our new reference-free metric, CLIPScore, achieves the highest correlation with human judgements. We also present a reference-augmented version, RefCLIPScore, which achieves even higher correlation.
  arXiv  Detail & Related papers  (2021-04-18T05:00:29Z)
• Intrinsic Image Captioning Evaluation [53.51379676690971]
  We propose a learning based metrics for image captioning, which we call Intrinsic Image Captioning Evaluation(I2CE) Experiment results show that our proposed method can keep robust performance and give more flexible scores to candidate captions when encountered with semantic similar expression or less aligned semantics.
  arXiv  Detail & Related papers  (2020-12-14T08:36:05Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.