OVFact: Measuring and Improving Open-Vocabulary Factuality for Long Caption Models

• URL: http://arxiv.org/abs/2507.19262v1
• Date: Fri, 25 Jul 2025 13:38:06 GMT
• Title: OVFact: Measuring and Improving Open-Vocabulary Factuality for Long Caption Models
• Authors: Monika Wysoczańska, Shyamal Buch, Anurag Arnab, Cordelia Schmid,
• Abstract summary: We introduce OV-Fact, a novel method for measuring caption factuality of long captions.<n>Our method improves agreement with human judgments and captures both captionness (recall) and factual precision in the same metric.<n>Unlike previous metrics, our reference-free method design enables new applications towards factuality-based data filtering.
• Score: 65.8015696586307
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Large vision-language models (VLMs) often struggle to generate long and factual captions. However, traditional measures for hallucination and factuality are not well suited for evaluating longer, more diverse captions and in settings where ground-truth human-annotated captions are unavailable. We introduce OV-Fact, a novel method for measuring caption factuality of long captions that leverages open-vocabulary visual grounding and tool-based verification without depending on human annotations. Our method improves agreement with human judgments and captures both caption descriptiveness (recall) and factual precision in the same metric. Furthermore, unlike previous metrics, our reference-free method design enables new applications towards factuality-based data filtering. We observe models trained on an OVFact-filtered (2.5-5x less) subset of a large-scale, noisy (VLM-generated) pretraining set meaningfully improve factuality precision without sacrificing caption descriptiveness across a range of downstream long caption benchmarks.

Related papers

• Painting with Words: Elevating Detailed Image Captioning with Benchmark and Alignment Learning [56.31096024472269]
  We introduce DeCapBench along with a novel metric, DCScore, specifically designed for detailed captioning tasks.<n>DCScore evaluates hallucinations and fine-grained comprehensiveness by deconstructing responses into the smallest self-sufficient units.<n>DeCapBench exhibits a high correlation with VLM arena results on descriptive tasks, surpassing existing benchmarks for vision-language models.
  arXiv  Detail & Related papers  (2025-03-10T22:53:56Z)
• CAPability: A Comprehensive Visual Caption Benchmark for Evaluating Both Correctness and Thoroughness [30.44039177018447]
  CAPability is a comprehensive benchmark for evaluating visual captioning across 12 dimensions spanning six critical views.<n>We curate nearly 11K human-annotated images and videos with visual element annotations to evaluate the generated captions.
  arXiv  Detail & Related papers  (2025-02-19T07:55:51Z)
• Toward Robust Hyper-Detailed Image Captioning: A Multiagent Approach and Dual Evaluation Metrics for Factuality and Coverage [50.84150600032693]
  Multimodal large language models (MLLMs) excel at generating highly detailed captions but often produce hallucinations.<n>We propose a multiagent approach that leverages LLM-MLLM collaboration to correct given captions.<n>Our proposed method significantly enhances the factual accuracy of captions, even improving those generated by GPT-4V.
  arXiv  Detail & Related papers  (2024-12-20T01:37:22Z)
• Bridging the Visual Gap: Fine-Tuning Multimodal Models with Knowledge-Adapted Captions [31.637204677787576]
  We introduce Knowledge Adapted (KnowAda) fine-tuning, a data-centric approach that automatically adapts training data with the model's existing knowledge and visual understanding.<n>KnowAda minimizes hallucinations while preserving high descriptiveness.<n>Our results show that KnowAda outperforms various baselines in both automatic metrics and human evaluations.
  arXiv  Detail & Related papers  (2024-11-13T20:50:04Z)
• VoLTA: Vision-Language Transformer with Weakly-Supervised Local-Feature Alignment [52.489874804051304]
  VoLTA is a new vision-language pre-training paradigm that only utilizes image-caption data but fine-grained region-level image understanding. VoLTA pushes multi-modal fusion deep into the uni-modal backbones during pre-training. Experiments on a wide range of vision- and vision-language downstream tasks demonstrate the effectiveness of VoLTA.
  arXiv  Detail & Related papers  (2022-10-09T01:49:58Z)
• Paraphrasing Is All You Need for Novel Object Captioning [126.66301869607656]
  Novel object captioning (NOC) aims to describe images containing objects without observing their ground truth captions during training. We present Paraphrasing-to-Captioning (P2C), a two-stage learning framework for NOC, which wouldally optimize the output captions via paraphrasing.
  arXiv  Detail & Related papers  (2022-09-25T22:56:04Z)
• Egoshots, an ego-vision life-logging dataset and semantic fidelity metric to evaluate diversity in image captioning models [63.11766263832545]
  We present a new image captioning dataset, Egoshots, consisting of 978 real life images with no captions. In order to evaluate the quality of the generated captions, we propose a new image captioning metric, object based Semantic Fidelity (SF)
  arXiv  Detail & Related papers  (2020-03-26T04:43:30Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.