Human-Grounded Calibration for Long-Text Image-Text Congruence in Vision-Language Models

• URL: http://arxiv.org/abs/2609.15640v1
• Date: Mon, 14 Sep 2026 14:26:53 GMT
• Title: Human-Grounded Calibration for Long-Text Image-Text Congruence in Vision-Language Models
• Abstract summary: Long-text image--text congruence scoring is increasingly important for vision-language systems.<n>This paper proposes Congruency Score (CS), a lightweight calibration layer that maps image--text similarity evidence into a bounded score.
• Score: 46.28810845443601
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Long-text image--text congruence scoring is increasingly important for vision-language systems that must evaluate whether detailed textual descriptions match visual content. However, raw similarity scores from dual-encoder models are difficult to interpret as calibrated congruence measures, especially under the modality gap between image and text embeddings. This paper proposes Congruency Score (CS), a lightweight calibration layer that maps image--text similarity evidence into a bounded score. Using DOCCI and Urban1k, we evaluate four frozen vision-language backbones and show that observed reductions in post-projection centroid distance do not uniformly improve image--text retrieval performance. Human-grounded evaluations on DOCCI further reveal a trade-off: direct post-hoc calibration preserves high association with human judgments, whereas selected projection-based configurations can reduce threshold-relevant slope and intercept distortions at the cost of retrieval performance and association strength. These results establish long-text image--text congruence scoring as a calibrated score-estimation problem, where retrieval performance, human association, and threshold calibration must be evaluated as distinct objectives. CS provides a lightweight way to expose and operationalize this separation.

Related papers

• Vision-Free Retrieval: Rethinking Multimodal Search with Textual Scene Descriptions [81.33113485830711]
  We introduce a vision-free, single-encoder retrieval pipeline for vision-language models.<n>We migrate to a text-to-text paradigm with the assistance of VLLM-generated structured image descriptions.<n>Our approach achieves state-of-the-art zero-shot performance on multiple retrieval and compositionality benchmarks.
  arXiv  Detail & Related papers  (2025-09-23T16:22:27Z)
• TokBench: Evaluating Your Visual Tokenizer before Visual Generation [75.38270351179018]
  We analyze text and face reconstruction quality across various scales for different image tokenizers and VAEs.<n>Our results show modern visual tokenizers still struggle to preserve fine-grained features, especially at smaller scales.
  arXiv  Detail & Related papers  (2025-05-23T17:52:16Z)
• Redemption Score: A Multi-Modal Evaluation Framework for Image Captioning via Distributional, Perceptual, and Linguistic Signal Triangulation [3.4998703934432682]
  Redemption Score(RS) is a novel framework that ranks image captions by triangulating three complementary signals.<n>On the Flickr8k benchmark, RS achieves a Kendall-$tau$ of 58.42, outperforming most prior methods.
  arXiv  Detail & Related papers  (2025-05-22T03:35:12Z)
• Evaluating Text-to-Image and Text-to-Video Synthesis with a Conditional Fréchet Distance [13.599366291595672]
  We propose cFreD, a metric that unifies the assessment of visual fidelity and text-prompt consistency into a single score.<n>Our findings validate cFreD as a robust, future-proof metric for the systematic evaluation of text conditioned models.
  arXiv  Detail & Related papers  (2025-03-27T17:35:14Z)
• Image2Text2Image: A Novel Framework for Label-Free Evaluation of Image-to-Text Generation with Text-to-Image Diffusion Models [16.00576040281808]
  We propose a novel framework called Image2Text2Image to evaluate image captioning models. A high similarity score suggests that the model has produced a faithful textual description, while a low score highlights discrepancies. Our framework does not rely on human-annotated captions reference, making it a valuable tool for assessing image captioning models.
  arXiv  Detail & Related papers  (2024-11-08T17:07:01Z)
• A Novel Evaluation Framework for Image2Text Generation [15.10524860121122]
  We propose an evaluation framework rooted in a modern large language model (LLM) capable of image generation. A high similarity score suggests that the image captioning model has accurately generated textual descriptions. A low similarity score indicates discrepancies, revealing potential shortcomings in the model's performance.
  arXiv  Detail & Related papers  (2024-08-03T09:27:57Z)
• FINEMATCH: Aspect-based Fine-grained Image and Text Mismatch Detection and Correction [66.98008357232428]
  We propose FineMatch, a new aspect-based fine-grained text and image matching benchmark. FineMatch focuses on text and image mismatch detection and correction. We show that models trained on FineMatch demonstrate enhanced proficiency in detecting fine-grained text and image mismatches.
  arXiv  Detail & Related papers  (2024-04-23T03:42:14Z)
• Vision Language Model-based Caption Evaluation Method Leveraging Visual Context Extraction [27.00018283430169]
  This paper presents VisCE$2$, a vision language model-based caption evaluation method. Our method focuses on visual context, which refers to the detailed content of images, including objects, attributes, and relationships.
  arXiv  Detail & Related papers  (2024-02-28T01:29:36Z)
• Mismatch Quest: Visual and Textual Feedback for Image-Text Misalignment [64.49170817854942]
  We present a method to provide detailed explanation of detected misalignments between text-image pairs. We leverage large language models and visual grounding models to automatically construct a training set that holds plausible captions for a given image. We also publish a new human curated test set comprising ground-truth textual and visual misalignment annotations.
  arXiv  Detail & Related papers  (2023-12-05T20:07:34Z)
• Transparent Human Evaluation for Image Captioning [70.03979566548823]
  We develop a rubric-based human evaluation protocol for image captioning models. We show that human-generated captions show substantially higher quality than machine-generated ones. We hope that this work will promote a more transparent evaluation protocol for image captioning.
  arXiv  Detail & Related papers  (2021-11-17T07:09:59Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.