Seeing Isn't Believing: Uncovering Blind Spots in Evaluator Vision-Language Models

• URL: http://arxiv.org/abs/2604.21523v1
• Date: Thu, 23 Apr 2026 10:36:50 GMT
• Title: Seeing Isn't Believing: Uncovering Blind Spots in Evaluator Vision-Language Models
• Abstract summary: We systematically evaluate the reliability of Evaluator VLMs across both I2T and T2I tasks.<n>We introduce perturbations that degrade output quality along key error dimensions, including object hallucinations, spatial reasoning, factual grounding, and visual fidelity.
• Score: 18.001586760420484
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Large Vision-Language Models (VLMs) are increasingly used to evaluate outputs of other models, for image-to-text (I2T) tasks such as visual question answering, and text-to-image (T2I) generation tasks. Despite this growing reliance, the reliability of these Evaluator VLMs remains under explored. In this work, we systematically evaluate the reliability of Evaluator VLMs across both I2T and T2I tasks. We introduce targeted perturbations that degrade output quality along key error dimensions, including object hallucinations, spatial reasoning, factual grounding, and visual fidelity. These perturbations test whether Evaluator VLMs can reliably account for these quality degrading errors in their evaluations. Using a comprehensive benchmark of over 4000 perturbed instances spanning 40 perturbation dimensions, we evaluate 4 prominent VLMs using single-answer scoring, pairwise comparison, and reference-guided paradigms. Our findings reveal that current VLM evaluators exhibit substantial blind spots: they often fail to detect perturbed outputs - in some cases exceeding 50%, struggle particularly with fine-grained compositional and spatial errors, and are often insensitive to hallucinated content that contradicts the input image. Pairwise comparison proves more reliable, though failure rates persist. These results highlight the unreliable nature of current Evaluator VLMs and urge caution in their deployment for benchmarking and development decisions. Code and data have been made publicly available.

Related papers

• Are Reasoning Vision-Language Models Robust to Semantic Visual Distractions? [34.294060886423985]
  Reasoning Vision-Language Models (VLMs) achieve strong performance on complex multimodal tasks.<n>We introduce textbfDistract-Bench, a benchmark for evaluating VLM robustness to textbfsemantic visual distractions<n>Our results show that Distract-Bench exposes a robustness failure distinct from vision corruptions.
  arXiv  Detail & Related papers  (2026-06-08T00:40:32Z)
• VLM-RobustBench: A Comprehensive Benchmark for Robustness of Vision-Language Models [64.56065206447788]
  Vision-language models (VLMs) achieve strong performance on standard, high-quality datasets.<n>We present VLM-RobustBench, a benchmark spanning 49 augmentation types across noise, blur, weather, digital, and geometrics.<n>We show that low-severity spatial perturbations often degrade performance more than visually severe photometric corruptions.
  arXiv  Detail & Related papers  (2026-03-06T10:58:02Z)
• Same Answer, Different Representations: Hidden instability in VLMs [65.36933543377346]
  We introduce a representation-aware and frequency-aware evaluation framework that measures internal embedding drift, spectral sensitivity, and structural smoothness.<n>We apply this framework to modern Vision Language Models (VLMs) across the SEEDBench, MMMU, and POPE datasets.
  arXiv  Detail & Related papers  (2026-02-06T12:24:26Z)
• FineGRAIN: Evaluating Failure Modes of Text-to-Image Models with Vision Language Model Judges [85.24983823102262]
  We propose a structured methodology for evaluating text-to-image (T2I) models and vision language models (VLMs)<n>We test whether VLMs can identify 27 specific failure modes in the images generated by T2I models conditioned on challenging prompts.<n>Our findings suggest that current metrics are insufficient to capture these nuanced errors.
  arXiv  Detail & Related papers  (2025-12-01T19:46:03Z)
• [De|Re]constructing VLMs' Reasoning in Counting [2.1856941852799134]
  We study the reasoning skills of seven state-of-the-art Vision-Language Models (VLMs) in the counting task under controlled experimental conditions.<n>A layer-wise analysis reveals that errors are due to incorrect mapping of the last-layer representation into the output space.<n>Our targeted training shows that fine-tuning just the output layer improves accuracy by up to 21%.
  arXiv  Detail & Related papers  (2025-10-22T13:08:47Z)
• Retention Score: Quantifying Jailbreak Risks for Vision Language Models [60.48306899271866]
  Vision-Language Models (VLMs) are integrated with Large Language Models (LLMs) to enhance multi-modal machine learning capabilities.<n>This paper aims to assess the resilience of VLMs against jailbreak attacks that can compromise model safety compliance and result in harmful outputs.<n>To evaluate a VLM's ability to maintain its robustness against adversarial input perturbations, we propose a novel metric called the textbfRetention Score.
  arXiv  Detail & Related papers  (2024-12-23T13:05:51Z)
• VHELM: A Holistic Evaluation of Vision Language Models [75.88987277686914]
  We present the Holistic Evaluation of Vision Language Models (VHELM) VHELM aggregates various datasets to cover one or more of the 9 aspects: visual perception, knowledge, reasoning, bias, fairness, multilinguality, robustness, toxicity, and safety. Our framework is designed to be lightweight and automatic so that evaluation runs are cheap and fast.
  arXiv  Detail & Related papers  (2024-10-09T17:46:34Z)
• DARE: Diverse Visual Question Answering with Robustness Evaluation [16.87867803628065]
  Vision Language Models (VLMs) extend remarkable capabilities of text-only large language models and vision-only models.<n>They struggle with a number of crucial vision-language (VL) reasoning abilities such as counting and spatial reasoning.<n>We introduce DARE, Diverse Visual Question Answering with Robustness Evaluation.
  arXiv  Detail & Related papers  (2024-09-26T16:31:50Z)
• Overconfidence is Key: Verbalized Uncertainty Evaluation in Large Language and Vision-Language Models [6.9060054915724]
  Language and Vision-Language Models (LLMs/VLMs) have revolutionized the field of AI by their ability to generate human-like text and understand images, but ensuring their reliability is crucial. This paper aims to evaluate the ability of LLMs (GPT4, GPT-3.5, LLaMA2, and PaLM 2) and VLMs (GPT4V and Gemini Pro Vision) to estimate their verbalized uncertainty via prompting. We propose the new Japanese Uncertain Scenes dataset aimed at testing VLM capabilities via difficult queries and object counting, and the Net Error dataset to measure direction of miscalibration.
  arXiv  Detail & Related papers  (2024-05-05T12:51:38Z)
• VALOR-EVAL: Holistic Coverage and Faithfulness Evaluation of Large Vision-Language Models [57.43276586087863]
  Large Vision-Language Models (LVLMs) suffer from hallucination issues, wherein the models generate plausible-sounding but factually incorrect outputs. Existing benchmarks are often limited in scope, focusing mainly on object hallucinations. We introduce a multi-dimensional benchmark covering objects, attributes, and relations, with challenging images selected based on associative biases.
  arXiv  Detail & Related papers  (2024-04-22T04:49:22Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.