Seeing What Is Actually There: PriVE-Bench and PriVE-Tools for Counterfactual Evaluation of Agentic Visual Evidence in VLMs

• URL: http://arxiv.org/abs/2607.16311v1
• Date: Tue, 14 Jul 2026 16:45:02 GMT
• Title: Seeing What Is Actually There: PriVE-Bench and PriVE-Tools for Counterfactual Evaluation of Agentic Visual Evidence in VLMs
• Abstract summary: Vision-language models (VLMs) often answer visual questions using learned language and category priors rather than grounding their predictions in the image itself.<n>We introduce PriVE-Bench, a Prior-vs-Visual Evidence Benchmark that uses paired original and counterfactual images to distinguish visually grounded answers from prior-consistent errors.<n>We also introduce PriVE-Tools, a controlled agentic-vision-inspired extension that evaluates whether tool-derived visual evidence improves grounding under the same counterfactual conflicts.
• Score: 14.028133377242701
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Vision-language models (VLMs) often answer visual questions using learned language and category priors rather than grounding their predictions in the image itself. Counterfactual images provide a natural diagnostic setting for this failure mode: when visible evidence contradicts what is usually true, a grounded model should answer from the pixels, while a prior-following model will produce a canonical but visually incorrect response. However, existing counterfactual benchmarks mainly ask whether such prior-following behavior exists. In this paper, we ask a further question motivated by the rise of tool-augmented and agentic vision systems: can additional visual evidence views help VLMs reason against their priors? We introduce PriVE-Bench, a Prior-vs-Visual Evidence Benchmark that uses paired original and counterfactual images to distinguish visually grounded answers from prior-consistent errors. We further introduce PriVE-Tools, a controlled agentic-vision-inspired extension that evaluates whether tool-derived visual evidence -- including bounding boxes, crops, zoom panels, and contours -- improves grounding under the same counterfactual conflicts. Across open- and closed-source VLMs, we compare raw, paired-image, and tool-conditioned inputs using accuracy, prior-following error rate, and other-response rate. Our results show that visual evidence tools can help in some settings, especially when models can use localized evidence effectively, but they are not a universal remedy: several models continue to follow language and category priors even when relevant visual evidence is explicitly provided.

Related papers

• Seeing or Knowing? Visual Context Sensitivity in Multimodal Large Language Models [14.5102059776721]
  Multimodal Large Language Models (MLLMs) achieve strong performance by integrating visual inputs with the rich priors of pretrained language models.<n>However, they often fail on vision-centric tasks, especially when visual evidence conflicts with pretrained knowledge.<n>We explore these failures using two diagnostic paradigms: (1) probing whether visual information is available, via image reconstruction, and (2) measuring multimodal context sensitivity.
  arXiv  Detail & Related papers  (2026-07-28T22:52:34Z)
• MissingBench-Verified: Probing Vision-Language Models' Inability to Detect Missing Object Parts [10.608824155356377]
  We present MissingBench-Verified, a benchmark designed to evaluate a specific and practically relevant scenario.<n>We observe consistent and significant failure rates that persist even when external tool evidence explicitly contradicts the model's visual perception.<n>We find that existing mitigation strategies, including tool-assisted verification, autonomous visual reasoning, longer reasoning durations, and fine-tuning on an easier dataset, provide negligible improvement.
  arXiv  Detail & Related papers  (2026-07-21T03:43:50Z)
• Does It Fail to See or Fail to Know? Attributing Errors in Vision-Language Models [33.564453692157095]
  Uncertainty should not only indicate whether the model is likely to fail but also diagnose why it is uncertain.<n>We propose a unified framework for disentangling failure modes and investigate whether pre-generation signals can predict these failure sources.<n>Our main finding is that recognition-related failures are best captured by visual-token representations, while failures that remain after recognition are better captured by prompt-conditioned hidden states.
  arXiv  Detail & Related papers  (2026-07-06T05:11:45Z)
• CounterCount: A Diagnostic Framework for Counting Bias in Vision Language Models [44.585250047959924]
  Vision-Language Models (VLMs) excel at multimodal reasoning, yet it remains unclear whether their answers are grounded in visual evidence or driven by learned language and world priors.<n>We introduce CounterCount, a diagnostic framework for counterfactual counting in VLMs.<n>We show that these failures are not solely due to missing or ambiguous visual evidence, but to models underweighting attention to count-relevant visual tokens.
  arXiv  Detail & Related papers  (2026-05-18T04:00:05Z)
• Are VLMs Seeing or Just Saying? Uncovering the Illusion of Visual Re-examination [39.713386491199884]
  Vision-Language Models (VLMs) often produce self-reflective statements like "let me check the figure again" during reasoning.<n>Do such statements trigger genuine visual re-examination, or are they merely learned textual patterns?<n>We investigate this via VisualSwap, an image-swap probing framework.
  arXiv  Detail & Related papers  (2026-05-15T11:31:14Z)
• ReVSI: Rebuilding Visual Spatial Intelligence Evaluation for Accurate Assessment of VLM 3D Reasoning [59.558706734431276]
  Current evaluations of spatial intelligence can be systematically invalid under modern vision-language model (VLM) settings.<n>We introduce ReVSI, a benchmark and protocol that ensures each QA pair is answerable and correct under the model's actual inputs.
  arXiv  Detail & Related papers  (2026-04-27T10:45:51Z)
• Improving Visual Reasoning with Iterative Evidence Refinement [41.18410988040976]
  Vision language models (VLMs) are increasingly capable of reasoning over images.<n>But robust visual reasoning often requires re-grounding intermediate steps in the underlying visual evidence.<n>We propose an end-to-end self-revisit framework, SIEVE, that trains models to re-engage image evidence through internal representations.
  arXiv  Detail & Related papers  (2026-03-14T21:08:21Z)
• Through the Lens of Contrast: Self-Improving Visual Reasoning in VLMs [60.93949629734977]
  We propose Visual Contrastive Self-Taught Reasoner (VC-STaR) to mitigate hallucinations in model-generated rationales.<n>We collect a diverse suite of VQA datasets, curate contrastive pairs according to multi-modal similarity, and generate rationales using VC-STaR.<n>Extensive experiments show that VC-STaR not only outperforms existing self-improving approaches but also surpasses models finetuned on the SoTA visual reasoning datasets.
  arXiv  Detail & Related papers  (2026-03-03T03:18:31Z)
• Thinking with Deltas: Incentivizing Reinforcement Learning via Differential Visual Reasoning Policy [75.66913260900726]
  Reinforcement Learning with Verifiable Rewards has significantly advanced reasoning capabilities in Large Language Models.<n>Existing paradigms, driven by text-centric outcome rewards, encourage models to bypass visual perception.<n>We propose textbfThinking with Deltas, a framework driven by a textbfDifferential Visual Reasoning Policy.
  arXiv  Detail & Related papers  (2026-01-11T08:25:34Z)
• Visually Prompted Benchmarks Are Surprisingly Fragile [82.98001690512461]
  Key challenge in evaluating VLMs is testing their ability to analyze visual content independently from their textual priors.<n>We demonstrate how details in benchmark setup, including visual marker design and dataset size, have a significant influence on model performance and leaderboard rankings.<n>To mitigate this instability, we curate existing datasets to create VPBench, a larger visually prompted benchmark with 16 visual marker variants.
  arXiv  Detail & Related papers  (2025-12-19T18:26:58Z)
• Seeing to Act, Prompting to Specify: A Bayesian Factorization of Vision Language Action Policy [59.44168425139687]
  BayesVLA is a Bayesian factorization that decomposes the policy into a visual-action prior, supporting seeing-to-act, and a language-conditioned likelihood, enabling prompt-to-specify.<n>Experiments show superior generalization to unseen instructions, objects, and environments compared to existing methods.
  arXiv  Detail & Related papers  (2025-12-12T01:59:23Z)
• Draft and Refine with Visual Experts [18.983324604452118]
  Large Vision-Language Models (LVLMs) produce ungrounded or hallucinated responses because they rely too heavily on linguistic priors instead of visual evidence.<n>We propose Draft and Refine (DnR), an agent framework driven by a question-conditioned utilization metric.
  arXiv  Detail & Related papers  (2025-11-14T06:50:14Z)
• VisRAG 2.0: Evidence-Guided Multi-Image Reasoning in Visual Retrieval-Augmented Generation [64.82775032985485]
  Visual retrieval-augmented generation (VRAG) augments vision-language models (VLMs) with external visual knowledge to ground reasoning and reduce hallucinations.<n>Yet current VRAG systems often fail to reliably perceive and integrate evidence across multiple images, leading to weak grounding and erroneous conclusions.<n>We propose EVisRAG, an end-to-end framework that learns to reason with evidence-guided multi-image to address this issue.
  arXiv  Detail & Related papers  (2025-10-10T13:34:23Z)
• BLINK-Twice: You see, but do you observe? A Reasoning Benchmark on Visual Perception [67.89135437537179]
  We introduce BLINK-Twice, a vision-centric reasoning benchmark grounded in challenging perceptual tasks.<n>Instead of relying on external knowledge, our tasks require models to reason from visual content alone.<n>Compared to prior perception benchmarks, it moves beyond shallow perception and requires fine-grained observation and analytical reasoning.
  arXiv  Detail & Related papers  (2025-10-10T13:14:13Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.