LLM-as-Judge Framework for Evaluating Tone-Induced Hallucination in Vision-Language Models

• URL: http://arxiv.org/abs/2604.18803v3
• Date: Sat, 25 Apr 2026 21:48:41 GMT
• Title: LLM-as-Judge Framework for Evaluating Tone-Induced Hallucination in Vision-Language Models
• Abstract summary: Ghost-100 is a benchmark of 800 synthetically generated images spanning eight categories across three task families.<n>Every image is paired with five prompts drawn from a structured 5-Level Prompt Intensity Framework.<n>We adopt a dual-track evaluation protocol: a rule-based H-Rate measuring the proportion of responses in which a model crosses from grounded refusal into unsupported positive commitment.
• Score: 9.095828883716907
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Vision-Language Models (VLMs) are increasingly deployed in settings where reliable visual grounding carries operational consequences, yet their behavior under progressively coercive prompt phrasing remains undercharacterized. Existing hallucination benchmarks predominantly rely on neutral prompts and binary detection, leaving open how both the incidence and the intensity of fabrication respond to graded linguistic pressure across structurally distinct task types. We present Ghost-100, a procedurally constructed benchmark of 800 synthetically generated images spanning eight categories across three task families: text-illegibility, time-reading, and object-absence, each designed under a negative-ground-truth principle that guarantees the queried target is absent, illegible, or indeterminate by construction. Every image is paired with five prompts drawn from a structured 5-Level Prompt Intensity Framework, holding the image and task identity fixed while varying only directive force, so that tone is isolated as the sole independent variable. We adopt a dual-track evaluation protocol: a rule-based H-Rate measuring the proportion of responses in which a model crosses from grounded refusal into unsupported positive commitment, and a GPT-4o-mini-judged H-Score on a 1-5 scale characterizing the confidence and specificity of fabrication once it occurs. We additionally release a three-stage automated validation workflow, which retrospectively confirms 717 of 800 images as strictly compliant. Evaluating nine open-weight VLMs, we find that H-Rate and H-Score dissociate substantially across model families, reading-style and presence-detection subsets respond to prompt pressure in qualitatively different ways, and several models exhibit non-monotonic sensitivity peaking at intermediate tone levels: patterns that aggregate metrics obscure.

Related papers

• FabriMAE I Trust Myself? Self-Evaluating VLA Action Generation with Markov Attention Entropy [63.26759696630558]
  Vision-Language-Action models (VLAs) integrate visual perception, language instruction, and action generation into end-to-end policies.<n>Existing methods either rely on expert annotations or estimate uncertainty only from output statistics.<n>We propose MAE (Markov Attention Entropy), a self-evaluation framework that converts internal attention signals into architecture-aware reliability scores.
  arXiv  Detail & Related papers  (2026-08-17T15:14:53Z)
• Hijacking Robots with a Piece of Paper: A Systematic Study of Physical Prompt Injection in VLM-Controlled Robots [15.891078191181697]
  Vision-Language Models (VLMs) are increasingly deployed as planners in robotic systems.<n> adversarial text placed within the robot's visual field can act as an indirect prompt injection into the VLM's reasoning stack.<n>We present a systematic study of physical prompt injection attacks against VLM-controlled sorting.
  arXiv  Detail & Related papers  (2026-08-06T07:52:55Z)
• Role-Break in Attention Heads: Understanding and Detecting Hallucinations in VLMs [49.743008490747094]
  Vision-Language Models (VLMs) remain prone to hallucinations, producing content that is inconsistent with or unsupported by the input image.<n>Under a unified head-level view, we find that hallucination-induced changes manifest as localized deviations from each head's faithful contextual behavior.<n> Detailed analysis reveals that these deviations are systematically organized across attention heads, contextual sources, and deviation directions, and that the resulting signal is linearly readable once head identity is preserved.
  arXiv  Detail & Related papers  (2026-07-31T13:33:51Z)
• Zero-Shot Vision-Language Models for Classroom Engagement Recognition: A Benchmark Study of Prompt Sensitivity and Cross-Dataset Generalization [0.44787896002954924]
  We present a systematic benchmark that evaluates five widely-used Vision-Language Models (VLMs)<n>Our experiments reveal three primary failure modes of zero-shot VLMs for engagement recognition.
  arXiv  Detail & Related papers  (2026-06-20T03:53:40Z)
• When Eyes Betray AI: Social Gaze Consistency as a Semantic Cue for AI-Generated Image Detection [12.446807294893638]
  We introduce Social Gaze Consistency, a high-level semantic cue defined as the mutual coherence of gaze direction, head-eye alignment, and pupil placement between interacting individuals.<n>We show that it constitutes a previously underutilized detection axis to existing low-level paradigms.<n>A four-step account explains why training on a single inpainter (FLUX.1-Fill) transfers to multi-generator suites.
  arXiv  Detail & Related papers  (2026-05-26T17:50:17Z)
• VISTAQA: Benchmarking Joint Visual Question Answering and Pixel-Level Evidence [26.0945130521806]
  Existing benchmarks assess either textual answer correctness or pixel-level localization in isolation.<n>We introduce VISTAQA, a benchmark for joint evaluation of free-form answer correctness and pixel-level evidence grounding in visual question answering.<n>We introduce GROVE, a metric that enforces joint correctness by combining textual accuracy and grounding quality via a per-sample geometric mean.
  arXiv  Detail & Related papers  (2026-05-20T03:44:06Z)
• TAME: Test-Time Adversarial Prompt Tuning via Mixture-of-Experts for Vision-Language Models [76.92890872338235]
  Large-scale pre-trained Vision-Language models (VLMs) exhibit strong zero-shot generalization, yet are highly vulnerable to imperceptible adversarial perturbations.<n>To enhance robustness without requiring downstream task-specific retraining, we propose TAME, a novel test-time defense.
  arXiv  Detail & Related papers  (2026-05-17T18:07:08Z)
• Beyond Standard Benchmarks: A Systematic Audit of Vision-Language Model's Robustness to Natural Semantic Variation Across Diverse Tasks [11.064940886724257]
  This paper presents a systematic evaluation framework for vision-language models (VLMs) under natural adversarial scenarios.<n>We measure the natural adversarial performance of selected VLMs for zero-shot image classification, semantic segmentation, and visual question answering.<n>Our analysis reveals that robust CLIP models can amplify natural adversarial vulnerabilities, and CLIP models significantly reduce performance for natural language-induced adversarial examples.
  arXiv  Detail & Related papers  (2026-04-06T06:48:32Z)
• Bi-Predictability: A Real-Time Signal for Monitoring LLM Interaction Integrity [0.0]
  We show that multi-turn interaction integrity can be continuously monitored using bi-predictability (P)<n>We introduce the Information Digital Twin (IDT), a lightweight architecture that estimates P across the context, response, next prompt loop without secondary inference or embeddings.
  arXiv  Detail & Related papers  (2026-03-18T18:10:37Z)
• Pressure Reveals Character: Behavioural Alignment Evaluation at Depth [3.634215320925722]
  We introduce an alignment benchmark spanning 904 scenarios across six categories -- Honesty, Safety, Non-Manipulation, Robustness, Corrigibility, and Scheming.<n>Our scenarios place models under conflicting instructions, simulated tool access, and multi-turn escalation to reveal behavioural tendencies that single-turn evaluations miss.<n>We find that even top-performing models exhibit gaps in specific categories, while the majority of models show consistent weaknesses.
  arXiv  Detail & Related papers  (2026-02-24T11:52:17Z)
• Same Answer, Different Representations: Hidden instability in VLMs [65.36933543377346]
  We introduce a representation-aware and frequency-aware evaluation framework that measures internal embedding drift, spectral sensitivity, and structural smoothness.<n>We apply this framework to modern Vision Language Models (VLMs) across the SEEDBench, MMMU, and POPE datasets.
  arXiv  Detail & Related papers  (2026-02-06T12:24:26Z)
• VIRO: Robust and Efficient Neuro-Symbolic Reasoning with Verification for Referring Expression Comprehension [51.76841625486355]
  Referring Expression (REC) aims to localize the image region corresponding to a natural-language query.<n>Recent neuro-symbolic REC approaches leverage large language models (LLMs) and vision-language models (VLMs) to perform compositional reasoning.<n>We introduce VIRO, a neuro-symbolic framework that embeds lightweight operator-level verifiers within reasoning steps.
  arXiv  Detail & Related papers  (2026-01-19T07:21:19Z)
• SSVP: Synergistic Semantic-Visual Prompting for Industrial Zero-Shot Anomaly Detection [55.54007781679915]
  We propose Synergistic Semantic-Visual Prompting (SSVP), that efficiently fuses diverse visual encodings to elevate model's fine-grained perception.<n>SSVP achieves state-of-the-art performance with 93.0% Image-AUROC and 92.2% Pixel-AUROC on MVTec-AD, significantly outperforming existing zero-shot approaches.
  arXiv  Detail & Related papers  (2026-01-14T04:42:19Z)
• ViPER: Empowering the Self-Evolution of Visual Perception Abilities in Vision-Language Model [61.29164681694533]
  ViPER is a self-bootstrapping framework specifically designed to enable iterative evolution through self-critiquing and self-prediction.<n>Qwen-Viper consistently demonstrates superior performance across different vision-language scenarios while maintaining generalizability.
  arXiv  Detail & Related papers  (2025-10-28T10:42:57Z)
• CPT: Colorful Prompt Tuning for Pre-trained Vision-Language Models [101.5066760592534]
  We present Cross-modal Prompt Tuning (CPT), a novel paradigm for tuning Vision-Language Models (VL-PTMs) CPT reformulates visual grounding into a fill-in-the-blank problem with color-based co-referential markers in image and text, maximally mitigating the gap. Comprehensive experimental results show that prompt tuned VL-PTMs outperform their fine-tuned counterparts by a large margin.
  arXiv  Detail & Related papers  (2021-09-24T08:07:29Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.