When Seeing Is Not Enough: Benchmarking Interactive Visual Grounding in LVLMs

• URL: http://arxiv.org/abs/2608.23978v2
• Date: Sun, 30 Aug 2026 22:39:24 GMT
• Title: When Seeing Is Not Enough: Benchmarking Interactive Visual Grounding in LVLMs
• Abstract summary: Visual grounding is evaluated as a one-shot mapping from an informative referring expression to a visual target.<n>We introduce a controlled evaluation framework for interactive visual grounding in large vision-language models.
• Score: 13.266692327291748
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Visual grounding is typically evaluated as a one-shot mapping from an informative referring expression to a visual target. This formulation misses a central property of real-world reference: initial referring expressions are often incomplete or ambiguous, requiring participants to establish shared understanding through interaction. We introduce a controlled evaluation framework for interactive visual grounding in large vision-language models (LVLMs), varying how much target information is provided upfront and how much must be acquired through dialogue. Across four human-grounded visual contexts and four interaction protocols, current LVLMs perform significantly below task-level human baselines. Interaction can help when follow-up questions refine or repair an initial target description. Performance is lowest when no initial description is provided and target information must be acquired through questions, indicating that proactive question-driven grounding remains difficult. LVLMs are also poorly calibrated, often reporting confidence that exceeds their empirical accuracy. Follow-up studies confirm these patterns across varied description sources (human versus AI), reasoning efforts, repeated interactions, description providers, and visual contexts. Overall, interactive visual grounding remains challenging, requiring visual matching, information seeking and synthesis.

Related papers

• Is Visual Prompting All You Need? Studying VLM Spatial Reasoning under Progressive Visual Scaffolds [12.841957617988148]
  We study how the visual presentation of a task shapes model performance and failure modes when the underlying reasoning problem is unchanged.<n>We introduce lightweight input-side scaffolds that preserve the visual modality while making spatial structure more accessible.<n>We find that visual presentation is a central factor that determines whether VLM benchmarks measure grounded perception, downstream reasoning, or a mixture of both.
  arXiv  Detail & Related papers  (2026-08-21T14:40:26Z)
• Diagnosing Visual Ignorance in Vision-Language Models [29.2851901986069]
  Vision-Language Models (VLMs) frequently rely on language priors, producing confident answers that are weakly grounded in visual evidence.<n>In this work, we study language-prior reliance from both mechanistic and behavioral perspectives.<n>We find that language-prior reliance is a systematic routing failure affecting both model internals and benchmark validity.
  arXiv  Detail & Related papers  (2026-06-05T04:16:01Z)
• Seek-and-Solve: Benchmarking MLLMs for Visual Clue-Driven Reasoning in Daily Scenarios [56.703500104126014]
  DailyClue is a benchmark designed for visual clue-driven reasoning in daily scenarios.<n>Our construction is guided by two core principles: (1) strict grounding in authentic daily activities, and (2) challenging query design.<n>Our analysis reveals several critical insights, emphasizing that the accurate identification of visual clues is essential for robust reasoning.
  arXiv  Detail & Related papers  (2026-04-15T16:22:10Z)
• Responses Fall Short of Understanding: Revealing the Gap between Internal Representations and Responses in Visual Document Understanding [13.93076527078901]
  We investigate how information required to solve VDU tasks is represented across different layers of large vision language models.<n>Our study reveals that there is a clear gap between internal representations and generated responses.<n>Experiments show that fine-tuning intermediate layers improves both linear probing accuracy and response accuracy while narrowing the gap.
  arXiv  Detail & Related papers  (2026-04-06T04:25:52Z)
• ACT Now: Preempting LVLM Hallucinations via Adaptive Context Integration [71.21097024566285]
  Large-Language Models (LVLMs) frequently suffer from severe hallucination issues.<n>Existing mitigation strategies rely on linguistic, single-step states to enhance visual focus or suppress strong priors.<n>We propose Adaptive Context in VisionTegration (ACT), a training-free inference intervention method that mitigates hallucination through the adaptive integration of contextual information.
  arXiv  Detail & Related papers  (2026-04-01T14:49:50Z)
• Teaching Language Models To Gather Information Proactively [53.85419549904644]
  Large language models (LLMs) are increasingly expected to function as collaborative partners.<n>In this work, we introduce a new task paradigm: proactive information gathering.<n>We design a scalable framework that generates partially specified, real-world tasks, masking key information.<n>Within this setup, our core innovation is a reinforcement finetuning strategy that rewards questions that elicit genuinely new, implicit user information.
  arXiv  Detail & Related papers  (2025-07-28T23:50:09Z)
• INTER: Mitigating Hallucination in Large Vision-Language Models by Interaction Guidance Sampling [22.022620124352603]
  Hallucinations in large vision-language models (LVLMs) pose significant challenges for real-world applications.<n>We propose textbfInter: textbfInteraction Guidance Sampling, a novel training-free algorithm that mitigates hallucinations without requiring additional data.
  arXiv  Detail & Related papers  (2025-07-07T14:38:53Z)
• ICT: Image-Object Cross-Level Trusted Intervention for Mitigating Object Hallucination in Large Vision-Language Models [32.24716280370563]
  ICT is a lightweight, training-free method that calculates an intervention direction to shift the model's focus towards different levels of visual information. It achieves strong performance with a small amount of data and generalizes well across different datasets and models.
  arXiv  Detail & Related papers  (2024-11-22T12:22:21Z)
• VAGUE: Visual Contexts Clarify Ambiguous Expressions [26.190504250419547]
  VAGUE is a benchmark evaluating multimodal AI systems' ability to integrate visual context for intent.<n>Our experiments reveal that existing multimodal AI models struggle to infer the speaker's true intent.<n>Analysis of failure cases demonstrates that current models fail to distinguish true intent from superficial correlations in the visual scene.
  arXiv  Detail & Related papers  (2024-11-21T14:01:42Z)
• Evaluating and Analyzing Relationship Hallucinations in Large Vision-Language Models [69.79709804046325]
  We introduce R-Bench, a novel benchmark for evaluating Vision Relationship Hallucination. R-Bench features image-level questions that focus on the existence of relationships and instance-level questions that assess local visual comprehension. We identify three types of relationship co-occurrences that lead to hallucinations: relationship-relationship, subject-relationship, and relationship-object.
  arXiv  Detail & Related papers  (2024-06-24T08:42:42Z)
• Behind the Magic, MERLIM: Multi-modal Evaluation Benchmark for Large Image-Language Models [50.653838482083614]
  This paper introduces a scalable test-bed to assess the capabilities of IT-LVLMs on fundamental computer vision tasks.<n> MERLIM contains over 300K image-question pairs and has a strong focus on detecting cross-modal "hallucination" events in IT-LVLMs.
  arXiv  Detail & Related papers  (2023-12-03T16:39:36Z)
• Understanding ME? Multimodal Evaluation for Fine-grained Visual Commonsense [98.70218717851665]
  It is unclear whether the models really understand the visual scene and underlying commonsense knowledge due to limited evaluation data resources. We present a Multimodal Evaluation (ME) pipeline to automatically generate question-answer pairs to test models' understanding of the visual scene, text, and related knowledge. We then take a step further to show that training with the ME data boosts the model's performance in standard VCR evaluation.
  arXiv  Detail & Related papers  (2022-11-10T21:44:33Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.