Seeing Is Not Sharing: Some Vision-Language Models Overestimate Common Ground in Asymmetric Dialogue

• URL: http://arxiv.org/abs/2606.31719v1
• Date: Tue, 30 Jun 2026 14:22:48 GMT
• Title: Seeing Is Not Sharing: Some Vision-Language Models Overestimate Common Ground in Asymmetric Dialogue
• Abstract summary: In collaborative dialogue, shared perception does not guarantee shared interpretation. Mutual understanding must be established through interaction.<n>We evaluate vision-language models (VLMs) under systematically controlled manipulations of dialogue context and map-information access.<n>Our results show that providing authentic map images improves overall performance but shifts models toward over-predicting alignment.
• Score: 8.925029287648345
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: In collaborative dialogue, shared perception does not guarantee shared interpretation. Mutual understanding must be established through interaction. We investigate whether vision-language models (VLMs) can distinguish what could be shared from what has been shared between dialogue participants through grounding. We formulate this as an interpretation-matching task on 13,077 annotated reference expressions from HCRC MapTask dialogues, and evaluate VLMs under systematically controlled manipulations of dialogue context and map-information access. Our results show that providing authentic map images improves overall performance but shifts models toward over-predicting alignment. Textual descriptions of the same map content reproduce this bias, while non-informative images suppress alignment predictions entirely, indicating that the bias is driven by task-relevant map content, not the visual channel. This improvement comes at the cost of degraded accuracy on non-aligned cases. Calibration analysis and reference-chain tracking further suggest that models rely on static referential cues on the maps rather than tracking how grounding unfolds through dialogue history. We observe these patterns most clearly in Qwen3-VL-8B-Instruct and, to varying degrees, in four additional models from two architecture families. In models that exhibit the bias, map content, whether presented visually or textually, is treated as evidence of mutual understanding, conflating potential with established common ground.

Related papers

• When Seeing Is Not Enough: Benchmarking Interactive Visual Grounding in LVLMs [13.266692327291748]
  Visual grounding is evaluated as a one-shot mapping from an informative referring expression to a visual target.<n>We introduce a controlled evaluation framework for interactive visual grounding in large vision-language models.
  arXiv  Detail & Related papers  (2026-08-25T02:12:31Z)
• Attention-Guided Saliency Maps for Interpreting Visualization Literacy in VLMs [4.507919939159203]
  We introduce a diagnostic saliency map method tailored for text generation over images using transformer models.<n>Our approach aggregates the language model's attention over the visual tokens across all heads and layers, then maps this attention back onto the vision encoder's patch grid to localise it over the image.
  arXiv  Detail & Related papers  (2026-07-17T16:40:23Z)
• Mirage Probes: How Vision Models Fake Visual Understanding [13.299409525312491]
  Vision-language models (VLMs) can answer image-based questions confidently, and often correctly, even when no image is provided.<n>We show that mirage behavior is linearly decodable from internal activations across residual stream, post-attention, and attention-head sites.
  arXiv  Detail & Related papers  (2026-06-11T19:51:44Z)
• Grounded Misunderstandings in Asymmetric Dialogue: A Perspectivist Annotation Scheme for MapTask [8.925029287648345]
  Collaborative dialogue relies on participants incrementally establishing common ground, yet in asymmetric settings they may believe they agree while referring to different entities.<n>We introduce a perspectivist annotation scheme for the HCRC MapTask corpus that separately captures speaker and addressee grounded interpretations for each reference expression.<n>We obtain 13k annotated reference expressions with reliability estimates and analyze the resulting understanding states.
  arXiv  Detail & Related papers  (2025-11-05T18:52:28Z)
• Mind-the-Glitch: Visual Correspondence for Detecting Inconsistencies in Subject-Driven Generation [120.23172120151821]
  We propose a novel approach for disentangling visual and semantic features from the backbones of pre-trained diffusion models.<n>We introduce an automated pipeline that constructs image pairs with annotated semantic and visual correspondences.<n>We propose a new metric, Visual Semantic Matching, that quantifies visual inconsistencies in subject-driven image generation.
  arXiv  Detail & Related papers  (2025-09-26T07:11:55Z)
• Correspondence Matters for Video Referring Expression Comprehension [64.60046797561455]
  Video Referring Expression (REC) aims to localize the referent objects described in the sentence to visual regions in the video frames. Existing methods suffer from two problems: 1) inconsistent localization results across video frames; 2) confusion between the referent and contextual objects. We propose a novel Dual Correspondence Network (dubbed as DCNet) which explicitly enhances the dense associations in both the inter-frame and cross-modal manners.
  arXiv  Detail & Related papers  (2022-07-21T10:31:39Z)
• Can Visual Dialogue Models Do Scorekeeping? Exploring How Dialogue Representations Incrementally Encode Shared Knowledge [19.812562421377706]
  We propose a theory-based evaluation method for investigating to what degree models pretrained on the VisDial dataset incrementally build representations that appropriately do scorekeeping.<n>Our conclusion is that the ability to make the distinction between shared and privately known statements along the dialogue is moderately present in the analysed models, but not always incrementally consistent.
  arXiv  Detail & Related papers  (2022-04-14T13:52:11Z)
• Unpaired Referring Expression Grounding via Bidirectional Cross-Modal Matching [53.27673119360868]
  Referring expression grounding is an important and challenging task in computer vision. We propose a novel bidirectional cross-modal matching (BiCM) framework to address these challenges. Our framework outperforms previous works by 6.55% and 9.94% on two popular grounding datasets.
  arXiv  Detail & Related papers  (2022-01-18T01:13:19Z)
• $C^3$: Compositional Counterfactual Contrastive Learning for Video-grounded Dialogues [97.25466640240619]
  Video-grounded dialogue systems aim to integrate video understanding and dialogue understanding to generate responses relevant to both the dialogue and video context. Most existing approaches employ deep learning models and have achieved remarkable performance, given the relatively small datasets available. We propose a novel approach of Compositional Counterfactual Contrastive Learning to develop contrastive training between factual and counterfactual samples in video-grounded dialogues.
  arXiv  Detail & Related papers  (2021-06-16T16:05:27Z)
• Learning Reasoning Paths over Semantic Graphs for Video-grounded Dialogues [73.04906599884868]
  We propose a novel framework of Reasoning Paths in Dialogue Context (PDC) PDC model discovers information flows among dialogue turns through a semantic graph constructed based on lexical components in each question and answer. Our model sequentially processes both visual and textual information through this reasoning path and the propagated features are used to generate the answer.
  arXiv  Detail & Related papers  (2021-03-01T07:39:26Z)
• Modality-Balanced Models for Visual Dialogue [102.35406085738325]
  The Visual Dialog task requires a model to exploit both image and conversational context information to generate the next response to the dialogue. We show that previous joint-modality (history and image) models over-rely on and are more prone to memorizing the dialogue history. We present methods for this integration of the two models, via ensemble and consensus dropout fusion with shared parameters.
  arXiv  Detail & Related papers  (2020-01-17T14:57:12Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.