When Discourse Pressures Conflict: Information Structure in Vision-Language Model Outputs

• URL: http://arxiv.org/abs/2605.28346v1
• Date: Wed, 27 May 2026 11:50:56 GMT
• Title: When Discourse Pressures Conflict: Information Structure in Vision-Language Model Outputs
• Abstract summary: We test whether vision-language models (VLMs) distinguish discourse-old Topics from discourse-new Foci in visually grounded question answering.<n>We exploit Hungarian, a language in which Topic and Focus map onto dedicated syntactic positions, making IS choices observable in text.<n>We find that models produce IS-relevant constructions, but over-regularise this sensitivity.
• Score: 8.897315812066683
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Vision-language models (VLMs) are increasingly evaluated for whether they identify the right visual content, but little is known about whether they express such content in a discourse-appropriate form. We address this research gap using information structure (IS), testing whether VLMs distinguish discourse-old Topics from discourse-new Foci in visually grounded question answering. We exploit Hungarian, a language in which Topic and Focus map onto dedicated syntactic positions, making IS choices observable in text. Comparing six VLMs with human participants, we find that models produce IS-relevant constructions, but over-regularise this sensitivity. Under the interacting pressures of discourse status, grammatical role (preference for subject Topics) and definiteness (preference for indefinite Foci), humans choose variable strategies for IS realisation. VLMs, by contrast, collapse onto narrow response templates, resembling mode collapse (Kirk et al., 2024). Our findings suggest that VLM evaluation should look beyond content accuracy to how content is packaged for the discourse.

Related papers

• From Vision to Language: Investigating Causal Information Flow in Multimodal Decision-Making [45.85783584369188]
  We investigate how visual information contributes to language-based decisions.<n>Our results show that visual information is mainly integrated while the model processes the candidate answer options.<n>We identify a distinct pattern in temporal reasoning, suggesting that VLMs struggle to reconstruct sequential information across video frames.
  arXiv  Detail & Related papers  (2026-09-04T13:52:27Z)
• AlloEgo-VLM: Disambiguating Allocentric and Egocentric Reference Frames in Vision-Language Models [15.021275200375504]
  This study investigates the challenge of ambiguity faced by Vision-Language Models (VLMs) in understanding spatial semantics.<n>Natural language descriptions of spatial relations frequently omit explicit reference frames, leading to semantic ambiguity and potentially serious errors for embodied AI robots.<n>We construct a new dataset, AlloEgo-View, comprising (image, query, view-specific answer) triplets that capture key object relations from both allocentric and egocentric perspectives.<n>We develop AlloEgo-VLM, a framework to disambiguate allocentric and egocentric reference frames, even under ambiguous queries, and to be easily integrated
  arXiv  Detail & Related papers  (2026-08-16T08:06:20Z)
• Real Images, Worse Judgments: Evaluating Vision-Language Models on Concreteness and Imagery [21.95600321802019]
  We study whether vision-language models can distinguish useful visual evidence from incidental image context in lexical judgments.<n>We find that real-image contexts do not yield consistent gains and often hurt alignment with human ratings.<n>Our findings suggest that current instruction-tuned VLMs need better calibration of when visual context should inform lexical judgments.
  arXiv  Detail & Related papers  (2026-05-26T17:24:59Z)
• Seeing to Act, Prompting to Specify: A Bayesian Factorization of Vision Language Action Policy [59.44168425139687]
  BayesVLA is a Bayesian factorization that decomposes the policy into a visual-action prior, supporting seeing-to-act, and a language-conditioned likelihood, enabling prompt-to-specify.<n>Experiments show superior generalization to unseen instructions, objects, and environments compared to existing methods.
  arXiv  Detail & Related papers  (2025-12-12T01:59:23Z)
• Context Matters: Learning Global Semantics via Object-Centric Representation [8.195437248815802]
  Vision models have yet to exhibit comparable progress in in-context learning.<n>We argue that this gap could stem from the lack of semantic and contextual guidance in current vision transformer (ViT) training schemes.<n>We propose to directly model "object" as the visual equivalence of "word," pushing the model to learn the global context and semantics among visual elements.
  arXiv  Detail & Related papers  (2025-10-07T08:33:36Z)
• Talking Point based Ideological Discourse Analysis in News Events [62.18747509565779]
  We propose a framework motivated by the theory of ideological discourse analysis to analyze news articles related to real-world events.<n>Our framework represents the news articles using a relational structure - talking points, which captures the interaction between entities, their roles, and media frames along with a topic of discussion.<n>We evaluate our framework's ability to generate these perspectives through automated tasks - ideology and partisan classification tasks, supplemented by human validation.
  arXiv  Detail & Related papers  (2025-04-10T02:52:34Z)
• Teaching VLMs to Localize Specific Objects from In-context Examples [56.797110842152]
  We find that present-day Vision-Language Models (VLMs) lack a fundamental cognitive ability: learning to localize specific objects in a scene by taking into account the context.<n>This work is the first to explore and benchmark personalized few-shot localization for VLMs.
  arXiv  Detail & Related papers  (2024-11-20T13:34:22Z)
• SADL: An Effective In-Context Learning Method for Compositional Visual QA [22.0603596548686]
  Large vision-language models (LVLMs) offer a novel capability for performing in-context learning (ICL) in Visual QA. This paper introduces SADL, a new visual-linguistic prompting framework for the task.
  arXiv  Detail & Related papers  (2024-07-02T06:41:39Z)
• Integrating Self-supervised Speech Model with Pseudo Word-level Targets from Visually-grounded Speech Model [57.78191634042409]
  We propose Pseudo-Word HuBERT (PW-HuBERT), a framework that integrates pseudo word-level targets into the training process. Our experimental results on four spoken language understanding (SLU) benchmarks suggest the superiority of our model in capturing semantic information.
  arXiv  Detail & Related papers  (2024-02-08T16:55:21Z)
• Simple Linguistic Inferences of Large Language Models (LLMs): Blind Spots and Blinds [59.71218039095155]
  We evaluate language understanding capacities on simple inference tasks that most humans find trivial. We target (i) grammatically-specified entailments, (ii) premises with evidential adverbs of uncertainty, and (iii) monotonicity entailments. The models exhibit moderate to low performance on these evaluation sets.
  arXiv  Detail & Related papers  (2023-05-24T06:41:09Z)
• Perceptual Grouping in Contrastive Vision-Language Models [59.1542019031645]
  We show how vision-language models are able to understand where objects reside within an image and group together visually related parts of the imagery. We propose a minimal set of modifications that results in models that uniquely learn both semantic and spatial information.
  arXiv  Detail & Related papers  (2022-10-18T17:01:35Z)
• Language Models as Zero-shot Visual Semantic Learners [0.618778092044887]
  We propose a Visual Se-mantic Embedding Probe (VSEP) to probe the semantic information of contextualized word embeddings. The VSEP with contextual representations can distinguish word-level object representations in complicated scenes as a compositional zero-shot learner. We find that contextual representations in language mod-els outperform static word embeddings, when the compositional chain of object is short.
  arXiv  Detail & Related papers  (2021-07-26T08:22:55Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.