Context informs pragmatic interpretation in vision-language models

• URL: http://arxiv.org/abs/2511.03908v1
• Date: Wed, 05 Nov 2025 23:14:59 GMT
• Title: Context informs pragmatic interpretation in vision-language models
• Abstract summary: Iterated reference games present a test case for agents' ability to perform context-sensitive pragmatic reasoning.<n>We tested humans and vision-language models on trials from iterated reference games.
• Score: 1.3941150400549018
• License: http://creativecommons.org/licenses/by-sa/4.0/
• Abstract: Iterated reference games - in which players repeatedly pick out novel referents using language - present a test case for agents' ability to perform context-sensitive pragmatic reasoning in multi-turn linguistic environments. We tested humans and vision-language models on trials from iterated reference games, varying the given context in terms of amount, order, and relevance. Without relevant context, models were above chance but substantially worse than humans. However, with relevant context, model performance increased dramatically over trials. Few-shot reference games with abstract referents remain a difficult task for machine learning models.

Related papers

• Which one is banana man? Evaluating vision-language models in multi-turn pragmatic interpretation [1.0195618602298684]
  Iterated reference games present a test case for agents' ability to perform context-sensitive pragmatic reasoning.<n>We tested humans and vision--language models on their ability to identify the intended meaning of descriptions produced in iterated reference games.
  arXiv  Detail & Related papers  (2026-08-30T05:36:41Z)
• Evaluating Large Language Models on Multiword Expressions in Multilingual and Code-Switched Contexts [2.519319150166215]
  This study evaluates how state-of-the-art language models process the ambiguity of potentially idiomatic multiword expressions.<n>We find that large language models, despite their strengths, struggle with nuanced language.
  arXiv  Detail & Related papers  (2025-04-10T16:39:28Z)
• Visually Grounded Language Learning: a review of language games, datasets, tasks, and models [60.2604624857992]
  Many Vision+Language (V+L) tasks have been defined with the aim of creating models that can ground symbols in the visual modality. In this work, we provide a systematic literature review of several tasks and models proposed in the V+L field.
  arXiv  Detail & Related papers  (2023-12-05T02:17:29Z)
• Evaluating Large Language Models on Controlled Generation Tasks [92.64781370921486]
  We present an extensive analysis of various benchmarks including a sentence planning benchmark with different granularities. After comparing large language models against state-of-the-start finetuned smaller models, we present a spectrum showing large language models falling behind, are comparable, or exceed the ability of smaller models.
  arXiv  Detail & Related papers  (2023-10-23T03:48:24Z)
• Quantifying the Plausibility of Context Reliance in Neural Machine Translation [25.29330352252055]
  We introduce Plausibility Evaluation of Context Reliance (PECoRe) PECoRe is an end-to-end interpretability framework designed to quantify context usage in language models' generations. We use pecore to quantify the plausibility of context-aware machine translation models.
  arXiv  Detail & Related papers  (2023-10-02T13:26:43Z)
• RAVEN: In-Context Learning with Retrieval-Augmented Encoder-Decoder Language Models [57.12888828853409]
  RAVEN is a model that combines retrieval-augmented masked language modeling and prefix language modeling. Fusion-in-Context Learning enables the model to leverage more in-context examples without requiring additional training. Our work underscores the potential of retrieval-augmented encoder-decoder language models for in-context learning.
  arXiv  Detail & Related papers  (2023-08-15T17:59:18Z)
• MiQA: A Benchmark for Inference on Metaphorical Questions [5.32836690371986]
  We propose a benchmark to assess the capability of large language models to reason with conventional metaphors. We examine the performance of state-of-the-art pre-trained models on binary-choice tasks.
  arXiv  Detail & Related papers  (2022-10-14T17:46:05Z)
• Testing the Ability of Language Models to Interpret Figurative Language [69.59943454934799]
  Figurative and metaphorical language are commonplace in discourse. It remains an open question to what extent modern language models can interpret nonliteral phrases. We introduce Fig-QA, a Winograd-style nonliteral language understanding task.
  arXiv  Detail & Related papers  (2022-04-26T23:42:22Z)
• Probing Contextual Language Models for Common Ground with Visual Representations [76.05769268286038]
  We design a probing model that evaluates how effective are text-only representations in distinguishing between matching and non-matching visual representations. Our findings show that language representations alone provide a strong signal for retrieving image patches from the correct object categories. Visually grounded language models slightly outperform text-only language models in instance retrieval, but greatly under-perform humans.
  arXiv  Detail & Related papers  (2020-05-01T21:28:28Z)
• How Far are We from Effective Context Modeling? An Exploratory Study on Semantic Parsing in Context [59.13515950353125]
  We present a grammar-based decoding semantic parsing and adapt typical context modeling methods on top of it. We evaluate 13 context modeling methods on two large cross-domain datasets, and our best model achieves state-of-the-art performances.
  arXiv  Detail & Related papers  (2020-02-03T11:28:10Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.