Calibrated Ambiguity in Multimodal Language Models: Humans reach for cultural references, while models describe the picture

• URL: http://arxiv.org/abs/2609.12575v1
• Date: Fri, 11 Sep 2026 08:25:55 GMT
• Title: Calibrated Ambiguity in Multimodal Language Models: Humans reach for cultural references, while models describe the picture
• Abstract summary: We operationalise the notion of calibrated ambiguity with a task drawn from the parlour game Dixit.<n>We compare differences in clues generated by human vs multimodal language models, based on a novel coding rubric for calibrated ambiguity.<n>Unlike human clues, AI-generated clues also exhibit cultural flattening; they almost never make reference to culturally-situated knowledge.
• Score: 3.7696009147557867
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Ambiguity is often treated as a bug for AI systems to resolve---but in human communication and culture, ambiguity can also be a generative resource. From humour to politics to art, people express themselves in words and images that are open enough to invite different interpretations, yet constrained enough to be interpretable. We operationalise this notion of calibrated ambiguity with a task drawn from the parlour game Dixit. We compare differences in clues generated by human vs multimodal language models, based on a novel coding rubric for calibrated ambiguity, and find that models consistently exhibit ambiguity collapse (i.e., their outputs are over-specified, leaving no room for multiple legitimate interpretations). Unlike human clues, AI-generated clues also exhibit cultural flattening; they almost never make reference to culturally-situated knowledge, even when prompted to use allusion and figurative language.

Related papers

• BanglaMemeX: Advancing Cultural Metaphoric Image Interpretation in Bangla with a Multimodal Explainable Dataset [2.8823932597429196]
  Internet memes present a challenging setting where meaning emerges from implicit interactions between image, overlaid text, sarcasm, and socio-cultural knowledge.<n>We introduce BanglaMemeX, a culturally grounded multimodal benchmark comprising 3,000 Bangla memes annotated with multi-dimensional labels.<n>Our results highlight the need for culturally-aware multimodal systems capable of grounded reasoning under linguistic and cultural distribution shift.
  arXiv  Detail & Related papers  (2026-09-07T22:27:18Z)
• Which one is banana man? Evaluating vision-language models in multi-turn pragmatic interpretation [1.0195618602298684]
  Iterated reference games present a test case for agents' ability to perform context-sensitive pragmatic reasoning.<n>We tested humans and vision--language models on their ability to identify the intended meaning of descriptions produced in iterated reference games.
  arXiv  Detail & Related papers  (2026-08-30T05:36:41Z)
• IDIOLEX: Unified and Continuous Representations for Idiolectal and Stylistic Variation [88.98544786373212]
  Existing sentence representations primarily encode what a sentence says, rather than how it is expressed.<n>We develop sentence representations that capture style and dialect, decoupled from semantic content.<n>We introduce IDIOLEX, a framework for training models that combines supervision from a sentence's provenance with linguistic features of a sentence's content.
  arXiv  Detail & Related papers  (2026-04-06T14:17:24Z)
• Identifying & Interactively Refining Ambiguous User Goals for Data Visualization Code Generation [48.63200319578052]
  We develop a taxonomy of types of ambiguity that arise in this task and propose metrics to quantify them.<n>Our work also explores how multi-turn dialogue can reduce ambiguity, therefore, improve code accuracy by better matching user goals.
  arXiv  Detail & Related papers  (2025-10-10T13:44:40Z)
• Can visual language models resolve textual ambiguity with visual cues? Let visual puns tell you! [14.84123301554462]
  We present UNPIE, a novel benchmark designed to assess the impact of multimodal inputs in resolving lexical ambiguities. Our dataset includes 1,000 puns, each accompanied by an image that explains both meanings. The results indicate that various Socratic Models and Visual-Language Models improve over the text-only models when given visual context.
  arXiv  Detail & Related papers  (2024-10-01T19:32:57Z)
• IRFL: Image Recognition of Figurative Language [20.472997304393413]
  Figurative forms are often conveyed through multiple modalities (e.g., both text and images) We develop the Image Recognition of Figurative Language dataset. We introduce two novel tasks as a benchmark for multimodal figurative language understanding.
  arXiv  Detail & Related papers  (2023-03-27T17:59:55Z)
• Is the Elephant Flying? Resolving Ambiguities in Text-to-Image Generative Models [64.58271886337826]
  We study ambiguities that arise in text-to-image generative models. We propose a framework to mitigate ambiguities in the prompts given to the systems by soliciting clarifications from the user.
  arXiv  Detail & Related papers  (2022-11-17T17:12:43Z)
• Testing the Ability of Language Models to Interpret Figurative Language [69.59943454934799]
  Figurative and metaphorical language are commonplace in discourse. It remains an open question to what extent modern language models can interpret nonliteral phrases. We introduce Fig-QA, a Winograd-style nonliteral language understanding task.
  arXiv  Detail & Related papers  (2022-04-26T23:42:22Z)
• Iconary: A Pictionary-Based Game for Testing Multimodal Communication with Drawings and Text [70.14613727284741]
  Communicating with humans is challenging for AIs because it requires a shared understanding of the world, complex semantics, and at times multi-modal gestures. We investigate these challenges in the context of Iconary, a collaborative game of drawing and guessing based on Pictionary. We propose models to play Iconary and train them on over 55,000 games between human players.
  arXiv  Detail & Related papers  (2021-12-01T19:41:03Z)
• It's not Rocket Science : Interpreting Figurative Language in Narratives [48.84507467131819]
  We study the interpretation of two non-compositional figurative languages (idioms and similes) Our experiments show that models based solely on pre-trained language models perform substantially worse than humans on these tasks. We additionally propose knowledge-enhanced models, adopting human strategies for interpreting figurative language.
  arXiv  Detail & Related papers  (2021-08-31T21:46:35Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.