"I See What You Did There": Can Large Vision-Language Models Understand Multimodal Puns?

• URL: http://arxiv.org/abs/2604.05930v1
• Date: Tue, 07 Apr 2026 14:31:32 GMT
• Title: "I See What You Did There": Can Large Vision-Language Models Understand Multimodal Puns?
• Abstract summary: Puns are a common form of rhetorical wordplay that exploits polysemy and phonetic similarity to create humor.<n>Although Vision-Language Models are widely used in multimodal understanding and generation, their ability to understand puns has not been systematically studied.<n>We introduce MultiPun, a dataset comprising diverse types of puns alongside adversarial non-pun distractors.<n>Our evaluation reveals that most models struggle to distinguish genuine puns from these distractors.
• Score: 52.182269580349605
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Puns are a common form of rhetorical wordplay that exploits polysemy and phonetic similarity to create humor. In multimodal puns, visual and textual elements synergize to ground the literal sense and evoke the figurative meaning simultaneously. Although Vision-Language Models (VLMs) are widely used in multimodal understanding and generation, their ability to understand puns has not been systematically studied due to a scarcity of rigorous benchmarks. To address this, we first propose a multimodal pun generation pipeline. We then introduce MultiPun, a dataset comprising diverse types of puns alongside adversarial non-pun distractors. Our evaluation reveals that most models struggle to distinguish genuine puns from these distractors. Moreover, we propose both prompt-level and model-level strategies to enhance pun comprehension, with an average improvement of 16.5% in F1 scores. Our findings provide valuable insights for developing future VLMs that master the subtleties of human-like humor via cross-modal reasoning.

Related papers

• Words at Play: Benchmarking Audio Pun Understanding in Large Audio-Language Models [16.83865095484227]
  Puns represent a typical linguistic phenomenon that exploits polysemy and phonetic ambiguity to generate humour.<n>Within pun research, audio plays a central role in human communication except text and images.<n>We present APUN-Bench, the first benchmark dedicated to evaluating large audio language models (LALMs) on audio pun understanding.
  arXiv  Detail & Related papers  (2026-03-19T09:39:52Z)
• Reasoning Beyond Literal: Cross-style Multimodal Reasoning for Figurative Language Understanding [16.589552325416015]
  Vision-language models (VLMs) have demonstrated strong reasoning abilities in literal multimodal tasks.<n> figurative language, such as sarcasm, humor, and metaphor, remains a significant challenge.<n>We propose a three-step framework for developing efficient multimodal reasoning models.
  arXiv  Detail & Related papers  (2026-01-23T22:01:39Z)
• ThinkMorph: Emergent Properties in Multimodal Interleaved Chain-of-Thought Reasoning [76.95203056566191]
  Multimodal reasoning requires iterative coordination between language and vision, yet it remains unclear what constitutes a meaningful interleaved chain of thought.<n>We build ThinkMorph, a unified model fine-tuned on approximately 24K high-quality interleaved reasoning traces spanning tasks with varying visual engagement.<n>ThinkMorph learns to generate progressive text-image reasoning steps that concretely manipulate visual content while maintaining coherent verbal logic.
  arXiv  Detail & Related papers  (2025-10-30T17:51:38Z)
• Pun Unintended: LLMs and the Illusion of Humor Understanding [50.29407048003165]
  Puns are a form of humorous wordplay that exploits polysemy and phonetic similarity.<n>Our contributions include comprehensive and nuanced pun detection benchmarks, human evaluation across recent LLMs, and an analysis of the robustness challenges these models face in processing puns.
  arXiv  Detail & Related papers  (2025-09-15T17:22:30Z)
• Can visual language models resolve textual ambiguity with visual cues? Let visual puns tell you! [14.84123301554462]
  We present UNPIE, a novel benchmark designed to assess the impact of multimodal inputs in resolving lexical ambiguities. Our dataset includes 1,000 puns, each accompanied by an image that explains both meanings. The results indicate that various Socratic Models and Visual-Language Models improve over the text-only models when given visual context.
  arXiv  Detail & Related papers  (2024-10-01T19:32:57Z)
• "A good pun is its own reword": Can Large Language Models Understand Puns? [9.541689402830642]
  Puns play a vital role in academic research due to their distinct structure and clear definition. The understanding of puns in large language models (LLMs) has not been thoroughly examined.
  arXiv  Detail & Related papers  (2024-04-21T09:42:05Z)
• MMoE: Enhancing Multimodal Models with Mixtures of Multimodal Interaction Experts [92.76662894585809]
  We introduce an approach to enhance multimodal models, which we call Multimodal Mixtures of Experts (MMoE) MMoE is able to be applied to various types of models to gain improvement.
  arXiv  Detail & Related papers  (2023-11-16T05:31:21Z)
• Exploiting BERT For Multimodal Target SentimentClassification Through Input Space Translation [75.82110684355979]
  We introduce a two-stream model that translates images in input space using an object-aware transformer. We then leverage the translation to construct an auxiliary sentence that provides multimodal information to a language model. We achieve state-of-the-art performance on two multimodal Twitter datasets.
  arXiv  Detail & Related papers  (2021-08-03T18:02:38Z)
• "The Boating Store Had Its Best Sail Ever": Pronunciation-attentive Contextualized Pun Recognition [80.59427655743092]
  We propose Pronunciation-attentive Contextualized Pun Recognition (PCPR) to perceive human humor. PCPR derives contextualized representation for each word in a sentence by capturing the association between the surrounding context and its corresponding phonetic symbols. Results demonstrate that the proposed approach significantly outperforms the state-of-the-art methods in pun detection and location tasks.
  arXiv  Detail & Related papers  (2020-04-29T20:12:20Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.