PunchBench: Benchmarking MLLMs in Multimodal Punchline Comprehension

• URL: http://arxiv.org/abs/2412.11906v2
• Date: Tue, 17 Jun 2025 13:33:58 GMT
• Title: PunchBench: Benchmarking MLLMs in Multimodal Punchline Comprehension
• Authors: Kun Ouyang, Yuanxin Liu, Shicheng Li, Yi Liu, Hao Zhou, Fandong Meng, Jie Zhou, Xu Sun,
• Abstract summary: We introduce a multimodal textbfPunchline comprehension textbfPunchBenchmark, named textbfPunchBench, for accurate and comprehensive evaluation of punchline comprehension.<n>To enhance the evaluation accuracy, we generate synonymous and antonymous captions by modifying original captions.<n>On this basis, we conduct extensive evaluations and reveal a significant gap between state-of-the-art MLLMs and humans in punchline comprehension.
• Score: 69.73137587705646
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Multimodal punchlines, which involve humor or sarcasm conveyed in image-caption pairs, are a popular way of communication on online multimedia platforms. With the rapid development of multimodal large language models (MLLMs), it is essential to assess their ability to effectively comprehend these punchlines. However, existing benchmarks on punchline comprehension suffer from three major limitations: 1) language shortcuts that allow models to solely rely on text, 2) lack of question diversity, and 3) narrow focus on a specific domain of multimodal content (e.g., cartoon). To address these limitations, we introduce a multimodal \textbf{Punch}line comprehension \textbf{Bench}mark, named \textbf{PunchBench}, which is tailored for accurate and comprehensive evaluation of punchline comprehension. To enhance the evaluation accuracy, we generate synonymous and antonymous captions by modifying original captions, which mitigates the impact of shortcuts in the captions. To provide a comprehensive evaluation, PunchBench incorporates diverse question formats and image-captions from various domains. On this basis, we conduct extensive evaluations and reveal a significant gap between state-of-the-art MLLMs and humans in punchline comprehension. To improve punchline comprehension, we propose Simple-to-Complex Chain-of-Question (SC-CoQ) strategy, enabling the models to incrementally address complicated questions by first mastering simple ones. SC-CoQ effectively enhances the performance of various MLLMs on PunchBench, surpassing in-context learning and chain-of-thought.

Related papers

• Q-BERT4Rec: Quantized Semantic-ID Representation Learning for Multimodal Recommendation [5.699357781063521]
  We propose Q-Bert4Rec, a sequential recommendation framework that unifies semantic representation and quantized modeling.<n>We validate our model on public Amazon benchmarks and demonstrate that Q-Bert4Rec significantly outperforms many strong existing methods.<n>Our source code will be publicly available on GitHub after publishing.
  arXiv  Detail & Related papers  (2025-12-02T07:06:44Z)
• SEPS: Semantic-enhanced Patch Slimming Framework for fine-grained cross-modal alignment [8.657941729790599]
  We introduce the Semantic-Enhanced Patch Slimming (SEPS) framework, which systematically addresses patch redundancy and ambiguity.<n>Our approach employs a two-stage mechanism to integrate unified semantics from both dense and sparse texts, enabling the identification of salient visual patches.<n>Experiments on Flickr30K and MS-COCO datasets validate that SEPS achieves superior performance.
  arXiv  Detail & Related papers  (2025-11-03T09:41:32Z)
• MOMENTS: A Comprehensive Multimodal Benchmark for Theory of Mind [41.188841829937466]
  MoMentS (Multimodal Mental States) is a benchmark for building socially intelligent multimodal agents.<n>MoMentS includes over 2,300 multiple-choice questions spanning seven distinct ToM categories.<n>We evaluate several MLLMs and find that although vision generally improves performance, models still struggle to integrate it effectively.
  arXiv  Detail & Related papers  (2025-07-06T15:06:30Z)
• Seeing is Understanding: Unlocking Causal Attention into Modality-Mutual Attention for Multimodal LLMs [7.03771340666549]
  Vision-language misalignment in Multimodal Large Language Models (MLLMs) is a critical challenge. We propose MapleLeaf AKI, a novel MLLM that unlocks causal attention into modality-mutual attention (MMA) to enable image tokens to attend to text tokens. Our MMA design is intended to be generic, allowing for application across various modalities, and scalable to accommodate diverse multimodal scenarios.
  arXiv  Detail & Related papers  (2025-03-04T13:18:33Z)
• Knowing Where to Focus: Attention-Guided Alignment for Text-based Person Search [64.15205542003056]
  We introduce Attention-Guided Alignment (AGA) framework featuring two innovative components: Attention-Guided Mask (AGM) Modeling and Text Enrichment Module (TEM) AGA achieves new state-of-the-art results with Rank-1 accuracy reaching 78.36%, 67.31%, and 67.4% on CUHK-PEDES, ICFG-PEDES, and RSTP, respectively.
  arXiv  Detail & Related papers  (2024-12-19T17:51:49Z)
• Order Matters: Exploring Order Sensitivity in Multimodal Large Language Models [15.622219099903067]
  We find that changing the order of multimodal input can cause the model's performance to fluctuate between advanced performance and random guessing. This phenomenon exists in both single-modality (text-only or image-only) and mixed-modality (image-text-pair) contexts. We propose a new metric, Position-Invariant Accuracy (PIA), to address order bias in MLLM evaluation.
  arXiv  Detail & Related papers  (2024-10-22T13:05:11Z)
• MC-Bench: A Benchmark for Multi-Context Visual Grounding in the Era of MLLMs [61.56904387052982]
  This paper proposes a new visual grounding task called multi-context visual grounding.<n>It aims to localize instances of interest across multiple images based on open-ended text prompts.<n>We benchmark over 20 state-of-the-art MLLMs and foundation models with potential multi-context visual grounding capabilities.
  arXiv  Detail & Related papers  (2024-10-16T07:52:57Z)
• Parrot: Multilingual Visual Instruction Tuning [66.65963606552839]
  Existing methods mainly focus on aligning vision encoders with Multimodal Large Language Models (MLLMs) We introduce Parrot, a novel method that utilizes textual guidance to drive visual token alignment at the language level. Our method not only demonstrates state-of-the-art performance on multilingual MMBench and MMMB, but also excels across a broad range of multimodal tasks.
  arXiv  Detail & Related papers  (2024-06-04T17:56:28Z)
• OTSeg: Multi-prompt Sinkhorn Attention for Zero-Shot Semantic Segmentation [57.84148140637513]
  Multi-Prompts Sinkhorn Attention (MPSA) effectively replaces cross-attention mechanisms within Transformer framework in multimodal settings. OTSeg achieves state-of-the-art (SOTA) performance with significant gains on Zero-Shot Semantic (ZS3) tasks.
  arXiv  Detail & Related papers  (2024-03-21T07:15:37Z)
• Mixture-of-Prompt-Experts for Multi-modal Semantic Understanding [7.329728566839757]
  We propose Mixture-of-Prompt-Experts with Block-Aware Prompt Fusion (MoPE-BAF) MoPE-BAF is a novel multi-modal soft prompt framework based on the unified vision-language model (VLM)
  arXiv  Detail & Related papers  (2024-03-17T19:12:26Z)
• Browse and Concentrate: Comprehending Multimodal Content via prior-LLM Context Fusion [70.9767518332692]
  Multimodal Large Language Models (MLLMs) that incorporate LLMs with pre-trained vision models have recently demonstrated impressive performance across diverse vision-language tasks. However, they fall short to comprehend context involving multiple images. We propose a two phase paradigm, browse-and-concentrate, to enable in-depth multimodal context fusion.
  arXiv  Detail & Related papers  (2024-02-19T14:59:07Z)
• DialCLIP: Empowering CLIP as Multi-Modal Dialog Retriever [83.33209603041013]
  We propose a parameter-efficient prompt-tuning method named DialCLIP for multi-modal dialog retrieval. Our approach introduces a multi-modal context generator to learn context features which are distilled into prompts within the pre-trained vision-language model CLIP. To facilitate various types of retrieval, we also design multiple experts to learn mappings from CLIP outputs to multi-modal representation space.
  arXiv  Detail & Related papers  (2024-01-02T07:40:12Z)
• SCMM: Calibrating Cross-modal Representations for Text-Based Person Search [43.17325362167387]
  Text-Based Person Search (TBPS) is a crucial task in the Internet of Things (IoT) domain.<n>For cross-modal TBPS tasks, it is critical to obtain well-distributed representation in the common space.<n>We present Sew embedding and Masked Modeling (SCMM) that calibrates cross-modal representations by learning compact and well-aligned embeddings.
  arXiv  Detail & Related papers  (2023-04-05T07:50:16Z)
• MaPLe: Multi-modal Prompt Learning [54.96069171726668]
  We propose Multi-modal Prompt Learning (MaPLe) for both vision and language branches to improve alignment between the vision and language representations. Compared with the state-of-the-art method Co-CoOp, MaPLe exhibits favorable performance and achieves an absolute gain of 3.45% on novel classes.
  arXiv  Detail & Related papers  (2022-10-06T17:59:56Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.