M-QUEST -- Meme Question-Understanding Evaluation on Semantics and Toxicity

• URL: http://arxiv.org/abs/2603.03315v1
• Date: Mon, 09 Feb 2026 16:56:39 GMT
• Title: M-QUEST -- Meme Question-Understanding Evaluation on Semantics and Toxicity
• Abstract summary: We present a semantic framework and a corresponding benchmark for automatic knowledge extraction from memes.<n>The framework guides a semi-automatic process of generating a benchmark with commonsense question-answer pairs about meme toxicity assessment.<n>The resulting benchmark M-QUEST consists of 609 question-answer pairs for 307 memes.
• Score: 10.944605467795848
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Internet memes are a powerful form of online communication, yet their nature and reliance on commonsense knowledge make toxicity detection challenging. Identifying key features for meme interpretation and understanding, is a crucial task. Previous work has been focused on some elements contributing to the meaning, such as the Textual dimension via OCR, the Visual dimension via object recognition, upper layers of meaning like the Emotional dimension, Toxicity detection via proxy variables, such as hate speech detection, and sentiment analysis. Nevertheless, there is still a lack of an overall architecture able to formally identify elements contributing to the meaning of a meme, and be used in the sense-making process. In this work, we present a semantic framework and a corresponding benchmark for automatic knowledge extraction from memes. First, we identify the necessary dimensions to understand and interpret a meme: Textual material, Visual material, Scene, Background Knowledge, Emotion, Semiotic Projection, Analogical Mapping, Overall Intent, Target Community, and Toxicity Assessment. Second, the framework guides a semi-automatic process of generating a benchmark with commonsense question-answer pairs about meme toxicity assessment and its underlying reason. The resulting benchmark M-QUEST consists of 609 question-answer pairs for 307 memes. Thirdly, we evaluate eight open-source large language models on their ability to correctly solve M-QUEST. Our results show that current models' commonsense reasoning capabilities for toxic meme interpretation vary depending on the dimension and architecture. Models with instruction tuning and reasoning capabilities significantly outperform the others, though pragmatic inference questions remain challenging. We release code, benchmark, and prompts to support future research intersecting multimodal content safety and commonsense reasoning.

Related papers

• Deeply Interleaved Text-Image Contexts for Multimodal LLMs Assessment [57.41731607488101]
  TIC-Bench is designed to evaluate the capability of models to integrate text-image clues and recover the ground truth facts within deeply interleaved contexts.<n>We evaluated 10 state-of-the-art MLLMs and observed a substantial performance gap compared to human experts.
  arXiv  Detail & Related papers  (2026-09-02T13:19:51Z)
• Multimodal QUD: Inquisitive Questions from Scientific Figures [63.41049609329304]
  We generate inquisitive questions that reach the depth of questions humans generate when engaging with scientific papers.<n>We extend the linguistic theory of Questions Under Discussion (QUD) from being text-only to multimodal.<n>We show that fine-tuning a VLM on MQUD shifts the model from generating generic low-level visual questions to content-specific grounding.
  arXiv  Detail & Related papers  (2026-04-26T14:25:23Z)
• I Came, I Saw, I Explained: Benchmarking Multimodal LLMs on Figurative Meaning in Memes [53.717874073948984]
  We evaluate eight state-of-the-art generative large language models (MLLMs) on their ability to detect and explain six types of figurative meaning.<n>Our findings indicate that all models exhibit a strong bias to associate a meme with figurative meaning, even when no such meaning is present.
  arXiv  Detail & Related papers  (2026-03-24T14:02:48Z)
• Can Unified Generation and Understanding Models Maintain Semantic Equivalence Across Different Output Modalities? [61.533560295383786]
  Unified Multimodal Large Language Models (U-MLLMs) integrate understanding and generation within a single architecture.<n>We observe that U-MLLMs fail to maintain semantic equivalence when required to render the same results in the image modality.<n>We introduce VGUBench, a framework to decouple reasoning logic from generation fidelity.
  arXiv  Detail & Related papers  (2026-02-27T06:23:56Z)
• Meme Similarity and Emotion Detection using Multimodal Analysis [0.0]
  This study employs a multimodal methodological approach, analyzing both the visual and textual elements of memes.<n>We extract low-level visual features and high-level semantic features to identify similar meme pairs.<n>Results indicate that anger and joy are the dominant emotions in memes, with motivational memes eliciting stronger emotional responses.
  arXiv  Detail & Related papers  (2025-03-21T19:07:16Z)
• Just KIDDIN: Knowledge Infusion and Distillation for Detection of INdecent Memes [8.42736066868944]
  We propose a novel framework that integrates Knowledge Distillation (KD) from Large Visual Language Models (LVLMs) and knowledge infusion to enhance the performance of toxicity detection in hateful memes.<n>Our approach extracts sub-knowledge graphs from ConceptNet, a large-scale commonsense Knowledge Graph (KG) to be infused within a compact VLM framework.<n> Experimental results from our study on two hate speech benchmark datasets demonstrate superior performance over the state-of-the-art baselines.
  arXiv  Detail & Related papers  (2024-11-19T02:39:28Z)
• Toxic Memes: A Survey of Computational Perspectives on the Detection and Explanation of Meme Toxicities [0.26624014064407714]
  Studies on the computational analyses of toxic memes have significantly grown over the past five years. The only three surveys on computational toxic meme analysis cover only work published until 2022. Our work fills this gap by surveying content-based computational perspectives on toxic memes.
  arXiv  Detail & Related papers  (2024-06-11T15:22:48Z)
• MemeMQA: Multimodal Question Answering for Memes via Rationale-Based Inferencing [53.30190591805432]
  We introduce MemeMQA, a multimodal question-answering framework to solicit accurate responses to structured questions. We also propose ARSENAL, a novel two-stage multimodal framework to address MemeMQA.
  arXiv  Detail & Related papers  (2024-05-18T07:44:41Z)
• SOK-Bench: A Situated Video Reasoning Benchmark with Aligned Open-World Knowledge [60.76719375410635]
  We propose a new benchmark (SOK-Bench) consisting of 44K questions and 10K situations with instance-level annotations depicted in the videos. The reasoning process is required to understand and apply situated knowledge and general knowledge for problem-solving. We generate associated question-answer pairs and reasoning processes, finally followed by manual reviews for quality assurance.
  arXiv  Detail & Related papers  (2024-05-15T21:55:31Z)
• DiPlomat: A Dialogue Dataset for Situated Pragmatic Reasoning [89.92601337474954]
  Pragmatic reasoning plays a pivotal role in deciphering implicit meanings that frequently arise in real-life conversations. We introduce a novel challenge, DiPlomat, aiming at benchmarking machines' capabilities on pragmatic reasoning and situated conversational understanding.
  arXiv  Detail & Related papers  (2023-06-15T10:41:23Z)
• What do you MEME? Generating Explanations for Visual Semantic Role Labelling in Memes [42.357272117919464]
  We introduce a novel task - EXCLAIM, generating explanations for visual semantic role labeling in memes. To this end, we curate ExHVV, a novel dataset that offers natural language explanations of connotative roles for three types of entities. We also posit LUMEN, a novel multimodal, multi-task learning framework that endeavors to address EXCLAIM optimally.
  arXiv  Detail & Related papers  (2022-12-01T18:21:36Z)
• A computational model implementing subjectivity with the 'Room Theory'. The case of detecting Emotion from Text [68.8204255655161]
  This work introduces a new method to consider subjectivity and general context dependency in text analysis. By using similarity measure between words, we are able to extract the relative relevance of the elements in the benchmark. This method could be applied to all the cases where evaluating subjectivity is relevant to understand the relative value or meaning of a text.
  arXiv  Detail & Related papers  (2020-05-12T21:26:04Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.