MIND: A Multi-agent Framework for Zero-shot Harmful Meme Detection

• URL: http://arxiv.org/abs/2507.06908v1
• Date: Wed, 09 Jul 2025 14:46:32 GMT
• Title: MIND: A Multi-agent Framework for Zero-shot Harmful Meme Detection
• Authors: Ziyan Liu, Chunxiao Fan, Haoran Lou, Yuexin Wu, Kaiwei Deng,
• Abstract summary: We propose MIND, a multi-agent framework for zero-shot harmful meme detection that does not rely on annotated data.<n>MIND implements three key strategies: 1) We retrieve similar memes from an unannotated reference set to provide contextual information; 2) We propose a bi-directional insight mechanism to extract a comprehensive understanding of similar memes; and 3) We employ a multi-agent debate mechanism to ensure robust decision-making through reasoned arbitration.
• Score: 3.7336554275205898
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: The rapid expansion of memes on social media has highlighted the urgent need for effective approaches to detect harmful content. However, traditional data-driven approaches struggle to detect new memes due to their evolving nature and the lack of up-to-date annotated data. To address this issue, we propose MIND, a multi-agent framework for zero-shot harmful meme detection that does not rely on annotated data. MIND implements three key strategies: 1) We retrieve similar memes from an unannotated reference set to provide contextual information. 2) We propose a bi-directional insight derivation mechanism to extract a comprehensive understanding of similar memes. 3) We then employ a multi-agent debate mechanism to ensure robust decision-making through reasoned arbitration. Extensive experiments on three meme datasets demonstrate that our proposed framework not only outperforms existing zero-shot approaches but also shows strong generalization across different model architectures and parameter scales, providing a scalable solution for harmful meme detection. The code is available at https://github.com/destroy-lonely/MIND.

Related papers

• Detecting Harmful Memes with Decoupled Understanding and Guided CoT Reasoning [26.546646866501735]
  We introduce U-CoT+, a novel framework for harmful meme detection.<n>We first develop a high-fidelity meme-to-text pipeline that converts visual memes into detail-preserving textual descriptions.<n>This design decouples meme interpretation from meme classification, thus avoiding immediate reasoning over complex raw visual content.
  arXiv  Detail & Related papers  (2025-06-10T06:10:45Z)
• Demystifying Hateful Content: Leveraging Large Multimodal Models for Hateful Meme Detection with Explainable Decisions [4.649093665157263]
  In this paper, we introduce IntMeme, a novel framework that leverages Large Multimodal Models (LMMs) for hateful meme classification with explainable decisions.<n>IntMeme addresses the dual challenges of improving both accuracy and explainability in meme moderation.<n>Our approach addresses the opacity and misclassification issues associated with PT-VLMs, optimizing the use of LMMs for hateful meme detection.
  arXiv  Detail & Related papers  (2025-02-16T10:45:40Z)
• Towards Low-Resource Harmful Meme Detection with LMM Agents [13.688955830843973]
  We propose an agency-driven framework for low-resource harmful meme detection. We first retrieve relative memes with annotations to leverage label information as auxiliary signals for the LMM agent. We elicit knowledge-revising behavior within the LMM agent to derive well-generalized insights into meme harmfulness.
  arXiv  Detail & Related papers  (2024-11-08T07:43:15Z)
• MemeMQA: Multimodal Question Answering for Memes via Rationale-Based Inferencing [53.30190591805432]
  We introduce MemeMQA, a multimodal question-answering framework to solicit accurate responses to structured questions. We also propose ARSENAL, a novel two-stage multimodal framework to address MemeMQA.
  arXiv  Detail & Related papers  (2024-05-18T07:44:41Z)
• Towards Explainable Harmful Meme Detection through Multimodal Debate between Large Language Models [18.181154544563416]
  The age of social media is flooded with Internet memes, necessitating a clear grasp and effective identification of harmful ones. Existing harmful meme detection methods do not present readable explanations that unveil such implicit meaning to support their detection decisions. We propose an explainable approach to detect harmful memes, achieved through reasoning over conflicting rationales from both harmless and harmful positions.
  arXiv  Detail & Related papers  (2024-01-24T08:37:16Z)
• Meme-ingful Analysis: Enhanced Understanding of Cyberbullying in Memes Through Multimodal Explanations [48.82168723932981]
  We introduce em MultiBully-Ex, the first benchmark dataset for multimodal explanation from code-mixed cyberbullying memes. A Contrastive Language-Image Pretraining (CLIP) approach has been proposed for visual and textual explanation of a meme.
  arXiv  Detail & Related papers  (2024-01-18T11:24:30Z)
• Beneath the Surface: Unveiling Harmful Memes with Multimodal Reasoning Distilled from Large Language Models [17.617187709968242]
  Existing harmful meme detection approaches only recognize superficial harm-indicative signals in an end-to-end classification manner. We propose a novel generative framework to learn reasonable thoughts from Large Language Models for better multimodal fusion. Our proposed approach achieves superior performance than state-of-the-art methods on the harmful meme detection task.
  arXiv  Detail & Related papers  (2023-12-09T01:59:11Z)
• A Template Is All You Meme [76.03172165923058]
  We create a knowledge base composed of more than 5,200 meme templates, information about them, and 54,000 examples of template instances.<n>To investigate the semantic signal of meme templates, we show that we can match memes in datasets to base templates contained in our knowledge base with a distance-based lookup.<n>Our examination of meme templates results in state-of-the-art performance for every dataset we consider, paving the way for analysis grounded in templateness.
  arXiv  Detail & Related papers  (2023-11-11T19:38:14Z)
• DisinfoMeme: A Multimodal Dataset for Detecting Meme Intentionally Spreading Out Disinformation [72.18912216025029]
  We present DisinfoMeme to help detect disinformation memes. The dataset contains memes mined from Reddit covering three current topics: the COVID-19 pandemic, the Black Lives Matter movement, and veganism/vegetarianism.
  arXiv  Detail & Related papers  (2022-05-25T09:54:59Z)
• Detecting and Understanding Harmful Memes: A Survey [48.135415967633676]
  We offer a comprehensive survey with a focus on harmful memes. One interesting finding is that many types of harmful memes are not really studied, e.g., such featuring self-harm and extremism. Another observation is that memes can propagate globally through repackaging in different languages and that they can also be multilingual.
  arXiv  Detail & Related papers  (2022-05-09T13:43:27Z)
• Memes in the Wild: Assessing the Generalizability of the Hateful Memes Challenge Dataset [47.65948529524281]
  We collect hateful and non-hateful memes from Pinterest to evaluate out-of-sample performance on models pre-trained on the Facebook dataset. We find that memes in the wild differ in two key aspects: 1) Captions must be extracted via OCR, and 2) Memes are more diverse than traditional memes', including screenshots of conversations or text on a plain background.
  arXiv  Detail & Related papers  (2021-07-09T09:04:05Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.