Towards Comprehensive Detection of Chinese Harmful Memes

• URL: http://arxiv.org/abs/2410.02378v1
• Date: Thu, 3 Oct 2024 10:51:02 GMT
• Title: Towards Comprehensive Detection of Chinese Harmful Memes
• Abstract summary: ToxiCN MM, the first Chinese harmful meme dataset, consists of 12,000 samples with fine-grained annotations for various meme types. We propose a baseline detector, Multimodal Knowledge Enhancement (MKE), incorporating contextual information of meme content. The experimental results indicate that detecting Chinese harmful memes is challenging for existing models.
• Score: 22.738223098517096
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: This paper has been accepted in the NeurIPS 2024 D & B Track. Harmful memes have proliferated on the Chinese Internet, while research on detecting Chinese harmful memes significantly lags behind due to the absence of reliable datasets and effective detectors. To this end, we focus on the comprehensive detection of Chinese harmful memes. We construct ToxiCN MM, the first Chinese harmful meme dataset, which consists of 12,000 samples with fine-grained annotations for various meme types. Additionally, we propose a baseline detector, Multimodal Knowledge Enhancement (MKE), incorporating contextual information of meme content generated by the LLM to enhance the understanding of Chinese memes. During the evaluation phase, we conduct extensive quantitative experiments and qualitative analyses on multiple baselines, including LLMs and our MKE. The experimental results indicate that detecting Chinese harmful memes is challenging for existing models while demonstrating the effectiveness of MKE. The resources for this paper are available at https://github.com/DUT-lujunyu/ToxiCN_MM.

Related papers

• CHILLGuard: Towards Fine-Grained Chinese LLM Safety Guardrail with Scalable Data Construction and Model-aware Preference Alignment [55.74660714572696]
  Malicious content generated from large language models (LLMs) could pose severe safety risks and ethical concerns.<n>Existing LLM safety guardrails excel in English or multilingual settings, but lack adaptation to Chinese-specific regulatory policies, cultural context and linguistic nuances.<n>We introduce a 5-macro, 31-micro category fine-grained risk taxonomy for Chinese scenarios, and build CHILLGuard: a dedicated Chinese LLM content safety guardrail.
  arXiv  Detail & Related papers  (2026-06-13T16:57:51Z)
• Distinguishing Right from Wrong in Debates: Attribution Analysis of Chinese Harmful Memes [33.60719903789302]
  We focus on the interpretable detection of Chinese harmful memes by constructing the first Chinese harmful meme explanation dataset, Ex-ToxiCN-MM.<n>This dataset offers opposing interpretations, categorized as "harmful" and "non-harmful", for each meme.<n>We have developed a comprehensive attribution analysis framework, RIKE, which includes an Attribution Knowledge Enhancement module (AKE) and a Relative Intent Reasoning module (RIR)
  arXiv  Detail & Related papers  (2026-05-23T02:06:34Z)
• Are Large Language Models Chronically Online Surfers? A Dataset for Chinese Internet Meme Explanation [4.797464143831952]
  Large language models (LLMs) are trained on vast amounts of text from the Internet, but do they truly understand the viral content that rapidly spreads online?<n>ChiME is a dataset for CHinese Internet Meme Explanation.<n>The dataset comprises popular phrase-based memes from the Chinese Internet, annotated with detailed information on their meaning, origin, example sentences, types, etc.
  arXiv  Detail & Related papers  (2025-10-01T06:41:46Z)
• MemeMind: A Large-Scale Multimodal Dataset with Chain-of-Thought Reasoning for Harmful Meme Detection [4.09109557328609]
  Harmful memes pose significant challenges for automated detection due to implicit semantics and complex multimodal interactions.<n>MemeMind is a novel dataset featuring scientifically rigorous standards, large scale, diversity, bilingual support (Chinese and English), and detailed Chain-of-Thought (CoT) annotations.<n>We propose an innovative detection framework, MemeGuard, which effectively integrates multimodal information with reasoning process modeling.
  arXiv  Detail & Related papers  (2025-06-15T13:45:30Z)
• ChineseHarm-Bench: A Chinese Harmful Content Detection Benchmark [50.89916747049978]
  Existing resources for harmful content detection are predominantly focused on English, with Chinese datasets remaining scarce and often limited in scope.<n>We present a comprehensive, professionally annotated benchmark for Chinese content harm detection, which covers six representative categories and is constructed entirely from real-world data.<n>We propose a knowledge-augmented baseline that integrates both human-annotated knowledge rules and implicit knowledge from large language models, enabling smaller models to achieve performance comparable to state-of-the-art LLMs.
  arXiv  Detail & Related papers  (2025-06-12T17:57:05Z)
• Detecting Harmful Memes with Decoupled Understanding and Guided CoT Reasoning [26.546646866501735]
  We introduce U-CoT+, a novel framework for harmful meme detection.<n>We first develop a high-fidelity meme-to-text pipeline that converts visual memes into detail-preserving textual descriptions.<n>This design decouples meme interpretation from meme classification, thus avoiding immediate reasoning over complex raw visual content.
  arXiv  Detail & Related papers  (2025-06-10T06:10:45Z)
• Exploring Multimodal Challenges in Toxic Chinese Detection: Taxonomy, Benchmark, and Findings [48.841514684592426]
  We highlight the multimodal nature of Chinese language as a key challenge for deploying language models in toxic Chinese detection.<n>First, we propose a taxonomy of 3 perturbation strategies and 8 specific approaches in toxic Chinese content.<n>Then, we curate a dataset based on this taxonomy, and benchmark 9 SOTA LLMs (from both the US and China) to assess if they can detect perturbed toxic Chinese text.
  arXiv  Detail & Related papers  (2025-05-30T08:32:45Z)
• MemeBLIP2: A novel lightweight multimodal system to detect harmful memes [10.174106475035689]
  We introduce MemeBLIP2, a light weight multimodal system that detects harmful memes by combining image and text features effectively. We build on previous studies by adding modules that align image and text representations into a shared space and fuse them for better classification. The results show that MemeBLIP2 can capture subtle cues in both modalities, even in cases with ironic or culturally specific content.
  arXiv  Detail & Related papers  (2025-04-29T23:41:06Z)
• Improving Multimodal Hateful Meme Detection Exploiting LMM-Generated Knowledge [11.801596051153725]
  detecting hateful content in memes has emerged as a task of critical importance. We propose to address the task leveraging knowledge encoded in powerful Large Multimodal Models (LMM) Specifically, we propose to exploit LMMs in a two-fold manner. First, by extracting knowledge oriented to the hateful meme detection task in order to build strong meme representations.
  arXiv  Detail & Related papers  (2025-04-14T06:23:44Z)
• Towards Low-Resource Harmful Meme Detection with LMM Agents [13.688955830843973]
  We propose an agency-driven framework for low-resource harmful meme detection. We first retrieve relative memes with annotations to leverage label information as auxiliary signals for the LMM agent. We elicit knowledge-revising behavior within the LMM agent to derive well-generalized insights into meme harmfulness.
  arXiv  Detail & Related papers  (2024-11-08T07:43:15Z)
• OSPC: Detecting Harmful Memes with Large Language Model as a Catalyst [23.806625629458946]
  This study presents a novel approach to detecting harmful memes, particularly within the multicultural and multilingual context of Singapore. Our methodology integrates image captioning, Optical Character Recognition (OCR), and Large Language Model (LLM) analysis. Our framework achieves top-1 at the public leaderboard of the Online Safety Prize Challenge hosted by AI Singapore.
  arXiv  Detail & Related papers  (2024-06-14T07:28:02Z)
• MemeMQA: Multimodal Question Answering for Memes via Rationale-Based Inferencing [53.30190591805432]
  We introduce MemeMQA, a multimodal question-answering framework to solicit accurate responses to structured questions. We also propose ARSENAL, a novel two-stage multimodal framework to address MemeMQA.
  arXiv  Detail & Related papers  (2024-05-18T07:44:41Z)
• A Chinese Dataset for Evaluating the Safeguards in Large Language Models [46.43476815725323]
  Large language models (LLMs) can produce harmful responses. This paper introduces a dataset for the safety evaluation of Chinese LLMs. We then extend it to two other scenarios that can be used to better identify false negative and false positive examples.
  arXiv  Detail & Related papers  (2024-02-19T14:56:18Z)
• Meme-ingful Analysis: Enhanced Understanding of Cyberbullying in Memes Through Multimodal Explanations [48.82168723932981]
  We introduce em MultiBully-Ex, the first benchmark dataset for multimodal explanation from code-mixed cyberbullying memes. A Contrastive Language-Image Pretraining (CLIP) approach has been proposed for visual and textual explanation of a meme.
  arXiv  Detail & Related papers  (2024-01-18T11:24:30Z)
• Explainable Multimodal Sentiment Analysis on Bengali Memes [0.0]
  Understanding and interpreting the sentiment underlying memes has become crucial in the age of information. This study employed a multimodal approach using ResNet50 and BanglishBERT and achieved a satisfactory result of 0.71 weighted F1-score.
  arXiv  Detail & Related papers  (2023-12-20T17:15:10Z)
• Beneath the Surface: Unveiling Harmful Memes with Multimodal Reasoning Distilled from Large Language Models [17.617187709968242]
  Existing harmful meme detection approaches only recognize superficial harm-indicative signals in an end-to-end classification manner. We propose a novel generative framework to learn reasonable thoughts from Large Language Models for better multimodal fusion. Our proposed approach achieves superior performance than state-of-the-art methods on the harmful meme detection task.
  arXiv  Detail & Related papers  (2023-12-09T01:59:11Z)
• A Template Is All You Meme [83.05919383106715]
  We release a knowledge base of memes and information found on www.knowyourmeme.com, composed of more than 54,000 images. We hypothesize that meme templates can be used to inject models with the context missing from previous approaches.
  arXiv  Detail & Related papers  (2023-11-11T19:38:14Z)
• DisinfoMeme: A Multimodal Dataset for Detecting Meme Intentionally Spreading Out Disinformation [72.18912216025029]
  We present DisinfoMeme to help detect disinformation memes. The dataset contains memes mined from Reddit covering three current topics: the COVID-19 pandemic, the Black Lives Matter movement, and veganism/vegetarianism.
  arXiv  Detail & Related papers  (2022-05-25T09:54:59Z)
• Detecting and Understanding Harmful Memes: A Survey [48.135415967633676]
  We offer a comprehensive survey with a focus on harmful memes. One interesting finding is that many types of harmful memes are not really studied, e.g., such featuring self-harm and extremism. Another observation is that memes can propagate globally through repackaging in different languages and that they can also be multilingual.
  arXiv  Detail & Related papers  (2022-05-09T13:43:27Z)
• COLD: A Benchmark for Chinese Offensive Language Detection [54.60909500459201]
  We use COLDataset, a Chinese offensive language dataset with 37k annotated sentences. We also propose textscCOLDetector to study output offensiveness of popular Chinese language models. Our resources and analyses are intended to help detoxify the Chinese online communities and evaluate the safety performance of generative language models.
  arXiv  Detail & Related papers  (2022-01-16T11:47:23Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.