OSPC: Detecting Harmful Memes with Large Language Model as a Catalyst

• URL: http://arxiv.org/abs/2406.09779v1
• Date: Fri, 14 Jun 2024 07:28:02 GMT
• Title: OSPC: Detecting Harmful Memes with Large Language Model as a Catalyst
• Abstract summary: This study presents a novel approach to detecting harmful memes, particularly within the multicultural and multilingual context of Singapore. Our methodology integrates image captioning, Optical Character Recognition (OCR), and Large Language Model (LLM) analysis. Our framework achieves top-1 at the public leaderboard of the Online Safety Prize Challenge hosted by AI Singapore.
• Score: 23.806625629458946
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Memes, which rapidly disseminate personal opinions and positions across the internet, also pose significant challenges in propagating social bias and prejudice. This study presents a novel approach to detecting harmful memes, particularly within the multicultural and multilingual context of Singapore. Our methodology integrates image captioning, Optical Character Recognition (OCR), and Large Language Model (LLM) analysis to comprehensively understand and classify harmful memes. Utilizing the BLIP model for image captioning, PP-OCR and TrOCR for text recognition across multiple languages, and the Qwen LLM for nuanced language understanding, our system is capable of identifying harmful content in memes created in English, Chinese, Malay, and Tamil. To enhance the system's performance, we fine-tuned our approach by leveraging additional data labeled using GPT-4V, aiming to distill the understanding capability of GPT-4V for harmful memes to our system. Our framework achieves top-1 at the public leaderboard of the Online Safety Prize Challenge hosted by AI Singapore, with the AUROC as 0.7749 and accuracy as 0.7087, significantly ahead of the other teams. Notably, our approach outperforms previous benchmarks, with FLAVA achieving an AUROC of 0.5695 and VisualBERT an AUROC of 0.5561.

Related papers

• MemeCULT-1K: Benchmarking South Asian Cultural Context and Humor Understanding of Multimodal Models [7.9152408875072275]
  We introduce MemeCULT-1K, a benchmark of 1,000 South Asian memes in Bengali, English, and Hindi.<n>We evaluate thirteen popular Vision Language Models (VLMs) under two settings: meme-only and context-aware.<n>Fine-grained error analysis reveals that closed-source models fail mainly on entity and reference misidentification, while open-source models are bottlenecked by broader cultural knowledge gaps.
  arXiv  Detail & Related papers  (2026-09-01T18:43:03Z)
• AHA-Memes: A Fine-Grained Multimodal Benchmark for Understanding Hate in Arabic Memes [7.015289934068985]
  We introduce AHA-Memes (Arabic HAteful Memes), the first large-scale Arabic hateful meme benchmark with fine-grained, multi-label annotations.<n>The dataset includes 5K manually annotated memes using a taxonomy that captures hate types, i.e., attack strategies.<n>We benchmark text-only, image-only, and late-fusion multimodal models, as well as few-shot in-context learning (ICL) and open- and closed-weight Vision-Language Models (VLMs) under zero-shot and fine-tuning settings.
  arXiv  Detail & Related papers  (2026-07-29T19:04:33Z)
• Adapting Reinforcement Learning with Chain-of-Thought Supervision for Explainable Detection of Hateful and Propagandistic Memes [7.77199523320035]
  Hateful and propagandistic memes exploit the interplay between images and text to convey harmful intent.<n>We propose a reinforcement learning-based post-training method that improves classification performance and reference-based explanation quality.
  arXiv  Detail & Related papers  (2026-06-13T13:51:54Z)
• Can Thinking Models Think to Detect Hateful Memes? [7.77199523320035]
  Thinking-based multimodal large language models (MLLMs) have recently advanced vision-language understanding.<n>We propose a reinforcement learning based post-training framework that improves reasoning in thinking-based MLLMs.<n>Our approach achieves state-of-the-art performance, improving accuracy and F1 by approximately 1 percent and explanation quality by approximately 3 percent.
  arXiv  Detail & Related papers  (2026-03-01T18:41:52Z)
• Lingua-SafetyBench: A Benchmark for Safety Evaluation of Multilingual Vision-Language Models [54.10540442330978]
  Existing benchmarks are typically multilingual but text-only, or multimodal but monolingual.<n>Recent multilingual red-teaming efforts render harmful prompts into images, yet rely heavily on typography-style visuals.<n>We introduce a benchmark of 100,440 harmful image-text pairs across 10 languages, explicitly partitioned into image-dominant and text-dominant subsets.
  arXiv  Detail & Related papers  (2026-01-30T09:18:13Z)
• Improving Methodologies for LLM Evaluations Across Global Languages [19.63570354411416]
  The exercise shows how safety behaviours can vary across languages.<n>It also generated insights for improving multilingual safety evaluations.<n>This work represents an initial step toward a shared framework for multilingual safety testing of advanced AI systems.
  arXiv  Detail & Related papers  (2026-01-22T07:18:08Z)
• Toxicity Red-Teaming: Benchmarking LLM Safety in Singapore's Low-Resource Languages [57.059267233093465]
  Large Language Models (LLMs) have transformed natural language processing, but their safety mechanisms remain under-explored in low-resource, multilingual settings.<n>We introduce textsfSGToxicGuard, a novel dataset and evaluation framework for benchmarking LLM safety in Singapore's diverse linguistic context.<n>We conduct extensive experiments with state-of-the-art multilingual LLMs, and the results uncover critical gaps in their safety guardrails.
  arXiv  Detail & Related papers  (2025-09-18T08:14:34Z)
• Meta CLIP 2: A Worldwide Scaling Recipe [112.4690561863437]
  We present Meta CLIP 2, the first recipe training CLIP from scratch on worldwide web-scale image-text pairs.<n>In zero-shot ImageNet classification, Meta CLIP 2 ViT-H/14 surpasses its English-only counterpart by 0.8% and mSigLIP by 0.7%.
  arXiv  Detail & Related papers  (2025-07-29T17:59:58Z)
• Detecting Harmful Memes with Decoupled Understanding and Guided CoT Reasoning [26.546646866501735]
  We introduce U-CoT+, a novel framework for harmful meme detection.<n>We first develop a high-fidelity meme-to-text pipeline that converts visual memes into detail-preserving textual descriptions.<n>This design decouples meme interpretation from meme classification, thus avoiding immediate reasoning over complex raw visual content.
  arXiv  Detail & Related papers  (2025-06-10T06:10:45Z)
• OMNIGUARD: An Efficient Approach for AI Safety Moderation Across Modalities [54.152681077418805]
  Current detection approaches are fallible, and are particularly susceptible to attacks that exploit mismatched generalizations of model capabilities.<n>We propose OMNIGUARD, an approach for detecting harmful prompts across languages and modalities.<n>Our approach improves harmful prompt classification accuracy by 11.57% over the strongest baseline in a multilingual setting.
  arXiv  Detail & Related papers  (2025-05-29T05:25:27Z)
• Are Vision-Language Models Safe in the Wild? A Meme-Based Benchmark Study [14.308220140623247]
  This study asks: How safe are current vision-language models when confronted with meme images that ordinary users share?<n>We introduce MemeSafetyBench, a benchmark pairing real meme images with both harmful and benign instructions.<n>We find that vision-language models show greater vulnerability to meme-based harmful prompts than to synthetic or typographic images.
  arXiv  Detail & Related papers  (2025-05-21T11:26:40Z)
• CAMU: Context Augmentation for Meme Understanding [9.49890289676001]
  Social media memes are a challenging domain for hate detection because they intertwine visual and textual cues into culturally nuanced messages. We introduce a novel framework, CAMU, which leverages large vision-language models to generate more descriptive captions. Our approach attains high accuracy (0.807) and F1-score (0.806) on the Hateful Memes dataset, at par with the existing SoTA framework.
  arXiv  Detail & Related papers  (2025-04-24T19:27:55Z)
• TULIP: Towards Unified Language-Image Pretraining [60.99500935831526]
  We introduce T, an open-source, drop-in replacement for existing CLIP-like models. Our method leverages generative data augmentation, enhanced image-image and text-text contrastive learning, and image/text reconstruction regularization to learn fine-grained visual features. Our approach, scaling to over 1B parameters, outperforms existing state-of-the-art (SOTA) models across benchmarks.
  arXiv  Detail & Related papers  (2025-03-19T17:58:57Z)
• Detecting Offensive Memes with Social Biases in Singapore Context Using Multimodal Large Language Models [0.0]
  Traditional online content moderation systems struggle to classify modern multimodal means of communication, such as memes. We curate a large collection of 112K memes labeled by GPT-4V for fine-tuning a VLM to classify offensive memes in Singapore context. Our solutions reach 80.62% accuracy and 0.8192 AUROC on a held-out test set, and can greatly aid human in moderating online contents.
  arXiv  Detail & Related papers  (2025-02-25T11:15:49Z)
• Just KIDDIN: Knowledge Infusion and Distillation for Detection of INdecent Memes [8.337745035712311]
  We propose a novel framework that integrates Knowledge Distillation (KD) from Large Visual Language Models (LVLMs) and knowledge infusion to enhance the performance of toxicity detection in hateful memes. Our approach extracts sub-knowledge graphs from ConceptNet, a large-scale commonsense Knowledge Graph (KG) to be infused within a compact VLM framework. Experimental results from our study on two hate speech benchmark datasets demonstrate superior performance over the state-of-the-art baselines.
  arXiv  Detail & Related papers  (2024-11-19T02:39:28Z)
• GPT-4o System Card [211.87336862081963]
  GPT-4o is an autoregressive omni model that accepts as input any combination of text, audio, image, and video. It's trained end-to-end across text, vision, and audio, meaning all inputs and outputs are processed by the same neural network. It matches GPT-4 Turbo performance on text in English and code, with significant improvement on text in non-English languages.
  arXiv  Detail & Related papers  (2024-10-25T17:43:01Z)
• XMeCap: Meme Caption Generation with Sub-Image Adaptability [53.2509590113364]
  Humor, deeply rooted in societal meanings and cultural details, poses a unique challenge for machines. We introduce the textscXMeCap framework, which adopts supervised fine-tuning and reinforcement learning. textscXMeCap achieves an average evaluation score of 75.85 for single-image memes and 66.32 for multi-image memes, outperforming the best baseline by 3.71% and 4.82%, respectively.
  arXiv  Detail & Related papers  (2024-07-24T10:51:46Z)
• OSPC: Artificial VLM Features for Hateful Meme Detection [0.0]
  This paper introduces a solution developed by team 'Baseline' for the AI Singapore Online Safety Prize Challenge. Focusing on computational efficiency and feature engineering, the solution achieved an AUROC of 0.76 and an accuracy of 0.69 on the test dataset.
  arXiv  Detail & Related papers  (2024-07-03T21:35:52Z)
• White-box Multimodal Jailbreaks Against Large Vision-Language Models [61.97578116584653]
  We propose a more comprehensive strategy that jointly attacks both text and image modalities to exploit a broader spectrum of vulnerability within Large Vision-Language Models. Our attack method begins by optimizing an adversarial image prefix from random noise to generate diverse harmful responses in the absence of text input. An adversarial text suffix is integrated and co-optimized with the adversarial image prefix to maximize the probability of eliciting affirmative responses to various harmful instructions.
  arXiv  Detail & Related papers  (2024-05-28T07:13:30Z)
• MemeMQA: Multimodal Question Answering for Memes via Rationale-Based Inferencing [53.30190591805432]
  We introduce MemeMQA, a multimodal question-answering framework to solicit accurate responses to structured questions. We also propose ARSENAL, a novel two-stage multimodal framework to address MemeMQA.
  arXiv  Detail & Related papers  (2024-05-18T07:44:41Z)
• Beneath the Surface: Unveiling Harmful Memes with Multimodal Reasoning Distilled from Large Language Models [17.617187709968242]
  Existing harmful meme detection approaches only recognize superficial harm-indicative signals in an end-to-end classification manner. We propose a novel generative framework to learn reasonable thoughts from Large Language Models for better multimodal fusion. Our proposed approach achieves superior performance than state-of-the-art methods on the harmful meme detection task.
  arXiv  Detail & Related papers  (2023-12-09T01:59:11Z)
• MLLMs-Augmented Visual-Language Representation Learning [70.5293060238008]
  We demonstrate that Multi-modal Large Language Models (MLLMs) can enhance visual-language representation learning. Our approach is simple, utilizing MLLMs to extend multiple diverse captions for each image. We propose "text shearing" to maintain the quality and availability of extended captions.
  arXiv  Detail & Related papers  (2023-11-30T18:05:52Z)
• What do you MEME? Generating Explanations for Visual Semantic Role Labelling in Memes [42.357272117919464]
  We introduce a novel task - EXCLAIM, generating explanations for visual semantic role labeling in memes. To this end, we curate ExHVV, a novel dataset that offers natural language explanations of connotative roles for three types of entities. We also posit LUMEN, a novel multimodal, multi-task learning framework that endeavors to address EXCLAIM optimally.
  arXiv  Detail & Related papers  (2022-12-01T18:21:36Z)
• Caption Enriched Samples for Improving Hateful Memes Detection [78.5136090997431]
  The hateful meme challenge demonstrates the difficulty of determining whether a meme is hateful or not. Both unimodal language models and multimodal vision-language models cannot reach the human level of performance.
  arXiv  Detail & Related papers  (2021-09-22T10:57:51Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.