Zero shot VLMs for hate meme detection: Are we there yet?

• URL: http://arxiv.org/abs/2402.12198v2
• Date: Mon, 5 Aug 2024 12:20:49 GMT
• Title: Zero shot VLMs for hate meme detection: Are we there yet?
• Authors: Naquee Rizwan, Paramananda Bhaskar, Mithun Das, Swadhin Satyaprakash Majhi, Punyajoy Saha, Animesh Mukherjee,
• Abstract summary: This study investigates the efficacy of visual language models in handling intricate tasks such as hate meme detection. We observe that large VLMs are still vulnerable for zero-shot hate meme detection.
• Score: 9.970031080934003
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Multimedia content on social media is rapidly evolving, with memes gaining prominence as a distinctive form. Unfortunately, some malicious users exploit memes to target individuals or vulnerable communities, making it imperative to identify and address such instances of hateful memes. Extensive research has been conducted to address this issue by developing hate meme detection models. However, a notable limitation of traditional machine/deep learning models is the requirement for labeled datasets for accurate classification. Recently, the research community has witnessed the emergence of several visual language models that have exhibited outstanding performance across various tasks. In this study, we aim to investigate the efficacy of these visual language models in handling intricate tasks such as hate meme detection. We use various prompt settings to focus on zero-shot classification of hateful/harmful memes. Through our analysis, we observe that large VLMs are still vulnerable for zero-shot hate meme detection.

Related papers

• Meme-ingful Analysis: Enhanced Understanding of Cyberbullying in Memes Through Multimodal Explanations [48.82168723932981]
  We introduce em MultiBully-Ex, the first benchmark dataset for multimodal explanation from code-mixed cyberbullying memes. A Contrastive Language-Image Pretraining (CLIP) approach has been proposed for visual and textual explanation of a meme.
  arXiv  Detail & Related papers  (2024-01-18T11:24:30Z)
• Improving Hateful Meme Detection through Retrieval-Guided Contrastive Learning [13.690436954062015]
  We propose constructing a hatefulness-aware embedding space through retrieval-guided contrastive training. Our approach achieves state-of-the-art performance on the HatefulMemes dataset with an AUROC of 87.0, outperforming much larger fine-tuned large multimodal models.
  arXiv  Detail & Related papers  (2023-11-14T12:14:54Z)
• A Template Is All You Meme [83.05919383106715]
  We release a knowledge base of memes and information found on www.knowyourmeme.com, composed of more than 54,000 images. We hypothesize that meme templates can be used to inject models with the context missing from previous approaches.
  arXiv  Detail & Related papers  (2023-11-11T19:38:14Z)
• DisinfoMeme: A Multimodal Dataset for Detecting Meme Intentionally Spreading Out Disinformation [72.18912216025029]
  We present DisinfoMeme to help detect disinformation memes. The dataset contains memes mined from Reddit covering three current topics: the COVID-19 pandemic, the Black Lives Matter movement, and veganism/vegetarianism.
  arXiv  Detail & Related papers  (2022-05-25T09:54:59Z)
• DISARM: Detecting the Victims Targeted by Harmful Memes [49.12165815990115]
  DISARM is a framework that uses named entity recognition and person identification to detect harmful memes. We show that DISARM significantly outperforms ten unimodal and multimodal systems. It can reduce the relative error rate for harmful target identification by up to 9 points absolute over several strong multimodal rivals.
  arXiv  Detail & Related papers  (2022-05-11T19:14:26Z)
• Detecting and Understanding Harmful Memes: A Survey [48.135415967633676]
  We offer a comprehensive survey with a focus on harmful memes. One interesting finding is that many types of harmful memes are not really studied, e.g., such featuring self-harm and extremism. Another observation is that memes can propagate globally through repackaging in different languages and that they can also be multilingual.
  arXiv  Detail & Related papers  (2022-05-09T13:43:27Z)
• Feels Bad Man: Dissecting Automated Hateful Meme Detection Through the Lens of Facebook's Challenge [10.775419935941008]
  We assess the efficacy of current state-of-the-art multimodal machine learning models toward hateful meme detection. We use two benchmark datasets comprising 12,140 and 10,567 images from 4chan's "Politically Incorrect" board (/pol/) and Facebook's Hateful Memes Challenge dataset. We conduct three experiments to determine the importance of multimodality on classification performance, the influential capacity of fringe Web communities on mainstream social platforms and vice versa.
  arXiv  Detail & Related papers  (2022-02-17T07:52:22Z)
• Detecting Harmful Memes and Their Targets [27.25262711136056]
  We present HarMeme, the first benchmark dataset, containing 3,544 memes related to COVID-19. In the first stage, we labeled a meme as very harmful, partially harmful, or harmless; in the second stage, we further annotated the type of target(s) that each harmful meme points to. The evaluation results using ten unimodal and multimodal models highlight the importance of using multimodal signals for both tasks.
  arXiv  Detail & Related papers  (2021-09-24T17:11:42Z)
• MOMENTA: A Multimodal Framework for Detecting Harmful Memes and Their Targets [28.877314859737197]
  We aim to solve two novel tasks: detecting harmful memes and identifying the social entities they target. In particular, we aim to solve two novel tasks: detecting harmful memes and identifying the social entities they target. We propose MOMENTA, a novel multimodal (text + image) deep neural model, which uses global and local perspectives to detect harmful memes.
  arXiv  Detail & Related papers  (2021-09-11T04:29:32Z)
• Memes in the Wild: Assessing the Generalizability of the Hateful Memes Challenge Dataset [47.65948529524281]
  We collect hateful and non-hateful memes from Pinterest to evaluate out-of-sample performance on models pre-trained on the Facebook dataset. We find that memes in the wild differ in two key aspects: 1) Captions must be extracted via OCR, and 2) Memes are more diverse than traditional memes', including screenshots of conversations or text on a plain background.
  arXiv  Detail & Related papers  (2021-07-09T09:04:05Z)
• Multimodal Learning for Hateful Memes Detection [6.6881085567421605]
  We propose a novel method that incorporates the image captioning process into the memes detection process. Our model achieves promising results on the Hateful Memes Detection Challenge.
  arXiv  Detail & Related papers  (2020-11-25T16:49:15Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.