Multimodal Learning for Hateful Memes Detection

• URL: http://arxiv.org/abs/2011.12870v3
• Date: Sun, 6 Dec 2020 22:16:30 GMT
• Title: Multimodal Learning for Hateful Memes Detection
• Authors: Yi Zhou, Zhenhao Chen
• Abstract summary: We propose a novel method that incorporates the image captioning process into the memes detection process. Our model achieves promising results on the Hateful Memes Detection Challenge.
• Score: 6.6881085567421605
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Memes are used for spreading ideas through social networks. Although most memes are created for humor, some memes become hateful under the combination of pictures and text. Automatically detecting the hateful memes can help reduce their harmful social impact. Unlike the conventional multimodal tasks, where the visual and textual information is semantically aligned, the challenge of hateful memes detection lies in its unique multimodal information. The image and text in memes are weakly aligned or even irrelevant, which requires the model to understand the content and perform reasoning over multiple modalities. In this paper, we focus on multimodal hateful memes detection and propose a novel method that incorporates the image captioning process into the memes detection process. We conduct extensive experiments on multimodal meme datasets and illustrated the effectiveness of our approach. Our model achieves promising results on the Hateful Memes Detection Challenge.

Related papers

• XMeCap: Meme Caption Generation with Sub-Image Adaptability [53.2509590113364]
  Humor, deeply rooted in societal meanings and cultural details, poses a unique challenge for machines. We introduce the textscXMeCap framework, which adopts supervised fine-tuning and reinforcement learning. textscXMeCap achieves an average evaluation score of 75.85 for single-image memes and 66.32 for multi-image memes, outperforming the best baseline by 3.71% and 4.82%, respectively.
  arXiv  Detail & Related papers  (2024-07-24T10:51:46Z)
• What Makes a Meme a Meme? Identifying Memes for Memetics-Aware Dataset Creation [0.9217021281095907]
  Multimodal Internet Memes are now a ubiquitous fixture in online discourse. Memetics are the process by which memes are imitated and transformed into symbols. We develop a meme identification protocol which distinguishes meme from non-memetic content by recognising the memetics within it.
  arXiv  Detail & Related papers  (2024-07-16T15:48:36Z)
• Meme-ingful Analysis: Enhanced Understanding of Cyberbullying in Memes Through Multimodal Explanations [48.82168723932981]
  We introduce em MultiBully-Ex, the first benchmark dataset for multimodal explanation from code-mixed cyberbullying memes. A Contrastive Language-Image Pretraining (CLIP) approach has been proposed for visual and textual explanation of a meme.
  arXiv  Detail & Related papers  (2024-01-18T11:24:30Z)
• A Template Is All You Meme [83.05919383106715]
  We release a knowledge base of memes and information found on www.knowyourmeme.com, composed of more than 54,000 images. We hypothesize that meme templates can be used to inject models with the context missing from previous approaches.
  arXiv  Detail & Related papers  (2023-11-11T19:38:14Z)
• Mapping Memes to Words for Multimodal Hateful Meme Classification [26.101116761577796]
  Some memes take a malicious turn, promoting hateful content and perpetuating discrimination. We propose a novel approach named ISSUES for multimodal hateful meme classification. Our method achieves state-of-the-art results on the Hateful Memes Challenge and HarMeme datasets.
  arXiv  Detail & Related papers  (2023-10-12T14:38:52Z)
• On the Evolution of (Hateful) Memes by Means of Multimodal Contrastive Learning [18.794226796466962]
  We study how hateful memes are created by combining visual elements from multiple images or fusing textual information with a hateful image. Using our framework on a dataset extracted from 4chan, we find 3.3K variants of the Happy Merchant meme. We envision that our framework can be used to aid human moderators by flagging new variants of hateful memes.
  arXiv  Detail & Related papers  (2022-12-13T13:38:04Z)
• DisinfoMeme: A Multimodal Dataset for Detecting Meme Intentionally Spreading Out Disinformation [72.18912216025029]
  We present DisinfoMeme to help detect disinformation memes. The dataset contains memes mined from Reddit covering three current topics: the COVID-19 pandemic, the Black Lives Matter movement, and veganism/vegetarianism.
  arXiv  Detail & Related papers  (2022-05-25T09:54:59Z)
• Detecting and Understanding Harmful Memes: A Survey [48.135415967633676]
  We offer a comprehensive survey with a focus on harmful memes. One interesting finding is that many types of harmful memes are not really studied, e.g., such featuring self-harm and extremism. Another observation is that memes can propagate globally through repackaging in different languages and that they can also be multilingual.
  arXiv  Detail & Related papers  (2022-05-09T13:43:27Z)
• Detecting Harmful Memes and Their Targets [27.25262711136056]
  We present HarMeme, the first benchmark dataset, containing 3,544 memes related to COVID-19. In the first stage, we labeled a meme as very harmful, partially harmful, or harmless; in the second stage, we further annotated the type of target(s) that each harmful meme points to. The evaluation results using ten unimodal and multimodal models highlight the importance of using multimodal signals for both tasks.
  arXiv  Detail & Related papers  (2021-09-24T17:11:42Z)
• Memes in the Wild: Assessing the Generalizability of the Hateful Memes Challenge Dataset [47.65948529524281]
  We collect hateful and non-hateful memes from Pinterest to evaluate out-of-sample performance on models pre-trained on the Facebook dataset. We find that memes in the wild differ in two key aspects: 1) Captions must be extracted via OCR, and 2) Memes are more diverse than traditional memes', including screenshots of conversations or text on a plain background.
  arXiv  Detail & Related papers  (2021-07-09T09:04:05Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.