Mapping Memes to Words for Multimodal Hateful Meme Classification
- URL: http://arxiv.org/abs/2310.08368v1
- Date: Thu, 12 Oct 2023 14:38:52 GMT
- Title: Mapping Memes to Words for Multimodal Hateful Meme Classification
- Authors: Giovanni Burbi, Alberto Baldrati, Lorenzo Agnolucci, Marco Bertini,
Alberto Del Bimbo
- Abstract summary: Some memes take a malicious turn, promoting hateful content and perpetuating discrimination.
We propose a novel approach named ISSUES for multimodal hateful meme classification.
Our method achieves state-of-the-art results on the Hateful Memes Challenge and HarMeme datasets.
- Score: 26.101116761577796
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: Multimodal image-text memes are prevalent on the internet, serving as a
unique form of communication that combines visual and textual elements to
convey humor, ideas, or emotions. However, some memes take a malicious turn,
promoting hateful content and perpetuating discrimination. Detecting hateful
memes within this multimodal context is a challenging task that requires
understanding the intertwined meaning of text and images. In this work, we
address this issue by proposing a novel approach named ISSUES for multimodal
hateful meme classification. ISSUES leverages a pre-trained CLIP
vision-language model and the textual inversion technique to effectively
capture the multimodal semantic content of the memes. The experiments show that
our method achieves state-of-the-art results on the Hateful Memes Challenge and
HarMeme datasets. The code and the pre-trained models are publicly available at
https://github.com/miccunifi/ISSUES.
Related papers
- XMeCap: Meme Caption Generation with Sub-Image Adaptability [53.2509590113364]
Humor, deeply rooted in societal meanings and cultural details, poses a unique challenge for machines.
We introduce the textscXMeCap framework, which adopts supervised fine-tuning and reinforcement learning.
textscXMeCap achieves an average evaluation score of 75.85 for single-image memes and 66.32 for multi-image memes, outperforming the best baseline by 3.71% and 4.82%, respectively.
arXiv Detail & Related papers (2024-07-24T10:51:46Z) - Meme-ingful Analysis: Enhanced Understanding of Cyberbullying in Memes
Through Multimodal Explanations [48.82168723932981]
We introduce em MultiBully-Ex, the first benchmark dataset for multimodal explanation from code-mixed cyberbullying memes.
A Contrastive Language-Image Pretraining (CLIP) approach has been proposed for visual and textual explanation of a meme.
arXiv Detail & Related papers (2024-01-18T11:24:30Z) - Beneath the Surface: Unveiling Harmful Memes with Multimodal Reasoning
Distilled from Large Language Models [17.617187709968242]
Existing harmful meme detection approaches only recognize superficial harm-indicative signals in an end-to-end classification manner.
We propose a novel generative framework to learn reasonable thoughts from Large Language Models for better multimodal fusion.
Our proposed approach achieves superior performance than state-of-the-art methods on the harmful meme detection task.
arXiv Detail & Related papers (2023-12-09T01:59:11Z) - A Template Is All You Meme [83.05919383106715]
We release a knowledge base of memes and information found on www.knowyourmeme.com, composed of more than 54,000 images.
We hypothesize that meme templates can be used to inject models with the context missing from previous approaches.
arXiv Detail & Related papers (2023-11-11T19:38:14Z) - On the Evolution of (Hateful) Memes by Means of Multimodal Contrastive
Learning [18.794226796466962]
We study how hateful memes are created by combining visual elements from multiple images or fusing textual information with a hateful image.
Using our framework on a dataset extracted from 4chan, we find 3.3K variants of the Happy Merchant meme.
We envision that our framework can be used to aid human moderators by flagging new variants of hateful memes.
arXiv Detail & Related papers (2022-12-13T13:38:04Z) - DisinfoMeme: A Multimodal Dataset for Detecting Meme Intentionally
Spreading Out Disinformation [72.18912216025029]
We present DisinfoMeme to help detect disinformation memes.
The dataset contains memes mined from Reddit covering three current topics: the COVID-19 pandemic, the Black Lives Matter movement, and veganism/vegetarianism.
arXiv Detail & Related papers (2022-05-25T09:54:59Z) - Detecting and Understanding Harmful Memes: A Survey [48.135415967633676]
We offer a comprehensive survey with a focus on harmful memes.
One interesting finding is that many types of harmful memes are not really studied, e.g., such featuring self-harm and extremism.
Another observation is that memes can propagate globally through repackaging in different languages and that they can also be multilingual.
arXiv Detail & Related papers (2022-05-09T13:43:27Z) - Memes in the Wild: Assessing the Generalizability of the Hateful Memes
Challenge Dataset [47.65948529524281]
We collect hateful and non-hateful memes from Pinterest to evaluate out-of-sample performance on models pre-trained on the Facebook dataset.
We find that memes in the wild differ in two key aspects: 1) Captions must be extracted via OCR, and 2) Memes are more diverse than traditional memes', including screenshots of conversations or text on a plain background.
arXiv Detail & Related papers (2021-07-09T09:04:05Z) - Detecting Hate Speech in Memes Using Multimodal Deep Learning
Approaches: Prize-winning solution to Hateful Memes Challenge [0.0]
The Hateful Memes Challenge is a first-of-its-kind competition which focuses on detecting hate speech in multimodal memes.
We utilize VisualBERT -- which meant to be the BERT of vision and language -- that was trained multimodally on images and captions.
Our approach achieves 0.811 AUROC with an accuracy of 0.765 on the challenge test set and placed third out of 3,173 participants in the Hateful Memes Challenge.
arXiv Detail & Related papers (2020-12-23T21:09:52Z) - Multimodal Learning for Hateful Memes Detection [6.6881085567421605]
We propose a novel method that incorporates the image captioning process into the memes detection process.
Our model achieves promising results on the Hateful Memes Detection Challenge.
arXiv Detail & Related papers (2020-11-25T16:49:15Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.