On Explaining Multimodal Hateful Meme Detection Models
- URL: http://arxiv.org/abs/2204.01734v2
- Date: Wed, 6 Apr 2022 08:56:40 GMT
- Title: On Explaining Multimodal Hateful Meme Detection Models
- Authors: Ming Shan Hee, Roy Ka-Wei Lee, Wen-Haw Chong
- Abstract summary: It is unclear if these models are able to capture the derogatory or slurs references in multimodality.
We found that the image modality contributes more to the hateful meme classification task.
Our error analysis shows that the visual-linguistic models have acquired biases, which resulted in false-positive predictions.
- Score: 4.509263496823139
- License: http://creativecommons.org/licenses/by/4.0/
- Abstract: Hateful meme detection is a new multimodal task that has gained significant
traction in academic and industry research communities. Recently, researchers
have applied pre-trained visual-linguistic models to perform the multimodal
classification task, and some of these solutions have yielded promising
results. However, what these visual-linguistic models learn for the hateful
meme classification task remains unclear. For instance, it is unclear if these
models are able to capture the derogatory or slurs references in multimodality
(i.e., image and text) of the hateful memes. To fill this research gap, this
paper propose three research questions to improve our understanding of these
visual-linguistic models performing the hateful meme classification task. We
found that the image modality contributes more to the hateful meme
classification task, and the visual-linguistic models are able to perform
visual-text slurs grounding to a certain extent. Our error analysis also shows
that the visual-linguistic models have acquired biases, which resulted in
false-positive predictions.
Related papers
- Multimodal and Explainable Internet Meme Classification [3.4690152926833315]
We design and implement a modular and explainable architecture for Internet meme understanding.
We study the relevance of our modular and explainable models in detecting harmful memes on two existing tasks: Hate Speech Detection and Misogyny Classification.
We devise a user-friendly interface that facilitates the comparative analysis of examples retrieved by all of our models for any given meme.
arXiv Detail & Related papers (2022-12-11T21:52:21Z) - Localization vs. Semantics: Visual Representations in Unimodal and
Multimodal Models [57.08925810659545]
We conduct a comparative analysis of the visual representations in existing vision-and-language models and vision-only models.
Our empirical observations suggest that vision-and-language models are better at label prediction tasks.
We hope our study sheds light on the role of language in visual learning, and serves as an empirical guide for various pretrained models.
arXiv Detail & Related papers (2022-12-01T05:00:18Z) - MultiViz: An Analysis Benchmark for Visualizing and Understanding
Multimodal Models [103.9987158554515]
MultiViz is a method for analyzing the behavior of multimodal models by scaffolding the problem of interpretability into 4 stages.
We show that the complementary stages in MultiViz together enable users to simulate model predictions, assign interpretable concepts to features, perform error analysis on model misclassifications, and use insights from error analysis to debug models.
arXiv Detail & Related papers (2022-06-30T18:42:06Z) - On Advances in Text Generation from Images Beyond Captioning: A Case
Study in Self-Rationalization [89.94078728495423]
We show that recent advances in each modality, CLIP image representations and scaling of language models, do not consistently improve multimodal self-rationalization of tasks with multimodal inputs.
Our findings call for a backbone modelling approach that can be built on to advance text generation from images and text beyond image captioning.
arXiv Detail & Related papers (2022-05-24T00:52:40Z) - Caption Enriched Samples for Improving Hateful Memes Detection [78.5136090997431]
The hateful meme challenge demonstrates the difficulty of determining whether a meme is hateful or not.
Both unimodal language models and multimodal vision-language models cannot reach the human level of performance.
arXiv Detail & Related papers (2021-09-22T10:57:51Z) - Dependent Multi-Task Learning with Causal Intervention for Image
Captioning [10.6405791176668]
In this paper, we propose a dependent multi-task learning framework with the causal intervention (DMTCI)
Firstly, we involve an intermediate task, bag-of-categories generation, before the final task, image captioning.
Secondly, we apply Pearl's do-calculus on the model, cutting off the link between the visual features and possible confounders.
Finally, we use a multi-agent reinforcement learning strategy to enable end-to-end training and reduce the inter-task error accumulations.
arXiv Detail & Related papers (2021-05-18T14:57:33Z) - Counterfactual Generative Networks [59.080843365828756]
We propose to decompose the image generation process into independent causal mechanisms that we train without direct supervision.
By exploiting appropriate inductive biases, these mechanisms disentangle object shape, object texture, and background.
We show that the counterfactual images can improve out-of-distribution with a marginal drop in performance on the original classification task.
arXiv Detail & Related papers (2021-01-15T10:23:12Z) - Detecting Hate Speech in Multi-modal Memes [14.036769355498546]
We focus on hate speech detection in multi-modal memes wherein memes pose an interesting multi-modal fusion problem.
We aim to solve the Facebook Meme Challenge citekiela 2020hateful which aims to solve a binary classification problem of predicting whether a meme is hateful or not.
arXiv Detail & Related papers (2020-12-29T18:30:00Z) - A Multimodal Framework for the Detection of Hateful Memes [16.7604156703965]
We aim to develop a framework for the detection of hateful memes.
We show the effectiveness of upsampling of contrastive examples to encourage multimodality and ensemble learning.
Our best approach comprises an ensemble of UNITER-based models and achieves an AUROC score of 80.53, placing us 4th on phase 2 of the 2020 Hateful Memes Challenge organized by Facebook.
arXiv Detail & Related papers (2020-12-23T18:37:11Z) - Exploring Simple Siamese Representation Learning [68.37628268182185]
We show that simple Siamese networks can learn meaningful representations even using none of the following: (i) negative sample pairs, (ii) large batches, (iii) momentum encoders.
Our experiments show that collapsing solutions do exist for the loss and structure, but a stop-gradient operation plays an essential role in preventing collapsing.
arXiv Detail & Related papers (2020-11-20T18:59:33Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.