MEMEX: Detecting Explanatory Evidence for Memes via Knowledge-Enriched Contextualization

• URL: http://arxiv.org/abs/2305.15913v2
• Date: Sat, 27 May 2023 13:09:46 GMT
• Title: MEMEX: Detecting Explanatory Evidence for Memes via Knowledge-Enriched Contextualization
• Authors: Shivam Sharma, Ramaneswaran S, Udit Arora, Md. Shad Akhtar and Tanmoy Chakraborty
• Abstract summary: We propose a novel task, MEMEX, given a meme and a related document, the aim is to mine the context that succinctly explains the background of the meme. To benchmark MCC, we propose MIME, a multimodal neural framework that uses common sense enriched meme representation and a layered approach to capture the cross-modal semantic dependencies between the meme and the context.
• Score: 31.209594252045566
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Memes are a powerful tool for communication over social media. Their affinity for evolving across politics, history, and sociocultural phenomena makes them an ideal communication vehicle. To comprehend the subtle message conveyed within a meme, one must understand the background that facilitates its holistic assimilation. Besides digital archiving of memes and their metadata by a few websites like knowyourmeme.com, currently, there is no efficient way to deduce a meme's context dynamically. In this work, we propose a novel task, MEMEX - given a meme and a related document, the aim is to mine the context that succinctly explains the background of the meme. At first, we develop MCC (Meme Context Corpus), a novel dataset for MEMEX. Further, to benchmark MCC, we propose MIME (MultImodal Meme Explainer), a multimodal neural framework that uses common sense enriched meme representation and a layered approach to capture the cross-modal semantic dependencies between the meme and the context. MIME surpasses several unimodal and multimodal systems and yields an absolute improvement of ~ 4% F1-score over the best baseline. Lastly, we conduct detailed analyses of MIME's performance, highlighting the aspects that could lead to optimal modeling of cross-modal contextual associations.

Related papers

• Decoding Memes: A Comparative Study of Machine Learning Models for Template Identification [0.0]
  "meme template" is a layout or format that is used to create memes. Despite extensive research on meme virality, the task of automatically identifying meme templates remains a challenge. This paper presents a comprehensive comparison and evaluation of existing meme template identification methods.
  arXiv  Detail & Related papers  (2024-08-15T12:52:06Z)
• XMeCap: Meme Caption Generation with Sub-Image Adaptability [53.2509590113364]
  Humor, deeply rooted in societal meanings and cultural details, poses a unique challenge for machines. We introduce the textscXMeCap framework, which adopts supervised fine-tuning and reinforcement learning. textscXMeCap achieves an average evaluation score of 75.85 for single-image memes and 66.32 for multi-image memes, outperforming the best baseline by 3.71% and 4.82%, respectively.
  arXiv  Detail & Related papers  (2024-07-24T10:51:46Z)
• MemeMQA: Multimodal Question Answering for Memes via Rationale-Based Inferencing [53.30190591805432]
  We introduce MemeMQA, a multimodal question-answering framework to solicit accurate responses to structured questions. We also propose ARSENAL, a novel two-stage multimodal framework to address MemeMQA.
  arXiv  Detail & Related papers  (2024-05-18T07:44:41Z)
• Meme-ingful Analysis: Enhanced Understanding of Cyberbullying in Memes Through Multimodal Explanations [48.82168723932981]
  We introduce em MultiBully-Ex, the first benchmark dataset for multimodal explanation from code-mixed cyberbullying memes. A Contrastive Language-Image Pretraining (CLIP) approach has been proposed for visual and textual explanation of a meme.
  arXiv  Detail & Related papers  (2024-01-18T11:24:30Z)
• PromptMTopic: Unsupervised Multimodal Topic Modeling of Memes using Large Language Models [7.388466146105024]
  We propose textPromptMTopic, a novel multimodal prompt-based model to learn topics from both text and visual modalities. Our model effectively extracts and clusters topics learned from memes, considering the semantic interaction between the text and visual modalities. Our work contributes to the understanding of the topics and themes of memes, a crucial form of communication in today's society.
  arXiv  Detail & Related papers  (2023-12-11T03:36:50Z)
• A Template Is All You Meme [83.05919383106715]
  We release a knowledge base of memes and information found on www.knowyourmeme.com, composed of more than 54,000 images. We hypothesize that meme templates can be used to inject models with the context missing from previous approaches.
  arXiv  Detail & Related papers  (2023-11-11T19:38:14Z)
• Mapping Memes to Words for Multimodal Hateful Meme Classification [26.101116761577796]
  Some memes take a malicious turn, promoting hateful content and perpetuating discrimination. We propose a novel approach named ISSUES for multimodal hateful meme classification. Our method achieves state-of-the-art results on the Hateful Memes Challenge and HarMeme datasets.
  arXiv  Detail & Related papers  (2023-10-12T14:38:52Z)
• What do you MEME? Generating Explanations for Visual Semantic Role Labelling in Memes [42.357272117919464]
  We introduce a novel task - EXCLAIM, generating explanations for visual semantic role labeling in memes. To this end, we curate ExHVV, a novel dataset that offers natural language explanations of connotative roles for three types of entities. We also posit LUMEN, a novel multimodal, multi-task learning framework that endeavors to address EXCLAIM optimally.
  arXiv  Detail & Related papers  (2022-12-01T18:21:36Z)
• Do Images really do the Talking? Analysing the significance of Images in Tamil Troll meme classification [0.16863755729554888]
  We try to explore the significance of visual features of images in classifying memes. We try to incorporate the memes as troll and non-trolling memes based on the images and the text on them.
  arXiv  Detail & Related papers  (2021-08-09T09:04:42Z)
• Memes in the Wild: Assessing the Generalizability of the Hateful Memes Challenge Dataset [47.65948529524281]
  We collect hateful and non-hateful memes from Pinterest to evaluate out-of-sample performance on models pre-trained on the Facebook dataset. We find that memes in the wild differ in two key aspects: 1) Captions must be extracted via OCR, and 2) Memes are more diverse than traditional memes', including screenshots of conversations or text on a plain background.
  arXiv  Detail & Related papers  (2021-07-09T09:04:05Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.