Meme Similarity and Emotion Detection using Multimodal Analysis

• URL: http://arxiv.org/abs/2503.17493v1
• Date: Fri, 21 Mar 2025 19:07:16 GMT
• Title: Meme Similarity and Emotion Detection using Multimodal Analysis
• Authors: Aidos Konyspay, Pakizar Shamoi, Malika Ziyada, Zhusup Smambayev,
• Abstract summary: This study employs a multimodal methodological approach, analyzing both the visual and textual elements of memes.<n>We extract low-level visual features and high-level semantic features to identify similar meme pairs.<n>Results indicate that anger and joy are the dominant emotions in memes, with motivational memes eliciting stronger emotional responses.
• Score: 0.0
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Internet memes are a central element of online culture, blending images and text. While substantial research has focused on either the visual or textual components of memes, little attention has been given to their interplay. This gap raises a key question: What methodology can effectively compare memes and the emotions they elicit? Our study employs a multimodal methodological approach, analyzing both the visual and textual elements of memes. Specifically, we perform a multimodal CLIP (Contrastive Language-Image Pre-training) model for grouping similar memes based on text and visual content embeddings, enabling robust similarity assessments across modalities. Using the Reddit Meme Dataset and Memotion Dataset, we extract low-level visual features and high-level semantic features to identify similar meme pairs. To validate these automated similarity assessments, we conducted a user study with 50 participants, asking them to provide yes/no responses regarding meme similarity and their emotional reactions. The comparison of experimental results with human judgments showed a 67.23\% agreement, suggesting that the computational approach aligns well with human perception. Additionally, we implemented a text-based classifier using the DistilBERT model to categorize memes into one of six basic emotions. The results indicate that anger and joy are the dominant emotions in memes, with motivational memes eliciting stronger emotional responses. This research contributes to the study of multimodal memes, enhancing both language-based and visual approaches to analyzing and improving online visual communication and user experiences. Furthermore, it provides insights for better content moderation strategies in online platforms.

Related papers

• Large Vision-Language Models for Knowledge-Grounded Data Annotation of Memes [5.243460995467895]
  This study introduces ClassicMemes-50-templates (CM50), a large-scale dataset consisting of over 33,000 memes, centered around 50 popular meme templates.<n>We also present an automated knowledge-grounded annotation pipeline leveraging large vision-language models to produce high-quality image captions, meme captions, and literary device labels.
  arXiv  Detail & Related papers  (2025-01-23T17:18:30Z)
• XMeCap: Meme Caption Generation with Sub-Image Adaptability [53.2509590113364]
  Humor, deeply rooted in societal meanings and cultural details, poses a unique challenge for machines. We introduce the textscXMeCap framework, which adopts supervised fine-tuning and reinforcement learning. textscXMeCap achieves an average evaluation score of 75.85 for single-image memes and 66.32 for multi-image memes, outperforming the best baseline by 3.71% and 4.82%, respectively.
  arXiv  Detail & Related papers  (2024-07-24T10:51:46Z)
• MemeMQA: Multimodal Question Answering for Memes via Rationale-Based Inferencing [53.30190591805432]
  We introduce MemeMQA, a multimodal question-answering framework to solicit accurate responses to structured questions. We also propose ARSENAL, a novel two-stage multimodal framework to address MemeMQA.
  arXiv  Detail & Related papers  (2024-05-18T07:44:41Z)
• Meme-ingful Analysis: Enhanced Understanding of Cyberbullying in Memes Through Multimodal Explanations [48.82168723932981]
  We introduce em MultiBully-Ex, the first benchmark dataset for multimodal explanation from code-mixed cyberbullying memes. A Contrastive Language-Image Pretraining (CLIP) approach has been proposed for visual and textual explanation of a meme.
  arXiv  Detail & Related papers  (2024-01-18T11:24:30Z)
• The Face of Populism: Examining Differences in Facial Emotional Expressions of Political Leaders Using Machine Learning [50.24983453990065]
  We use a deep-learning approach to process a sample of 220 YouTube videos of political leaders from 15 different countries. We observe statistically significant differences in the average score of negative emotions between groups of leaders with varying degrees of populist rhetoric.
  arXiv  Detail & Related papers  (2023-04-19T18:32:49Z)
• Seeking Subjectivity in Visual Emotion Distribution Learning [93.96205258496697]
  Visual Emotion Analysis (VEA) aims to predict people's emotions towards different visual stimuli. Existing methods often predict visual emotion distribution in a unified network, neglecting the inherent subjectivity in its crowd voting process. We propose a novel textitSubjectivity Appraise-and-Match Network (SAMNet) to investigate the subjectivity in visual emotion distribution.
  arXiv  Detail & Related papers  (2022-07-25T02:20:03Z)
• Detecting and Understanding Harmful Memes: A Survey [48.135415967633676]
  We offer a comprehensive survey with a focus on harmful memes. One interesting finding is that many types of harmful memes are not really studied, e.g., such featuring self-harm and extremism. Another observation is that memes can propagate globally through repackaging in different languages and that they can also be multilingual.
  arXiv  Detail & Related papers  (2022-05-09T13:43:27Z)
• Towards Building an Open-Domain Dialogue System Incorporated with Internet Memes [19.57042922215698]
  This paper presents our solutions for the Meme incorporated Open-domain Dialogue (MOD) Challenge of DSTC10. We leverage a large-scale pre-trained dialogue model for coherent and informative response generation. Based on interaction-based text-matching, our approach can retrieve appropriate memes with good generalization ability.
  arXiv  Detail & Related papers  (2022-03-08T03:54:02Z)
• Do Images really do the Talking? Analysing the significance of Images in Tamil Troll meme classification [0.16863755729554888]
  We try to explore the significance of visual features of images in classifying memes. We try to incorporate the memes as troll and non-trolling memes based on the images and the text on them.
  arXiv  Detail & Related papers  (2021-08-09T09:04:42Z)
• Multimodal Learning for Hateful Memes Detection [6.6881085567421605]
  We propose a novel method that incorporates the image captioning process into the memes detection process. Our model achieves promising results on the Hateful Memes Detection Challenge.
  arXiv  Detail & Related papers  (2020-11-25T16:49:15Z)
• SemEval-2020 Task 8: Memotion Analysis -- The Visuo-Lingual Metaphor! [20.55903557920223]
  The objective of this proposal is to bring the attention of the research community towards the automatic processing of Internet memes. The task Memotion analysis released approx 10K annotated memes, with human-annotated labels namely sentiment (positive, negative, neutral), type of emotion (sarcastic, funny, offensive, motivation) and corresponding intensity. The challenge consisted of three subtasks: sentiment (positive, negative, and neutral) analysis of memes, overall emotion (humour, sarcasm, offensive, and motivational) classification of memes, and classifying intensity of meme emotion.
  arXiv  Detail & Related papers  (2020-08-09T18:17:33Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.