Enhancing Meme Emotion Understanding with Multi-Level Modality Enhancement and Dual-Stage Modal Fusion

• URL: http://arxiv.org/abs/2511.11126v1
• Date: Fri, 14 Nov 2025 09:59:08 GMT
• Title: Enhancing Meme Emotion Understanding with Multi-Level Modality Enhancement and Dual-Stage Modal Fusion
• Authors: Yi Shi, Wenlong Meng, Zhenyuan Guo, Chengkun Wei, Wenzhi Chen,
• Abstract summary: We propose MemoDetector, a novel framework for advancing Meme Emotion Understanding (MEU)<n>Our method consistently outperforms state-of-the-art baselines. Specifically, MemoDetector improves F1 scores by 4.3% on MET-MEME and 3.4% on MOOD.
• Score: 18.557896531533043
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: With the rapid rise of social media and Internet culture, memes have become a popular medium for expressing emotional tendencies. This has sparked growing interest in Meme Emotion Understanding (MEU), which aims to classify the emotional intent behind memes by leveraging their multimodal contents. While existing efforts have achieved promising results, two major challenges remain: (1) a lack of fine-grained multimodal fusion strategies, and (2) insufficient mining of memes' implicit meanings and background knowledge. To address these challenges, we propose MemoDetector, a novel framework for advancing MEU. First, we introduce a four-step textual enhancement module that utilizes the rich knowledge and reasoning capabilities of Multimodal Large Language Models (MLLMs) to progressively infer and extract implicit and contextual insights from memes. These enhanced texts significantly enrich the original meme contents and provide valuable guidance for downstream classification. Next, we design a dual-stage modal fusion strategy: the first stage performs shallow fusion on raw meme image and text, while the second stage deeply integrates the enhanced visual and textual features. This hierarchical fusion enables the model to better capture nuanced cross-modal emotional cues. Experiments on two datasets, MET-MEME and MOOD, demonstrate that our method consistently outperforms state-of-the-art baselines. Specifically, MemoDetector improves F1 scores by 4.3\% on MET-MEME and 3.4\% on MOOD. Further ablation studies and in-depth analyses validate the effectiveness and robustness of our approach, highlighting its strong potential for advancing MEU. Our code is available at https://github.com/singing-cat/MemoDetector.

Related papers

• MEGC2025: Micro-Expression Grand Challenge on Spot Then Recognize and Visual Question Answering [54.0963629109064]
  Facial micro-expressions (MEs) are involuntary movements of the face that occur spontaneously when a person experiences an emotion.<n>In recent years, substantial advancements have been made in the areas of ME recognition, spotting, and generation.<n>The ME grand challenge (MEGC) 2025 introduces two tasks that reflect these evolving research directions.
  arXiv  Detail & Related papers  (2025-06-18T09:29:51Z)
• DeepMLF: Multimodal language model with learnable tokens for deep fusion in sentiment analysis [62.31018417955254]
  DeepMLF is a novel multimodal language model with learnable tokens tailored toward deep fusion.<n>Our results confirm that deeper fusion leads to better performance, with optimal fusion depths (5-7) exceeding those of existing approaches.
  arXiv  Detail & Related papers  (2025-04-15T11:28:02Z)
• Improving Multimodal Hateful Meme Detection Exploiting LMM-Generated Knowledge [11.801596051153725]
  detecting hateful content in memes has emerged as a task of critical importance.<n>We propose to address the task leveraging knowledge encoded in powerful Large Multimodal Models (LMM)<n>Specifically, we propose to exploit LMMs in a two-fold manner. First, by extracting knowledge oriented to the hateful meme detection task in order to build strong meme representations.
  arXiv  Detail & Related papers  (2025-04-14T06:23:44Z)
• Demystifying Hateful Content: Leveraging Large Multimodal Models for Hateful Meme Detection with Explainable Decisions [4.649093665157263]
  In this paper, we introduce IntMeme, a novel framework that leverages Large Multimodal Models (LMMs) for hateful meme classification with explainable decisions.<n>IntMeme addresses the dual challenges of improving both accuracy and explainability in meme moderation.<n>Our approach addresses the opacity and misclassification issues associated with PT-VLMs, optimizing the use of LMMs for hateful meme detection.
  arXiv  Detail & Related papers  (2025-02-16T10:45:40Z)
• Fine-tuning Multimodal Large Language Models for Product Bundling [53.01642741096356]
  We introduce Bundle-MLLM, a novel framework that fine-tunes large language models (LLMs) through a hybrid item tokenization approach.<n>Specifically, we integrate textual, media, and relational data into a unified tokenization, introducing a soft separation token to distinguish between textual and non-textual tokens.<n>We propose a progressive optimization strategy that fine-tunes LLMs for disentangled objectives: 1) learning bundle patterns and 2) enhancing multimodal semantic understanding specific to product bundling.
  arXiv  Detail & Related papers  (2024-07-16T13:30:14Z)
• NoteLLM-2: Multimodal Large Representation Models for Recommendation [71.87790090964734]
  Large Language Models (LLMs) have demonstrated exceptional proficiency in text understanding and embedding tasks.<n>Their potential in multimodal representation, particularly for item-to-item (I2I) recommendations, remains underexplored.<n>We propose an end-to-end fine-tuning method that customizes the integration of any existing LLMs and vision encoders for efficient multimodal representation.
  arXiv  Detail & Related papers  (2024-05-27T03:24:01Z)
• MemeMQA: Multimodal Question Answering for Memes via Rationale-Based Inferencing [53.30190591805432]
  We introduce MemeMQA, a multimodal question-answering framework to solicit accurate responses to structured questions. We also propose ARSENAL, a novel two-stage multimodal framework to address MemeMQA.
  arXiv  Detail & Related papers  (2024-05-18T07:44:41Z)
• AIMDiT: Modality Augmentation and Interaction via Multimodal Dimension Transformation for Emotion Recognition in Conversations [57.99479708224221]
  We propose a novel framework called AIMDiT to solve the problem of multimodal fusion of deep features. Experiments conducted using our AIMDiT framework on the public benchmark dataset MELD reveal 2.34% and 2.87% improvements in terms of the Acc-7 and w-F1 metrics.
  arXiv  Detail & Related papers  (2024-04-12T11:31:18Z)
• Emotion-Aware Multimodal Fusion for Meme Emotion Detection [37.86468816979694]
  MOOD (Meme emOtiOns dataset) embodies six basic emotions. ALFRED (emotion-Aware muLtimodal Fusion foR Emotion Detection) explicitly models emotion-enriched visual cues.
  arXiv  Detail & Related papers  (2024-03-15T13:20:38Z)
• MEMEX: Detecting Explanatory Evidence for Memes via Knowledge-Enriched Contextualization [31.209594252045566]
  We propose a novel task, MEMEX, given a meme and a related document, the aim is to mine the context that succinctly explains the background of the meme. To benchmark MCC, we propose MIME, a multimodal neural framework that uses common sense enriched meme representation and a layered approach to capture the cross-modal semantic dependencies between the meme and the context.
  arXiv  Detail & Related papers  (2023-05-25T10:19:35Z)
• MemeFier: Dual-stage Modality Fusion for Image Meme Classification [8.794414326545697]
  New forms of digital content such as image memes have given rise to spread of hate using multimodal means. We propose MemeFier, a deep learning-based architecture for fine-grained classification of Internet image memes.
  arXiv  Detail & Related papers  (2023-04-06T07:36:52Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.