MELLM: Exploring LLM-Powered Micro-Expression Understanding Enhanced by Subtle Motion Perception

• URL: http://arxiv.org/abs/2505.07007v2
• Date: Thu, 23 Oct 2025 17:59:16 GMT
• Title: MELLM: Exploring LLM-Powered Micro-Expression Understanding Enhanced by Subtle Motion Perception
• Authors: Sirui Zhao, Zhengye Zhang, Shifeng Liu, Xinglong Mao, Shukang Yin, Chaoyou Fu, Tong Xu, Enhong Chen,
• Abstract summary: Micro-expressions (MEs) are brief and low-intensity facial movements revealing concealed emotions.<n>We propose a ME Large Language Model (MELLM) that integrates optical flow-based sensitivity to subtle facial motions.<n>MELLM achieves state-of-the-art accuracy and generalization across multiple ME benchmarks.
• Score: 53.00485107136624
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Micro-expressions (MEs), brief and low-intensity facial movements revealing concealed emotions, are crucial for affective computing. Despite notable progress in ME recognition, existing methods are largely confined to discrete emotion classification, lacking the capacity for comprehensive ME Understanding (MEU), particularly in interpreting subtle facial dynamics and underlying emotional cues. While Multimodal Large Language Models (MLLMs) offer potential for MEU with their advanced reasoning abilities, they still struggle to perceive such subtle facial affective behaviors. To bridge this gap, we propose a ME Large Language Model (MELLM) that integrates optical flow-based sensitivity to subtle facial motions with the powerful inference ability of LLMs. Specifically, an iterative, warping-based optical-flow estimator, named MEFlowNet, is introduced to precisely capture facial micro-movements. For its training and evaluation, we construct MEFlowDataset, a large-scale optical-flow dataset with 54,611 onset-apex image pairs spanning diverse identities and subtle facial motions. Subsequently, we design a Flow-Guided Micro-Expression Understanding paradigm. Under this framework, the optical flow signals extracted by MEFlowNet are leveraged to build MEU-Instruct, an instruction-tuning dataset for MEU. MELLM is then fine-tuned on MEU-Instruct, enabling it to translate subtle motion patterns into human-readable descriptions and generate corresponding emotional inferences. Experiments demonstrate that MEFlowNet significantly outperforms existing optical flow methods in facial and ME-flow estimation, while MELLM achieves state-of-the-art accuracy and generalization across multiple ME benchmarks. To the best of our knowledge, this work presents two key contributions: MEFlowNet as the first dedicated ME flow estimator, and MELLM as the first LLM tailored for MEU.

Related papers

• E^2-LLM: Bridging Neural Signals and Interpretable Affective Analysis [54.763420895859035]
  We present ELLM2-EEG-to-Emotion Large Language Model, first MLLM framework for interpretable emotion analysis from EEG.<n>ELLM integrates a pretrained EEG encoder with Q-based LLMs through learnable projection layers, employing a multi-stage training pipeline.<n>Experiments on the dataset across seven emotion categories demonstrate that ELLM2-EEG-to-Emotion Large Language Model achieves excellent performance on emotion classification.
  arXiv  Detail & Related papers  (2026-01-11T13:21:20Z)
• DEFT-LLM: Disentangled Expert Feature Tuning for Micro-Expression Recognition [16.903294278064667]
  We propose DEFT-LLM, which achieves semantic alignment by multi-expert disentanglement.<n>We first introduce Uni-MER, a motion-driven instruction designed to align text with local facial motion.<n>We then design an architecture with three experts to decouple facial dynamics into independent representations.
  arXiv  Detail & Related papers  (2025-11-14T04:21:24Z)
• Emotion-Coherent Reasoning for Multimodal LLMs via Emotional Rationale Verifier [53.55996102181836]
  We propose the Emotional Rationale Verifier (ERV) and an Explanation Reward.<n>Our method guides the model to produce reasoning that is explicitly consistent with the target emotion.<n>We show that our approach not only enhances alignment between explanation and prediction but also empowers MLLMs to deliver emotionally coherent, trustworthy interactions.
  arXiv  Detail & Related papers  (2025-10-27T16:40:17Z)
• FMANet: A Novel Dual-Phase Optical Flow Approach with Fusion Motion Attention Network for Robust Micro-expression Recognition [0.0]
  micro-expression recognition is challenging due to the difficulty of capturing subtle facial movements.<n>We introduce a comprehensive motion representation, which integrates motion dynamics from both micro-expression phases into a unified descriptor.<n>We then propose FMANet, a novel end-to-end neural network architecture that internalizes the dual-phase analysis and magnitude modulation into learnable modules.
  arXiv  Detail & Related papers  (2025-10-09T05:36:40Z)
• MPT: Motion Prompt Tuning for Micro-Expression Recognition [47.62949098749473]
  This paper introduces Motion Prompt Tuning (MPT) as a novel approach to adapting pre-training models for micro-expression recognition (MER)<n>MPT represents a pioneering method for subtle motion prompt tuning. Particularly, we introduce motion prompt generation, including motion magnification and Gaussian tokenization, to extract subtle motions as prompts for LMs.<n>Extensive experiments conducted on three widely used MER datasets demonstrate that our proposed MPT consistently surpasses state-of-the-art approaches.
  arXiv  Detail & Related papers  (2025-08-13T02:57:43Z)
• MME-Emotion: A Holistic Evaluation Benchmark for Emotional Intelligence in Multimodal Large Language Models [108.61337743051483]
  We present MME-Emotion, a systematic benchmark that assesses both emotional understanding and reasoning capabilities of MLLMs.<n>MME-Emotion contains over 6,000 curated video clips with task-specific questioning-answering (QA) pairs, spanning broad scenarios to formulate eight emotional tasks.<n>It incorporates a holistic evaluation suite with hybrid metrics for emotion recognition and reasoning, analyzed through a multi-agent system framework.
  arXiv  Detail & Related papers  (2025-08-11T03:14:55Z)
• AU-LLM: Micro-Expression Action Unit Detection via Enhanced LLM-Based Feature Fusion [26.058143518505805]
  This paper introduces textbfAU-LLM, a novel framework that uses Large Language Models to detect micro-expression Action Units (AUs) in micro-expression datasets with subtle intensities and the scarcity of data.<n>We specifically address the critical vision-language semantic gap, the textbfEnhanced Fusion Projector (EFP). The EFP employs a Multi-Layer Perceptron (MLP) to intelligently fuse mid-level (local texture) and high-level (global semantics) visual features from a specialized 3D-CNN backbone into a single, information-dense token.
  arXiv  Detail & Related papers  (2025-07-29T13:01:59Z)
• MEGC2025: Micro-Expression Grand Challenge on Spot Then Recognize and Visual Question Answering [54.0963629109064]
  Facial micro-expressions (MEs) are involuntary movements of the face that occur spontaneously when a person experiences an emotion.<n>In recent years, substantial advancements have been made in the areas of ME recognition, spotting, and generation.<n>The ME grand challenge (MEGC) 2025 introduces two tasks that reflect these evolving research directions.
  arXiv  Detail & Related papers  (2025-06-18T09:29:51Z)
• V-MAGE: A Game Evaluation Framework for Assessing Vision-Centric Capabilities in Multimodal Large Language Models [84.27290155010533]
  We introduce Vision-centric Multiple Abilities Game Evaluation (V-MAGE), a novel game-based evaluation framework.<n>V-MAGE features five distinct video games comprising over 30 carefully constructed evaluation scenarios.<n>We show V-MAGE provides actionable insights for improving the visual and reasoning capabilities of MLLMs in dynamic, interactive settings.
  arXiv  Detail & Related papers  (2025-04-08T15:43:01Z)
• BeMERC: Behavior-Aware MLLM-based Framework for Multimodal Emotion Recognition in Conversation [29.514459004019024]
  We propose a behavior-aware MLLM-based framework (BeMERC) to incorporate speaker's behaviors into a vanilla MLLM-based MERC model.<n>BeMERC achieves superior performance than the state-of-the-art methods on two benchmark datasets.
  arXiv  Detail & Related papers  (2025-03-31T12:04:53Z)
• SegAgent: Exploring Pixel Understanding Capabilities in MLLMs by Imitating Human Annotator Trajectories [52.57696897619189]
  We introduce the Human-Like Mask Modeling Task (HLMAT), a new paradigm where MLLMs mimic human annotators using interactive segmentation tools.<n>HLMAT enables MLLMs to iteratively generate text-based click points, achieving high-quality masks without architectural changes or implicit tokens.<n>HLMAT provides a protocol for assessing fine-grained pixel understanding in MLLMs and introduces a vision-centric, multi-step decision-making task.
  arXiv  Detail & Related papers  (2025-03-11T17:08:54Z)
• MOSABench: Multi-Object Sentiment Analysis Benchmark for Evaluating Multimodal Large Language Models Understanding of Complex Image [16.040813949620958]
  We introduce MOSABench, a novel evaluation dataset designed specifically for multi-object sentiment analysis.<n>Key innovations in MOSABench include distance-based target annotation, post-processing for evaluation to standardize outputs, and an improved scoring mechanism.<n>This research underscores the need for MLLMs to enhance accuracy in complex, multi-object sentiment analysis tasks.
  arXiv  Detail & Related papers  (2024-11-25T09:00:36Z)
• Can MLLMs Guide Weakly-Supervised Temporal Action Localization Tasks? [6.7065734065794835]
  We introduce a novel learning paradigm termed MLLM4WTAL. It harnesses the potential of MLLM to offer temporal action key semantics and complete semantic priors. It achieves this by integrating two distinct modules: Key Semantic Matching (KSM) and Complete Semantic Reconstruction (CSR)
  arXiv  Detail & Related papers  (2024-11-13T09:37:24Z)
• A Comprehensive Review of Multimodal Large Language Models: Performance and Challenges Across Different Tasks [74.52259252807191]
  Multimodal Large Language Models (MLLMs) address the complexities of real-world applications far beyond the capabilities of single-modality systems. This paper systematically sorts out the applications of MLLM in multimodal tasks such as natural language, vision, and audio.
  arXiv  Detail & Related papers  (2024-08-02T15:14:53Z)
• X-Former: Unifying Contrastive and Reconstruction Learning for MLLMs [49.30255148577368]
  X-Former is a lightweight transformer module designed to exploit the complementary strengths of CL and MIM. X-Former first bootstraps vision-language representation learning and multimodal-to-multimodal generative learning from two frozen vision encoders. It further bootstraps vision-to-language generative learning from a frozen LLM to ensure visual features from X-Former can be interpreted by the LLM.
  arXiv  Detail & Related papers  (2024-07-18T18:39:54Z)
• From Macro to Micro: Boosting micro-expression recognition via pre-training on macro-expression videos [9.472210792839023]
  Micro-expression recognition (MER) has drawn increasing attention in recent years due to its potential applications in intelligent medical and lie detection. We propose a generalized transfer learning paradigm, called textbfMAcro-expression textbfTO textbfMIcro-expression (MA2MI) Under our paradigm, networks can learns the ability to represent subtle facial movement by reconstructing future frames.
  arXiv  Detail & Related papers  (2024-05-26T06:42:06Z)
• Multi-modal Instruction Tuned LLMs with Fine-grained Visual Perception [63.03288425612792]
  We propose bfAnyRef, a general MLLM model that can generate pixel-wise object perceptions and natural language descriptions from multi-modality references. Our model achieves state-of-the-art results across multiple benchmarks, including diverse modality referring segmentation and region-level referring expression generation.
  arXiv  Detail & Related papers  (2024-03-05T13:45:46Z)
• From Training-Free to Adaptive: Empirical Insights into MLLMs' Understanding of Detection Information [32.57246173437492]
  Vision detection models excel at recognizing fine-grained image details.<n>One effective strategy is to infuse detection information in text format, which has proven simple and effective.<n>This paper addresses the question: How does training impact MLLMs' understanding of infused textual detection information?
  arXiv  Detail & Related papers  (2024-01-31T16:38:32Z)
• HiLM-D: Enhancing MLLMs with Multi-Scale High-Resolution Details for Autonomous Driving [44.06475712570428]
  HiLM-D is a resource-efficient framework that enhances visual information processing in MLLMs for ROLISP.<n>Our method is motivated by the fact that the primary variations in autonomous driving scenarios are the motion trajectories.<n>Experiments show HiLM-D's significant improvements over current MLLMs, with a 3.7% in BLEU-4 for captioning and 8.7% in mIoU for detection.
  arXiv  Detail & Related papers  (2023-09-11T01:24:13Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.