MMME: A Spontaneous Multi-Modal Micro-Expression Dataset Enabling Visual-Physiological Fusion

• URL: http://arxiv.org/abs/2506.09834v2
• Date: Thu, 12 Jun 2025 03:33:02 GMT
• Title: MMME: A Spontaneous Multi-Modal Micro-Expression Dataset Enabling Visual-Physiological Fusion
• Authors: Chuang Ma, Yu Pei, Jianhang Zhang, Shaokai Zhao, Bowen Ji, Liang Xie, Ye Yan, Erwei Yin,
• Abstract summary: Micro-expressions (MEs) are subtle, fleeting nonverbal cues that reveal an individual's genuine emotional state.<n>Existing ME research is limited to single visual modality, overlooking the rich emotional information conveyed by other physiological modalities.<n>This study introduces a novel ME, MMME, which, for the first time, enables synchronized collection of facial action signals.
• Score: 2.719872133434811
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Micro-expressions (MEs) are subtle, fleeting nonverbal cues that reveal an individual's genuine emotional state. Their analysis has attracted considerable interest due to its promising applications in fields such as healthcare, criminal investigation, and human-computer interaction. However, existing ME research is limited to single visual modality, overlooking the rich emotional information conveyed by other physiological modalities, resulting in ME recognition and spotting performance far below practical application needs. Therefore, exploring the cross-modal association mechanism between ME visual features and physiological signals (PS), and developing a multimodal fusion framework, represents a pivotal step toward advancing ME analysis. This study introduces a novel ME dataset, MMME, which, for the first time, enables synchronized collection of facial action signals (MEs), central nervous system signals (EEG), and peripheral PS (PPG, RSP, SKT, EDA, and ECG). By overcoming the constraints of existing ME corpora, MMME comprises 634 MEs, 2,841 macro-expressions (MaEs), and 2,890 trials of synchronized multimodal PS, establishing a robust foundation for investigating ME neural mechanisms and conducting multimodal fusion-based analyses. Extensive experiments validate the dataset's reliability and provide benchmarks for ME analysis, demonstrating that integrating MEs with PS significantly enhances recognition and spotting performance. To the best of our knowledge, MMME is the most comprehensive ME dataset to date in terms of modality diversity. It provides critical data support for exploring the neural mechanisms of MEs and uncovering the visual-physiological synergistic effects, driving a paradigm shift in ME research from single-modality visual analysis to multimodal fusion. The dataset will be publicly available upon acceptance of this paper.

Related papers

• Platform for Representation and Integration of multimodal Molecular Embeddings [43.54912893426355]
  Existing machine learning methods for molecular embeddings are restricted to specific tasks or data modalities.<n>Existing embeddings capture largely non-overlapping molecular signals, highlighting the value of embedding integration.<n>We propose Platform for Representation and Integration of multimodal Molecular Embeddings (PRISME) to integrate heterogeneous embeddings into a unified multimodal representation.
  arXiv  Detail & Related papers  (2025-07-10T01:18:50Z)
• MEGC2025: Micro-Expression Grand Challenge on Spot Then Recognize and Visual Question Answering [55.30507585676142]
  Facial micro-expressions (MEs) are involuntary movements of the face that occur spontaneously when a person experiences an emotion.<n>In recent years, substantial advancements have been made in the areas of ME recognition, spotting, and generation.<n>The ME grand challenge (MEGC) 2025 introduces two tasks that reflect these evolving research directions.
  arXiv  Detail & Related papers  (2025-06-18T09:29:51Z)
• Milmer: a Framework for Multiple Instance Learning based Multimodal Emotion Recognition [16.616341358877243]
  This study addresses the challenges of emotion recognition by integrating facial expression analysis with electroencephalogram (EEG) signals.<n>The proposed framework employs a transformer-based fusion approach to effectively integrate visual and physiological modalities.<n>A key innovation of this work is the adoption of a multiple instance learning (MIL) approach, which extracts meaningful information from multiple facial expression images.
  arXiv  Detail & Related papers  (2025-02-01T20:32:57Z)
• Joint Multimodal Transformer for Emotion Recognition in the Wild [49.735299182004404]
  Multimodal emotion recognition (MMER) systems typically outperform unimodal systems. This paper proposes an MMER method that relies on a joint multimodal transformer (JMT) for fusion with key-based cross-attention.
  arXiv  Detail & Related papers  (2024-03-15T17:23:38Z)
• DFME: A New Benchmark for Dynamic Facial Micro-expression Recognition [51.26943074578153]
  Micro-expression (ME) is a spontaneous, subtle, and transient facial expression that reveals human beings genuine emotion.<n>The ME data scarcity has severely hindered the development of advanced data-driven MER models.<n>In this paper, we overcome the ME data scarcity problem by collecting and annotating a dynamic spontaneous ME database.
  arXiv  Detail & Related papers  (2023-01-03T07:33:33Z)
• SEMPAI: a Self-Enhancing Multi-Photon Artificial Intelligence for prior-informed assessment of muscle function and pathology [48.54269377408277]
  We introduce the Self-Enhancing Multi-Photon Artificial Intelligence (SEMPAI), that integrates hypothesis-driven priors in a data-driven Deep Learning approach. SEMPAI performs joint learning of several tasks to enable prediction for small datasets. SEMPAI outperforms state-of-the-art biomarkers in six of seven predictive tasks, including those with scarce data.
  arXiv  Detail & Related papers  (2022-10-28T17:03:04Z)
• Video-based Facial Micro-Expression Analysis: A Survey of Datasets, Features and Algorithms [52.58031087639394]
  micro-expressions are involuntary and transient facial expressions. They can provide important information in a broad range of applications such as lie detection, criminal detection, etc. Since micro-expressions are transient and of low intensity, their detection and recognition is difficult and relies heavily on expert experiences.
  arXiv  Detail & Related papers  (2022-01-30T05:14:13Z)
• iMiGUE: An Identity-free Video Dataset for Micro-Gesture Understanding and Emotion Analysis [23.261770969903065]
  iMiGUE is identity-free video dataset for Micro-Gesture Understanding and Emotion analysis (iMiGUE) iMiGUE focuses on micro-gesture, i.e., unintentional behaviors driven by inner feelings.
  arXiv  Detail & Related papers  (2021-07-01T08:15:14Z)
• Micro-expression spotting: A new benchmark [74.69928316848866]
  Micro-expressions (MEs) are brief and involuntary facial expressions that occur when people are trying to hide their true feelings or conceal their emotions. In the computer vision field, the study of MEs can be divided into two main tasks, spotting and recognition. This paper introduces an extension of the SMIC-E database, namely the SMIC-E-Long database, which is a new challenging benchmark for ME spotting.
  arXiv  Detail & Related papers  (2020-07-24T09:18:41Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.