ERIT Lightweight Multimodal Dataset for Elderly Emotion Recognition and Multimodal Fusion Evaluation

• URL: http://arxiv.org/abs/2407.17772v1
• Date: Thu, 25 Jul 2024 05:02:27 GMT
• Title: ERIT Lightweight Multimodal Dataset for Elderly Emotion Recognition and Multimodal Fusion Evaluation
• Abstract summary: ERIT is a novel multimodal dataset designed to facilitate research in a lightweight multimodal fusion. It contains text and image data collected from videos of elderly individuals reacting to various situations, as well as seven emotion labels for each data sample.
• Score: 35.10226782746507
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: ERIT is a novel multimodal dataset designed to facilitate research in a lightweight multimodal fusion. It contains text and image data collected from videos of elderly individuals reacting to various situations, as well as seven emotion labels for each data sample. Because of the use of labeled images of elderly users reacting emotionally, it is also facilitating research on emotion recognition in an underrepresented age group in machine learning visual emotion recognition. The dataset is validated through comprehensive experiments indicating its importance in neural multimodal fusion research.

Related papers

• MECO: A Multimodal Dataset for Emotion and Cognitive Understanding in Older Adults [19.96001371308951]
  We present MECO, a Multimodal dataset for Emotion and Cognitive understanding in Older adults.<n>MECO includes 42 participants and provides approximately 38 hours of multimodal signals, yielding 30,592 synchronized samples.<n>The dataset offers comprehensive annotations of emotional and cognitive states, including self-assessed valence, arousal, six basic emotions, and Mini-Mental State Examination cognitive scores.
  arXiv  Detail & Related papers  (2026-04-03T14:03:23Z)
• Smile on the Face, Sadness in the Eyes: Bridging the Emotion Gap with a Multimodal Dataset of Eye and Facial Behaviors [49.833812625518554]
  We introduce eye behaviors as an important emotional cue and construct an Eye-behavior-aided Multimodal Emotion Recognition dataset.<n>In the experiment, we introduce seven multimodal benchmark protocols for a variety of comprehensive evaluations of the EMER dataset.<n>The results show that the EMERT outperforms other state-of-the-art multimodal methods by a great margin, revealing the importance of modeling eye behaviors for robust ER.
  arXiv  Detail & Related papers  (2025-12-18T12:52:55Z)
• EmoVerse: A MLLMs-Driven Emotion Representation Dataset for Interpretable Visual Emotion Analysis [61.87711517626139]
  EmoVerse is a large-scale open-source dataset that enables interpretable visual emotion analysis.<n>With over 219k images, the dataset further includes dual annotations in Categorical Emotion States (CES) and Dimensional Emotion Space (DES)
  arXiv  Detail & Related papers  (2025-11-16T11:16:50Z)
• ASLSL: Adaptive shared latent structure learning with incomplete multi-modal physiological data for multi-dimensional emotional feature selection [6.558247638028055]
  We propose a novel method for incomplete multi-modal physiological signal feature selection called adaptive shared latent structure learning (ASLSL)<n>Based on the property that similar features share similar emotional labels, ASLSL employs adaptive shared latent structure learning to explore a common latent space shared for incomplete multi-modal physiological signals and multi-dimensional emotional labels.
  arXiv  Detail & Related papers  (2025-08-08T01:54:02Z)
• Bridging Cognition and Emotion: Empathy-Driven Multimodal Misinformation Detection [56.644686934050576]
  Social media has become a major conduit for information dissemination, yet it also facilitates the rapid spread of misinformation. Traditional misinformation detection methods primarily focus on surface-level features, overlooking the crucial roles of human empathy in the propagation process. We propose the Dual-Aspect Empathy Framework (DAE), which integrates cognitive and emotional empathy to analyze misinformation from both the creator and reader perspectives.
  arXiv  Detail & Related papers  (2025-04-24T07:48:26Z)
• CG-MER: A Card Game-based Multimodal dataset for Emotion Recognition [0.0]
  This paper introduces a comprehensive French multimodal dataset designed specifically for emotion recognition. The dataset encompasses three primary modalities: facial expressions, speech, and gestures, providing a holistic perspective on emotions. The dataset has the potential to incorporate additional modalities, such as Natural Language Processing (NLP) to expand the scope of emotion recognition research.
  arXiv  Detail & Related papers  (2025-01-14T15:08:56Z)
• Enriching Multimodal Sentiment Analysis through Textual Emotional Descriptions of Visual-Audio Content [56.62027582702816]
  Multimodal Sentiment Analysis seeks to unravel human emotions by amalgamating text, audio, and visual data. Yet, discerning subtle emotional nuances within audio and video expressions poses a formidable challenge. We introduce DEVA, a progressive fusion framework founded on textual sentiment descriptions.
  arXiv  Detail & Related papers  (2024-12-12T11:30:41Z)
• VR Based Emotion Recognition Using Deep Multimodal Fusion With Biosignals Across Multiple Anatomical Domains [3.303674512749726]
  We introduce a novel multi-scale attention-based LSTM architecture, combined with Squeeze-and-Excitation (SE) blocks. The proposed architecture, validated in a user study, demonstrates superior performance in classifying valance and arousal level.
  arXiv  Detail & Related papers  (2024-12-03T08:59:12Z)
• Smile upon the Face but Sadness in the Eyes: Emotion Recognition based on Facial Expressions and Eye Behaviors [63.194053817609024]
  We introduce eye behaviors as an important emotional cues for the creation of a new Eye-behavior-aided Multimodal Emotion Recognition dataset. For the first time, we provide annotations for both Emotion Recognition (ER) and Facial Expression Recognition (FER) in the EMER dataset. We specifically design a new EMERT architecture to concurrently enhance performance in both ER and FER.
  arXiv  Detail & Related papers  (2024-11-08T04:53:55Z)
• Hypergraph Multi-modal Large Language Model: Exploiting EEG and Eye-tracking Modalities to Evaluate Heterogeneous Responses for Video Understanding [25.4933695784155]
  Understanding of video creativity and content often varies among individuals, with differences in focal points and cognitive levels across different ages, experiences, and genders. To bridge the gap to real-world applications, we introduce a large-scale Subjective Response Indicators for Advertisement Videos dataset. We developed tasks and protocols to analyze and evaluate the extent of cognitive understanding of video content among different users.
  arXiv  Detail & Related papers  (2024-07-11T03:00:26Z)
• EIT-1M: One Million EEG-Image-Text Pairs for Human Visual-textual Recognition and More [10.918317402874665]
  We propose a novel large-scale multi-modal dataset, named EIT-1M, with over 1 million EEG-image-text pairs. Our dataset is superior in its capacity of reflecting brain activities in simultaneously processing multi-modal information. We provide an in-depth analysis of EEG data captured from multi-modal stimuli across different categories and participants, along with data quality scores for transparency.
  arXiv  Detail & Related papers  (2024-07-02T02:11:15Z)
• FAF: A novel multimodal emotion recognition approach integrating face, body and text [13.485538135494153]
  We develop a large multimodal emotion dataset, named "HED" dataset, to facilitate the emotion recognition task. To promote recognition accuracy, "Feature After Feature" framework was used to explore crucial emotional information. We employ various benchmarks to evaluate the "HED" dataset and compare the performance with our method.
  arXiv  Detail & Related papers  (2022-11-20T14:43:36Z)
• Vision+X: A Survey on Multimodal Learning in the Light of Data [64.03266872103835]
  multimodal machine learning that incorporates data from various sources has become an increasingly popular research area. We analyze the commonness and uniqueness of each data format mainly ranging from vision, audio, text, and motions. We investigate the existing literature on multimodal learning from both the representation learning and downstream application levels.
  arXiv  Detail & Related papers  (2022-10-05T13:14:57Z)
• VISTANet: VIsual Spoken Textual Additive Net for Interpretable Multimodal Emotion Recognition [21.247650660908484]
  This paper proposes a multimodal emotion recognition system, VIsual Textual Additive Net (VISTANet) The VISTANet fuses information from image, speech, and text modalities using a hybrid of early and late fusion. The KAAP technique computes the contribution of each modality and corresponding features toward predicting a particular emotion class.
  arXiv  Detail & Related papers  (2022-08-24T11:35:51Z)
• Multimodal Emotion Recognition with Modality-Pairwise Unsupervised Contrastive Loss [80.79641247882012]
  We focus on unsupervised feature learning for Multimodal Emotion Recognition (MER) We consider discrete emotions, and as modalities text, audio and vision are used. Our method, as being based on contrastive loss between pairwise modalities, is the first attempt in MER literature.
  arXiv  Detail & Related papers  (2022-07-23T10:11:24Z)
• Multimodal Image Synthesis and Editing: The Generative AI Era [131.9569600472503]
  multimodal image synthesis and editing has become a hot research topic in recent years. We comprehensively contextualize the advance of the recent multimodal image synthesis and editing. We describe benchmark datasets and evaluation metrics as well as corresponding experimental results.
  arXiv  Detail & Related papers  (2021-12-27T10:00:16Z)
• Attentive Cross-modal Connections for Deep Multimodal Wearable-based Emotion Recognition [7.559720049837459]
  We present a novel attentive cross-modal connection to share information between convolutional neural networks. Specifically, these connections improve emotion classification by sharing intermediate representations among EDA and ECG. Our experiments show that the proposed approach is capable of learning strong multimodal representations and outperforms a number of baselines methods.
  arXiv  Detail & Related papers  (2021-08-04T18:40:32Z)
• Affective Image Content Analysis: Two Decades Review and New Perspectives [132.889649256384]
  We will comprehensively review the development of affective image content analysis (AICA) in the recent two decades. We will focus on the state-of-the-art methods with respect to three main challenges -- the affective gap, perception subjectivity, and label noise and absence. We discuss some challenges and promising research directions in the future, such as image content and context understanding, group emotion clustering, and viewer-image interaction.
  arXiv  Detail & Related papers  (2021-06-30T15:20:56Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.