Bridging the Emotional Semantic Gap via Multimodal Relevance Estimation

• URL: http://arxiv.org/abs/2302.01555v1
• Date: Fri, 3 Feb 2023 05:27:52 GMT
• Title: Bridging the Emotional Semantic Gap via Multimodal Relevance Estimation
• Abstract summary: We propose a multimodal relevance estimation network to capture the relevant semantics among modalities in multimodal emotions. We make use of contrastive learning to optimize the similarity of category-level modality-relevant semantics across different modalities in feature space. In order to better reflect the emotional state in the real interactive scenarios, we collect a single-label discrete multimodal emotion dataset named SDME.
• Score: 9.632084332065523
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Human beings have rich ways of emotional expressions, including facial action, voice, and natural languages. Due to the diversity and complexity of different individuals, the emotions expressed by various modalities may be semantically irrelevant. Directly fusing information from different modalities may inevitably make the model subject to the noise from semantically irrelevant modalities. To tackle this problem, we propose a multimodal relevance estimation network to capture the relevant semantics among modalities in multimodal emotions. Specifically, we take advantage of an attention mechanism to reflect the semantic relevance weights of each modality. Moreover, we propose a relevant semantic estimation loss to weakly supervise the semantics of each modality. Furthermore, we make use of contrastive learning to optimize the similarity of category-level modality-relevant semantics across different modalities in feature space, thereby bridging the semantic gap between heterogeneous modalities. In order to better reflect the emotional state in the real interactive scenarios and perform the semantic relevance analysis, we collect a single-label discrete multimodal emotion dataset named SDME, which enables researchers to conduct multimodal semantic relevance research with large category bias. Experiments on continuous and discrete emotion datasets show that our model can effectively capture the relevant semantics, especially for the large deviations in modal semantics. The code and SDME dataset will be publicly available.

Related papers

• Dynamic Fusion-Aware Graph Convolutional Neural Network for Multimodal Emotion Recognition in Conversations [38.80171097773061]
  We propose a dynamic fusion-aware graph convolutional neural network (DF-GCN) for robust recognition of multimodal emotion features in conversations.<n> DF-GCN integrates ordinary differential equations into graph convolutional networks (GCNs) to capture the dynamic nature of emotional dependencies within utterance interaction networks.
  arXiv  Detail & Related papers  (2026-03-22T02:43:10Z)
• AMB-DSGDN: Adaptive Modality-Balanced Dynamic Semantic Graph Differential Network for Multimodal Emotion Recognition [35.370240758006936]
  Multimodal dialogue emotion recognition captures emotional cues by fusing text, visual, and audio modalities.<n>Existing approaches still suffer from notable limitations in modeling emotional dependencies and learning multimodal representations.<n>We propose an Adaptive Modality-Balanced Dynamic Semantic Graph Differential Network (AMB-DSGDN)
  arXiv  Detail & Related papers  (2026-03-07T11:23:23Z)
• Memory-guided Prototypical Co-occurrence Learning for Mixed Emotion Recognition [56.00118641432005]
  We propose a Memory-guided Prototypical Co-occurrence Learning framework that explicitly models emotion co-occurrence patterns.<n>Inspired by human cognitive memory systems, we introduce a memory retrieval strategy to extract semantic-level co-occurrence associations.<n>Our model learns affectively informative representations for accurate emotion distribution prediction.
  arXiv  Detail & Related papers  (2026-02-24T04:11:25Z)
• Multi-Modal Manipulation via Multi-Modal Policy Consensus [62.49978559936122]
  We propose a new approach to integrate diverse sensory modalities for robotic manipulation.<n>Our method factorizes the policy into a set of diffusion models, each specialized for a single representation.<n>We evaluate our approach on simulated manipulation tasks in RLBench, as well as real-world tasks such as occluded object picking, in-hand spoon reorientation, and puzzle insertion.
  arXiv  Detail & Related papers  (2025-09-27T19:43:04Z)
• Explaining multimodal LLMs via intra-modal token interactions [55.27436637894534]
  Multimodal Large Language Models (MLLMs) have achieved remarkable success across diverse vision-language tasks, yet their internal decision-making mechanisms remain insufficiently understood.<n>We propose enhancing interpretability by leveraging intra-modal interaction.
  arXiv  Detail & Related papers  (2025-09-26T14:39:13Z)
• Structures Meet Semantics: Multimodal Fusion via Graph Contrastive Learning [8.187594234413568]
  We propose a novel framework called the Structural-Semantic Unifier (SSU)<n>SSU integrates modality-specific structural information and cross-modal semantic grounding for enhanced multimodal representations.<n>SSU consistently achieves state-of-the-art performance while significantly reducing computational overhead.
  arXiv  Detail & Related papers  (2025-08-24T13:44:54Z)
• Unified Generative and Discriminative Training for Multi-modal Large Language Models [88.84491005030316]
  Generative training has enabled Vision-Language Models (VLMs) to tackle various complex tasks. Discriminative training, exemplified by models like CLIP, excels in zero-shot image-text classification and retrieval. This paper proposes a unified approach that integrates the strengths of both paradigms.
  arXiv  Detail & Related papers  (2024-11-01T01:51:31Z)
• AMuSE: Adaptive Multimodal Analysis for Speaker Emotion Recognition in Group Conversations [39.79734528362605]
  Multimodal Attention Network captures cross-modal interactions at various levels of spatial abstraction. AMuSE model condenses both spatial and temporal features into two dense descriptors: speaker-level and utterance-level.
  arXiv  Detail & Related papers  (2024-01-26T19:17:05Z)
• A Multi-Task, Multi-Modal Approach for Predicting Categorical and Dimensional Emotions [0.0]
  We propose a multi-task, multi-modal system that predicts categorical and dimensional emotions. Results emphasise the importance of cross-regularisation between the two types of emotions.
  arXiv  Detail & Related papers  (2023-12-31T16:48:03Z)
• Adversarial Representation with Intra-Modal and Inter-Modal Graph Contrastive Learning for Multimodal Emotion Recognition [14.639340916340801]
  We propose a novel Adversarial Representation with Intra-Modal and Inter-Modal Graph Contrastive for Multimodal Emotion Recognition (AR-IIGCN) method. Firstly, we input video, audio, and text features into a multi-layer perceptron (MLP) to map them into separate feature spaces. Secondly, we build a generator and a discriminator for the three modal features through adversarial representation. Thirdly, we introduce contrastive graph representation learning to capture intra-modal and inter-modal complementary semantic information.
  arXiv  Detail & Related papers  (2023-12-28T01:57:26Z)
• A Transformer-Based Model With Self-Distillation for Multimodal Emotion Recognition in Conversations [15.77747948751497]
  We propose a transformer-based model with self-distillation (SDT) for the task. The proposed model captures intra- and inter-modal interactions by utilizing intra- and inter-modal transformers. We introduce self-distillation to transfer knowledge of hard and soft labels from the proposed model to each modality.
  arXiv  Detail & Related papers  (2023-10-31T14:33:30Z)
• Syntax and Semantics Meet in the "Middle": Probing the Syntax-Semantics Interface of LMs Through Agentivity [68.8204255655161]
  We present the semantic notion of agentivity as a case study for probing such interactions. This suggests LMs may potentially serve as more useful tools for linguistic annotation, theory testing, and discovery.
  arXiv  Detail & Related papers  (2023-05-29T16:24:01Z)
• Disentangling Learnable and Memorizable Data via Contrastive Learning for Semantic Communications [81.10703519117465]
  A novel machine reasoning framework is proposed to disentangle source data so as to make it semantic-ready. In particular, a novel contrastive learning framework is proposed, whereby instance and cluster discrimination are performed on the data. Deep semantic clusters of highest confidence are considered learnable, semantic-rich data. Our simulation results showcase the superiority of our contrastive learning approach in terms of semantic impact and minimalism.
  arXiv  Detail & Related papers  (2022-12-18T12:00:12Z)
• Imitation Learning-based Implicit Semantic-aware Communication Networks: Multi-layer Representation and Collaborative Reasoning [68.63380306259742]
  Despite its promising potential, semantic communications and semantic-aware networking are still at their infancy. We propose a novel reasoning-based implicit semantic-aware communication network architecture that allows multiple tiers of CDC and edge servers to collaborate. We introduce a new multi-layer representation of semantic information taking into consideration both the hierarchical structure of implicit semantics as well as the personalized inference preference of individual users.
  arXiv  Detail & Related papers  (2022-10-28T13:26:08Z)
• Group Gated Fusion on Attention-based Bidirectional Alignment for Multimodal Emotion Recognition [63.07844685982738]
  This paper presents a new model named as Gated Bidirectional Alignment Network (GBAN), which consists of an attention-based bidirectional alignment network over LSTM hidden states. We empirically show that the attention-aligned representations outperform the last-hidden-states of LSTM significantly. The proposed GBAN model outperforms existing state-of-the-art multimodal approaches on the IEMOCAP dataset.
  arXiv  Detail & Related papers  (2022-01-17T09:46:59Z)
• Affect-DML: Context-Aware One-Shot Recognition of Human Affect using Deep Metric Learning [29.262204241732565]
  Existing methods assume that all emotions-of-interest are given a priori as annotated training examples. We conceptualize one-shot recognition of emotions in context -- a new problem aimed at recognizing human affect states in finer particle level from a single support sample. All variants of our model clearly outperform the random baseline, while leveraging the semantic scene context consistently improves the learnt representations.
  arXiv  Detail & Related papers  (2021-11-30T10:35:20Z)
• Multimodal Routing: Improving Local and Global Interpretability of Multimodal Language Analysis [103.69656907534456]
  Recent multimodal learning with strong performances on human-centric tasks are often black-box. We propose Multimodal Routing, which adjusts weights between input modalities and output representations differently for each input sample.
  arXiv  Detail & Related papers  (2020-04-29T13:42:22Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.