DLF: Disentangled-Language-Focused Multimodal Sentiment Analysis

• URL: http://arxiv.org/abs/2412.12225v3
• Date: Wed, 09 Apr 2025 00:52:30 GMT
• Title: DLF: Disentangled-Language-Focused Multimodal Sentiment Analysis
• Authors: Pan Wang, Qiang Zhou, Yawen Wu, Tianlong Chen, Jingtong Hu,
• Abstract summary: We propose a Disentangled-Language-Focused (DLF) multimodal representation learning framework.<n>It incorporates a feature disentanglement module to separate modality-shared and modality-specific information.<n>A Language-Focused Attractor (LFA) is further developed to strengthen language representation by leveraging complementary modality-specific information.
• Score: 41.29318462528406
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: Multimodal Sentiment Analysis (MSA) leverages heterogeneous modalities, such as language, vision, and audio, to enhance the understanding of human sentiment. While existing models often focus on extracting shared information across modalities or directly fusing heterogeneous modalities, such approaches can introduce redundancy and conflicts due to equal treatment of all modalities and the mutual transfer of information between modality pairs. To address these issues, we propose a Disentangled-Language-Focused (DLF) multimodal representation learning framework, which incorporates a feature disentanglement module to separate modality-shared and modality-specific information. To further reduce redundancy and enhance language-targeted features, four geometric measures are introduced to refine the disentanglement process. A Language-Focused Attractor (LFA) is further developed to strengthen language representation by leveraging complementary modality-specific information through a language-guided cross-attention mechanism. The framework also employs hierarchical predictions to improve overall accuracy. Extensive experiments on two popular MSA datasets, CMU-MOSI and CMU-MOSEI, demonstrate the significant performance gains achieved by the proposed DLF framework. Comprehensive ablation studies further validate the effectiveness of the feature disentanglement module, language-focused attractor, and hierarchical predictions. Our code is available at https://github.com/pwang322/DLF.

Related papers

• Relation-R1: Cognitive Chain-of-Thought Guided Reinforcement Learning for Unified Relational Comprehension [12.563060744760651]
  Relation-R1 is the first unified relational comprehension framework. It integrates cognitive chain-of-thought (CoT)-guided Supervised Fine-Tuning (SFT) and Group Relative Policy Optimization (GRPO) Experiments on widely-used PSG and SWiG datasets demonstrate that Relation-R1 achieves state-of-the-art performance in both binary and textitN-ary relation understanding.
  arXiv  Detail & Related papers  (2025-04-20T14:50:49Z)
• Distributional Vision-Language Alignment by Cauchy-Schwarz Divergence [83.15764564701706]
  We propose a novel framework that performs distributional vision-language alignment by integrating Cauchy-Schwarz divergence with mutual information. In the proposed framework, we find that the CS divergence and mutual information serve complementary roles in multimodal alignment, capturing both the global distribution information of each modality and the pairwise semantic relationships. Experiments on text-to-image generation and cross-modality retrieval tasks demonstrate the effectiveness of our method on vision-language alignment.
  arXiv  Detail & Related papers  (2025-02-24T10:29:15Z)
• Cross-domain Few-shot Object Detection with Multi-modal Textual Enrichment [21.36633828492347]
  Cross-Domain Multi-Modal Few-Shot Object Detection (CDMM-FSOD) We introduce a meta-learning-based framework designed to leverage rich textual semantics as an auxiliary modality to achieve effective domain adaptation. We evaluate the proposed method on common cross-domain object detection benchmarks and demonstrate that it significantly surpasses existing few-shot object detection approaches.
  arXiv  Detail & Related papers  (2025-02-23T06:59:22Z)
• A Survey on Mechanistic Interpretability for Multi-Modal Foundation Models [74.48084001058672]
  The rise of foundation models has transformed machine learning research. multimodal foundation models (MMFMs) pose unique interpretability challenges beyond unimodal frameworks. This survey explores two key aspects: (1) the adaptation of LLM interpretability methods to multimodal models and (2) understanding the mechanistic differences between unimodal language models and crossmodal systems.
  arXiv  Detail & Related papers  (2025-02-22T20:55:26Z)
• Toward Robust Incomplete Multimodal Sentiment Analysis via Hierarchical Representation Learning [21.127950337002776]
  Multimodal Sentiment Analysis (MSA) is an important research area that aims to understand and recognize human sentiment through multiple modalities. We propose a Hierarchical Representation Learning Framework (HRLF) for the task under uncertain missing modalities. We show that HRLF significantly improves MSA performance under uncertain modality missing cases.
  arXiv  Detail & Related papers  (2024-11-05T04:04:41Z)
• Cross-domain Multi-modal Few-shot Object Detection via Rich Text [21.36633828492347]
  Cross-modal feature extraction and integration have led to steady performance improvements in few-shot learning tasks. We study the Cross-Domain few-shot generalization of MM-OD (CDMM-FSOD) and propose a meta-learning based multi-modal few-shot object detection method.
  arXiv  Detail & Related papers  (2024-03-24T15:10:22Z)
• Learning Language-guided Adaptive Hyper-modality Representation for Multimodal Sentiment Analysis [22.012103941836838]
  We present Adaptive Language-guided Multimodal Transformer (ALMT) ALMT incorporates an Adaptive Hyper-modality Learning (AHL) module to learn an irrelevance/conflict-suppressing representation. ALMT achieves state-of-the-art performance on several popular datasets.
  arXiv  Detail & Related papers  (2023-10-09T15:43:07Z)
• Exploiting Modality-Specific Features For Multi-Modal Manipulation Detection And Grounding [54.49214267905562]
  We construct a transformer-based framework for multi-modal manipulation detection and grounding tasks. Our framework simultaneously explores modality-specific features while preserving the capability for multi-modal alignment. We propose an implicit manipulation query (IMQ) that adaptively aggregates global contextual cues within each modality.
  arXiv  Detail & Related papers  (2023-09-22T06:55:41Z)
• Multi-Grained Multimodal Interaction Network for Entity Linking [65.30260033700338]
  Multimodal entity linking task aims at resolving ambiguous mentions to a multimodal knowledge graph. We propose a novel Multi-GraIned Multimodal InteraCtion Network $textbf(MIMIC)$ framework for solving the MEL task.
  arXiv  Detail & Related papers  (2023-07-19T02:11:19Z)
• Incorporating Linguistic Knowledge for Abstractive Multi-document Summarization [20.572283625521784]
  We develop a neural network based abstractive multi-document summarization (MDS) model. We process the dependency information into the linguistic-guided attention mechanism. With the help of linguistic signals, sentence-level relations can be correctly captured.
  arXiv  Detail & Related papers  (2021-09-23T08:13:35Z)
• Learning Relation Alignment for Calibrated Cross-modal Retrieval [52.760541762871505]
  We propose a novel metric, Intra-modal Self-attention Distance (ISD), to quantify the relation consistency by measuring the semantic distance between linguistic and visual relations. We present Inter-modal Alignment on Intra-modal Self-attentions (IAIS), a regularized training method to optimize the ISD and calibrate intra-modal self-attentions mutually via inter-modal alignment.
  arXiv  Detail & Related papers  (2021-05-28T14:25:49Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.