Modality Disentangled Learning for Incomplete Multimodal Emotion Recognition: A Primitive Memory Distillation Perspective

• URL: http://arxiv.org/abs/2608.30563v1
• Date: Mon, 31 Aug 2026 10:38:47 GMT
• Title: Modality Disentangled Learning for Incomplete Multimodal Emotion Recognition: A Primitive Memory Distillation Perspective
• Abstract summary: We propose the Primitive Memory Distillation (PriMD) framework to solve the problem of missing modalities in Emotion Recognition systems.<n>PriMD disentangles cross-modal shared semantics from modality-specific representations, and then discretizes the latter into learnable semantic primitives.<n>It compensates for missing modality-specific information within a constrained memory space and aligns with the teacher model.
• Score: 69.4071488744115
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Multimodal Emotion Recognition (MER) systems often suffer from missing modalities in real-world scenarios. Existing methods usually generate, align, or distill missing modalities as a whole, overlooking the heterogeneous nature of the information carried by each modality. Such holistic treatment mixes inferable shared semantics with uncertain modality-specific details, yielding unstable representations and degrading robustness. To address this issue, we propose the Primitive Memory Distillation (PriMD) framework. Unlike existing methods, PriMD takes an intra-modal perspective and focuses on how different types of information within a modality differ in recoverability within each modality. PriMD first disentangles cross-modal shared semantics from modality-specific representations, and then discretizes the latter into learnable semantic primitives to construct modality-specific memory banks. When modalities are missing, PriMD is a teacher-student framework that the student model uses the shared semantics of available modalities as queries to dynamically retrieve primitives. It compensates for missing modality-specific information within a constrained memory space and aligns with the teacher model. Extensive experiments on IEMOCAP, CMU-MOSI, and CMU-MOSEI demonstrate that PriMD achieves state-of-the-art performance and consistently stronger robustness across a wide range of missing-modality settings, while mitigating the instability caused by holistic feature inference. Our code and project website are available at https://github.com/JiaqiZhang-Sengoku/PriMD and https://jiaqizhang-sengoku.github.io/PriMD/, respectively.

Related papers

• Resilient Vision-Tabular Multimodal Learning under Modality Missingness [1.696842238811138]
  Multimodal deep learning has shown strong potential in medical applications.<n>Most existing approaches implicitly assume complete modality availability.<n>We propose a multimodal transformer framework for joint vision-tabular learning.
  arXiv  Detail & Related papers  (2026-05-12T12:14:13Z)
• Beyond Surface Artifacts: Capturing Shared Latent Forgery Knowledge Across Modalities [52.189426539304065]
  This paper introduces a paradigm shift that redefines multimodal forensics from conventional "feature fusion" to "modality generalization"<n>We propose the first modality-agnostic forgery (MAF) detection framework. By explicitly decoupling modality-specific styles, MAF precisely extracts the essential, cross-modal latent forgery knowledge.
  arXiv  Detail & Related papers  (2026-04-09T03:35:21Z)
• MDReID: Modality-Decoupled Learning for Any-to-Any Multi-Modal Object Re-Identification [32.811775633963066]
  MDReID is a flexible ReID framework designed to operate under both modality-matched and modality-mismatched scenarios.<n> MDReID builds on the insight that modality information can be decomposed into two components: modality-shared features that are predictable and transferable, and modality-specific features that capture unique, modality-dependent characteristics.
  arXiv  Detail & Related papers  (2025-10-27T13:08:46Z)
• No Modality Left Behind: Adapting to Missing Modalities via Knowledge Distillation for Brain Tumor Segmentation [11.776176055438313]
  AdaMM is a multi-modal brain tumor segmentation framework tailored for missing-modality scenarios.<n>AdaMM consistently outperforms existing methods, exhibiting superior segmentation accuracy and robustness.
  arXiv  Detail & Related papers  (2025-09-18T14:47:20Z)
• MuSACo: Multimodal Subject-Specific Selection and Adaptation for Expression Recognition with Co-Training [52.99217736494484]
  We introduce MuSACo, a multi-modal subject-specific selection and adaptation method for personalized expression recognition.<n>This makes MuSACo relevant for affective computing applications in digital health, such as patient-specific assessment for stress or pain.<n>Our experimental results on challenging multimodal ER datasets: BioVid and StressID, show that MuSACo can outperform UDA (blending) and state-of-the-art MSDA methods.
  arXiv  Detail & Related papers  (2025-08-17T23:08:21Z)
• Deep Correlated Prompting for Visual Recognition with Missing Modalities [22.40271366031256]
  Large-scale multimodal models have shown excellent performance over a series of tasks powered by the large corpus of paired multimodal training data. However, this simple assumption may not always hold in the real world due to privacy constraints or collection difficulty. We refer to prompt learning to adapt large pretrained multimodal models to handle missing-modality scenarios by regarding different missing cases as different types of input.
  arXiv  Detail & Related papers  (2024-10-09T05:28:43Z)
• Learning Unseen Modality Interaction [54.23533023883659]
  Multimodal learning assumes all modality combinations of interest are available during training to learn cross-modal correspondences. We pose the problem of unseen modality interaction and introduce a first solution. It exploits a module that projects the multidimensional features of different modalities into a common space with rich information preserved.
  arXiv  Detail & Related papers  (2023-06-22T10:53:10Z)
• Exploiting modality-invariant feature for robust multimodal emotion recognition with missing modalities [76.08541852988536]
  We propose to use invariant features for a missing modality imagination network (IF-MMIN) We show that the proposed model outperforms all baselines and invariantly improves the overall emotion recognition performance under uncertain missing-modality conditions.
  arXiv  Detail & Related papers  (2022-10-27T12:16:25Z)
• MMD-ReID: A Simple but Effective Solution for Visible-Thermal Person ReID [20.08880264104061]
  We propose a simple but effective framework, MMD-ReID, that reduces the modality gap by an explicit discrepancy reduction constraint. We conduct extensive experiments to demonstrate both qualitatively and quantitatively the effectiveness of MMD-ReID. The proposed framework significantly outperforms the state-of-the-art methods on SYSU-MM01 and RegDB datasets.
  arXiv  Detail & Related papers  (2021-11-09T11:33:32Z)
• Robust Multimodal Brain Tumor Segmentation via Feature Disentanglement and Gated Fusion [71.87627318863612]
  We propose a novel multimodal segmentation framework which is robust to the absence of imaging modalities. Our network uses feature disentanglement to decompose the input modalities into the modality-specific appearance code. We validate our method on the important yet challenging multimodal brain tumor segmentation task with the BRATS challenge dataset.
  arXiv  Detail & Related papers  (2020-02-22T14:32:04Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.