Learning Invariant Modality Representation for Robust Multimodal Learning from a Causal Inference Perspective

• URL: http://arxiv.org/abs/2604.18460v1
• Date: Mon, 20 Apr 2026 16:16:36 GMT
• Title: Learning Invariant Modality Representation for Robust Multimodal Learning from a Causal Inference Perspective
• Abstract summary: We propose a causal modality-invariant representation (CmIR) learning framework for robust multimodal learning.<n>CmIR ensures that the learned invariant representations retain stable predictive relationships with labels across different environments.<n>CmIR excels on out-of-distribution data and noisy data, confirming its robustness and generalizability.
• Score: 6.3310165899037045
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Multimodal affective computing aims to predict humans' sentiment, emotion, intention, and opinion using language, acoustic, and visual modalities. However, current models often learn spurious correlations that harm generalization under distribution shifts or noisy modalities. To address this, we propose a causal modality-invariant representation (CmIR) learning framework for robust multimodal learning. At its core, we introduce a theoretically grounded disentanglement method that separates each modality into `causal invariant representation' and `environment-specific spurious representation' from a causal inference perspective. CmIR ensures that the learned invariant representations retain stable predictive relationships with labels across different environments while preserving sufficient information from the raw inputs via invariance constraint, mutual information constraint, and reconstruction constraint. Experiments across multiple multimodal benchmarks demonstrate that CmIR achieves state-of-the-art performance. CmIR particularly excels on out-of-distribution data and noisy data, confirming its robustness and generalizability.

Related papers

• From Sparse Decisions to Dense Reasoning: A Multi-attribute Trajectory Paradigm for Multimodal Moderation [59.27094165576015]
  We propose a novel learning paradigm (UniMod) that transitions from sparse decision-making to dense reasoning traces.<n>By constructing structured trajectories encompassing evidence grounding, modality assessment, risk mapping, policy decision, and response generation, we reformulate monolithic decision tasks into a multi-dimensional boundary learning process.<n>We introduce specialized optimization strategies to decouple task-specific parameters and rebalance training dynamics, effectively resolving interference between diverse objectives in multi-task learning.
  arXiv  Detail & Related papers  (2026-01-28T09:29:40Z)
• Dual-Stream Cross-Modal Representation Learning via Residual Semantic Decorrelation [5.272868130772015]
  Cross-modal representations often suffer from modality dominance, redundant information coupling, and spurious cross-modal correlations.<n>We propose a Dual-Stream Residual Semantic Decorrelation Network (DSRSD-Net) to disentangle modality-specific and modality-shared information.
  arXiv  Detail & Related papers  (2025-12-08T14:01:16Z)
• Explaining multimodal LLMs via intra-modal token interactions [55.27436637894534]
  Multimodal Large Language Models (MLLMs) have achieved remarkable success across diverse vision-language tasks, yet their internal decision-making mechanisms remain insufficiently understood.<n>We propose enhancing interpretability by leveraging intra-modal interaction.
  arXiv  Detail & Related papers  (2025-09-26T14:39:13Z)
• Deceptive Risk Minimization: Out-of-Distribution Generalization by Deceiving Distribution Shift Detectors [13.676316138121395]
  This paper proposes deception as a mechanism for out-of-distribution generalization.<n>By learning data representations that make training data appear independent and identically distributed to an observer, we can identify stable features that eliminate spurious correlations.
  arXiv  Detail & Related papers  (2025-09-15T16:11:55Z)
• Robust Multimodal Learning via Representation Decoupling [6.7678581401558295]
  Multimodal learning has attracted increasing attention due to its practicality. Existing methods tend to address it by learning a common subspace representation for different modality combinations. We propose a novel Decoupled Multimodal Representation Network (DMRNet) to assist robust multimodal learning.
  arXiv  Detail & Related papers  (2024-07-05T12:09:33Z)
• Beyond DAGs: A Latent Partial Causal Model for Multimodal Learning [80.44084021062105]
  We propose a novel latent partial causal model for multimodal data, featuring two latent coupled variables, connected by an undirected edge, to represent the transfer of knowledge across modalities.<n>Under specific statistical assumptions, we establish an identifiability result, demonstrating that representations learned by multimodal contrastive learning correspond to the latent coupled variables up to a trivial transformation.<n>Experiments on a pre-trained CLIP model embodies disentangled representations, enabling few-shot learning and improving domain generalization across diverse real-world datasets.
  arXiv  Detail & Related papers  (2024-02-09T07:18:06Z)
• Causal Flow-based Variational Auto-Encoder for Disentangled Causal Representation Learning [1.4875602190483512]
  Disentangled representation learning aims to learn representations where each dimension corresponds to an underlying generative factor.<n>We propose the Disentangled Causal Variational Auto-Encoder (DCVAE), a novel supervised VAE framework that integrates causal flows into the representation learning process.<n>We evaluate DCVAE on both synthetic and real-world datasets, demonstrating its superior ability in causal disentanglement and intervention experiments.
  arXiv  Detail & Related papers  (2023-04-18T14:26:02Z)
• Correlation Information Bottleneck: Towards Adapting Pretrained Multimodal Models for Robust Visual Question Answering [63.87200781247364]
  Correlation Information Bottleneck (CIB) seeks a tradeoff between compression and redundancy in representations. We derive a tight theoretical upper bound for the mutual information between multimodal inputs and representations.
  arXiv  Detail & Related papers  (2022-09-14T22:04:10Z)
• Variational Distillation for Multi-View Learning [104.17551354374821]
  We design several variational information bottlenecks to exploit two key characteristics for multi-view representation learning. Under rigorously theoretical guarantee, our approach enables IB to grasp the intrinsic correlation between observations and semantic labels.
  arXiv  Detail & Related papers  (2022-06-20T03:09:46Z)
• Invariant Causal Prediction for Block MDPs [106.63346115341862]
  Generalization across environments is critical to the successful application of reinforcement learning algorithms to real-world challenges. We propose a method of invariant prediction to learn model-irrelevance state abstractions (MISA) that generalize to novel observations in the multi-environment setting.
  arXiv  Detail & Related papers  (2020-03-12T21:03:01Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.