Unbiased Dynamic Multimodal Fusion

• URL: http://arxiv.org/abs/2603.19681v1
• Date: Fri, 20 Mar 2026 06:29:59 GMT
• Title: Unbiased Dynamic Multimodal Fusion
• Abstract summary: Traditional multimodal methods often assume static modality quality, which limits their adaptability in dynamic real-world scenarios.<n>We introduce a noise-aware uncertainty estimator that adds controlled noise to the modality data and predicts its intensity from the modality feature.<n>This forces the model to learn a clear correspondence between feature corruption and noise level, allowing accurate uncertainty measure across both low- and high-noise conditions.
• Score: 17.111704025557376
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Traditional multimodal methods often assume static modality quality, which limits their adaptability in dynamic real-world scenarios. Thus, dynamical multimodal methods are proposed to assess modality quality and adjust their contribution accordingly. However, they typically rely on empirical metrics, failing to measure the modality quality when noise levels are extremely low or high. Moreover, existing methods usually assume that the initial contribution of each modality is the same, neglecting the intrinsic modality dependency bias. As a result, the modality hard to learn would be doubly penalized, and the performance of dynamical fusion could be inferior to that of static fusion. To address these challenges, we propose the Unbiased Dynamic Multimodal Learning (UDML) framework. Specifically, we introduce a noise-aware uncertainty estimator that adds controlled noise to the modality data and predicts its intensity from the modality feature. This forces the model to learn a clear correspondence between feature corruption and noise level, allowing accurate uncertainty measure across both low- and high-noise conditions. Furthermore, we quantify the inherent modality reliance bias within multimodal networks via modality dropout and incorporate it into the weighting mechanism. This eliminates the dual suppression effect on the hard-to-learn modality. Extensive experiments across diverse multimodal benchmark tasks validate the effectiveness, versatility, and generalizability of the proposed UDML. The code is available at https://github.com/shicaiwei123/UDML.

Related papers

• General Incomplete Multimodal Learning via Dynamic Quality Perception [5.340979083445884]
  We propose a unified framework that simultaneously handles both inter-modality missing and intra-modality degradation.<n>Specifically, GIML models heterogeneous missing patterns as continuous modality information degradation.<n>To achieve reliable quality perception, we introduce a Noise-aware Quality Estor that learns the mapping from corrupted features to noise intensity.
  arXiv  Detail & Related papers  (2026-07-08T03:13:43Z)
• Label What Matters: Modality-Balanced and Difficulty-Aware Multimodal Active Learning [13.136629024689432]
  We propose RL-MBA, a reinforcement-learning framework for modality-balanced, difficulty-aware active learning.<n>Experiments on Food101, KineticsSound, and VGGSound demonstrate that RL-MBA consistently outperforms strong baselines.
  arXiv  Detail & Related papers  (2026-03-26T07:26:48Z)
• Test-time Adaptive Hierarchical Co-enhanced Denoising Network for Reliable Multimodal Classification [55.56234913868664]
  We propose Test-time Adaptive Hierarchical Co-enhanced Denoising Network (TAHCD) for reliable learning on multimodal data.<n>The proposed method achieves superior classification performance, robustness, and generalization compared with state-of-the-art reliable multimodal learning approaches.
  arXiv  Detail & Related papers  (2026-01-12T03:14:12Z)
• Multimodal Negative Learning [55.67017420486548]
  We propose a new learning paradigm: "Learning Not to be" (Negative Learning)<n>Instead of enhancing weak modalities' target-class predictions, the dominant modalities dynamically guide the weak modality to suppress non-target classes.<n>This stabilizes the decision space and preserves modality-specific information.
  arXiv  Detail & Related papers  (2025-10-23T11:47:11Z)
• BTW: A Non-Parametric Variance Stabilization Framework for Multimodal Model Integration [20.600001069987318]
  We propose Beyond Two-modality Weighting (BTW) to dynamically adjust modality importance during training.<n>BTW computes per-example KL weights by measuring the divergence between each unimodal and the current multimodal prediction.<n>Our method significantly improves regression performance and multiclass classification accuracy.
  arXiv  Detail & Related papers  (2025-08-25T23:00:38Z)
• Dynamic Modality Scheduling for Multimodal Large Models via Confidence, Uncertainty, and Semantic Consistency [0.0]
  We propose Dynamic Modality Scheduling (DMS), a novel framework that adaptively adjusts the contribution of each modality at a per-sample level.<n> Experimental results on VQA, image-text retrieval, and captioning tasks show that DMS significantly improves both clean and robust performance.
  arXiv  Detail & Related papers  (2025-06-15T05:15:52Z)
• Improving Multimodal Learning Balance and Sufficiency through Data Remixing [14.282792733217653]
  Methods for enforcing the weak modality fail to achieve unimodal sufficiency and multimodal balance.<n>We propose multimodal Data Remixing, including decoupling multimodal data and filtering hard samples for each modality to mitigate modality imbalance.<n>Our method can be seamlessly integrated with existing approaches, improving accuracy by approximately 6.50%$uparrow$ on CREMAD and 3.41%$uparrow$ on Kinetic-Sounds.
  arXiv  Detail & Related papers  (2025-06-13T08:01:29Z)
• A Study of Dropout-Induced Modality Bias on Robustness to Missing Video Frames for Audio-Visual Speech Recognition [53.800937914403654]
  Advanced Audio-Visual Speech Recognition (AVSR) systems have been observed to be sensitive to missing video frames. While applying the dropout technique to the video modality enhances robustness to missing frames, it simultaneously results in a performance loss when dealing with complete data input. We propose a novel Multimodal Distribution Approximation with Knowledge Distillation (MDA-KD) framework to reduce over-reliance on the audio modality.
  arXiv  Detail & Related papers  (2024-03-07T06:06:55Z)
• Provable Dynamic Fusion for Low-Quality Multimodal Data [94.39538027450948]
  Dynamic multimodal fusion emerges as a promising learning paradigm. Despite its widespread use, theoretical justifications in this field are still notably lacking. This paper provides theoretical understandings to answer this question under a most popular multimodal fusion framework from the generalization perspective. A novel multimodal fusion framework termed Quality-aware Multimodal Fusion (QMF) is proposed, which can improve the performance in terms of classification accuracy and model robustness.
  arXiv  Detail & Related papers  (2023-06-03T08:32:35Z)
• Cross-Attention is Not Enough: Incongruity-Aware Dynamic Hierarchical Fusion for Multimodal Affect Recognition [69.32305810128994]
  Incongruity between modalities poses a challenge for multimodal fusion, especially in affect recognition. We propose the Hierarchical Crossmodal Transformer with Dynamic Modality Gating (HCT-DMG), a lightweight incongruity-aware model. HCT-DMG: 1) outperforms previous multimodal models with a reduced size of approximately 0.8M parameters; 2) recognizes hard samples where incongruity makes affect recognition difficult; 3) mitigates the incongruity at the latent level in crossmodal attention.
  arXiv  Detail & Related papers  (2023-05-23T01:24:15Z)
• Trustworthy Multimodal Regression with Mixture of Normal-inverse Gamma Distributions [91.63716984911278]
  We introduce a novel Mixture of Normal-Inverse Gamma distributions (MoNIG) algorithm, which efficiently estimates uncertainty in principle for adaptive integration of different modalities and produces a trustworthy regression result. Experimental results on both synthetic and different real-world data demonstrate the effectiveness and trustworthiness of our method on various multimodal regression tasks.
  arXiv  Detail & Related papers  (2021-11-11T14:28:12Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.