DAIF: A Data-Driven Intermediate Fusion Framework for Multimodal Supervised Learning via Approximate Message Passing

• URL: http://arxiv.org/abs/2608.02769v1
• Date: Mon, 03 Aug 2026 18:10:58 GMT
• Title: DAIF: A Data-Driven Intermediate Fusion Framework for Multimodal Supervised Learning via Approximate Message Passing
• Abstract summary: DAIF is a data adaptive intermediate fusion framework to learn fusion structure directly from data.<n>We operate under a Bayesian multimodal factor model where the prior on the latent factors determines the cross-modal dependence.<n>Our method clusters modalities based on estimated intermodal dependence, then performs clusterwise empirical Bayes estimation of the priors.
• Score: 11.222071637855079
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Multimodal supervised learning seeks to leverage multiple heterogeneous data sources to improve predictive performance. A central challenge is determining the fusion granularity across modalities: over-integration may amplify noise while under-integration fails to exploit cross-modal dependence. Existing approaches rely on pre-specified fusion architectures, from early to late fusion, that may not adapt to the underlying dependence structure among modalities. We propose DAIF, a data adaptive intermediate fusion framework that combines random matrix theory and non-parametric dependence measures to learn fusion structure directly from data. We operate under a Bayesian multimodal factor model where the prior on the latent factors determines the cross-modal dependence. Our method clusters modalities based on estimated intermodal dependence, then performs clusterwise empirical Bayes estimation of the priors. These estimated priors are used to construct denoisers within an approximate message passing (AMP) framework, yielding denoised low-dimensional features that borrow strength across related modalities while preserving modality-specific signal. The resulting embeddings are used for downstream supervised prediction. We evaluate the framework through simulations under varying dependence structures and signal regimes, comparing against several benchmark methods, and demonstrate its practical utility on two multimodal datasets, namely a trimodal TEA-seq dataset (Swanson et al., 2021) and TCGA-BRCA dataset (Goldman et al., 2020). In the first example, we predict the expression level of a T-cell differentiation marker protein and in the second case we analyze patient survival prediction based on multimodal information. Our method competes with or outperforms the state-of-the-art techniques in both prediction problems, demonstrating its versatility across diverse supervised learning tasks.

Related papers

• Beyond Modality Fusion: Deep Ensembles for Multimodal Classification [0.38059763597999]
  In multimodal classification, late-fusion approaches classifyd modality-specific features extracted by unimodal neural networks.<n>We show that multimodal data can be effectively classified without any explicit modality fusion, using deep ensembles of unimodal networks.<n>We propose a synthetic multimodal framework that allows control over both the number of modalities and their predictive strength.
  arXiv  Detail & Related papers  (2026-07-06T12:59:45Z)
• Attention-Based Multimodal Survival Prediction with Cross-Modal Bilinear Fusion [1.4203297056262238]
  We propose a novel deep learning framework for patient-level survival prediction.<n>It integrates whole-slide histology features, RNA-seq expression profiles, and clinical variables.<n>Our architecture combines an ABMIL moduleciteilse2018attention for slide-level representation with feedforward encoders for RNA and clinical data.
  arXiv  Detail & Related papers  (2026-05-12T13:09:25Z)
• Omnimodal Dataset Distillation via High-order Proxy Alignment [59.74134776126595]
  We propose HoPA, a unified method that captures high-order cross-modal alignments via a proxy structure.<n>Our method avoids the complexity of pairwise modality modeling and enables scalable proxy analysis across heterogeneous modalities.
  arXiv  Detail & Related papers  (2026-04-12T14:47:41Z)
• AlignMamba-2: Enhancing Multimodal Fusion and Sentiment Analysis with Modality-Aware Mamba [56.52470564147458]
  We propose textbfAlignMamba-2, an effective and efficient framework for multimodal fusion and sentiment analysis.<n>Our approach introduces a dual alignment strategy that regularizes the model using both Optimal Transport distance and Maximum Mean Discrepancy.<n>More importantly, we design a Modality-Aware Mamba layer, which employs a Mixture-of-Experts architecture with modality-specific and modality-shared experts.
  arXiv  Detail & Related papers  (2026-03-19T03:47:21Z)
• Explainable Multimodal Regression via Information Decomposition [27.157278306251772]
  We propose a novel multimodal regression framework grounded in Partial Information Decomposition (PID)<n>Our framework outperforms state-of-the-art methods in both predictive accuracy and interpretability, while also enabling informed modality selection for efficient inference.
  arXiv  Detail & Related papers  (2025-12-26T18:07:18Z)
• Multi-modal Bayesian Neural Network Surrogates with Conjugate Last-Layer Estimation [0.30586855806896046]
  We develop two multi-modal neural network surrogate models and leverage conditionally conjugate distributions in the last layer to estimate model parameters.<n>We demonstrate improved prediction accuracy and uncertainty compared to uni-modal surrogate models for both scalar and time series data.
  arXiv  Detail & Related papers  (2025-09-26T00:13:57Z)
• Learning Discrete Bayesian Networks with Hierarchical Dirichlet Shrinkage [52.914168158222765]
  We detail a comprehensive Bayesian framework for learning DBNs.<n>We give a novel Markov chain Monte Carlo (MCMC) algorithm utilizing parallel Langevin proposals to generate exact posterior samples.<n>We apply our methodology to uncover prognostic network structure from primary breast cancer samples.
  arXiv  Detail & Related papers  (2025-09-16T17:24:35Z)
• NSF-MAP: Neurosymbolic Multimodal Fusion for Robust and Interpretable Anomaly Prediction in Assembly Pipelines [0.0]
  This paper proposes a neurosymbolic AI and fusion-based approach for multimodal anomaly prediction in assembly pipelines.<n>We introduce a time series and image-based fusion model that leverages decision-level fusion techniques.<n>The results demonstrate that a neurosymbolic AI-based fusion approach that uses transfer learning can effectively harness the complementary strengths of time series and image data.
  arXiv  Detail & Related papers  (2025-05-09T16:50:42Z)
• Ensemble Modeling for Multimodal Visual Action Recognition [50.38638300332429]
  We propose an ensemble modeling approach for multimodal action recognition. We independently train individual modality models using a variant of focal loss tailored to handle the long-tailed distribution of the MECCANO [21] dataset.
  arXiv  Detail & Related papers  (2023-08-10T08:43:20Z)
• Quantifying & Modeling Multimodal Interactions: An Information Decomposition Framework [89.8609061423685]
  We propose an information-theoretic approach to quantify the degree of redundancy, uniqueness, and synergy relating input modalities with an output task. To validate PID estimation, we conduct extensive experiments on both synthetic datasets where the PID is known and on large-scale multimodal benchmarks. We demonstrate their usefulness in (1) quantifying interactions within multimodal datasets, (2) quantifying interactions captured by multimodal models, (3) principled approaches for model selection, and (4) three real-world case studies.
  arXiv  Detail & Related papers  (2023-02-23T18:59:05Z)
• Trustworthy Multimodal Regression with Mixture of Normal-inverse Gamma Distributions [91.63716984911278]
  We introduce a novel Mixture of Normal-Inverse Gamma distributions (MoNIG) algorithm, which efficiently estimates uncertainty in principle for adaptive integration of different modalities and produces a trustworthy regression result. Experimental results on both synthetic and different real-world data demonstrate the effectiveness and trustworthiness of our method on various multimodal regression tasks.
  arXiv  Detail & Related papers  (2021-11-11T14:28:12Z)
• Bi-Bimodal Modality Fusion for Correlation-Controlled Multimodal Sentiment Analysis [96.46952672172021]
  Bi-Bimodal Fusion Network (BBFN) is a novel end-to-end network that performs fusion on pairwise modality representations. Model takes two bimodal pairs as input due to known information imbalance among modalities.
  arXiv  Detail & Related papers  (2021-07-28T23:33:42Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.