Fuse It or Lose It: Deep Fusion for Multimodal Simulation-Based Inference

• URL: http://arxiv.org/abs/2311.10671v3
• Date: Mon, 04 Nov 2024 11:24:51 GMT
• Title: Fuse It or Lose It: Deep Fusion for Multimodal Simulation-Based Inference
• Authors: Marvin Schmitt, Leona Odole, Stefan T. Radev, Paul-Christian Bürkner,
• Abstract summary: MultiNPE is a method to integrate heterogeneous data from different sources in simulation-based inference with neural networks. We consider three fusion approaches for MultiNPE and evaluate their performance in three challenging experiments.
• Score: 3.045422936743619
• License: http://creativecommons.org/licenses/by-sa/4.0/
• Abstract: We present multimodal neural posterior estimation (MultiNPE), a method to integrate heterogeneous data from different sources in simulation-based inference with neural networks. Inspired by advances in deep fusion, it allows researchers to analyze data from different domains and infer the parameters of complex mathematical models with increased accuracy. We consider three fusion approaches for MultiNPE (early, late, hybrid) and evaluate their performance in three challenging experiments. MultiNPE not only outperforms single-source baselines on a reference task, but also achieves superior inference on scientific models from cognitive neuroscience and cardiology. We systematically investigate the impact of partially missing data on the different fusion strategies. Across our experiments, late and hybrid fusion techniques emerge as the methods of choice for practical applications of multimodal simulation-based inference.

Related papers

• NSF-MAP: Neurosymbolic Multimodal Fusion for Robust and Interpretable Anomaly Prediction in Assembly Pipelines [0.0]
  This paper proposes a neurosymbolic AI and fusion-based approach for multimodal anomaly prediction in assembly pipelines.<n>We introduce a time series and image-based fusion model that leverages decision-level fusion techniques.<n>The results demonstrate that a neurosymbolic AI-based fusion approach that uses transfer learning can effectively harness the complementary strengths of time series and image data.
  arXiv  Detail & Related papers  (2025-05-09T16:50:42Z)
• Uncertainty Quantification for Multi-fidelity Simulations [0.0]
  The work focuses on gathering high-fidelity and low-fidelity numerical simulations data using Nektar++ and XFOIL respectively. The utilization of the higher distribution in calculating the Coefficient of lift and drag has demonstrated superior accuracy and precision. To minimize the reliability on high-fidelity numerical simulations in Uncertainty Quantification, a multi-fidelity strategy has been adopted.
  arXiv  Detail & Related papers  (2025-03-11T13:11:18Z)
• Application of Multimodal Fusion Deep Learning Model in Disease Recognition [14.655086303102575]
  This paper introduces an innovative multi-modal fusion deep learning approach to overcome the drawbacks of traditional single-modal recognition techniques. During the feature extraction stage, cutting-edge deep learning models are applied to distill advanced features from image-based, temporal, and structured data sources. The findings demonstrate significant advantages of the multimodal fusion model across multiple evaluation metrics.
  arXiv  Detail & Related papers  (2024-05-22T23:09:49Z)
• Diffusion posterior sampling for simulation-based inference in tall data settings [53.17563688225137]
  Simulation-based inference ( SBI) is capable of approximating the posterior distribution that relates input parameters to a given observation. In this work, we consider a tall data extension in which multiple observations are available to better infer the parameters of the model. We compare our method to recently proposed competing approaches on various numerical experiments and demonstrate its superiority in terms of numerical stability and computational cost.
  arXiv  Detail & Related papers  (2024-04-11T09:23:36Z)
• Coupled generator decomposition for fusion of electro- and magnetoencephalography data [1.7102695043811291]
  Data fusion modeling can identify common features across diverse data sources while accounting for source-specific variability. We introduce the concept of a textitcoupled generator decomposition and demonstrate how it generalizes sparse principal component analysis for data fusion.
  arXiv  Detail & Related papers  (2024-03-02T12:09:16Z)
• A Multi-Grained Symmetric Differential Equation Model for Learning Protein-Ligand Binding Dynamics [73.35846234413611]
  In drug discovery, molecular dynamics (MD) simulation provides a powerful tool for predicting binding affinities, estimating transport properties, and exploring pocket sites. We propose NeuralMD, the first machine learning (ML) surrogate that can facilitate numerical MD and provide accurate simulations in protein-ligand binding dynamics. We demonstrate the efficiency and effectiveness of NeuralMD, achieving over 1K$times$ speedup compared to standard numerical MD simulations.
  arXiv  Detail & Related papers  (2024-01-26T09:35:17Z)
• Joint Self-Supervised and Supervised Contrastive Learning for Multimodal MRI Data: Towards Predicting Abnormal Neurodevelopment [5.771221868064265]
  We present a novel joint self-supervised and supervised contrastive learning method to learn the robust latent feature representation from multimodal MRI data. Our method has the capability to facilitate computer-aided diagnosis within clinical practice, harnessing the power of multimodal data.
  arXiv  Detail & Related papers  (2023-12-22T21:05:51Z)
• Multimodal Multi-loss Fusion Network for Sentiment Analysis [3.8611070161950902]
  This paper investigates the optimal selection and fusion of feature encoders across multiple modalities to improve sentiment detection. We compare different fusion methods and examine the impact of multi-loss training within the multi-modality fusion network. We have found that integrating context significantly enhances model performance.
  arXiv  Detail & Related papers  (2023-08-01T03:54:27Z)
• Quantifying & Modeling Multimodal Interactions: An Information Decomposition Framework [89.8609061423685]
  We propose an information-theoretic approach to quantify the degree of redundancy, uniqueness, and synergy relating input modalities with an output task. To validate PID estimation, we conduct extensive experiments on both synthetic datasets where the PID is known and on large-scale multimodal benchmarks. We demonstrate their usefulness in (1) quantifying interactions within multimodal datasets, (2) quantifying interactions captured by multimodal models, (3) principled approaches for model selection, and (4) three real-world case studies.
  arXiv  Detail & Related papers  (2023-02-23T18:59:05Z)
• Investigating Neuron Disturbing in Fusing Heterogeneous Neural Networks [6.389882065284252]
  In this paper, we reveal the phenomenon of neuron disturbing, where neurons from heterogeneous local models interfere with each other mutually. We propose an experimental method that excludes neuron disturbing and fuses neural networks via adaptively selecting a local model, called AMS, to execute the prediction.
  arXiv  Detail & Related papers  (2022-10-24T06:47:48Z)
• Multi-fidelity Hierarchical Neural Processes [79.0284780825048]
  Multi-fidelity surrogate modeling reduces the computational cost by fusing different simulation outputs. We propose Multi-fidelity Hierarchical Neural Processes (MF-HNP), a unified neural latent variable model for multi-fidelity surrogate modeling. We evaluate MF-HNP on epidemiology and climate modeling tasks, achieving competitive performance in terms of accuracy and uncertainty estimation.
  arXiv  Detail & Related papers  (2022-06-10T04:54:13Z)
• Mixed Effects Neural ODE: A Variational Approximation for Analyzing the Dynamics of Panel Data [50.23363975709122]
  We propose a probabilistic model called ME-NODE to incorporate (fixed + random) mixed effects for analyzing panel data. We show that our model can be derived using smooth approximations of SDEs provided by the Wong-Zakai theorem. We then derive Evidence Based Lower Bounds for ME-NODE, and develop (efficient) training algorithms.
  arXiv  Detail & Related papers  (2022-02-18T22:41:51Z)
• A deep learning driven pseudospectral PCE based FFT homogenization algorithm for complex microstructures [68.8204255655161]
  It is shown that the proposed method is able to predict central moments of interest while being magnitudes faster to evaluate than traditional approaches. It is shown, that the proposed method is able to predict central moments of interest while being magnitudes faster to evaluate than traditional approaches.
  arXiv  Detail & Related papers  (2021-10-26T07:02:14Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.