Uncertainty-aware Multi-modal Learning via Cross-modal Random Network Prediction

• URL: http://arxiv.org/abs/2207.10851v1
• Date: Fri, 22 Jul 2022 03:00:10 GMT
• Title: Uncertainty-aware Multi-modal Learning via Cross-modal Random Network Prediction
• Authors: Hu Wang, Jianpeng Zhang, Yuanhong Chen, Congbo Ma, Jodie Avery, Louise Hull, Gustavo Carneiro
• Abstract summary: We propose a new Uncertainty-aware Multi-modal Learner that estimates uncertainty by measuring feature density via Cross-modal Random Network Prediction (CRNP) CRNP is designed to require little adaptation to translate between different prediction tasks, while having a stable training process.
• Score: 22.786774541083652
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Multi-modal learning focuses on training models by equally combining multiple input data modalities during the prediction process. However, this equal combination can be detrimental to the prediction accuracy because different modalities are usually accompanied by varying levels of uncertainty. Using such uncertainty to combine modalities has been studied by a couple of approaches, but with limited success because these approaches are either designed to deal with specific classification or segmentation problems and cannot be easily translated into other tasks, or suffer from numerical instabilities. In this paper, we propose a new Uncertainty-aware Multi-modal Learner that estimates uncertainty by measuring feature density via Cross-modal Random Network Prediction (CRNP). CRNP is designed to require little adaptation to translate between different prediction tasks, while having a stable training process. From a technical point of view, CRNP is the first approach to explore random network prediction to estimate uncertainty and to combine multi-modal data. Experiments on two 3D multi-modal medical image segmentation tasks and three 2D multi-modal computer vision classification tasks show the effectiveness, adaptability and robustness of CRNP. Also, we provide an extensive discussion on different fusion functions and visualization to validate the proposed model.

Related papers

• Confidence-aware multi-modality learning for eye disease screening [58.861421804458395]
  We propose a novel multi-modality evidential fusion pipeline for eye disease screening. It provides a measure of confidence for each modality and elegantly integrates the multi-modality information. Experimental results on both public and internal datasets demonstrate that our model excels in robustness.
  arXiv  Detail & Related papers  (2024-05-28T13:27:30Z)
• Calibrating Multimodal Learning [94.65232214643436]
  We propose a novel regularization technique, i.e., Calibrating Multimodal Learning (CML) regularization, to calibrate the predictive confidence of previous methods. This technique could be flexibly equipped by existing models and improve the performance in terms of confidence calibration, classification accuracy, and model robustness.
  arXiv  Detail & Related papers  (2023-06-02T04:29:57Z)
• Channel Exchanging Networks for Multimodal and Multitask Dense Image Prediction [125.18248926508045]
  We propose Channel-Exchanging-Network (CEN) which is self-adaptive, parameter-free, and more importantly, applicable for both multimodal fusion and multitask learning. CEN dynamically exchanges channels betweenworks of different modalities. For the application of dense image prediction, the validity of CEN is tested by four different scenarios.
  arXiv  Detail & Related papers  (2021-12-04T05:47:54Z)
• Randomized ReLU Activation for Uncertainty Estimation of Deep Neural Networks [8.541875999755593]
  Deep neural networks (DNNs) have successfully learned useful data representations in various tasks. Deep Ensemble is widely considered the state-of-the-art method for uncertainty estimation, but it is very expensive to train and test. We introduce Randomized ReLU Activation framework to get more diverse predictions in less time.
  arXiv  Detail & Related papers  (2021-07-15T08:54:41Z)
• Multivariate Deep Evidential Regression [77.34726150561087]
  A new approach with uncertainty-aware neural networks shows promise over traditional deterministic methods. We discuss three issues with a proposed solution to extract aleatoric and epistemic uncertainties from regression-based neural networks.
  arXiv  Detail & Related papers  (2021-04-13T12:20:18Z)
• Multi-Loss Sub-Ensembles for Accurate Classification with Uncertainty Estimation [1.2891210250935146]
  We propose an efficient method for uncertainty estimation in deep neural networks (DNNs) achieving high accuracy. We keep our inference time relatively low by leveraging the advantage proposed by the Deep-Sub-Ensembles method. Our results show improved accuracy on the classification task and competitive results on several uncertainty measures.
  arXiv  Detail & Related papers  (2020-10-05T10:59:11Z)
• M2Net: Multi-modal Multi-channel Network for Overall Survival Time Prediction of Brain Tumor Patients [151.4352001822956]
  Early and accurate prediction of overall survival (OS) time can help to obtain better treatment planning for brain tumor patients. Existing prediction methods rely on radiomic features at the local lesion area of a magnetic resonance (MR) volume. We propose an end-to-end OS time prediction model; namely, Multi-modal Multi-channel Network (M2Net)
  arXiv  Detail & Related papers  (2020-06-01T05:21:37Z)
• Diversity inducing Information Bottleneck in Model Ensembles [73.80615604822435]
  In this paper, we target the problem of generating effective ensembles of neural networks by encouraging diversity in prediction. We explicitly optimize a diversity inducing adversarial loss for learning latent variables and thereby obtain diversity in the output predictions necessary for modeling multi-modal data. Compared to the most competitive baselines, we show significant improvements in classification accuracy, under a shift in the data distribution.
  arXiv  Detail & Related papers  (2020-03-10T03:10:41Z)
• Uncertainty Estimation for End-To-End Learned Dense Stereo Matching via Probabilistic Deep Learning [0.0]
  A novel probabilistic neural network is presented for the task of joint depth and uncertainty estimation from epipolar rectified stereo image pairs. The network learns a probability distribution from which parameters are sampled for every prediction. The quality of the estimated depth and uncertainty information is assessed in an extensive evaluation on three different datasets.
  arXiv  Detail & Related papers  (2020-02-10T11:27:52Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.