Explaining Multimodal Data Fusion: Occlusion Analysis for Wilderness
Mapping
- URL: http://arxiv.org/abs/2304.02407v1
- Date: Wed, 5 Apr 2023 12:35:02 GMT
- Title: Explaining Multimodal Data Fusion: Occlusion Analysis for Wilderness
Mapping
- Authors: Burak Ekim and Michael Schmitt
- Abstract summary: This study proposes a deep learning framework for the modality-level interpretation of multimodal earth observation data.
We show that the task of wilderness mapping largely benefits from auxiliary data such as land cover and night time light data.
- Score: 2.123635308480885
- License: http://creativecommons.org/licenses/by-nc-nd/4.0/
- Abstract: Jointly harnessing complementary features of multi-modal input data in a
common latent space has been found to be beneficial long ago. However, the
influence of each modality on the models decision remains a puzzle. This study
proposes a deep learning framework for the modality-level interpretation of
multimodal earth observation data in an end-to-end fashion. While leveraging an
explainable machine learning method, namely Occlusion Sensitivity, the proposed
framework investigates the influence of modalities under an early-fusion
scenario in which the modalities are fused before the learning process. We show
that the task of wilderness mapping largely benefits from auxiliary data such
as land cover and night time light data.
Related papers
- One-stage Modality Distillation for Incomplete Multimodal Learning [7.791488931628906]
This paper presents a one-stage modality distillation framework that unifies the privileged knowledge transfer and modality information fusion.
The proposed framework can overcome the problem of incomplete modality input in various scenes and achieve state-of-the-art performance.
arXiv Detail & Related papers (2023-09-15T07:12:27Z) - Ensemble Modeling for Multimodal Visual Action Recognition [50.38638300332429]
We propose an ensemble modeling approach for multimodal action recognition.
We independently train individual modality models using a variant of focal loss tailored to handle the long-tailed distribution of the MECCANO [21] dataset.
arXiv Detail & Related papers (2023-08-10T08:43:20Z) - Learning Unseen Modality Interaction [54.23533023883659]
Multimodal learning assumes all modality combinations of interest are available during training to learn cross-modal correspondences.
We pose the problem of unseen modality interaction and introduce a first solution.
It exploits a module that projects the multidimensional features of different modalities into a common space with rich information preserved.
arXiv Detail & Related papers (2023-06-22T10:53:10Z) - Multimodal Explainability via Latent Shift applied to COVID-19 stratification [0.7831774233149619]
We present a deep architecture, which jointly learns modality reconstructions and sample classifications.
We validate our approach in the context of COVID-19 pandemic using the AIforCOVID dataset.
arXiv Detail & Related papers (2022-12-28T20:07:43Z) - Self-Attention Neural Bag-of-Features [103.70855797025689]
We build on the recently introduced 2D-Attention and reformulate the attention learning methodology.
We propose a joint feature-temporal attention mechanism that learns a joint 2D attention mask highlighting relevant information.
arXiv Detail & Related papers (2022-01-26T17:54:14Z) - Learning Neural Causal Models with Active Interventions [83.44636110899742]
We introduce an active intervention-targeting mechanism which enables a quick identification of the underlying causal structure of the data-generating process.
Our method significantly reduces the required number of interactions compared with random intervention targeting.
We demonstrate superior performance on multiple benchmarks from simulated to real-world data.
arXiv Detail & Related papers (2021-09-06T13:10:37Z) - Enhancing ensemble learning and transfer learning in multimodal data
analysis by adaptive dimensionality reduction [10.646114896709717]
In multimodal data analysis, not all observations would show the same level of reliability or information quality.
We propose an adaptive approach for dimensionality reduction to overcome this issue.
We test our approach on multimodal datasets acquired in diverse research fields.
arXiv Detail & Related papers (2021-05-08T11:53:12Z) - OR-Net: Pointwise Relational Inference for Data Completion under Partial
Observation [51.083573770706636]
This work uses relational inference to fill in the incomplete data.
We propose Omni-Relational Network (OR-Net) to model the pointwise relativity in two aspects.
arXiv Detail & Related papers (2021-05-02T06:05:54Z) - Deep Co-Attention Network for Multi-View Subspace Learning [73.3450258002607]
We propose a deep co-attention network for multi-view subspace learning.
It aims to extract both the common information and the complementary information in an adversarial setting.
In particular, it uses a novel cross reconstruction loss and leverages the label information to guide the construction of the latent representation.
arXiv Detail & Related papers (2021-02-15T18:46:44Z) - Survey on Deep Multi-modal Data Analytics: Collaboration, Rivalry and
Fusion [6.225190099424806]
Multi-modal or multi-view data has surged as a major stream for big data, where each modal/view encodes individual property of data objects.
Most of the existing state-of-the-art focused on how to fuse the energy or information from multi-modal spaces to deliver a superior performance.
Deep neural networks have exhibited as a powerful architecture to well capture the nonlinear distribution of high-dimensional multimedia data.
arXiv Detail & Related papers (2020-06-15T06:42:04Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.