Multimodal Explainable Artificial Intelligence: A Comprehensive Review of Methodological Advances and Future Research Directions

• URL: http://arxiv.org/abs/2306.05731v2
• Date: Sun, 30 Jun 2024 15:24:05 GMT
• Title: Multimodal Explainable Artificial Intelligence: A Comprehensive Review of Methodological Advances and Future Research Directions
• Authors: Nikolaos Rodis, Christos Sardianos, Panagiotis Radoglou-Grammatikis, Panagiotis Sarigiannidis, Iraklis Varlamis, Georgios Th. Papadopoulos,
• Abstract summary: This study focuses on analyzing the recent advances in the area of Multimodal XAI (MXAI) MXAI comprises methods that involve multiple modalities in the primary prediction and explanation tasks.
• Score: 2.35574869517894
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Despite the fact that Artificial Intelligence (AI) has boosted the achievement of remarkable results across numerous data analysis tasks, however, this is typically accompanied by a significant shortcoming in the exhibited transparency and trustworthiness of the developed systems. In order to address the latter challenge, the so-called eXplainable AI (XAI) research field has emerged, which aims, among others, at estimating meaningful explanations regarding the employed model reasoning process. The current study focuses on systematically analyzing the recent advances in the area of Multimodal XAI (MXAI), which comprises methods that involve multiple modalities in the primary prediction and explanation tasks. In particular, the relevant AI-boosted prediction tasks and publicly available datasets used for learning/evaluating explanations in multimodal scenarios are initially described. Subsequently, a systematic and comprehensive analysis of the MXAI methods of the literature is provided, taking into account the following key criteria: a) The number of the involved modalities (in the employed AI module), b) The processing stage at which explanations are generated, and c) The type of the adopted methodology (i.e. the actual mechanism and mathematical formalization) for producing explanations. Then, a thorough analysis of the metrics used for MXAI methods evaluation is performed. Finally, an extensive discussion regarding the current challenges and future research directions is provided.

Related papers

• Data Analysis in the Era of Generative AI [56.44807642944589]
  This paper explores the potential of AI-powered tools to reshape data analysis, focusing on design considerations and challenges. We explore how the emergence of large language and multimodal models offers new opportunities to enhance various stages of data analysis workflow. We then examine human-centered design principles that facilitate intuitive interactions, build user trust, and streamline the AI-assisted analysis workflow across multiple apps.
  arXiv  Detail & Related papers  (2024-09-27T06:31:03Z)
• A Survey on Multimodal Benchmarks: In the Era of Large AI Models [13.299775710527962]
  Multimodal Large Language Models (MLLMs) have brought substantial advancements in artificial intelligence. This survey systematically reviews 211 benchmarks that assess MLLMs across four core domains: understanding, reasoning, generation, and application.
  arXiv  Detail & Related papers  (2024-09-21T15:22:26Z)
• Unified Explanations in Machine Learning Models: A Perturbation Approach [0.0]
  Inconsistencies between XAI and modeling techniques can have the undesirable effect of casting doubt upon the efficacy of these explainability approaches. We propose a systematic, perturbation-based analysis against a popular, model-agnostic method in XAI, SHapley Additive exPlanations (Shap) We devise algorithms to generate relative feature importance in settings of dynamic inference amongst a suite of popular machine learning and deep learning methods, and metrics that allow us to quantify how well explanations generated under the static case hold.
  arXiv  Detail & Related papers  (2024-05-30T16:04:35Z)
• Gradient based Feature Attribution in Explainable AI: A Technical Review [13.848675695545909]
  Surge in black-box AI models has prompted the need to explain the internal mechanism and justify their reliability. gradient based explanations can be directly adopted for neural network models. We introduce both human and quantitative evaluations to measure algorithm performance.
  arXiv  Detail & Related papers  (2024-03-15T15:49:31Z)
• Opening the Black-Box: A Systematic Review on Explainable AI in Remote Sensing [51.524108608250074]
  Black-box machine learning approaches have become a dominant modeling paradigm for knowledge extraction in remote sensing. We perform a systematic review to identify the key trends in the field and shed light on novel explainable AI approaches. We also give a detailed outlook on the challenges and promising research directions.
  arXiv  Detail & Related papers  (2024-02-21T13:19:58Z)
• Interpretable and Explainable Machine Learning Methods for Predictive Process Monitoring: A Systematic Literature Review [1.3812010983144802]
  This paper presents a systematic review on the explainability and interpretability of machine learning (ML) models within the context of predictive process mining. We provide a comprehensive overview of the current methodologies and their applications across various application domains. Our findings aim to equip researchers and practitioners with a deeper understanding of how to develop and implement more trustworthy, transparent, and effective intelligent systems for process analytics.
  arXiv  Detail & Related papers  (2023-12-29T12:43:43Z)
• A Survey on Interpretable Cross-modal Reasoning [64.37362731950843]
  Cross-modal reasoning (CMR) has emerged as a pivotal area with applications spanning from multimedia analysis to healthcare diagnostics. This survey delves into the realm of interpretable cross-modal reasoning (I-CMR) This survey presents a comprehensive overview of the typical methods with a three-level taxonomy for I-CMR.
  arXiv  Detail & Related papers  (2023-09-05T05:06:48Z)
• Bridging MDE and AI: A Systematic Review of Domain-Specific Languages and Model-Driven Practices in AI Software Systems Engineering [1.4853133497896698]
  This study aims to investigate the existing model-driven approaches relying on DSL in support of the engineering of AI software systems. The use of MDE for AI is still in its early stages, and there is no single tool or method that is widely used.
  arXiv  Detail & Related papers  (2023-07-10T14:38:38Z)
• A Survey of Contextual Optimization Methods for Decision Making under Uncertainty [47.73071218563257]
  This review article identifies three main frameworks for learning policies from data and discusses their strengths and limitations. We present the existing models and methods under a uniform notation and terminology and classify them according to the three main frameworks.
  arXiv  Detail & Related papers  (2023-06-17T15:21:02Z)
• Beyond Explaining: Opportunities and Challenges of XAI-Based Model Improvement [75.00655434905417]
  Explainable Artificial Intelligence (XAI) is an emerging research field bringing transparency to highly complex machine learning (ML) models. This paper offers a comprehensive overview over techniques that apply XAI practically for improving various properties of ML models. We show empirically through experiments on toy and realistic settings how explanations can help improve properties such as model generalization ability or reasoning.
  arXiv  Detail & Related papers  (2022-03-15T15:44:28Z)
• DIME: Fine-grained Interpretations of Multimodal Models via Disentangled Local Explanations [119.1953397679783]
  We focus on advancing the state-of-the-art in interpreting multimodal models. Our proposed approach, DIME, enables accurate and fine-grained analysis of multimodal models.
  arXiv  Detail & Related papers  (2022-03-03T20:52:47Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.