Multimodal Methods for Analyzing Learning and Training Environments: A Systematic Literature Review

• URL: http://arxiv.org/abs/2408.14491v1
• Date: Thu, 22 Aug 2024 22:42:23 GMT
• Title: Multimodal Methods for Analyzing Learning and Training Environments: A Systematic Literature Review
• Authors: Clayton Cohn, Eduardo Davalos, Caleb Vatral, Joyce Horn Fonteles, Hanchen David Wang, Meiyi Ma, Gautam Biswas,
• Abstract summary: This literature review proposes a taxonomy and framework that encapsulates recent methodological advances in this field. We introduce a novel data fusion category -- mid fusion -- and a graph-based technique for refining literature reviews, termed citation graph pruning. There remains a need for further research to bridge the divide between multimodal learning and training studies and foundational AI research.
• Score: 3.0712840129998513
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Recent technological advancements have enhanced our ability to collect and analyze rich multimodal data (e.g., speech, video, and eye gaze) to better inform learning and training experiences. While previous reviews have focused on parts of the multimodal pipeline (e.g., conceptual models and data fusion), a comprehensive literature review on the methods informing multimodal learning and training environments has not been conducted. This literature review provides an in-depth analysis of research methods in these environments, proposing a taxonomy and framework that encapsulates recent methodological advances in this field and characterizes the multimodal domain in terms of five modality groups: Natural Language, Video, Sensors, Human-Centered, and Environment Logs. We introduce a novel data fusion category -- mid fusion -- and a graph-based technique for refining literature reviews, termed citation graph pruning. Our analysis reveals that leveraging multiple modalities offers a more holistic understanding of the behaviors and outcomes of learners and trainees. Even when multimodality does not enhance predictive accuracy, it often uncovers patterns that contextualize and elucidate unimodal data, revealing subtleties that a single modality may miss. However, there remains a need for further research to bridge the divide between multimodal learning and training studies and foundational AI research.

Related papers

• Multimodal Alignment and Fusion: A Survey [7.250878248686215]
  Multimodal integration enables improved model accuracy and broader applicability. We systematically categorize and analyze existing alignment and fusion techniques. This survey focuses on applications in domains like social media analysis, medical imaging, and emotion recognition.
  arXiv  Detail & Related papers  (2024-11-26T02:10:27Z)
• A Systematic Review of Intermediate Fusion in Multimodal Deep Learning for Biomedical Applications [0.7831774233149619]
  This systematic review aims to analyze and formalize current intermediate fusion methods in biomedical applications. We introduce a structured notation to enhance the understanding and application of these methods beyond the biomedical domain. Our findings are intended to support researchers, healthcare professionals, and the broader deep learning community in developing more sophisticated and insightful multimodal models.
  arXiv  Detail & Related papers  (2024-08-02T11:48:04Z)
• Ontology Embedding: A Survey of Methods, Applications and Resources [54.3453925775069]
  Ontologies are widely used for representing domain knowledge and meta data. One straightforward solution is to integrate statistical analysis and machine learning. Numerous papers have been published on embedding, but a lack of systematic reviews hinders researchers from gaining a comprehensive understanding of this field.
  arXiv  Detail & Related papers  (2024-06-16T14:49:19Z)
• Recent Advances in Hate Speech Moderation: Multimodality and the Role of Large Models [52.24001776263608]
  This comprehensive survey delves into the recent strides in HS moderation. We highlight the burgeoning role of large language models (LLMs) and large multimodal models (LMMs) We identify existing gaps in research, particularly in the context of underrepresented languages and cultures.
  arXiv  Detail & Related papers  (2024-01-30T03:51:44Z)
• Multimodality Representation Learning: A Survey on Evolution, Pretraining and Its Applications [47.501121601856795]
  Multimodality Representation Learning is a technique of learning to embed information from different modalities and their correlations. Cross-modal interaction and complementary information from different modalities are crucial for advanced models to perform any multimodal task. This survey presents the literature on the evolution and enhancement of deep learning multimodal architectures.
  arXiv  Detail & Related papers  (2023-02-01T11:48:34Z)
• Vision+X: A Survey on Multimodal Learning in the Light of Data [64.03266872103835]
  multimodal machine learning that incorporates data from various sources has become an increasingly popular research area. We analyze the commonness and uniqueness of each data format mainly ranging from vision, audio, text, and motions. We investigate the existing literature on multimodal learning from both the representation learning and downstream application levels.
  arXiv  Detail & Related papers  (2022-10-05T13:14:57Z)
• Foundations and Recent Trends in Multimodal Machine Learning: Principles, Challenges, and Open Questions [68.6358773622615]
  This paper provides an overview of the computational and theoretical foundations of multimodal machine learning. We propose a taxonomy of 6 core technical challenges: representation, alignment, reasoning, generation, transference, and quantification. Recent technical achievements will be presented through the lens of this taxonomy, allowing researchers to understand the similarities and differences across new approaches.
  arXiv  Detail & Related papers  (2022-09-07T19:21:19Z)
• Multimodal Image Synthesis and Editing: The Generative AI Era [131.9569600472503]
  multimodal image synthesis and editing has become a hot research topic in recent years. We comprehensively contextualize the advance of the recent multimodal image synthesis and editing. We describe benchmark datasets and evaluation metrics as well as corresponding experimental results.
  arXiv  Detail & Related papers  (2021-12-27T10:00:16Z)
• Multimodality in Meta-Learning: A Comprehensive Survey [34.69292359136745]
  This survey provides a comprehensive overview of the multimodality-based meta-learning landscape. We first formalize the definition of meta-learning and multimodality, along with the research challenges in this growing field. We then propose a new taxonomy to systematically discuss typical meta-learning algorithms combined with multimodal tasks.
  arXiv  Detail & Related papers  (2021-09-28T09:16:12Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.