Impact of Multimodal and Conversational AI on Learning Outcomes and Experience

• URL: http://arxiv.org/abs/2604.02221v1
• Date: Thu, 02 Apr 2026 16:12:00 GMT
• Title: Impact of Multimodal and Conversational AI on Learning Outcomes and Experience
• Abstract summary: Multimodal Large Language Models (MLLMs) offer an opportunity to support multimedia learning through conversational systems grounded in educational content.<n>While conversational AI is known to boost engagement, its impact on learning in visually-rich STEM domains remains under-explored.<n>This work reports findings from a randomized controlled online study comparing three approaches to learning biology from textbook content.
• Score: 6.455101027580352
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Multimodal Large Language Models (MLLMs) offer an opportunity to support multimedia learning through conversational systems grounded in educational content. However, while conversational AI is known to boost engagement, its impact on learning in visually-rich STEM domains remains under-explored. Moreover, there is limited understanding of how multimodality and conversationality jointly influence learning in generative AI systems. This work reports findings from a randomized controlled online study (N = 124) comparing three approaches to learning biology from textbook content: (1) a document-grounded conversational AI with interleaved text-and-image responses (MuDoC), (2) a document-grounded conversational AI with text-only responses (TexDoC), and (3) a textbook interface with semantic search and highlighting (DocSearch). Learners using MuDoC achieved the highest post-test scores and reported the most positive learning experience. Notably, while TexDoC was rated as significantly more engaging and easier to use than DocSearch, it led to the lowest post-test scores, revealing a disconnect between student perceptions and learning outcomes. Interpreted through the lens of the Cognitive Load Theory, these findings suggest that conversationality reduces extraneous load, while visual-verbal integration induced by multimodality increases germane load, leading to better learning outcomes. When conversationality is not complemented by multimodality, reduced cognitive effort may instead inflate perceived understanding without improving learning outcomes.

Related papers

• Socrates went Nuclear: Comparing Interaction Strategies for AI systems in a Learning Context using Brain Sensing [66.03354630509942]
  This paper reports on a study where we compare three designs for user-AI interaction in a learning context.<n>Fifty study participants were tasked with learning about nuclear safety protocols.
  arXiv  Detail & Related papers  (2026-09-01T02:24:35Z)
• Understanding the Dilemma of Unlearning for Large Language Models [50.54260066313032]
  Unlearning seeks to remove specific knowledge from large language models (LLMs)<n>We propose unPact, an interpretable framework for unlearning via prompt attribution and contribution tracking.
  arXiv  Detail & Related papers  (2025-09-29T12:15:19Z)
• Dynamic Scoring with Enhanced Semantics for Training-Free Human-Object Interaction Detection [51.52749744031413]
  Human-Object Interaction (HOI) detection aims to identify humans and objects within images and interpret their interactions.<n>Existing HOI methods rely heavily on large datasets with manual annotations to learn interactions from visual cues.<n>We propose a novel training-free HOI detection framework for Dynamic Scoring with enhanced semantics.
  arXiv  Detail & Related papers  (2025-07-23T12:30:19Z)
• Towards a Multimodal Document-grounded Conversational AI System for Education [5.228830802958218]
  We present MuDoC, a Multimodal Document-grounded Conversational AI system based on GPT-4o.<n>Its interface allows verification of AI generated content through seamless navigation to the source.<n>Our findings indicate that both visuals and verifiability of content foster learner engagement and trust; however, no significant impact on performance was observed.
  arXiv  Detail & Related papers  (2025-04-04T00:04:19Z)
• Student-AI Interaction in an LLM-Empowered Learning Environment: A Cluster Analysis of Engagement Profiles [28.794946431719392]
  This study explored diverse learner profiles within a multi-agent, LLM-empowered learning environment.<n>Students exhibit varied behavioral, cognitive, and emotional engagement tendencies.
  arXiv  Detail & Related papers  (2025-03-03T16:08:28Z)
• RA-BLIP: Multimodal Adaptive Retrieval-Augmented Bootstrapping Language-Image Pre-training [55.54020926284334]
  Multimodal Large Language Models (MLLMs) have recently received substantial interest, which shows their emerging potential as general-purpose models for various vision-language tasks. Retrieval augmentation techniques have proven to be effective plugins for both LLMs and MLLMs. In this study, we propose multimodal adaptive Retrieval-Augmented Bootstrapping Language-Image Pre-training (RA-BLIP), a novel retrieval-augmented framework for various MLLMs.
  arXiv  Detail & Related papers  (2024-10-18T03:45:19Z)
• Crafting a Good Prompt or Providing Exemplary Dialogues? A Study of In-Context Learning for Persona-based Dialogue Generation [15.143135611057309]
  We systematically investigate the ICL capabilities of large language models (LLMs) in persona-based dialogue generation. From experimental results, we draw three conclusions: 1) adjusting prompt instructions is the most direct, effective, and economical way to improve generation quality; 2) randomly retrieving demonstrations (demos) achieves the best results; and 3) even when we destroy the multi-turn associations and single-turn semantics in the demos, increasing the number of demos still improves dialogue performance.
  arXiv  Detail & Related papers  (2024-02-15T14:03:33Z)
• A Survey on Deep Multi-modal Learning for Body Language Recognition and Generation [5.8522989442606566]
  Body language (BL) refers to the non-verbal communication expressed through physical movements, gestures, facial expressions, and postures. Deep multi-modal learning techniques have shown promise in understanding and analyzing these diverse aspects of BL. Several common BLs are considered i.e., Sign Language (SL), Cued Speech (CS), Co-speech (CoS), and Talking Head (TH)
  arXiv  Detail & Related papers  (2023-08-17T08:15:51Z)
• Re-mine, Learn and Reason: Exploring the Cross-modal Semantic Correlations for Language-guided HOI detection [57.13665112065285]
  Human-Object Interaction (HOI) detection is a challenging computer vision task. We present a framework that enhances HOI detection by incorporating structured text knowledge.
  arXiv  Detail & Related papers  (2023-07-25T14:20:52Z)
• Multimodal Lecture Presentations Dataset: Understanding Multimodality in Educational Slides [57.86931911522967]
  We test the capabilities of machine learning models in multimodal understanding of educational content. Our dataset contains aligned slides and spoken language, for 180+ hours of video and 9000+ slides, with 10 lecturers from various subjects. We introduce PolyViLT, a multimodal transformer trained with a multi-instance learning loss that is more effective than current approaches.
  arXiv  Detail & Related papers  (2022-08-17T05:30:18Z)
• Knowledge Augmented BERT Mutual Network in Multi-turn Spoken Dialogues [6.4144180888492075]
  We propose to equip a BERT-based joint model with a knowledge attention module to mutually leverage dialogue contexts between two SLU tasks. A gating mechanism is further utilized to filter out irrelevant knowledge triples and to circumvent distracting comprehension. Experimental results in two complicated multi-turn dialogue datasets have demonstrate by mutually modeling two SLU tasks with filtered knowledge and dialogue contexts.
  arXiv  Detail & Related papers  (2022-02-23T04:03:35Z)
• Few-Shot Bot: Prompt-Based Learning for Dialogue Systems [58.27337673451943]
  Learning to converse using only a few examples is a great challenge in conversational AI. The current best conversational models are either good chit-chatters (e.g., BlenderBot) or goal-oriented systems (e.g., MinTL) We propose prompt-based few-shot learning which does not require gradient-based fine-tuning but instead uses a few examples as the only source of learning.
  arXiv  Detail & Related papers  (2021-10-15T14:36:45Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.