Comparing the Impact of Pedagogy-Informed Custom and General-Purpose GAI Chatbots on Students' Science Problem-Solving Processes and Performance Using Heterogeneous Interaction Network Analysis

• URL: http://arxiv.org/abs/2604.03022v1
• Date: Fri, 03 Apr 2026 13:13:28 GMT
• Title: Comparing the Impact of Pedagogy-Informed Custom and General-Purpose GAI Chatbots on Students' Science Problem-Solving Processes and Performance Using Heterogeneous Interaction Network Analysis
• Abstract summary: This study employed a within-subjects counterbalanced design in which 48 secondary school students used both custom and general-purpose chatbots.<n>3297 student-chatbot dialogues were collected and analyzed using Heterogeneous Interaction Network Analysis (HINA)
• Score: 0.03540145348020375
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Problem solving plays an essential role in science education, and generative AI (GAI) chatbots have emerged as a promising tool for supporting students' science problem solving. However, general-purpose chatbots (e.g., ChatGPT), which often provide direct, ready-made answers, may lead to students' cognitive offloading. Prior research has rarely focused on custom chatbots for facilitating students' science problem solving, nor has it examined how they differently influence problem-solving processes and performance compared to general-purpose chatbots. To address this gap, we developed a pedagogy-informed custom GAI chatbot grounded in the Socratic questioning method, which supports students by prompting them with guiding questions. This study employed a within-subjects counterbalanced design in which 48 secondary school students used both custom and general-purpose chatbot to complete two science problem-solving tasks. 3297 student-chatbot dialogues were collected and analyzed using Heterogeneous Interaction Network Analysis (HINA). The results showed that: (1) students demonstrated significantly higher interaction intensity and cognitive interaction diversity when using custom chatbot than using general-purpose chatbot; (2) students were more likely to follow custom chatbot's guidance to think and reflect, whereas they tended to request general-purpose chatbot to execute specific commands; and (3) no statistically significant difference was observed in students' problem-solving performance evaluated by solution quality between two chatbot conditions. This study provides novel theoretical insights and empirical evidence that custom chatbots are less likely to induce cognitive offloading and instead foster greater cognitive engagement compared to general-purpose chatbots. This study also offers insights into the design and integration of GAI chatbots in science education.

Related papers

• Socrates went Nuclear: Comparing Interaction Strategies for AI systems in a Learning Context using Brain Sensing [66.03354630509942]
  This paper reports on a study where we compare three designs for user-AI interaction in a learning context.<n>Fifty study participants were tasked with learning about nuclear safety protocols.
  arXiv  Detail & Related papers  (2026-09-01T02:24:35Z)
• Exploring Teacher-Chatbot Interaction and Affect in Block-Based Programming [5.281501249826119]
  AI-based chatbots have the potential to accelerate learning and teaching, but may also have counterproductive consequences without thoughtful design and scaffolding.<n>We conducted a study with 11 teams of middle school teachers using chatbots for a science and computational thinking activity within a block-based programming environment.<n>We found that teachers perceived chatbots benefits such as building prompting skills and self-confidence alongside risks including potential declines in learning and critical thinking.
  arXiv  Detail & Related papers  (2026-03-02T15:42:27Z)
• Using the SOCIO Chatbot for UML Modelling: A Family of Experiments [0.1957338076370071]
  We compare the usability of SOCIO for collaborative modelling (i.e., SOCIO) and an online web tool (i.e., Creately) in academic settings. The student participants were faster at building class diagrams using SOCIO than with the online collaborative tool. Our study has helped us to shed light on the future direction for experimentation in this field.
  arXiv  Detail & Related papers  (2024-08-26T08:12:11Z)
• Analyzing Large language models chatbots: An experimental approach using a probability test [0.0]
  This study consists of qualitative empirical research, conducted through exploratory tests with two different Large Language Models (LLMs) The methodological procedure involved exploratory tests based on prompts designed with a probability question. The "Linda Problem", widely recognized in cognitive psychology, was used as a basis to create the tests, along with the development of a new problem specifically for this experiment, the "Mary Problem"
  arXiv  Detail & Related papers  (2024-07-10T15:49:40Z)
• Developing generative AI chatbots conceptual framework for higher education [0.0]
  This study aims to comprehend the implications of AIgeneratives on higher education and pinpoint critical elements for their efficacious implementation. The results demonstrate how much AI chatbots can do to improve student engagement, streamline the educational process, and support administrative and research duties. But there are also clear difficulties, such as unfavorable student sentiments, doubts about the veracity of material produced by AI, and unease and nervousness with new technologies.
  arXiv  Detail & Related papers  (2024-03-28T10:40:26Z)
• Evaluating Chatbots to Promote Users' Trust -- Practices and Open Problems [11.427175278545517]
  This paper reviews current practices for testing chatbots. It identifies gaps as open problems in pursuit of user trust. It outlines a path forward to mitigate issues of trust related to service or product performance, user satisfaction and long-term unintended consequences for society.
  arXiv  Detail & Related papers  (2023-09-09T22:40:30Z)
• Developing Effective Educational Chatbots with ChatGPT prompts: Insights from Preliminary Tests in a Case Study on Social Media Literacy (with appendix) [43.55994393060723]
  Recent advances in language learning models with zero-shot learning capabilities, such as ChatGPT, suggest a new possibility for developing educational chatbots. We present a case study with a simple system that enables mixed-turn chatbots interactions. We examine ChatGPT's ability to pursue multiple interconnected learning objectives, adapt the educational activity to users' characteristics, such as culture, age, and level of education, and its ability to use diverse educational strategies and conversational styles.
  arXiv  Detail & Related papers  (2023-06-18T22:23:18Z)
• Chatbots put to the test in math and logic problems: A preliminary comparison and assessment of ChatGPT-3.5, ChatGPT-4, and Google Bard [68.8204255655161]
  We use 30 questions that are clear, without any ambiguities, fully described with plain text only, and have a unique, well defined correct answer. The answers are recorded and discussed, highlighting their strengths and weaknesses. It was found that ChatGPT-4 outperforms ChatGPT-3.5 in both sets of questions.
  arXiv  Detail & Related papers  (2023-05-30T11:18:05Z)
• Implementing a Chatbot Solution for Learning Management System [0.0]
  One of the main problem that chatbots face today is to mimic human language. Extreme programming methodology was chosen to use integrate ChatterBot, Pyside2, web scraping and Tampermonkey into Blackboard. We showed the plausibility of integrating an AI bot in an educational setting.
  arXiv  Detail & Related papers  (2022-06-27T11:04:42Z)
• CheerBots: Chatbots toward Empathy and Emotionusing Reinforcement Learning [60.348822346249854]
  This study presents a framework whereby several empathetic chatbots are based on understanding users' implied feelings and replying empathetically for multiple dialogue turns. We call these chatbots CheerBots. CheerBots can be retrieval-based or generative-based and were finetuned by deep reinforcement learning. To respond in an empathetic way, we develop a simulating agent, a Conceptual Human Model, as aids for CheerBots in training with considerations on changes in user's emotional states in the future to arouse sympathy.
  arXiv  Detail & Related papers  (2021-10-08T07:44:47Z)
• Put Chatbot into Its Interlocutor's Shoes: New Framework to Learn Chatbot Responding with Intention [55.77218465471519]
  This paper proposes an innovative framework to train chatbots to possess human-like intentions. Our framework included a guiding robot and an interlocutor model that plays the role of humans. We examined our framework using three experimental setups and evaluate the guiding robot with four different metrics to demonstrated flexibility and performance advantages.
  arXiv  Detail & Related papers  (2021-03-30T15:24:37Z)
• CASS: Towards Building a Social-Support Chatbot for Online Health Community [67.45813419121603]
  The CASS architecture is based on advanced neural network algorithms. It can handle new inputs from users and generate a variety of responses to them. With a follow-up field experiment, CASS is proven useful in supporting individual members who seek emotional support.
  arXiv  Detail & Related papers  (2021-01-04T05:52:03Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.