What Don't You Understand? Using Large Language Models to Identify and Characterize Student Misconceptions About Challenging Topics

• URL: http://arxiv.org/abs/2605.00294v1
• Date: Thu, 30 Apr 2026 23:32:50 GMT
• Title: What Don't You Understand? Using Large Language Models to Identify and Characterize Student Misconceptions About Challenging Topics
• Abstract summary: This study presents a systematic approach to identifying and characterizing student misconceptions in online learning environments.<n>We analyzed data from 9 course periods across 5 online biomedical science courses, encompassing 3,802 medical student enrollments.
• Score: 0.0
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: This study presents a systematic approach to identifying and characterizing student misconceptions in online learning environments through a novel combination of quantitative performance analysis and large language model (LLM) assessment. We analyzed data from 9 course periods across 5 online biomedical science courses, encompassing 3,802 medical student enrollments. Using data from 40-50 topic-focused quizzes per course, we developed a two-stage methodology. First, we identified challenging central topics using quiz-level performance metrics. Second, we employed LLMs to characterize the underlying misconceptions in these high-priority areas. By examining student performance on first attempts across primarily multiple-choice questions (MCQs), we identified consistently challenging topics that were also central to course objectives. We then leveraged recent advances in generative AI to analyze three distinct data sources in combination: quiz question content, student response patterns, and lecture transcripts. This approach revealed actionable insights about student misconceptions that were not apparent from performance data alone. The quality of the LLM-identified misconceptions was rated as excellent by subject matter experts. We also conducted teacher interviews to assess the perceived utility of our topic identification method. Faculty found that data-driven identification of challenging topics was valuable and corroborated their own classroom observations. This methodology provides a scalable approach to characterizing student difficulties in learning environments where quizzes are used. Our findings demonstrate the potential for targeted and potentially personalized interventions in future course iterations, with clear pathways for measuring intervention effectiveness through follow-up quiz performance.

Related papers

• "How Do I ...?": Procedural Questions Predominate Student-LLM Chatbot Conversations [39.146761527401424]
  This paper focuses on such student questions from two datasets of distinct learning contexts: formative self-study, and summative assessed coursework.<n>We analysed 6,113 messages from both learning contexts using 11 different Large Language Models (LLM) and three human raters.<n>Results show that 'procedural' questions predominated in both learning contexts, but more so when students prepare for summative assessment.
  arXiv  Detail & Related papers  (2026-02-20T17:27:41Z)
• Mining the Gold: Student-AI Chat Logs as Rich Sources for Automated Knowledge Gap Detection [2.6808104662419097]
  In large lectures, instructors face challenges on identifying students' knowledge gaps timely.<n>We propose QueryQuilt, a multi-agent LLM framework that automatically detects common knowledge gaps in large-scale lectures by analyzing students' chat logs with AI assistants.<n>Our evaluation demonstrates promising results, with QueryQuilt achieving 100% accuracy in identifying knowledge gaps among simulated students and 95% completeness when tested on real student-AI dialogue data.
  arXiv  Detail & Related papers  (2025-12-26T23:04:04Z)
• EduVidQA: Generating and Evaluating Long-form Answers to Student Questions based on Lecture Videos [8.393656792613]
  This paper explores using Multimodal Large Language Models (MLLMs) to automatically respond to student questions from online lectures.<n>We introduce the EduVidQA dataset with 5252 question-answer pairs (both synthetic and real-world) from 296 computer science videos covering diverse topics and difficulty levels.
  arXiv  Detail & Related papers  (2025-09-28T23:33:58Z)
• Teaching Language Models To Gather Information Proactively [53.85419549904644]
  Large language models (LLMs) are increasingly expected to function as collaborative partners.<n>In this work, we introduce a new task paradigm: proactive information gathering.<n>We design a scalable framework that generates partially specified, real-world tasks, masking key information.<n>Within this setup, our core innovation is a reinforcement finetuning strategy that rewards questions that elicit genuinely new, implicit user information.
  arXiv  Detail & Related papers  (2025-07-28T23:50:09Z)
• Early Detection of At-Risk Students Using Machine Learning [0.0]
  We aim to tackle the persistent challenges of higher education retention and student dropout rates by screening for at-risk students.<n>This work considers several machine learning models, including Support Vector Machines (SVM), Naive Bayes, K-nearest neighbors (KNN), Decision Trees, Logistic Regression, and Random Forest.<n>Our analysis indicates that all algorithms generate an acceptable outcome for at-risk student predictions, while Naive Bayes performs best overall.
  arXiv  Detail & Related papers  (2024-12-12T17:33:06Z)
• Benchmarking Vision Language Model Unlearning via Fictitious Facial Identity Dataset [92.99416966226724]
  We introduce Facial Identity Unlearning Benchmark (FIUBench), a novel VLM unlearning benchmark designed to robustly evaluate the effectiveness of unlearning algorithms.<n>We apply a two-stage evaluation pipeline that is designed to precisely control the sources of information and their exposure levels.<n>Through the evaluation of four baseline VLM unlearning algorithms within FIUBench, we find that all methods remain limited in their unlearning performance.
  arXiv  Detail & Related papers  (2024-11-05T23:26:10Z)
• Dataset resulting from the user study on comprehensibility of explainable AI algorithms [5.775094401949666]
  This paper introduces a dataset that is the result of a user study on the comprehensibility of explainable artificial intelligence (XAI) algorithms.<n>The study participants were recruited from 149 candidates to form three groups representing experts in the domain of mycology (DE)<n>The main part of the dataset contains 39 transcripts of interviews during which participants were asked to complete a series of tasks and questions related to the interpretation of decisions of a machine learning model trained to distinguish between edible and inedible mushrooms.
  arXiv  Detail & Related papers  (2024-10-21T11:37:58Z)
• Towards Effective Evaluations and Comparisons for LLM Unlearning Methods [97.2995389188179]
  This paper seeks to refine the evaluation of machine unlearning for large language models.<n>It addresses two key challenges -- the robustness of evaluation metrics and the trade-offs between competing goals.
  arXiv  Detail & Related papers  (2024-06-13T14:41:00Z)
• Explainable Few-shot Knowledge Tracing [48.877979333221326]
  We propose a cognition-guided framework that can track the student knowledge from a few student records while providing natural language explanations. Experimental results from three widely used datasets show that LLMs can perform comparable or superior to competitive deep knowledge tracing methods.
  arXiv  Detail & Related papers  (2024-05-23T10:07:21Z)
• Deep Learning-Based Object Pose Estimation: A Comprehensive Survey [73.74933379151419]
  We discuss the recent advances in deep learning-based object pose estimation. Our survey also covers multiple input data modalities, degrees-of-freedom of output poses, object properties, and downstream tasks.
  arXiv  Detail & Related papers  (2024-05-13T14:44:22Z)
• Bias and Fairness in Large Language Models: A Survey [73.87651986156006]
  We present a comprehensive survey of bias evaluation and mitigation techniques for large language models (LLMs) We first consolidate, formalize, and expand notions of social bias and fairness in natural language processing. We then unify the literature by proposing three intuitive, two for bias evaluation, and one for mitigation.
  arXiv  Detail & Related papers  (2023-09-02T00:32:55Z)
• Automated Distractor and Feedback Generation for Math Multiple-choice Questions via In-context Learning [43.83422798569986]
  Multiple-choice questions (MCQs) are ubiquitous in almost all levels of education since they are easy to administer, grade, and reliable form of assessment. To date, the task of crafting high-quality distractors has largely remained a labor-intensive process for teachers and learning content designers. We propose a simple, in-context learning-based solution for automated distractor and corresponding feedback message generation.
  arXiv  Detail & Related papers  (2023-08-07T01:03:04Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.