Mining the Gold: Student-AI Chat Logs as Rich Sources for Automated Knowledge Gap Detection

• URL: http://arxiv.org/abs/2512.22404v1
• Date: Fri, 26 Dec 2025 23:04:04 GMT
• Title: Mining the Gold: Student-AI Chat Logs as Rich Sources for Automated Knowledge Gap Detection
• Authors: Quanzhi Fu, Qiyu Wu, Dan Williams,
• Abstract summary: In large lectures, instructors face challenges on identifying students' knowledge gaps timely.<n>We propose QueryQuilt, a multi-agent LLM framework that automatically detects common knowledge gaps in large-scale lectures by analyzing students' chat logs with AI assistants.<n>Our evaluation demonstrates promising results, with QueryQuilt achieving 100% accuracy in identifying knowledge gaps among simulated students and 95% completeness when tested on real student-AI dialogue data.
• Score: 2.6808104662419097
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: With the significant increase in enrollment in computing-related programs over the past 20 years, lecture sizes have grown correspondingly. In large lectures, instructors face challenges on identifying students' knowledge gaps timely, which is critical for effective teaching. Existing classroom response systems rely on instructor-initiated interactions, which limits their ability to capture the spontaneous knowledge gaps that naturally emerge during lectures. With the widespread adoption of LLMs among students, we recognize these student-AI dialogues as a valuable, student-centered data source for identifying knowledge gaps. In this idea paper, we propose QueryQuilt, a multi-agent LLM framework that automatically detects common knowledge gaps in large-scale lectures by analyzing students' chat logs with AI assistants. QueryQuilt consists of two key components: (1) a Dialogue Agent that responds to student questions while employing probing questions to reveal underlying knowledge gaps, and (2) a Knowledge Gap Identification Agent that systematically analyzes these dialogues to identify knowledge gaps across the student population. By generating frequency distributions of identified gaps, instructors can gain comprehensive insights into class-wide understanding. Our evaluation demonstrates promising results, with QueryQuilt achieving 100% accuracy in identifying knowledge gaps among simulated students and 95% completeness when tested on real student-AI dialogue data. These initial findings indicate the system's potential for facilitate teaching in authentic learning environments. We plan to deploy QueryQuilt in actual classroom settings for comprehensive evaluation, measuring its detection accuracy and impact on instruction.

Related papers

• What Students Ask, How a Generative AI Assistant Responds: Exploring Higher Education Students' Dialogues on Learning Analytics Feedback [0.3562673545689596]
  Learning analytics dashboards (LADs) aim to support students' regulation of learning by translating complex data into feedback.<n>Students, especially those with lower self-regulated learning (SRL) competence, often struggle to engage with and interpret analytics feedback.<n>We explored authentic dialogues between students and GenAI assistant integrated into LAD during a 10-week semester.
  arXiv  Detail & Related papers  (2026-01-08T13:17:44Z)
• LLM4CD: Leveraging Large Language Models for Open-World Knowledge Augmented Cognitive Diagnosis [56.50378080174923]
  We propose LLM4CD, which Leverages Large Language Models for Open-World Knowledge Augmented Cognitive Diagnosis.<n>Our method utilizes the open-world knowledge of LLMs to construct cognitively expressive textual representations, which are encoded to introduce rich semantic information into the CD task.<n>This approach substitutes traditional ID embeddings with semantic representations, enabling the model to accommodate new students and exercises with open-world knowledge and address the cold-start problem.
  arXiv  Detail & Related papers  (2025-05-14T14:48:00Z)
• Investigating Large Language Models in Diagnosing Students' Cognitive Skills in Math Problem-solving [23.811625065982486]
  We investigate how state-of-the-art large language models diagnose students' cognitive skills in mathematics.<n>We constructed MathCog, a novel benchmark dataset comprising 639 student responses to 110 middle school math problems.<n>Our evaluation reveals that even the state-of-the-art LLMs struggle with the task, all F1 scores below 0.5, and tend to exhibit strong false confidence for incorrect cases.
  arXiv  Detail & Related papers  (2025-04-01T14:29:41Z)
• Exploring Knowledge Tracing in Tutor-Student Dialogues using LLMs [49.18567856499736]
  We investigate whether large language models (LLMs) can be supportive of open-ended dialogue tutoring.<n>We apply a range of knowledge tracing (KT) methods on the resulting labeled data to track student knowledge levels over an entire dialogue.<n>We conduct experiments on two tutoring dialogue datasets, and show that a novel yet simple LLM-based method, LLMKT, significantly outperforms existing KT methods in predicting student response correctness in dialogues.
  arXiv  Detail & Related papers  (2024-09-24T22:31:39Z)
• Explainable Few-shot Knowledge Tracing [48.877979333221326]
  We propose a cognition-guided framework that can track the student knowledge from a few student records while providing natural language explanations. Experimental results from three widely used datasets show that LLMs can perform comparable or superior to competitive deep knowledge tracing methods.
  arXiv  Detail & Related papers  (2024-05-23T10:07:21Z)
• Beyond Factuality: A Comprehensive Evaluation of Large Language Models as Knowledge Generators [78.63553017938911]
  Large language models (LLMs) outperform information retrieval techniques for downstream knowledge-intensive tasks. However, community concerns abound regarding the factuality and potential implications of using this uncensored knowledge. We introduce CONNER, designed to evaluate generated knowledge from six important perspectives.
  arXiv  Detail & Related papers  (2023-10-11T08:22:37Z)
• Quiz-based Knowledge Tracing [61.9152637457605]
  Knowledge tracing aims to assess individuals' evolving knowledge states according to their learning interactions. QKT achieves state-of-the-art performance compared to existing methods.
  arXiv  Detail & Related papers  (2023-04-05T12:48:42Z)
• RHO ($\ ho$): Reducing Hallucination in Open-domain Dialogues with Knowledge Grounding [57.46495388734495]
  This paper presents RHO ($rho$) utilizing the representations of linked entities and relation predicates from a knowledge graph (KG) We propose (1) local knowledge grounding to combine textual embeddings with the corresponding KG embeddings; and (2) global knowledge grounding to equip RHO with multi-hop reasoning abilities via the attention mechanism.
  arXiv  Detail & Related papers  (2022-12-03T10:36:34Z)
• GPT-based Open-Ended Knowledge Tracing [24.822739021636455]
  We study the new task of predicting students' exact open-ended responses to questions. Our work is grounded in the domain of computer science education with programming questions. We develop an initial solution to the OKT problem, a student knowledge-guided code generation approach.
  arXiv  Detail & Related papers  (2022-02-21T02:33:34Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.