ConceptKT: A Benchmark for Concept-Level Deficiency Prediction in Knowledge Tracing

• URL: http://arxiv.org/abs/2603.24073v1
• Date: Wed, 25 Mar 2026 08:27:01 GMT
• Title: ConceptKT: A Benchmark for Concept-Level Deficiency Prediction in Knowledge Tracing
• Abstract summary: We present ConceptKT, a dataset annotated with labels that capture both the concepts required to solve each question and the missing concepts underlying incorrect responses.<n>We investigate in-context learning approaches to Knowledge Tracing (KT) and evaluate the diagnostic capabilities of various Large Language Models (LLMs) and Large Reasoning Models (LRMs)<n> Experimental results demonstrate that selecting response histories based on conceptual alignment and semantic similarity leads to improved performance on both correctness prediction and concept-level deficiency identification.
• Score: 5.988618373867388
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Knowledge Tracing (KT) is a critical technique for modeling student knowledge to support personalized learning. However, most KT systems focus on binary correctness prediction and cannot diagnose the underlying conceptual misunderstandings that lead to errors. Such fine-grained diagnostic feedback is essential for designing targeted instruction and effective remediation. In this work, we introduce the task of concept-level deficiency prediction, which extends traditional KT by identifying the specific concepts a student is likely to struggle with on future problems. We present ConceptKT, a dataset annotated with labels that capture both the concepts required to solve each question and the missing concepts underlying incorrect responses. We investigate in-context learning approaches to KT and evaluate the diagnostic capabilities of various Large Language Models (LLMs) and Large Reasoning Models (LRMs). Different strategies for selecting informative historical records are explored. Experimental results demonstrate that selecting response histories based on conceptual alignment and semantic similarity leads to improved performance on both correctness prediction and concept-level deficiency identification.

Related papers

• Beyond ID Embeddings: Process-Grounded Language Modeling for Cognitive Diagnosis [77.02066500576528]
  Process-aware Language Cognitive Diagnosis (PLCD) framework uses language-derived structures as cognitive priors and response records to calibrate student posterior states.<n>PLCD not only outperforms traditional baselines in predicting student performance but also exhibits strong cognitive transfer capabilities.
  arXiv  Detail & Related papers  (2026-09-11T03:51:20Z)
• Rethinking Concept Bottleneck Models: From Pitfalls to Solutions [53.84388497227224]
  Concept Bottleneck Models (CBMs) ground predictions in human-understandable concepts.<n>CBM-Suite is a methodological framework to address these challenges.
  arXiv  Detail & Related papers  (2026-03-05T19:37:49Z)
• PICKT: Practical Interlinked Concept Knowledge Tracing for Personalized Learning using Knowledge Map Concept Relations [2.449909275410288]
  This paper focuses on the core technology of Knowledge Tracing models that analyze students' sequences of interactions to predict their knowledge acquisition levels.<n>A knowledge map structures the relationships among concepts considering the question and concept text information, thereby enabling effective knowledge tracing even in cold start situations.
  arXiv  Detail & Related papers  (2025-12-08T05:24:17Z)
• Understanding Catastrophic Interference: On the Identifibility of Latent Representations [67.05452287233122]
  Catastrophic interference, also known as catastrophic forgetting, is a fundamental challenge in machine learning.<n>We propose a novel theoretical framework that formulates catastrophic interference as an identification problem.<n>Our approach provides both theoretical guarantees and practical performance improvements across both synthetic and benchmark datasets.
  arXiv  Detail & Related papers  (2025-09-27T00:53:32Z)
• Harnessing Structured Knowledge: A Concept Map-Based Approach for High-Quality Multiple Choice Question Generation with Effective Distractors [0.0]
  This paper presents a hierarchical concept map-based framework that provides structured knowledge to guide LLMs in generating MCQs with distractors.<n>We evaluate our framework against two baseline approaches: a base LLM and a RAG-based generation.
  arXiv  Detail & Related papers  (2025-05-02T06:36:06Z)
• Language Guided Concept Bottleneck Models for Interpretable Continual Learning [62.09201360376577]
  Continual learning aims to enable learning systems to acquire new knowledge constantly without forgetting previously learned information.<n>Most existing CL methods focus primarily on preserving learned knowledge to improve model performance.<n>We introduce a novel framework that integrates language-guided Concept Bottleneck Models to address both challenges.
  arXiv  Detail & Related papers  (2025-03-30T02:41:55Z)
• Sample-efficient Learning of Concepts with Theoretical Guarantees: from Data to Concepts without Interventions [13.877511370053794]
  Concept Bottleneck Models (CBM) address some of these challenges by learning interpretable concepts from high-dimensional data.<n>We describe a framework that provides theoretical guarantees on the correctness of the learned concepts and on the number of required labels.<n>We evaluate our framework in synthetic and image benchmarks, showing that the learned concepts have less impurities and are often more accurate than other CBMs.
  arXiv  Detail & Related papers  (2025-02-10T15:01:56Z)
• Automated Knowledge Concept Annotation and Question Representation Learning for Knowledge Tracing [59.480951050911436]
  We present KCQRL, a framework for automated knowledge concept annotation and question representation learning.<n>We demonstrate the effectiveness of KCQRL across 15 KT algorithms on two large real-world Math learning datasets.
  arXiv  Detail & Related papers  (2024-10-02T16:37:19Z)
• Enhancing Knowledge Tracing with Concept Map and Response Disentanglement [5.201585012263761]
  We propose the Concept map-driven Response disentanglement method for enhancing Knowledge Tracing (CRKT) model. CRKT benefits KT by directly leveraging answer choices--beyond merely identifying correct or incorrect answers--to distinguish responses with different incorrect choices. We further introduce the novel use of unchosen responses by employing disentangled representations to get insights from options not selected by students.
  arXiv  Detail & Related papers  (2024-08-23T11:25:56Z)
• SINKT: A Structure-Aware Inductive Knowledge Tracing Model with Large Language Model [64.92472567841105]
  Knowledge Tracing (KT) aims to determine whether students will respond correctly to the next question. Structure-aware Inductive Knowledge Tracing model with large language model (dubbed SINKT) SINKT predicts the student's response to the target question by interacting with the student's knowledge state and the question representation.
  arXiv  Detail & Related papers  (2024-07-01T12:44:52Z)
• Differentiating Student Feedbacks for Knowledge Tracing [28.669001606806525]
  We propose a framework to reweight the contribution of different responses based on their discrimination in training.<n>We also introduce an adaptive predictive score fusion technique to maintain accuracy on less discriminative responses.
  arXiv  Detail & Related papers  (2022-12-16T13:55:07Z)
• DISSECT: Disentangled Simultaneous Explanations via Concept Traversals [33.65478845353047]
  DISSECT is a novel approach to explaining deep learning model inferences. By training a generative model from a classifier's signal, DISSECT offers a way to discover a classifier's inherent "notion" of distinct concepts. We show that DISSECT produces CTs that disentangle several concepts and are coupled to its reasoning due to joint training.
  arXiv  Detail & Related papers  (2021-05-31T17:11:56Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.