Analyzing the Difficulty of Programming Assignments with Interpretable Knowledge Component Metrics
- URL: http://arxiv.org/abs/2607.03419v1
- Date: Fri, 03 Jul 2026 15:28:58 GMT
- Title: Analyzing the Difficulty of Programming Assignments with Interpretable Knowledge Component Metrics
- Abstract summary: This research paper examines how Knowledge Components (KCs) can be used as interpretable signals for understanding assignment difficulty and student struggle.<n>We analyze KC-based metrics, including the number of KCs per assignment and changes in KC coverage between consecutive assignments.<n>We propose an interpretable framework for analyzing programming assignments using KC-based metrics, with practical implications for instructors and course designers.
- Score: 14.174837465294717
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: This research paper examines how Knowledge Components (KCs) - fine-grained concepts or skills required to solve programming tasks - can be used as interpretable signals for understanding assignment difficulty and student struggle in introductory programming courses. While prior work has focused on predictive models based on programming behavior, such models are often difficult to interpret and therefore hard to use for instructional decisions. We analyze KC-based metrics, including the number of KCs per assignment and changes in KC coverage between consecutive assignments. We examine correlations between the number of KCs and student performance on the assignment, and analyze changes in KCs across assignments to identify cases where performance declines without new concepts being introduced. Selected assignments are then qualitatively inspected to understand potential design issues. Our results on data from three introductory programming course datasets show that assignments involving more KCs are generally associated with lower performance, and that sudden shifts in required KCs can coincide with disruptions in learning progression. We also identify assignments where performance declines even though no new KCs are introduced, suggesting potential issues in task design or instruction, which could be examined with qualitative analysis. We propose an interpretable framework for analyzing programming assignments using KC-based metrics, with practical implications for instructors and course designers who want to better understand where and why students struggle, and how course materials might be improved. Our method can use KCs defined by an expert or extracted by an LLM, offering instructors an additional way to assess assignment quality beyond average correctness. It can be applied to any course with ordered assignments and measurable performance.
Related papers
- Personalized Worked Example Generation from Student Code Submissions Using Pattern-based Knowledge Components [8.40619529758373]
We present an approach for knowledge-component guided educational content generation using pattern-based KCs extracted from student code.<n>Given a problem statement and student submissions, our pipeline extracts recurring structural KC patterns from students' code through AST-based analysis and uses them to condition a generative model.<n>Results suggest that KC-conditioned generation improves topical focus and relevance to students' underlying logical errors.
arXiv Detail & Related papers (2026-04-27T17:56:56Z) - Using LLMs for Knowledge Component-level Correctness Labeling in Open-ended Coding Problems [0.4316506818580031]
We propose an automated framework to label KC-level correctness directly from student-written code.<n>We evaluate the resulting KC-level correctness labels in terms of learning curve fit and predictive performance.
arXiv Detail & Related papers (2026-02-19T16:58:34Z) - Enhancing Large Language Model for Knowledge Graph Completion via Structure-Aware Alignment-Tuning [52.78024385391959]
Knowledge graph completion (KGC) aims to infer new knowledge and make predictions from knowledge graphs.<n>Existing methods often ignore the inconsistent representation spaces between natural language and graph structures.<n>We propose SAT, a novel framework that enhances LLMs for KGC via structure-aware alignment-tuning.
arXiv Detail & Related papers (2025-09-01T06:38:11Z) - Pattern-based Knowledge Component Extraction from Student Code Using Representation Learning [2.726913697825415]
This work advances knowledge modeling in computer science education by providing an automated, scalable, and explainable framework for identifying granular code patterns and algorithmic constructs, essential for student learning.
arXiv Detail & Related papers (2025-08-12T18:42:15Z) - Illusion or Algorithm? Investigating Memorization, Emergence, and Symbolic Processing in In-Context Learning [50.53703102032562]
Large-scale Transformer language models (LMs) trained solely on next-token prediction with web-scale data can solve a wide range of tasks.<n>The mechanism behind this capability, known as in-context learning (ICL), remains both controversial and poorly understood.
arXiv Detail & Related papers (2025-05-16T08:50:42Z) - KCluster: An LLM-based Clustering Approach to Knowledge Component Discovery [0.26626950367610397]
We propose KCluster, a novel KC discovery algorithm based on identifying clusters of congruent questions.<n>We demonstrate in three datasets that an LLM can create an effective metric of question similarity.<n>KCluster generates descriptive KC labels and discovers KC models that predict student performance better than the best expert-designed models.
arXiv Detail & Related papers (2025-05-09T23:47:58Z) - Automated Knowledge Component Generation for Interpretable Knowledge Tracing in Coding Problems [2.801976382946474]
Knowledge components (KCs) mapped to problems help model student learning, tracking their mastery levels on fine-grained skills.<n>We present an automated, LLM-based pipeline for KC generation and tagging for open-ended programming problems.<n>We find that KCGen-KT outperforms existing KT methods and human-written KCs on future student response prediction.
arXiv Detail & Related papers (2025-02-25T20:40:51Z) - Learning Task Representations from In-Context Learning [67.66042137487287]
Large language models (LLMs) have demonstrated remarkable proficiency in in-context learning (ICL)<n>We introduce an automated formulation for encoding task information in ICL prompts as a function of attention heads.<n>The proposed method successfully extracts task-specific information from in-context demonstrations and excels in both text and regression tasks.
arXiv Detail & Related papers (2025-02-08T00:16:44Z) - Automated Knowledge Concept Annotation and Question Representation Learning for Knowledge Tracing [59.480951050911436]
We present KCQRL, a framework for automated knowledge concept annotation and question representation learning.<n>We demonstrate the effectiveness of KCQRL across 15 KT algorithms on two large real-world Math learning datasets.
arXiv Detail & Related papers (2024-10-02T16:37:19Z) - A Unified Causal View of Instruction Tuning [76.1000380429553]
We develop a meta Structural Causal Model (meta-SCM) to integrate different NLP tasks under a single causal structure of the data.
Key idea is to learn task-required causal factors and only use those to make predictions for a given task.
arXiv Detail & Related papers (2024-02-09T07:12:56Z) - Using Representation Expressiveness and Learnability to Evaluate
Self-Supervised Learning Methods [61.49061000562676]
We introduce Cluster Learnability (CL) to assess learnability.
CL is measured in terms of the performance of a KNN trained to predict labels obtained by clustering the representations with K-means.
We find that CL better correlates with in-distribution model performance than other competing recent evaluation schemes.
arXiv Detail & Related papers (2022-06-02T19:05:13Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.