Rule-Based Error Classification for Analyzing Differences in Frequent Errors

• URL: http://arxiv.org/abs/2311.00513v1
• Date: Wed, 1 Nov 2023 13:36:20 GMT
• Title: Rule-Based Error Classification for Analyzing Differences in Frequent Errors
• Authors: Atsushi Shirafuji, Taku Matsumoto, Md Faizul Ibne Amin, Yutaka Watanobe
• Abstract summary: We classify errors for 95,631 code pairs and identify 3.47 errors on average, which are submitted by various levels of programmers on an online judge system. The analyzed results show that, as for the same introductory problems, errors made by novices are due to the lack of knowledge in programming. On the other hand, errors made by experts are due to misunderstandings caused by the carelessness of reading problems or the challenges of solving problems differently than usual.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Finding and fixing errors is a time-consuming task not only for novice programmers but also for expert programmers. Prior work has identified frequent error patterns among various levels of programmers. However, the differences in the tendencies between novices and experts have yet to be revealed. From the knowledge of the frequent errors in each level of programmers, instructors will be able to provide helpful advice for each level of learners. In this paper, we propose a rule-based error classification tool to classify errors in code pairs consisting of wrong and correct programs. We classify errors for 95,631 code pairs and identify 3.47 errors on average, which are submitted by various levels of programmers on an online judge system. The classified errors are used to analyze the differences in frequent errors between novice and expert programmers. The analyzed results show that, as for the same introductory problems, errors made by novices are due to the lack of knowledge in programming, and the mistakes are considered an essential part of the learning process. On the other hand, errors made by experts are due to misunderstandings caused by the carelessness of reading problems or the challenges of solving problems differently than usual. The proposed tool can be used to create error-labeled datasets and for further code-related educational research.

Related papers

• From Bugs to Breakthroughs: Novice Errors in CS2 [1.0609815608017066]
  We conducted a longitudinal study of errors that students of a CS2 course made in subsequent programming assignments. We manually categorized 710 errors based on a modified version of an established error framework. Students have only little trouble with learning the programming language, but need more time to understand and express concepts in a programming language.
  arXiv  Detail & Related papers  (2025-02-20T10:41:44Z)
• Flow Graph-Based Classification of Defects4J Faults [49.38684825106323]
  Software fault datasets such as Defects4J provide for each individual fault its location and repair, but do not characterize the faults. We propose a new, direct fault classification scheme based on the control- and data-flow graph representations of the program.
  arXiv  Detail & Related papers  (2025-02-04T13:10:28Z)
• Rethinking Early Stopping: Refine, Then Calibrate [49.966899634962374]
  We show that calibration error and refinement error are not minimized simultaneously during training. We introduce a new metric for early stopping and hyper parameter tuning that makes it possible to minimize refinement error during training. Our method integrates seamlessly with any architecture and consistently improves performance across diverse classification tasks.
  arXiv  Detail & Related papers  (2025-01-31T15:03:54Z)
• Debugging Without Error Messages: How LLM Prompting Strategy Affects Programming Error Explanation Effectiveness [0.9014547127329643]
  We show how GPT-3.5 is prompted for error explanations on just the erroneous source code itself. We report the baseline results of how effective the error explanations are at providing feedback.
  arXiv  Detail & Related papers  (2025-01-10T04:32:19Z)
• Subtle Errors Matter: Preference Learning via Error-injected Self-editing [59.405145971637204]
  We propose a novel preference learning framework called eRror-Injected Self-Editing (RISE) RISE injects predefined subtle errors into partial tokens of correct solutions to construct hard pairs for error mitigation. Experiments validate the effectiveness of RISE, with preference learning on Qwen2-7B-Instruct yielding notable improvements of 3.0% on GSM8K and 7.9% on MATH.
  arXiv  Detail & Related papers  (2024-10-09T07:43:38Z)
• MC-MKE: A Fine-Grained Multimodal Knowledge Editing Benchmark Emphasizing Modality Consistency [50.40318712497071]
  Multimodal large language models (MLLMs) are prone to non-factual or outdated knowledge issues. We decompose multimodal knowledge into its visual and textual components. We present MC-MKE, a fine-grained Multimodal Knowledge Editing benchmark.
  arXiv  Detail & Related papers  (2024-06-19T05:15:21Z)
• Improving LLM Classification of Logical Errors by Integrating Error Relationship into Prompts [1.7095867620640115]
  A key aspect of programming education is understanding and dealing with error message. 'logical errors' in which the program operates against the programmer's intentions do not receive error messages from the compiler. We propose an effective approach for detecting logical errors with LLMs that makes use of relations among error types in the Chain-of-Thought and Tree-of-Thought prompts.
  arXiv  Detail & Related papers  (2024-04-30T08:03:22Z)
• Understanding and Mitigating Classification Errors Through Interpretable Token Patterns [58.91023283103762]
  Characterizing errors in easily interpretable terms gives insight into whether a classifier is prone to making systematic errors. We propose to discover those patterns of tokens that distinguish correct and erroneous predictions. We show that our method, Premise, performs well in practice.
  arXiv  Detail & Related papers  (2023-11-18T00:24:26Z)
• Some Grammatical Errors are Frequent, Others are Important [32.922128367314194]
  We show that some rare errors are considered disturbing while other common ones are not. This affects possible directions to improve both systems and their evaluation.
  arXiv  Detail & Related papers  (2022-05-11T18:59:20Z)
• Identifying Bug Patterns in Quantum Programs [4.282118876884235]
  Bug patterns are erroneous code idioms or bad coding practices that have been proved to fail time and time again. This paper identifies and categorizes some bug patterns in the quantum programming language Qiskit.
  arXiv  Detail & Related papers  (2021-03-16T13:43:45Z)
• On the Robustness of Language Encoders against Grammatical Errors [66.05648604987479]
  We collect real grammatical errors from non-native speakers and conduct adversarial attacks to simulate these errors on clean text data. Results confirm that the performance of all tested models is affected but the degree of impact varies.
  arXiv  Detail & Related papers  (2020-05-12T11:01:44Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.