Rule-Based Error Classification for Analyzing Differences in Frequent Errors

• URL: http://arxiv.org/abs/2311.00513v1
• Date: Wed, 1 Nov 2023 13:36:20 GMT
• Title: Rule-Based Error Classification for Analyzing Differences in Frequent Errors
• Authors: Atsushi Shirafuji, Taku Matsumoto, Md Faizul Ibne Amin, Yutaka Watanobe
• Abstract summary: We classify errors for 95,631 code pairs and identify 3.47 errors on average, which are submitted by various levels of programmers on an online judge system. The analyzed results show that, as for the same introductory problems, errors made by novices are due to the lack of knowledge in programming. On the other hand, errors made by experts are due to misunderstandings caused by the carelessness of reading problems or the challenges of solving problems differently than usual.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Finding and fixing errors is a time-consuming task not only for novice programmers but also for expert programmers. Prior work has identified frequent error patterns among various levels of programmers. However, the differences in the tendencies between novices and experts have yet to be revealed. From the knowledge of the frequent errors in each level of programmers, instructors will be able to provide helpful advice for each level of learners. In this paper, we propose a rule-based error classification tool to classify errors in code pairs consisting of wrong and correct programs. We classify errors for 95,631 code pairs and identify 3.47 errors on average, which are submitted by various levels of programmers on an online judge system. The classified errors are used to analyze the differences in frequent errors between novice and expert programmers. The analyzed results show that, as for the same introductory problems, errors made by novices are due to the lack of knowledge in programming, and the mistakes are considered an essential part of the learning process. On the other hand, errors made by experts are due to misunderstandings caused by the carelessness of reading problems or the challenges of solving problems differently than usual. The proposed tool can be used to create error-labeled datasets and for further code-related educational research.

Related papers

• MC-MKE: A Fine-Grained Multimodal Knowledge Editing Benchmark Emphasizing Modality Consistency [50.40318712497071]
  Multimodal large language models (MLLMs) are prone to non-factual or outdated knowledge issues. We decompose multimodal knowledge into its visual and textual components. We present MC-MKE, a fine-grained Multimodal Knowledge Editing benchmark.
  arXiv  Detail & Related papers  (2024-06-19T05:15:21Z)
• Improving LLM Classification of Logical Errors by Integrating Error Relationship into Prompts [1.7095867620640115]
  A key aspect of programming education is understanding and dealing with error message. 'logical errors' in which the program operates against the programmer's intentions do not receive error messages from the compiler. We propose an effective approach for detecting logical errors with LLMs that makes use of relations among error types in the Chain-of-Thought and Tree-of-Thought prompts.
  arXiv  Detail & Related papers  (2024-04-30T08:03:22Z)
• An Information-Theoretic Analysis of In-Context Learning [67.62099509406173]
  We introduce new information-theoretic tools that lead to an elegant and very general decomposition of error into three components: irreducible error, meta-learning error, and intra-task error. Our theoretical results characterizes how error decays in both the number of training sequences and sequence lengths.
  arXiv  Detail & Related papers  (2024-01-28T00:36:44Z)
• Understanding and Mitigating Classification Errors Through Interpretable Token Patterns [58.91023283103762]
  Characterizing errors in easily interpretable terms gives insight into whether a classifier is prone to making systematic errors. We propose to discover those patterns of tokens that distinguish correct and erroneous predictions. We show that our method, Premise, performs well in practice.
  arXiv  Detail & Related papers  (2023-11-18T00:24:26Z)
• Giving Feedback on Interactive Student Programs with Meta-Exploration [74.5597783609281]
  Developing interactive software, such as websites or games, is a particularly engaging way to learn computer science. Standard approaches require instructors to manually grade student-implemented interactive programs. Online platforms that serve millions, like Code.org, are unable to provide any feedback on assignments for implementing interactive programs.
  arXiv  Detail & Related papers  (2022-11-16T10:00:23Z)
• Using Large Language Models to Enhance Programming Error Messages [5.903720638984496]
  Large language models can be used to create useful enhancements to programming error messages. We discuss the benefits and downsides of large language models and highlight future streams of research for enhancing programming error messages.
  arXiv  Detail & Related papers  (2022-10-20T23:17:26Z)
• Fault-Aware Neural Code Rankers [64.41888054066861]
  We propose fault-aware neural code rankers that can predict the correctness of a sampled program without executing it. Our fault-aware rankers can significantly increase the pass@1 accuracy of various code generation models.
  arXiv  Detail & Related papers  (2022-06-04T22:01:05Z)
• Some Grammatical Errors are Frequent, Others are Important [32.922128367314194]
  We show that some rare errors are considered disturbing while other common ones are not. This affects possible directions to improve both systems and their evaluation.
  arXiv  Detail & Related papers  (2022-05-11T18:59:20Z)
• Repairing Group-Level Errors for DNNs Using Weighted Regularization [15.180437840817785]
  Deep Neural Networks (DNNs) have been widely used in software making decisions impacting people's lives. They have been found to exhibit severe erroneous behaviors that may lead to unfortunate outcomes. Previous work shows that such misbehaviors often occur due to class property violations rather than errors on a single image. Here, we propose a generic method called Weighted Regularization consisting of five concrete methods targeting the error-producing classes to fix the DNNs.
  arXiv  Detail & Related papers  (2022-03-24T15:45:23Z)
• Identifying Bug Patterns in Quantum Programs [4.282118876884235]
  Bug patterns are erroneous code idioms or bad coding practices that have been proved to fail time and time again. This paper identifies and categorizes some bug patterns in the quantum programming language Qiskit.
  arXiv  Detail & Related papers  (2021-03-16T13:43:45Z)
• On the Robustness of Language Encoders against Grammatical Errors [66.05648604987479]
  We collect real grammatical errors from non-native speakers and conduct adversarial attacks to simulate these errors on clean text data. Results confirm that the performance of all tested models is affected but the degree of impact varies.
  arXiv  Detail & Related papers  (2020-05-12T11:01:44Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.