Logic Error Localization in Student Programming Assignments Using Pseudocode and Graph Neural Networks

• URL: http://arxiv.org/abs/2410.21282v1
• Date: Fri, 11 Oct 2024 01:46:24 GMT
• Title: Logic Error Localization in Student Programming Assignments Using Pseudocode and Graph Neural Networks
• Authors: Zhenyu Xu, Kun Zhang, Victor S. Sheng,
• Abstract summary: We develop a system designed to localize logic errors within student programming assignments at the line level. We employ a graph neural network to both localize and suggest corrections for logic errors. Our experimental results are promising, demonstrating a localization accuracy of 99.2% for logic errors within the top-10 suspected lines.
• Score: 31.600659350609476
• License:
• Abstract: Pseudocode is extensively used in introductory programming courses to instruct computer science students in algorithm design, utilizing natural language to define algorithmic behaviors. This learning approach enables students to convert pseudocode into source code and execute it to verify their algorithms' correctness. This process typically introduces two types of errors: syntax errors and logic errors. Syntax errors are often accompanied by compiler feedback, which helps students identify incorrect lines. In contrast, logic errors are more challenging because they do not trigger compiler errors and lack immediate diagnostic feedback, making them harder to detect and correct. To address this challenge, we developed a system designed to localize logic errors within student programming assignments at the line level. Our approach utilizes pseudocode as a scaffold to build a code-pseudocode graph, connecting symbols from the source code to their pseudocode counterparts. We then employ a graph neural network to both localize and suggest corrections for logic errors. Additionally, we have devised a method to efficiently gather logic-error-prone programs during the syntax error correction process and compile these into a dataset that includes single and multiple line logic errors, complete with indices of the erroneous lines. Our experimental results are promising, demonstrating a localization accuracy of 99.2% for logic errors within the top-10 suspected lines, highlighting the effectiveness of our approach in enhancing students' coding proficiency and error correction skills.

Related papers

• Combining Logic with Large Language Models for Automatic Debugging and Repair of ASP Programs [1.0650780147044159]
  FormHe is a tool that combines logic-based techniques and Large Language Models to identify and correct issues in Answer Set Programming submissions. We show that FormHe accurately detects faults in 94% of cases and successfully repairs 58% of incorrect submissions.
  arXiv  Detail & Related papers  (2024-10-28T12:30:48Z)
• LLM Critics Help Catch Bugs in Mathematics: Towards a Better Mathematical Verifier with Natural Language Feedback [71.95402654982095]
  We propose Math-Minos, a natural language feedback-enhanced verifier. Our experiments reveal that a small set of natural language feedback can significantly boost the performance of the verifier.
  arXiv  Detail & Related papers  (2024-06-20T06:42:27Z)
• Improving LLM Classification of Logical Errors by Integrating Error Relationship into Prompts [1.7095867620640115]
  A key aspect of programming education is understanding and dealing with error message. 'logical errors' in which the program operates against the programmer's intentions do not receive error messages from the compiler. We propose an effective approach for detecting logical errors with LLMs that makes use of relations among error types in the Chain-of-Thought and Tree-of-Thought prompts.
  arXiv  Detail & Related papers  (2024-04-30T08:03:22Z)
• Language Models as Compilers: Simulating Pseudocode Execution Improves Algorithmic Reasoning in Language Models [17.76252625790628]
  This paper presents Think-and-Execute, a framework that decomposes the reasoning process of language models into two steps. With extensive experiments on seven algorithmic reasoning tasks, we demonstrate the effectiveness of Think-and-Execute.
  arXiv  Detail & Related papers  (2024-04-03T08:49:11Z)
• Decoding Logic Errors: A Comparative Study on Bug Detection by Students and Large Language Models [5.162225137921625]
  Large language models (LLMs) have recently demonstrated surprising performance for a range of computing tasks. We investigate the performance of two popular LLMs, GPT-3 and GPT-4, for detecting and providing a novice-friendly explanation of logic errors.
  arXiv  Detail & Related papers  (2023-11-27T17:28:33Z)
• Testing the Accuracy of Surface Code Decoders [55.616364225463066]
  Large-scale, fault-tolerant quantum computations will be enabled by quantum error-correcting codes (QECC) This work presents the first systematic technique to test the accuracy and effectiveness of different QECC decoding schemes.
  arXiv  Detail & Related papers  (2023-11-21T10:22:08Z)
• Understanding and Mitigating Classification Errors Through Interpretable Token Patterns [58.91023283103762]
  Characterizing errors in easily interpretable terms gives insight into whether a classifier is prone to making systematic errors. We propose to discover those patterns of tokens that distinguish correct and erroneous predictions. We show that our method, Premise, performs well in practice.
  arXiv  Detail & Related papers  (2023-11-18T00:24:26Z)
• Refining neural network predictions using background knowledge [68.35246878394702]
  We show we can use logical background knowledge in learning system to compensate for a lack of labeled training data. We introduce differentiable refinement functions that find a corrected prediction close to the original prediction. This algorithm finds optimal refinements on complex SAT formulas in significantly fewer iterations and frequently finds solutions where gradient descent can not.
  arXiv  Detail & Related papers  (2022-06-10T10:17:59Z)
• Scaling up reservoir engineering for error-correcting codes [0.0]
  We focus on a proposal for the repetition code that counters bit-flip errors, and how to scale up the network encoding a logical quantum bit. The challenge has been to devise a network architecture which allows to autonomously correct higher-order errors, while remaining realistic towards experimental realization.
  arXiv  Detail & Related papers  (2020-10-06T16:14:13Z)
• On the Robustness of Language Encoders against Grammatical Errors [66.05648604987479]
  We collect real grammatical errors from non-native speakers and conduct adversarial attacks to simulate these errors on clean text data. Results confirm that the performance of all tested models is affected but the degree of impact varies.
  arXiv  Detail & Related papers  (2020-05-12T11:01:44Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.