Error Understanding in Program Code With LLM-DL for Multi-label Classification

• URL: http://arxiv.org/abs/2603.25005v1
• Date: Thu, 26 Mar 2026 04:05:42 GMT
• Title: Error Understanding in Program Code With LLM-DL for Multi-label Classification
• Abstract summary: Large Language Models (LLMs) have shown remarkable capabilities in natural language understanding and generation tasks.<n>This study proposes a multi-label error classification framework for source code that leverages fine-tuned LLMs.<n>This work lays the foundation for developing intelligent, scalable tools for automated code feedback.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Programming is a core skill in computer science and software engineering (SE), yet identifying and resolving code errors remains challenging for both novice and experienced developers. While Large Language Models (LLMs) have shown remarkable capabilities in natural language understanding and generation tasks, their potential in domain-specific, complex scenarios, such as multi-label classification (MLC) of programming errors, remains underexplored. Recognizing this less-explored area, this study proposes a multi-label error classification (MLEC) framework for source code that leverages fine-tuned LLMs, including CodeT5-base, GraphCodeBERT, CodeT5+, UniXcoder, RoBERTa, PLBART, and CoTexT. These LLMs are integrated with deep learning (DL) architectures such as GRU, LSTM, BiLSTM, and BiLSTM with an additive attention mechanism (BiLSTM-A) to capture both syntactic and semantic features from a real-world student-written Python code error dataset. Extensive experiments across 32 model variants, optimized using Optuna-based hyperparameter tuning, have been evaluated using comprehensive multi-label metrics, including average accuracy, macro and weighted precision, recall, F1-score, exact match accuracy, One-error, Hamming loss, Jaccard similarity, and ROC-AUC (micro, macro, and weighted). Results show that the CodeT5+\_GRU model achieved the strongest performance, with a weighted F1-score of 0.8243, average accuracy of 91.84\%, exact match accuracy of 53.78\%, Hamming loss of 0.0816, and One error of 0.0708. These findings confirm the effectiveness of combining pretrained semantic encoders with efficient recurrent decoders. This work lays the foundation for developing intelligent, scalable tools for automated code feedback, with potential applications in programming education (PE) and broader SE domains.

Related papers

• From Code to Prediction: Fine-Tuning LLMs for Neural Network Performance Classification in NNGPT [48.83701310501069]
  We present a classification task integrated into the NNGPT framework.<n>A fine-tuned LLM predicts which of two image classification datasets a given neural network architecture achieves higher accuracy on.<n>The results establish that LLMs can be fine-tuned to predict cross-dataset suitability from neural network code.
  arXiv  Detail & Related papers  (2026-05-05T12:30:19Z)
• Evaluating Prompting and Execution-Based Methods for Deterministic Computation in LLMs [1.2957535240267326]
  Large Language Models (LLMs) have demonstrated strong capabilities in natural language understanding and reasoning.<n>We systematically evaluate multiple prompting strategies, including Chain-of-Thought (CoT), Least-to-Most decomposition, Program-of-Thought (PoT), and Self-Consistency (SC)<n>Our results show that standard prompting methods achieve only moderate accuracy on sequence-based tasks.
  arXiv  Detail & Related papers  (2026-05-04T23:32:37Z)
• Whitespaces Don't Lie: Feature-Driven and Embedding-Based Approaches for Detecting Machine-Generated Code [0.2624902795082451]
  Large language models (LLMs) have made it remarkably easy to synthesize plausible source code from natural language prompts.<n>This paper investigates the problem of distinguishing human-written from machine-generated code by comparing two complementary approaches.
  arXiv  Detail & Related papers  (2026-01-27T06:43:51Z)
• Anka: A Domain-Specific Language for Reliable LLM Code Generation [0.0]
  Large Language Models (LLMs) exhibit systematic errors on complex, multi-step programming tasks.<n>We introduce Anka, a domain-specific language () for data transformation pipelines designed with explicit, constrained syntax.<n>Anka achieves 99.9% parse success and 95.8% overall task accuracy across 100 benchmark problems.
  arXiv  Detail & Related papers  (2025-12-29T05:28:17Z)
• SWE-Compass: Towards Unified Evaluation of Agentic Coding Abilities for Large Language Models [59.90381306452982]
  evaluating large language models (LLMs) for software engineering has been limited by narrow task coverage, language bias, and insufficient alignment with real-world developer.<n>We introduce SWE-1, a comprehensive benchmark that unifies heterogeneous code-related evaluations into a structured and production-aligned framework.<n>SWE- spans 8 task types, 8 programming scenarios, and 10 programming languages, with 2000 high-quality instances curated from authentic GitHub pull requests.
  arXiv  Detail & Related papers  (2025-11-07T18:01:32Z)
• Is Compression Really Linear with Code Intelligence? [60.123628177110206]
  textitFormat Annealing is a lightweight, transparent training methodology designed to assess the intrinsic capabilities of pre-trained models equitably.<n>Our empirical results reveal a fundamental logarithmic relationship between measured code intelligence and bits-per-character (BPC)<n>Our work provides a more nuanced understanding of compression's role in developing code intelligence and contributes a robust evaluation framework in the code domain.
  arXiv  Detail & Related papers  (2025-05-16T16:59:14Z)
• Program Semantic Inequivalence Game with Large Language Models [20.43560028315856]
  Large Language Models (LLMs) can achieve strong performance on everyday coding tasks, but they can fail on complex tasks that require non-trivial reasoning about program semantics.<n>In this work, we explore a method to synthetically generate code reasoning training data based on a semantic inequivalence game SInQ.<n>We prove that this setup enables theoretically unlimited improvement through self-play in the limit of infinite computational resources.
  arXiv  Detail & Related papers  (2025-05-02T20:03:35Z)
• EquiBench: Benchmarking Large Language Models' Reasoning about Program Semantics via Equivalence Checking [58.15568681219339]
  We introduce EquiBench, a new benchmark for evaluating large language models (LLMs)<n>This task directly tests a model's ability to reason about program semantics.<n>We evaluate 19 state-of-the-art LLMs and find that in the most challenging categories, the best accuracies are 63.8% and 76.2%, only modestly above the 50% random baseline.
  arXiv  Detail & Related papers  (2025-02-18T02:54:25Z)
• Subtle Errors in Reasoning: Preference Learning via Error-injected Self-editing [59.405145971637204]
  We propose a novel preference learning framework called eRror-Injected Self-Editing (RISE)<n>RISE injects predefined subtle errors into pivotal tokens in reasoning or steps to construct hard pairs for error mitigation.<n>Experiments validate the effectiveness of RISE, with preference learning on Qwen2-7B-Instruct yielding notable improvements of 3.0% on GSM8K and 7.9% on MATH with only 4.5K training samples.
  arXiv  Detail & Related papers  (2024-10-09T07:43:38Z)
• BigCodeBench: Benchmarking Code Generation with Diverse Function Calls and Complex Instructions [72.56339136017759]
  We introduce BigCodeBench, a benchmark that challenges Large Language Models (LLMs) to invoke multiple function calls as tools from 139 libraries and 7 domains for 1,140 fine-grained tasks.<n>Our evaluation shows that LLMs are not yet capable of following complex instructions to use function calls precisely, with scores up to 60%, significantly lower than the human performance of 97%.<n>We propose a natural-language-oriented variant of BigCodeBench, BigCodeBench-Instruct, that automatically transforms the original docstrings into short instructions only with essential information.
  arXiv  Detail & Related papers  (2024-06-22T15:52:04Z)
• DeepCode AI Fix: Fixing Security Vulnerabilities with Large Language Models [3.1690235522182104]
  Large language models (LLMs) are increasingly used to solve various programming tasks. We show that the task is difficult as it requires the model to learn long-range code relationships. We propose a technique to address these challenges with a new approach for querying and fine-tuning LLMs.
  arXiv  Detail & Related papers  (2024-02-19T18:35:40Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.