Chatbot-Based Assessment of Code Understanding in Automated Programming Assessment Systems

• URL: http://arxiv.org/abs/2604.07304v1
• Date: Wed, 08 Apr 2026 17:11:30 GMT
• Title: Chatbot-Based Assessment of Code Understanding in Automated Programming Assessment Systems
• Abstract summary: This paper reports a saturation-based scoping review of conversational assessment approaches in programming education.<n>The review identifies three dominant architectural families: rule-based or template-driven systems, LLM-based systems, and hybrid systems.<n>The paper synthesizes these findings into a Hybrid Socratic Framework for integrating conversational verification into Automated Programming Assessment Systems.
• Score: 1.0705399532413615
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Large Language Models (LLMs) challenge conventional automated programming assessment because students can now produce functionally correct code without demonstrating corresponding understanding. This paper makes two contributions. First, it reports a saturation-based scoping review of conversational assessment approaches in programming education. The review identifies three dominant architectural families: rule-based or template-driven systems, LLM-based systems, and hybrid systems. Across the literature, conversational agents appear promising for scalable feedback and deeper probing of code understanding, but important limitations remain around hallucinations, over-reliance, privacy, integrity, and deployment constraints. Second, the paper synthesizes these findings into a Hybrid Socratic Framework for integrating conversational verification into Automated Programming Assessment Systems (APASs). The framework combines deterministic code analysis with a dual-agent conversational layer, knowledge tracking, scaffolded questioning, and guardrails that tie prompts to runtime facts. The paper also discusses practical safeguards against LLM-generated explanations, including proctored deployment modes, randomized trace questions, stepwise reasoning tied to concrete execution states, and local-model deployment options for privacy-sensitive settings. Rather than replacing conventional testing, the framework is intended as a complementary layer for verifying whether students understand the code they submit.

Related papers

• A Tri-Agent Framework for Evaluating and Aligning Question Clarification Capabilities of Large Language Models [2.332848751256598]
  Large Language Models (LLMs) are increasingly deployed in interactive systems where understanding user intent precisely is paramount.<n>This paper introduces a novel tri-agent framework for the robust evaluation of an LLM's ability to engage in clarifying dialogue.
  arXiv  Detail & Related papers  (2026-09-02T03:33:40Z)
• Prompt-Driven Code Summarization: A Systematic Literature Review [4.238464304669881]
  Large language models (LLMs) offer a promising solution by automatically generating natural language descriptions from source code.<n>This systematic review literature consolidates existing evidence, categorizes prompting paradigms, examines their effectiveness, and identifies gaps to guide future research and practical adoption.
  arXiv  Detail & Related papers  (2026-04-16T03:25:28Z)
• MuTSE: A Human-in-the-Loop Multi-use Text Simplification Evaluator [0.13048920509133807]
  textbfMuTSEfootnote is an interactive human-in-the-loop web application designed to streamline the evaluation of Large Language Models.<n>The system supports concurrent execution of $P times M$ prompt-model permutations, generating a comprehensive comparison matrix in real-time.
  arXiv  Detail & Related papers  (2026-04-10T04:31:43Z)
• Beyond Holistic Scores: Automatic Trait-Based Quality Scoring of Argumentative Essays [15.895792302323883]
  In educational contexts, teachers and learners require interpretable, trait-level feedback.<n>We study trait-based Automatic Argumentative Essay Scoring using two complementary modeling paradigms.<n>We show that explicitly modeling score ordinality substantially improves agreement with human raters.
  arXiv  Detail & Related papers  (2026-02-04T14:30:52Z)
• The Vibe-Check Protocol: Quantifying Cognitive Offloading in AI Programming [5.584060970507507]
  Vibe Coding'' is a paradigm where developers articulate high-level intent through natural language and delegate implementation to AI agents.<n>This paper proposes a theoretical framework to investigate the research question: textitIs Vibe Coding a better way to learn software engineering?
  arXiv  Detail & Related papers  (2026-01-02T06:13:41Z)
• Rethinking Testing for LLM Applications: Characteristics, Challenges, and a Lightweight Interaction Protocol [83.83217247686402]
  Large Language Models (LLMs) have evolved from simple text generators into complex software systems that integrate retrieval augmentation, tool invocation, and multi-turn interactions.<n>Their inherent non-determinism, dynamism, and context dependence pose fundamental challenges for quality assurance.<n>This paper decomposes LLM applications into a three-layer architecture: textbftextitSystem Shell Layer, textbftextitPrompt Orchestration Layer, and textbftextitLLM Inference Core.
  arXiv  Detail & Related papers  (2025-08-28T13:00:28Z)
• MERA Code: A Unified Framework for Evaluating Code Generation Across Tasks [56.34018316319873]
  We propose MERA Code, a benchmark for evaluating code for the latest code generation LLMs in Russian.<n>This benchmark includes 11 evaluation tasks that span 8 programming languages.<n>We evaluate open LLMs and frontier API models, analyzing their limitations in terms of practical coding tasks in non-English languages.
  arXiv  Detail & Related papers  (2025-07-16T14:31:33Z)
• Training Language Models to Generate Quality Code with Program Analysis Feedback [66.0854002147103]
  Code generation with large language models (LLMs) is increasingly adopted in production but fails to ensure code quality.<n>We propose REAL, a reinforcement learning framework that incentivizes LLMs to generate production-quality code.
  arXiv  Detail & Related papers  (2025-05-28T17:57:47Z)
• Evaluating LLM-based Agents for Multi-Turn Conversations: A Survey [64.08485471150486]
  This survey examines evaluation methods for large language model (LLM)-based agents in multi-turn conversational settings.<n>We systematically reviewed nearly 250 scholarly sources, capturing the state of the art from various venues of publication.
  arXiv  Detail & Related papers  (2025-03-28T14:08:40Z)
• ReDefining Code Comprehension: Function Naming as a Mechanism for Evaluating Code Comprehension [2.250363093539224]
  "Explain in Plain English" (EiPE) questions are widely used to assess code comprehension skills.<n>Recent approaches like Code Generation Based Grading (CGBG) leverage large language models to generate code.<n>We propose a modified approach where students generate function names, emphasizing the function's purpose over implementation details.
  arXiv  Detail & Related papers  (2025-03-15T17:22:14Z)
• Auto-PRE: An Automatic and Cost-Efficient Peer-Review Framework for Language Generation Evaluation [52.76508734756661]
  Auto-PRE is an automatic evaluation framework inspired by the peer review process.<n>Unlike previous approaches that rely on human annotations, Auto-PRE automatically selects evaluators based on three core traits.<n> Experiments on three representative tasks, including summarization, non-factoid QA, and dialogue generation, demonstrate that Auto-PRE achieves state-of-the-art performance.
  arXiv  Detail & Related papers  (2024-10-16T06:06:06Z)
• INDICT: Code Generation with Internal Dialogues of Critiques for Both Security and Helpfulness [110.6921470281479]
  We introduce INDICT: a new framework that empowers large language models with Internal Dialogues of Critiques for both safety and helpfulness guidance. The internal dialogue is a dual cooperative system between a safety-driven critic and a helpfulness-driven critic. We observed that our approach can provide an advanced level of critiques of both safety and helpfulness analysis, significantly improving the quality of output codes.
  arXiv  Detail & Related papers  (2024-06-23T15:55:07Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.