Related papers: Prompt Coach: An Empirical Evaluation of an Agentic Tutor for Learning Prompt Engineering in Software Development

Prompt Coach: An Empirical Evaluation of an Agentic Tutor for Learning Prompt Engineering in Software Development

URL: http://arxiv.org/abs/2607.06074v1
Date: Tue, 07 Jul 2026 09:44:46 GMT
Title: Prompt Coach: An Empirical Evaluation of an Agentic Tutor for Learning Prompt Engineering in Software Development
Abstract summary: Prompt Coach (PC) is an agentic tutor that helps developers learn how to craft high-quality code-generation prompts.<n>PC evaluates prompt quality across multiple dimensions and surfaces targeted questions to guide self-taught.
Score: 2.1480208860520533
License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
Abstract: Prompt engineering has emerged as a critical yet undertaught skill for software developers, one that traditional learning approaches are ill-equipped to support given its evolving, interactive, and context-dependent nature. In this paper, we introduce Prompt Coach (PC), an agentic tutor that helps developers learn how to craft high-quality code-generation prompts through Socratic guidance embedded in-flow within their IDE. PC evaluates prompt quality across multiple dimensions and surfaces targeted questions to guide self-correction, grounded in the developer's codebase and the behavior of the target LLM. We present an early empirical study with 15 professional developers combining quantitative prompt quality scoring with qualitative perception measures. Participants showed statistically significant improvements after a single 60-minute session, with the largest gains across dimensions commonly overlooked by developers. They also reported strong trust, high adoption readiness, and unanimous agreement that PC improved their prompt-writing skills.

Related papers

AgentForge: An Immersive Role-Playing Platform for Learning Agentic Software Engineering [4.609790123179083]
We present textitAgentForge, an immersive learning system in which novices take on one of four software-engineering roles.<n>In each practice session, novices perform their chosen role while AI agents perform the remaining three.<n>In a study with 37 novice developers, participants achieved high task-completion rates with AI-agent support.
arXiv Detail & Related papers (2026-08-04T18:58:18Z)
DeepTutor: Towards Agentic Personalized Tutoring [17.49651550264547]
DeepTutor is an agent-native open-source framework for personalized tutoring.<n>We propose a hybrid personalization engine that couples static knowledge grounding with dynamic multi-resolution memory.<n>We also introduce TutorBot, a proactive multi-agent layer that deploys tutoring capabilities through skills and unified multi-channel access.
arXiv Detail & Related papers (2026-04-10T17:57:00Z)
Evaluating LLMs for Answering Student Questions in Introductory Programming Courses [0.0]
This study investigates the capabilities of Large Language Models to safely and effectively assist educators in answering student questions within a CS1 programming course.<n>Because traditional text-matching metrics are insufficient for evaluating open-ended educational responses, we developed and validated a custom LLM-as-a-Judge metric.<n>Our findings demonstrate that models, such as Gemini 3 flash, can surpass the quality baseline of typical educator responses, achieving high alignment with expert pedagogical standards.
arXiv Detail & Related papers (2026-03-30T11:22:58Z)
SimpleDevQA: Benchmarking Large Language Models on Development Knowledge QA [58.75982433502236]
The Dev Knowledge QA task accounts for 39.6% of interactions.<n>Only 27.5% of real Dev Knowledge QA dialogues focus on code understanding.<n>Only 17.1% of real-world Dev Knowledge QA dialogues can be used for constructing a benchmark.
arXiv Detail & Related papers (2025-12-09T17:58:36Z)
EduAgentQG: A Multi-Agent Workflow Framework for Personalized Question Generation [56.43882334582494]
We propose EduAgentQG, a multi-agent collaborative framework for generating high-quality and diverse personalized questions.<n>The framework consists of five specialized agents and operates through an iterative feedback loop.<n>EduAgentQG outperforms existing single-agent and multi-agent methods in terms of question diversity, goal consistency, and overall quality.
arXiv Detail & Related papers (2025-11-08T12:25:31Z)
CATArena: Evaluation of LLM Agents through Iterative Tournament Competitions [49.02422075498554]
Large Language Model (LLM) agents have evolved from basic text generation to autonomously completing complex tasks through interaction with external tools.<n>In this work, we emphasize the importance of learning ability, including both self-improvement and peer-learning, as a core driver for agent evolution toward human-level intelligence.<n>We propose an iterative, competitive peer-learning framework, which allows agents to refine and optimize their strategies through repeated interactions and feedback.
arXiv Detail & Related papers (2025-10-30T15:22:53Z)
The AI Imperative: Scaling High-Quality Peer Review in Machine Learning [49.87236114682497]
We argue that AI-assisted peer review must become an urgent research and infrastructure priority.<n>We propose specific roles for AI in enhancing factual verification, guiding reviewer performance, assisting authors in quality improvement, and supporting ACs in decision-making.
arXiv Detail & Related papers (2025-06-09T18:37:14Z)
Design of AI-Powered Tool for Self-Regulation Support in Programming Education [4.171227316909729]
Large Language Model (LLM) tools have demonstrated their potential to deliver high-quality assistance.<n>However, many of these tools operate independently from institutional Learning Management Systems.<n>This isolation limits the ability to leverage learning materials and exercise context for generating tailored, context-aware feedback.
arXiv Detail & Related papers (2025-04-03T22:47:33Z)
MathTutorBench: A Benchmark for Measuring Open-ended Pedagogical Capabilities of LLM Tutors [82.91830877219822]
We present MathTutorBench, an open-source benchmark for holistic tutoring model evaluation.<n>MathTutorBench contains datasets and metrics that broadly cover tutor abilities as defined by learning sciences research in dialog-based teaching.<n>We evaluate a wide set of closed- and open-weight models and find that subject expertise, indicated by solving ability, does not immediately translate to good teaching.
arXiv Detail & Related papers (2025-02-26T08:43:47Z)
From Critique to Clarity: A Pathway to Faithful and Personalized Code Explanations with Large Language Models [3.5176327869934747]
This paper presents an innovative approach to generate faithful and personalized code explanations.<n>Our methodology integrates prompt enhancement, self-consuming mechanisms, personalized content customization, and interaction with external tools.<n>Our findings suggest that this approach significantly improves the quality and relevance of code explanations.
arXiv Detail & Related papers (2024-12-08T09:02:04Z)
Modelling Assessment Rubrics through Bayesian Networks: a Pragmatic Approach [40.06500618820166]
This paper presents an approach to deriving a learner model directly from an assessment rubric. We illustrate how the approach can be applied to automatize the human assessment of an activity developed for testing computational thinking skills.
arXiv Detail & Related papers (2022-09-07T10:09:12Z)
ProtoTransformer: A Meta-Learning Approach to Providing Student Feedback [54.142719510638614]
In this paper, we frame the problem of providing feedback as few-shot classification. A meta-learner adapts to give feedback to student code on a new programming question from just a few examples by instructors. Our approach was successfully deployed to deliver feedback to 16,000 student exam-solutions in a programming course offered by a tier 1 university.
arXiv Detail & Related papers (2021-07-23T22:41:28Z)

This list is automatically generated from the titles and abstracts of the papers in this site.