Accelerating Accurate Assignment Authoring Using Solution-Generated Autograders

• URL: http://arxiv.org/abs/2608.06572v1
• Date: Thu, 06 Aug 2026 20:29:21 GMT
• Title: Accelerating Accurate Assignment Authoring Using Solution-Generated Autograders
• Abstract summary: We present solution-generated autograding: a faster, more accurate, and more enjoyable way to create autograders.<n>We describe Questioner, an implementation of solution-generated autograding for Java and Kotlin.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Students learning to program benefit from access to large numbers of practice problems. Autograders are commonly used to support programming questions by providing quick feedback on submissions. But authoring accurate autograders remains challenging. Autograders are frequently created by enumerating test cases--a tedious process that can produce inaccurate autograders that fail to correctly classify submissions. When authoring accurate autograders is slow, it is difficult to create large banks of practice problems to support beginning programmers. We present solution-generated autograding: a faster, more accurate, and more enjoyable way to create autograders. Our approach leverages a key difference between software testing and autograding: The question author can provide a solution. By starting with a solution, we can eliminate the need to manually enumerate test cases, validate the autograder's accuracy, and evaluate other aspects of submission code quality beyond behavioral correctness. We describe Questioner, an implementation of solution-generated autograding for Java and Kotlin, and share experiences from four years using Questioner to support a large CS1 course: authoring nearly 800 programming questions used by thousands of students to evaluate millions of submissions.

Related papers

• AutoCode: LLMs as Problem Setters for Competitive Programming [94.71566758494787]
  We introduce AutoCode, which uses multiple rounds of validation to yield competition-grade problem statements and test cases.<n>On held-out problems, AutoCode test suites approach 99% consistency with official judgments.
  arXiv  Detail & Related papers  (2025-09-29T17:59:03Z)
• Self-Questioning Language Models [58.73276539661649]
  We propose an asymmetric self-play framework where a proposer is given the topic and generates a question for a solver.<n>Both the proposer and solver are trained via reinforcement learning.<n>We study this asymmetric self-play framework on three benchmarks: three-digit multiplication, algebra problems from the OMEGA benchmark, and programming problems from Codeforces.
  arXiv  Detail & Related papers  (2025-08-05T17:51:33Z)
• Automating Autograding: Large Language Models as Test Suite Generators for Introductory Programming [1.9121661610146587]
  We evaluate the effectiveness of using Large Language Models to automatically generate test suites for programming problems. We compare the performance of the LLM-generated test suites against the instructor-created test suites for each problem.
  arXiv  Detail & Related papers  (2024-11-14T07:58:44Z)
• Code quality assessment using transformers [0.0]
  In this work we investigate the use of CodeBERT to automatically assign quality score to Java code. We explore the accuracy of the models on a novel dataset for code quality assessment. We find that code quality to some extent is predictable and that transformer based models using task adapted pre-training can solve the task more efficiently than other techniques.
  arXiv  Detail & Related papers  (2023-09-17T12:59:59Z)
• Giving Feedback on Interactive Student Programs with Meta-Exploration [74.5597783609281]
  Developing interactive software, such as websites or games, is a particularly engaging way to learn computer science. Standard approaches require instructors to manually grade student-implemented interactive programs. Online platforms that serve millions, like Code.org, are unable to provide any feedback on assignments for implementing interactive programs.
  arXiv  Detail & Related papers  (2022-11-16T10:00:23Z)
• Write a Line: Tests with Answer Templates and String Completion Hints for Self-Learning in a CS1 Course [0.0]
  This paper reports the results of using regular-expression-based questions with string completion hints in a CS1 course for 4 years with 497 students. The evaluation results show that Perl-compatible regular expressions provide good precision and recall (more than 99%) when used for questions requiring writing a single line of code.
  arXiv  Detail & Related papers  (2022-04-19T17:53:35Z)
• Continuous Examination by Automatic Quiz Assessment Using Spiral Codes and Image Processing [69.35569554213679]
  Paper quizzes are affordable and within reach of campus education in classrooms. correction of the quiz is a considerable obstacle. We suggest mitigating the issue by a novel image processing technique.
  arXiv  Detail & Related papers  (2022-01-26T22:58:15Z)
• An Analysis of Programming Course Evaluations Before and After the Introduction of an Autograder [1.329950749508442]
  This paper studies the answers to the standardized university evaluation questionnaires of foundational computer science courses which recently introduced autograding. We hypothesize how the autograder might have contributed to the significant changes in the data, such as, improved interactions between tutors and students, improved overall course quality, improved learning success, increased time spent, and reduced difficulty. The autograder technology can be validated as a teaching method to improve student satisfaction with programming courses.
  arXiv  Detail & Related papers  (2021-10-28T14:09:44Z)
• ProtoTransformer: A Meta-Learning Approach to Providing Student Feedback [54.142719510638614]
  In this paper, we frame the problem of providing feedback as few-shot classification. A meta-learner adapts to give feedback to student code on a new programming question from just a few examples by instructors. Our approach was successfully deployed to deliver feedback to 16,000 student exam-solutions in a programming course offered by a tier 1 university.
  arXiv  Detail & Related papers  (2021-07-23T22:41:28Z)
• Retrieve, Program, Repeat: Complex Knowledge Base Question Answering via Alternate Meta-learning [56.771557756836906]
  We present a novel method that automatically learns a retrieval model alternately with the programmer from weak supervision. Our system leads to state-of-the-art performance on a large-scale task for complex question answering over knowledge bases.
  arXiv  Detail & Related papers  (2020-10-29T18:28:16Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.