Is Solving Better Than Evaluating GenAI Solutions?

• URL: http://arxiv.org/abs/2607.27586v1
• Date: Thu, 30 Jul 2026 02:08:29 GMT
• Title: Is Solving Better Than Evaluating GenAI Solutions?
• Abstract summary: This study compares evaluating GenAI-generated solutions with traditional problem solving in a junior-level algorithms course.<n>We found no statistically significant differences between groups in midterm scores, final exam scores, overall course grades, or exam problems structurally aligned with the homework interventions.<n>Students received significantly higher homework scores when evaluating GenAI-generated solutions, but this localized advantage did not translate into downstream gains.
• Score: 45.714140455691485
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: As Generative AI (GenAI) tools become increasingly capable of generating solutions to computing assignments, the computing education community is exploring pedagogical approaches that emphasize solution evaluation, verification, and critique alongside traditional solution generation. However, evidence regarding the impact of such evaluation-centered tasks on student learning remains limited, particularly in upper-division, theory-heavy courses. We conducted a randomized A/B crossover study (N=220) in a junior-level algorithms course to compare evaluating GenAI-generated solutions with traditional problem solving. Across six assignments, student working groups either solved challenging algorithmic problems directly or evaluated often-flawed GenAI-generated solutions, with roles reversed midway through the semester. We found no statistically significant differences between groups in midterm scores, final exam scores, overall course grades, or exam problems structurally aligned with the homework interventions. Students received significantly higher homework scores when evaluating GenAI-generated solutions, but this localized advantage did not translate into downstream summative gains. Survey data further indicated that most students reported no change in study habits in response to the intervention; however, those who reported adapting their study strategies rated the GenAI-evaluation assignments as significantly more helpful. These findings suggest that GenAI evaluation redistributes student effort from open-ended solution construction toward verification, diagnosis, and judgment, but does not automatically produce stronger conceptual transfer. We conclude that GenAI-evaluation activities can be incorporated into algorithms coursework without broad performance losses, but meaningful learning gains may require deliberate scaffolding that pushes students beyond simple error diagnosis.

Related papers

• Generative AI Availability, Grades, and Student Satisfaction at a Large University [6.872059640265074]
  The spread of generative AI (GenAI) in higher education has raised concerns that students offload cognitive effort to AI, earning high grades without learning.<n>If this "GenAI substitution hypothesis" is true, grades should rise disproportionately in GenAI-susceptible courses.<n>We find no significant differential effect of GenAI availability on grades overall or among previously lower-performing students.
  arXiv  Detail & Related papers  (2026-07-23T17:19:55Z)
• Bridging Qualitative Rubrics and AI: A Binary Question Framework for Criterion-Referenced Grading in Engineering [5.378302855328016]
  This study investigates how GenAI can be integrated with a criterion-referenced grading framework to improve the efficiency and quality of grading for mathematical assessments in engineering.
  arXiv  Detail & Related papers  (2026-01-22T03:57:47Z)
• Examining the Usage of Generative AI Models in Student Learning Activities for Software Programming [3.2383459424229835]
  This study investigates how GenAI assistance compares to conventional online resources in supporting knowledge gains across different proficiency levels.<n>We conducted a controlled user experiment with 24 undergraduate students of two different levels of programming experience.<n>Our findings reveal that generating complete solutions with GenAI significantly improves task performance, especially for beginners, but does not consistently result in knowledge gains.
  arXiv  Detail & Related papers  (2025-11-17T11:42:24Z)
• Barbarians at the Gate: How AI is Upending Systems Research [58.95406995634148]
  We argue that systems research, long focused on designing and evaluating new performance-oriented algorithms, is particularly well-suited for AI-driven solution discovery.<n>We term this approach as AI-Driven Research for Systems ( ADRS), which iteratively generates, evaluates, and refines solutions.<n>Our results highlight both the disruptive potential and the urgent need to adapt systems research practices in the age of AI.
  arXiv  Detail & Related papers  (2025-10-07T17:49:24Z)
• Evaluating undergraduate mathematics examinations in the era of generative AI: a curriculum-level case study [0.0]
  We generate, transcribe, and blind-mark GenAI submissions to eight undergraduate mathematics examinations at a Russell Group university.<n>We find that GenAI attainment is at the level of a first-class degree, though current performance can vary between modules.
  arXiv  Detail & Related papers  (2025-09-15T10:34:31Z)
• Evaluating the AI-Lab Intervention: Impact on Student Perception and Use of Generative AI in Early Undergraduate Computer Science Courses [0.0]
  Generative AI (GenAI) is rapidly entering computer science education.<n>Concerns about overreliance coexist with a gap in research on structured scaffolding to guide tool use in formal courses.<n>This study examines the impact of a dedicated "AI-Lab" intervention on undergraduate students.
  arXiv  Detail & Related papers  (2025-04-30T18:12:42Z)
• Are we making progress in unlearning? Findings from the first NeurIPS unlearning competition [70.60872754129832]
  First NeurIPS competition on unlearning sought to stimulate the development of novel algorithms. Nearly 1,200 teams from across the world participated. We analyze top solutions and delve into discussions on benchmarking unlearning.
  arXiv  Detail & Related papers  (2024-06-13T12:58:00Z)
• A Reinforcement Learning-assisted Genetic Programming Algorithm for Team Formation Problem Considering Person-Job Matching [70.28786574064694]
  A reinforcement learning-assisted genetic programming algorithm (RL-GP) is proposed to enhance the quality of solutions. The hyper-heuristic rules obtained through efficient learning can be utilized as decision-making aids when forming project teams.
  arXiv  Detail & Related papers  (2023-04-08T14:32:12Z)
• Distantly-Supervised Named Entity Recognition with Adaptive Teacher Learning and Fine-grained Student Ensemble [56.705249154629264]
  Self-training teacher-student frameworks are proposed to improve the robustness of NER models. In this paper, we propose an adaptive teacher learning comprised of two teacher-student networks. Fine-grained student ensemble updates each fragment of the teacher model with a temporal moving average of the corresponding fragment of the student, which enhances consistent predictions on each model fragment against noise.
  arXiv  Detail & Related papers  (2022-12-13T12:14:09Z)
• Towards Explainable Metaheuristic: Mining Surrogate Fitness Models for Importance of Variables [69.02115180674885]
  We use four benchmark problems to train a surrogate model and investigate the learning of the search space by the surrogate model. We show that the surrogate model picks out key characteristics of the problem as it is trained on population data from each generation.
  arXiv  Detail & Related papers  (2022-05-31T09:16:18Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.