AGACCI : Affiliated Grading Agents for Criteria-Centric Interface in Educational Coding Contexts

• URL: http://arxiv.org/abs/2507.05321v1
• Date: Mon, 07 Jul 2025 15:50:46 GMT
• Title: AGACCI : Affiliated Grading Agents for Criteria-Centric Interface in Educational Coding Contexts
• Authors: Kwangsuk Park, Jiwoong Yang,
• Abstract summary: We introduce AGACCI, a multi-agent system that distributes specialized evaluation roles across collaborative agents.<n>AGACCI outperforms a single GPT-based baseline in terms of rubric and feedback accuracy, relevance, consistency, and coherence.
• Score: 0.6050976240234864
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Recent advances in AI-assisted education have encouraged the integration of vision-language models (VLMs) into academic assessment, particularly for tasks that require both quantitative and qualitative evaluation. However, existing VLM based approaches struggle with complex educational artifacts, such as programming tasks with executable components and measurable outputs, that require structured reasoning and alignment with clearly defined evaluation criteria. We introduce AGACCI, a multi-agent system that distributes specialized evaluation roles across collaborative agents to improve accuracy, interpretability, and consistency in code-oriented assessment. To evaluate the framework, we collected 360 graduate-level code-based assignments from 60 participants, each annotated by domain experts with binary rubric scores and qualitative feedback. Experimental results demonstrate that AGACCI outperforms a single GPT-based baseline in terms of rubric and feedback accuracy, relevance, consistency, and coherence, while preserving the instructional intent and evaluative depth of expert assessments. Although performance varies across task types, AGACCI highlights the potential of multi-agent systems for scalable and context-aware educational evaluation.

Related papers

• Rethinking Machine Unlearning in Image Generation Models [59.697750585491264]
  CatIGMU is a novel hierarchical task categorization framework.<n>EvalIGMU is a comprehensive evaluation framework.<n>We construct DataIGM, a high-quality unlearning dataset.
  arXiv  Detail & Related papers  (2025-06-03T11:25:14Z)
• From Coders to Critics: Empowering Students through Peer Assessment in the Age of AI Copilots [3.3094795918443634]
  This paper presents an empirical study of a rubric based, anonymized peer review process implemented in a large programming course.<n>Students evaluated each other's final projects (2D game) and their assessments were compared to instructor grades using correlation, mean absolute error, and root mean square error (RMSE)<n>Results show that peer review can approximate instructor evaluation with moderate accuracy and foster student engagement, evaluative thinking, and interest in providing good feedback to their peers.
  arXiv  Detail & Related papers  (2025-05-28T08:17:05Z)
• Can LLMs Be Trusted for Evaluating RAG Systems? A Survey of Methods and Datasets [0.0]
  Retrieval-Augmented Generation (RAG) has advanced significantly in recent years.<n>RAG complexity poses substantial challenges for systematic evaluation and quality enhancement.<n>This study systematically reviews 63 academic articles to provide a comprehensive overview of state-of-the-art RAG evaluation methodologies.
  arXiv  Detail & Related papers  (2025-04-28T08:22:19Z)
• Learning to Align Multi-Faceted Evaluation: A Unified and Robust Framework [61.38174427966444]
  Large Language Models (LLMs) are being used more and more extensively for automated evaluation in various scenarios.<n>Previous studies have attempted to fine-tune open-source LLMs to replicate the evaluation explanations and judgments of powerful proprietary models.<n>We propose a novel evaluation framework, ARJudge, that adaptively formulates evaluation criteria and synthesizes both text-based and code-driven analyses.
  arXiv  Detail & Related papers  (2025-02-26T06:31:45Z)
• HREF: Human Response-Guided Evaluation of Instruction Following in Language Models [61.273153125847166]
  We develop a new evaluation benchmark, Human Response-Guided Evaluation of Instruction Following (HREF)<n>In addition to providing reliable evaluation, HREF emphasizes individual task performance and is free from contamination.<n>We study the impact of key design choices in HREF, including the size of the evaluation set, the judge model, the baseline model, and the prompt template.
  arXiv  Detail & Related papers  (2024-12-20T03:26:47Z)
• Towards Flexible Evaluation for Generative Visual Question Answering [17.271448204525612]
  This paper proposes the use of semantics-based evaluators for assessing unconstrained open-ended responses on Visual Question Answering (VQA) datasets. In addition, this paper proposes a Semantically Flexible VQA Evaluator (SFVE) with meticulous design based on the unique features of VQA evaluation.
  arXiv  Detail & Related papers  (2024-08-01T05:56:34Z)
• HD-Eval: Aligning Large Language Model Evaluators Through Hierarchical Criteria Decomposition [92.17397504834825]
  HD-Eval is a framework that iteratively aligns large language models evaluators with human preference. HD-Eval inherits the essence from the evaluation mindset of human experts and enhances the alignment of LLM-based evaluators. Extensive experiments on three evaluation domains demonstrate the superiority of HD-Eval in further aligning state-of-the-art evaluators.
  arXiv  Detail & Related papers  (2024-02-24T08:01:32Z)
• AgentBoard: An Analytical Evaluation Board of Multi-turn LLM Agents [74.16170899755281]
  We introduce AgentBoard, a pioneering comprehensive benchmark and accompanied open-source evaluation framework tailored to analytical evaluation of LLM agents.<n>AgentBoard offers a fine-grained progress rate metric that captures incremental advancements as well as a comprehensive evaluation toolkit.<n>This not only sheds light on the capabilities and limitations of LLM agents but also propels the interpretability of their performance to the forefront.
  arXiv  Detail & Related papers  (2024-01-24T01:51:00Z)
• Evaluating General-Purpose AI with Psychometrics [43.85432514910491]
  We discuss the need for a comprehensive and accurate evaluation of general-purpose AI systems such as large language models. Current evaluation methodology, mostly based on benchmarks of specific tasks, falls short of adequately assessing these versatile AI systems. To tackle these challenges, we suggest transitioning from task-oriented evaluation to construct-oriented evaluation.
  arXiv  Detail & Related papers  (2023-10-25T05:38:38Z)
• Hierarchical Evaluation Framework: Best Practices for Human Evaluation [17.91641890651225]
  The absence of widely accepted human evaluation metrics in NLP hampers fair comparisons among different systems and the establishment of universal assessment standards. We develop our own hierarchical evaluation framework to provide a more comprehensive representation of the NLP system's performance. In future work, we will investigate the potential time-saving benefits of our proposed framework for evaluators assessing NLP systems.
  arXiv  Detail & Related papers  (2023-10-03T09:46:02Z)
• Modelling Assessment Rubrics through Bayesian Networks: a Pragmatic Approach [40.06500618820166]
  This paper presents an approach to deriving a learner model directly from an assessment rubric. We illustrate how the approach can be applied to automatize the human assessment of an activity developed for testing computational thinking skills.
  arXiv  Detail & Related papers  (2022-09-07T10:09:12Z)
• Towards a multi-stakeholder value-based assessment framework for algorithmic systems [76.79703106646967]
  We develop a value-based assessment framework that visualizes closeness and tensions between values. We give guidelines on how to operationalize them, while opening up the evaluation and deliberation process to a wide range of stakeholders.
  arXiv  Detail & Related papers  (2022-05-09T19:28:32Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.