Human-in-the-Loop Benchmarking of Heterogeneous LLMs for Automated Competency Assessment in Secondary Level Mathematics

• URL: http://arxiv.org/abs/2604.26607v1
• Date: Wed, 29 Apr 2026 12:36:19 GMT
• Title: Human-in-the-Loop Benchmarking of Heterogeneous LLMs for Automated Competency Assessment in Secondary Level Mathematics
• Abstract summary: This paper suggests a "Human-in-the-Loop" benchmarking framework to assess the effectiveness of multiple LLMs in secondary-level mathematics assessment.<n>Based on the Grade 10 Optional Mathematics curriculum in Nepal, we created a multi-dimensional rubric for four topics and four cross-cutting competencies.<n>Although the Gemini-based Mixture-of-Experts (Sparse MoE) models achieved "Fair Agreement" (kappa_w 0.38), the larger Orion (70B) model exhibited "No Agreement" (kappa_w = -0.0261)
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: As Competency-Based Education (CBE) is gaining traction around the world, the shift from marks-based assessment to qualitative competency mapping is a manual challenge for educators. This paper tackles the bottleneck issue by suggesting a "Human-in-the-Loop" benchmarking framework to assess the effectiveness of multiple LLMs in automating secondary-level mathematics assessment. Based on the Grade 10 Optional Mathematics curriculum in Nepal, we created a multi-dimensional rubric for four topics and four cross-cutting competencies: Comprehension, Knowledge, Operational Fluency, and Behavior and Correlation. The multi-provider ensemble, consisted of open-weight models -- Eagle (Llama 3.1-8B) and Orion (Llama 3.3-70B) -- and proprietary frontier models Nova (Gemini 2.5 Flash) and Lyra (Gemini 3 Pro), was benchmarked against a ground truth defined by two senior mathematics faculty members (kappa_w = 0.8652). The findings show a marked "Architecture-compatibility gap". Although the Gemini-based Mixture-of-Experts (Sparse MoE) models achieved "Fair Agreement" (kappa_w ~ 0.38), the larger Orion (70B) model exhibited "No Agreement" (kappa_w = -0.0261), suggesting that architectural compliance with instruction constraints outweighs the scale of raw parameters in rubric-constrained tasks. We conclude that while LLMs are not yet suitable for autonomous certification, they provide high-value assistive support for preliminary evidence extraction within a "Human-in-the-Loop" framework.

Related papers

• GTBench: A Curriculum-Grounded Benchmark for Evaluating LLMs as Mathematical Research Assistants in Graph Theory [0.0134610399747818]
  GTBench is a curriculum-grounded benchmark for evaluating large language models (LLMs) as mathematical research assistants in graph theory.<n>We evaluate GPT-5, Claude Sonnet 4.6, Gemini 2.5 Flash-Lite, Llama 3.3 70B, and Mistral Large 3 under zero-shot and chain-of-thought prompting.
  arXiv  Detail & Related papers  (2026-06-02T04:40:25Z)
• Creating and Evaluating K-12 GenAI Assessment Graders Through Context Engineering [6.131107680009006]
  The integration of large language models (LLMs) into educational assessment represents a transformative shift in classroom grading practices.<n>This paper examines the theoretical foundations and evaluates an LLM grader that uses commercially available foundation models with context and prompt engineering to score student work against a rubric.
  arXiv  Detail & Related papers  (2026-05-08T16:32:43Z)
• Beyond Grading Accuracy: Exploring Alignment of TAs and LLMs [1.529342790344802]
  We investigate the potential of open-source Large Language Models (LLMs) for grading Unified Language (UML) class diagrams.<n>Our results show per-criterion accuracy of up to 88.56% and a Pearson correlation coefficient of up to 0.78, representing a substantial improvement over previous work.
  arXiv  Detail & Related papers  (2026-03-17T10:40:35Z)
• RankLLM: Weighted Ranking of LLMs by Quantifying Question Difficulty [102.02839046225468]
  RankLLM is a novel framework designed to quantify both question difficulty and model competency.<n>We evaluate 30 models on 35,550 questions across multiple domains.
  arXiv  Detail & Related papers  (2026-02-12T21:28:46Z)
• RPC-Bench: A Fine-grained Benchmark for Research Paper Comprehension [65.81339691942757]
  RPC-Bench is a large-scale question-answering benchmark built from review-rebuttal exchanges of high-quality computer science papers.<n>We design a fine-grained taxonomy aligned with the scientific research flow to assess models' ability to understand and answer why, what, and how questions in scholarly contexts.
  arXiv  Detail & Related papers  (2026-01-14T11:37:00Z)
• TutorBench: A Benchmark To Assess Tutoring Capabilities Of Large Language Models [10.963195858672627]
  TutorBench is a dataset and evaluation benchmark designed to rigorously evaluate the core tutoring skills of large language models (LLMs)<n>Samples are drawn from three common tutoring tasks: (i) generating adaptive explanations tailored to a student's confusion, (ii) providing actionable feedback on a student's work, and (iii) promoting active learning through effective hint generation.<n>We evaluate 16 frontier LLMs on TutorBench and present a detailed analysis of their performance and behavior.
  arXiv  Detail & Related papers  (2025-10-03T01:41:09Z)
• ASyMOB: Algebraic Symbolic Mathematical Operations Benchmark [0.0]
  Large language models (LLMs) are rapidly approaching the level of proficiency in university-level symbolic mathematics.<n>We introduce ASyMOB, a novel assessment framework focused exclusively on symbolic manipulation.
  arXiv  Detail & Related papers  (2025-05-28T23:11:14Z)
• GeoBenchX: Benchmarking LLMs in Agent Solving Multistep Geospatial Tasks [0.11458853556386796]
  This paper establishes a benchmark for evaluating tool-calling capabilities of large language models (LLMs)<n>We assess eight commercial LLMs (Claude Sonnet 3.5 and 4, Claude Haiku 3.5, Gemini 2.0 Flash, Gemini 2.5 Pro Preview, GPT-4o, GPT-4.1 and o4-mini) using a simple tool-calling agent equipped with 23 geospatial functions.<n>Results show o4-mini and Claude 3.5 Sonnet achieve the best overall performance, OpenAI's GPT-4.1, GPT-4o and Google's Gemini 2.5 Pro Preview do not fall far behind, but the last two are more efficient in
  arXiv  Detail & Related papers  (2025-03-23T16:20:14Z)
• Leveraging Online Olympiad-Level Math Problems for LLMs Training and Contamination-Resistant Evaluation [55.21013307734612]
  AoPS-Instruct is a dataset of more than 600,000 high-quality QA pairs.<n>LiveAoPSBench is an evolving evaluation set with timestamps, derived from the latest forum data.<n>Our work presents a scalable approach to creating and maintaining large-scale, high-quality datasets for advanced math reasoning.
  arXiv  Detail & Related papers  (2025-01-24T06:39:38Z)
• PATCH! {P}sychometrics-{A}ssis{T}ed Ben{CH}marking of Large Language Models against Human Populations: A Case Study of Proficiency in 8th Grade Mathematics [3.9362370389588834]
  Many existing benchmarks of large (multimodal) language models (LLMs) focus on measuring LLMs' academic proficiency, often with also an interest in comparing model performance with human test takers'<n>LLMs suffer from several limitations, including questionable measurement quality, lack of quality assessment and unclear human population reference.<n>We show that adopting a psychometrics-based approach yields evaluation outcomes that diverge from those based on current benchmarking practices.
  arXiv  Detail & Related papers  (2024-04-02T09:58:57Z)
• KoLA: Carefully Benchmarking World Knowledge of Large Language Models [87.96683299084788]
  We construct a Knowledge-oriented LLM Assessment benchmark (KoLA) We mimic human cognition to form a four-level taxonomy of knowledge-related abilities, covering $19$ tasks. We use both Wikipedia, a corpus prevalently pre-trained by LLMs, along with continuously collected emerging corpora, to evaluate the capacity to handle unseen data and evolving knowledge.
  arXiv  Detail & Related papers  (2023-06-15T17:20:46Z)
• Evaluating Language Models for Mathematics through Interactions [116.67206980096513]
  We introduce CheckMate, a prototype platform for humans to interact with and evaluate large language models (LLMs) We conduct a study with CheckMate to evaluate three language models (InstructGPT, ChatGPT, and GPT-4) as assistants in proving undergraduate-level mathematics. We derive a taxonomy of human behaviours and uncover that despite a generally positive correlation, there are notable instances of divergence between correctness and perceived helpfulness.
  arXiv  Detail & Related papers  (2023-06-02T17:12:25Z)
• Modality-Aware Triplet Hard Mining for Zero-shot Sketch-Based Image Retrieval [51.42470171051007]
  This paper tackles the Zero-Shot Sketch-Based Image Retrieval (ZS-SBIR) problem from the viewpoint of cross-modality metric learning. By combining two fundamental learning approaches in DML, e.g., classification training and pairwise training, we set up a strong baseline for ZS-SBIR. We show that Modality-Aware Triplet Hard Mining (MATHM) enhances the baseline with three types of pairwise learning.
  arXiv  Detail & Related papers  (2021-12-15T08:36:44Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.