Evaluating Vision-Language and Large Language Models for Automated Student Assessment in Indonesian Classrooms

• URL: http://arxiv.org/abs/2506.04822v1
• Date: Thu, 05 Jun 2025 09:41:09 GMT
• Title: Evaluating Vision-Language and Large Language Models for Automated Student Assessment in Indonesian Classrooms
• Authors: Nurul Aisyah, Muhammad Dehan Al Kautsar, Arif Hidayat, Raqib Chowdhury, Fajri Koto,
• Abstract summary: This study evaluated the performance of a state-of-the-art VLM and several LLMs on 646 handwritten exam responses from grade 4 students in six Indonesian schools.<n>Our findings show that the VLM often struggles to accurately recognize student handwriting, leading to error propagation in downstream LLM grading.
• Score: 5.076445852481039
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Although vision-language and large language models (VLM and LLM) offer promising opportunities for AI-driven educational assessment, their effectiveness in real-world classroom settings, particularly in underrepresented educational contexts, remains underexplored. In this study, we evaluated the performance of a state-of-the-art VLM and several LLMs on 646 handwritten exam responses from grade 4 students in six Indonesian schools, covering two subjects: Mathematics and English. These sheets contain more than 14K student answers that span multiple choice, short answer, and essay questions. Assessment tasks include grading these responses and generating personalized feedback. Our findings show that the VLM often struggles to accurately recognize student handwriting, leading to error propagation in downstream LLM grading. Nevertheless, LLM-generated feedback retains some utility, even when derived from imperfect input, although limitations in personalization and contextual relevance persist.

Related papers

• LLMs vs. Traditional Sentiment Tools in Psychology: An Evaluation on Belgian-Dutch Narratives [0.0]
  Large Language Models (LLMs) promise enhanced context understanding.<n>We evaluated three Dutch-specific LLMs against LIWC and Pattern.<n>Surprisingly, despite architectural advancements, the Dutch-tuned LLMs underperformed compared to traditional methods.
  arXiv  Detail & Related papers  (2025-11-10T21:26:13Z)
• CEA-LIST at CheckThat! 2025: Evaluating LLMs as Detectors of Bias and Opinion in Text [3.9845507207125967]
  This paper presents a competitive approach to multilingual subjectivity detection using large language models (LLMs) with few-shot prompting.<n>We show that LLMs, when paired with carefully designed prompts, can match or outperform fine-tuned smaller language models (SLMs)<n>Our system achieved top rankings across multiple languages in the CheckThat! 2025 subjectivity detection task.
  arXiv  Detail & Related papers  (2025-07-10T08:35:05Z)
• An Empirical Study of Many-to-Many Summarization with Large Language Models [82.10000188179168]
  Large language models (LLMs) have shown strong multi-lingual abilities, giving them the potential to perform Many-to-many summarization (M2MS) in real applications.<n>This work presents a systematic empirical study on LLMs' M2MS ability.
  arXiv  Detail & Related papers  (2025-05-19T11:18:54Z)
• PolyMath: Evaluating Mathematical Reasoning in Multilingual Contexts [79.84059473102778]
  PolyMath is a multilingual mathematical reasoning benchmark covering 18 languages and 4 easy-to-hard difficulty levels.<n>Our benchmark ensures difficulty comprehensiveness, language diversity, and high-quality translation.
  arXiv  Detail & Related papers  (2025-04-25T15:39:04Z)
• On the Consistency of Multilingual Context Utilization in Retrieval-Augmented Generation [7.478369203246005]
  Retrieval-augmented generation (RAG) with large language models (LLMs) has demonstrated strong performance in multilingual question-answering tasks.<n>In multilingual RAG, retrieved passages can be written in languages other than that of the query entered by the user.
  arXiv  Detail & Related papers  (2025-04-01T09:55:23Z)
• Automated Assignment Grading with Large Language Models: Insights From a Bioinformatics Course [0.0]
  Natural language processing and large language models (LLMs) offer a promising solution by enabling the efficient delivery of personalized feedback.<n>Recent advances in natural language processing and large language models (LLMs) offer a promising solution by enabling the efficient delivery of personalized feedback.<n>Our results show that with well-designed prompts, LLMs can achieve grading accuracy and feedback quality comparable to human graders.
  arXiv  Detail & Related papers  (2025-01-24T13:59:14Z)
• Truth or Mirage? Towards End-to-End Factuality Evaluation with LLM-Oasis [78.07225438556203]
  We introduce LLM-Oasis, the largest resource for training end-to-end factuality evaluators.<n>It is constructed by extracting claims from Wikipedia, falsifying a subset of these claims, and generating pairs of factual and unfactual texts.<n>We then rely on human annotators to both validate the quality of our dataset and to create a gold standard test set for factuality evaluation systems.
  arXiv  Detail & Related papers  (2024-11-29T12:21:15Z)
• Embracing AI in Education: Understanding the Surge in Large Language Model Use by Secondary Students [53.20318273452059]
  Large language models (LLMs) like OpenAI's ChatGPT have opened up new avenues in education.<n>Despite school restrictions, our survey of over 300 middle and high school students revealed that a remarkable 70% of students have utilized LLMs.<n>We propose a few ideas to address such issues, including subject-specific models, personalized learning, and AI classrooms.
  arXiv  Detail & Related papers  (2024-11-27T19:19:34Z)
• Hate Personified: Investigating the role of LLMs in content moderation [64.26243779985393]
  For subjective tasks such as hate detection, where people perceive hate differently, the Large Language Model's (LLM) ability to represent diverse groups is unclear. By including additional context in prompts, we analyze LLM's sensitivity to geographical priming, persona attributes, and numerical information to assess how well the needs of various groups are reflected.
  arXiv  Detail & Related papers  (2024-10-03T16:43:17Z)
• LLM-as-a-Judge & Reward Model: What They Can and Cannot Do [2.2469442203227863]
  We conduct a comprehensive analysis of automated evaluators, reporting several key findings on their behavior. We discover that English evaluation capabilities significantly influence language-specific evaluation capabilities, enabling evaluators trained in English to easily transfer their skills to other languages. We find that state-of-the-art evaluators struggle with challenging prompts, in either English or Korean, underscoring their limitations in assessing or generating complex reasoning questions.
  arXiv  Detail & Related papers  (2024-09-17T14:40:02Z)
• Large Language Model as an Assignment Evaluator: Insights, Feedback, and Challenges in a 1000+ Student Course [49.296957552006226]
  Using large language models (LLMs) for automatic evaluation has become an important evaluation method in NLP research. This report shares how we use GPT-4 as an automatic assignment evaluator in a university course with 1,028 students.
  arXiv  Detail & Related papers  (2024-07-07T00:17:24Z)
• Crossing Linguistic Horizons: Finetuning and Comprehensive Evaluation of Vietnamese Large Language Models [11.563813473794013]
  Open-sourced large language models (LLMs) exhibit limited effectiveness in processing Vietnamese. To mitigate these issues, we have finetuned LLMs specifically for Vietnamese. Our evaluation results reveal that the fine-tuned LLMs exhibit enhanced comprehension and generative capabilities in Vietnamese.
  arXiv  Detail & Related papers  (2024-03-05T07:13:28Z)
• Automated Assessment of Students' Code Comprehension using LLMs [0.3293989832773954]
  Large Language Models (LLMs) and encoder-based Semantic Textual Similarity (STS) models are assessed. Our findings indicate that LLMs, when prompted in few-shot and chain-of-thought setting, perform comparable to fine-tuned encoder-based models in evaluating students' short answers in programming domain.
  arXiv  Detail & Related papers  (2023-12-19T20:39:12Z)
• Behind the Magic, MERLIM: Multi-modal Evaluation Benchmark for Large Image-Language Models [50.653838482083614]
  This paper introduces a scalable test-bed to assess the capabilities of IT-LVLMs on fundamental computer vision tasks.<n> MERLIM contains over 300K image-question pairs and has a strong focus on detecting cross-modal "hallucination" events in IT-LVLMs.
  arXiv  Detail & Related papers  (2023-12-03T16:39:36Z)
• TouchStone: Evaluating Vision-Language Models by Language Models [91.69776377214814]
  We propose an evaluation method that uses strong large language models as judges to comprehensively evaluate the various abilities of LVLMs. We construct a comprehensive visual dialogue dataset TouchStone, consisting of open-world images and questions, covering five major categories of abilities and 27 subtasks. We demonstrate that powerful LVLMs, such as GPT-4, can effectively score dialogue quality by leveraging their textual capabilities alone.
  arXiv  Detail & Related papers  (2023-08-31T17:52:04Z)
• Benchmarking Zero-Shot Recognition with Vision-Language Models: Challenges on Granularity and Specificity [45.86789047206224]
  This paper presents novel benchmarks for evaluating vision-language models (VLMs) in zero-shot recognition. Our benchmarks test VLMs' consistency in understanding concepts across semantic granularity levels and their response to varying text specificity. Findings show that VLMs favor moderately fine-grained concepts and struggle with specificity, often misjudging texts that differ from their training data.
  arXiv  Detail & Related papers  (2023-06-28T09:29:06Z)
• Sentiment Analysis in the Era of Large Language Models: A Reality Check [69.97942065617664]
  This paper investigates the capabilities of large language models (LLMs) in performing various sentiment analysis tasks. We evaluate performance across 13 tasks on 26 datasets and compare the results against small language models (SLMs) trained on domain-specific datasets.
  arXiv  Detail & Related papers  (2023-05-24T10:45:25Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.