Pensieve Grader: An AI-Powered, Ready-to-Use Platform for Effortless Handwritten STEM Grading

• URL: http://arxiv.org/abs/2507.01431v2
• Date: Mon, 07 Jul 2025 05:10:47 GMT
• Title: Pensieve Grader: An AI-Powered, Ready-to-Use Platform for Effortless Handwritten STEM Grading
• Abstract summary: We introduce Pensieve, an AI-assisted grading platform for STEM courses.<n>Pensieve supports the entire grading pipeline-from scanned student submissions to final feedback-within a human-in-the-loop interface.<n>Our findings show that Pensieve reduces grading time by an average of 65%, while maintaining a 95.4% agreement rate with instructor-assigned grades for high-confidence predictions.
• Score: 1.1666234644810896
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Grading handwritten, open-ended responses remains a major bottleneck in large university STEM courses. We introduce Pensieve (https://www.pensieve.co), an AI-assisted grading platform that leverages large language models (LLMs) to transcribe and evaluate student work, providing instructors with rubric-aligned scores, transcriptions, and confidence ratings. Unlike prior tools that focus narrowly on specific tasks like transcription or rubric generation, Pensieve supports the entire grading pipeline-from scanned student submissions to final feedback-within a human-in-the-loop interface. Pensieve has been deployed in real-world courses at over 20 institutions and has graded more than 300,000 student responses. We present system details and empirical results across four core STEM disciplines: Computer Science, Mathematics, Physics, and Chemistry. Our findings show that Pensieve reduces grading time by an average of 65%, while maintaining a 95.4% agreement rate with instructor-assigned grades for high-confidence predictions.

Related papers

• Deep Academic Survey: Stateful Agentic Closed-Loop Paradigm for Academic Survey Automation [60.151050016855834]
  DAS is a stateful agentic framework for generating publication-oriented academic surveys.<n>Its key idea is to separate reusable paper analysis from topic-specific manuscript construction.<n>DAS achieves the highest average in all four dimensions, with an overall score of 4.34 compared with 4.03 for the strongest competitor.
  arXiv  Detail & Related papers  (2026-08-18T17:29:00Z)
• Evaluating AI Grading on Real-World Handwritten College Mathematics: A Large-Scale Study Toward a Benchmark [9.922581736690159]
  We present a large-scale empirical study of AI grading on real, handwritten calculus work from UC Irvine.<n>Using OCR-conditioned large language models with structured, rubric-guided prompting, our system produces scores and formative feedback for thousands of free-response quiz submissions.<n>In a setting with no single ground-truth label, we evaluate performance against official teaching-assistant grades, student surveys, and independent human review.
  arXiv  Detail & Related papers  (2026-03-01T03:32:51Z)
• Grading Handwritten Engineering Exams with Multimodal Large Language Models [0.0]
  Handwritten STEM exams capture open-ended reasoning and diagrams, but manual grading is slow and difficult to scale.<n>We present an end-to-end workflow for grading scanned handwritten engineering quizzes with multimodal large language models (LLMs)<n>The lecturer provides only a handwritten reference solution (100%) and a short set of grading rules; the reference is converted into a text-only summary that conditions grading without exposing the reference scan.
  arXiv  Detail & Related papers  (2026-01-02T16:10:08Z)
• CoTAL: Human-in-the-Loop Prompt Engineering, Chain-of-Thought Reasoning, and Active Learning for Generalizable Formative Assessment Scoring [2.249916681499244]
  Methods such as chain-of-thought (CoT) prompting enable teachers to grade formative assessments in science.<n>The extent to which these methods generalize across curricula in multiple domains remains largely untested.<n>We introduce Chain-of-Thought Prompting + Active Learning (CoTAL), an LLM-based approach to formative assessment scoring.
  arXiv  Detail & Related papers  (2025-04-03T06:53:34Z)
• PyEvalAI: AI-assisted evaluation of Jupyter Notebooks for immediate personalized feedback [43.56788158589046]
  PyEvalAI scores Jupyter notebooks using a combination of unit tests and a locally hosted language model to preserve privacy.<n>A case study demonstrates its effectiveness in improving feedback speed and grading efficiency for exercises in a university-level course on numerics.
  arXiv  Detail & Related papers  (2025-02-25T18:20:20Z)
• Evaluating GPT-4 at Grading Handwritten Solutions in Math Exams [48.99818550820575]
  We leverage state-of-the-art multi-modal AI models, in particular GPT-4o, to automatically grade handwritten responses to college-level math exams.<n>Using real student responses to questions in a probability theory exam, we evaluate GPT-4o's alignment with ground-truth scores from human graders using various prompting techniques.
  arXiv  Detail & Related papers  (2024-11-07T22:51:47Z)
• Could ChatGPT get an Engineering Degree? Evaluating Higher Education Vulnerability to AI Assistants [176.39275404745098]
  We evaluate whether two AI assistants, GPT-3.5 and GPT-4, can adequately answer assessment questions.<n>GPT-4 answers an average of 65.8% of questions correctly, and can even produce the correct answer across at least one prompting strategy for 85.1% of questions.<n>Our results call for revising program-level assessment design in higher education in light of advances in generative AI.
  arXiv  Detail & Related papers  (2024-08-07T12:11:49Z)
• "I understand why I got this grade": Automatic Short Answer Grading with Feedback [33.63970664152288]
  We introduce Engineering Short Answer Feedback (EngSAF), a dataset designed for automatic short-answer grading with feedback.<n>We incorporate feedback into our dataset by leveraging the generative capabilities of state-of-the-art large language models (LLMs) using our Label-Aware Synthetic Feedback Generation (LASFG) strategy.<n>The best-performing model (Mistral-7B) achieves an overall accuracy of 75.4% and 58.7% on unseen answers and unseen question test sets, respectively.
  arXiv  Detail & Related papers  (2024-06-30T15:42:18Z)
• SciRIFF: A Resource to Enhance Language Model Instruction-Following over Scientific Literature [97.31347312130119]
  SciRIFF (Scientific Resource for Instruction-Following and Finetuning) is a dataset of 137K instruction-following instances for training and evaluation, covering 54 tasks.<n>These tasks span five core scientific literature understanding capabilities: information extraction, summarization, question answering, claim verification, and classification.<n> SciRIFF is unique in being entirely expert-written, high-quality instruction-following dataset for extracting and synthesizing information from research literature across diverse scientific fields.
  arXiv  Detail & Related papers  (2024-06-10T21:22:08Z)
• Grade Like a Human: Rethinking Automated Assessment with Large Language Models [11.442433408767583]
  Large language models (LLMs) have been used for automated grading, but they have not yet achieved the same level of performance as humans. We propose an LLM-based grading system that addresses the entire grading procedure, including the following key components.
  arXiv  Detail & Related papers  (2024-05-30T05:08:15Z)
• Empowering Private Tutoring by Chaining Large Language Models [87.76985829144834]
  This work explores the development of a full-fledged intelligent tutoring system powered by state-of-the-art large language models (LLMs) The system is into three inter-connected core processes-interaction, reflection, and reaction. Each process is implemented by chaining LLM-powered tools along with dynamically updated memory modules.
  arXiv  Detail & Related papers  (2023-09-15T02:42:03Z)
• Automated Grading and Feedback Tools for Programming Education: A Systematic Review [7.776434991976473]
  Most papers assess the correctness of assignments in object-oriented languages. Few tools assess the maintainability, readability or documentation of the source code. Most tools offered fully automated assessment to allow for near-instantaneous feedback.
  arXiv  Detail & Related papers  (2023-06-20T17:54:50Z)
• SETSum: Summarization and Visualization of Student Evaluations of Teaching [74.76373136325032]
  Student Evaluations of Teaching (SETs) are widely used in colleges and universities. SETSum provides organized illustrations of SET findings to instructors and other reviewers.
  arXiv  Detail & Related papers  (2022-07-08T01:40:11Z)
• ProtoTransformer: A Meta-Learning Approach to Providing Student Feedback [54.142719510638614]
  In this paper, we frame the problem of providing feedback as few-shot classification. A meta-learner adapts to give feedback to student code on a new programming question from just a few examples by instructors. Our approach was successfully deployed to deliver feedback to 16,000 student exam-solutions in a programming course offered by a tier 1 university.
  arXiv  Detail & Related papers  (2021-07-23T22:41:28Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.