Human or AI? Comparing Design Thinking Assessments by Teaching Assistants and Bots

• URL: http://arxiv.org/abs/2510.16069v1
• Date: Fri, 17 Oct 2025 07:09:21 GMT
• Title: Human or AI? Comparing Design Thinking Assessments by Teaching Assistants and Bots
• Abstract summary: This study investigates the reliability and perceived accuracy of AI-assisted assessment compared to TA-assisted assessment in evaluating student posters in design thinking education.<n>Results showed low statistical agreement between instructor and AI scores for empathy and pain points, with slightly higher alignment for visual communication.<n>The study underscores the need for hybrid assessment models that integrate computational efficiency with human insights.
• Score: 0.38233569758620045
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: As design thinking education grows in secondary and tertiary contexts, educators face the challenge of evaluating creative artefacts that combine visual and textual elements. Traditional rubric-based assessment is laborious, time-consuming, and inconsistent due to reliance on Teaching Assistants (TA) in large, multi-section cohorts. This paper presents an exploratory study investigating the reliability and perceived accuracy of AI-assisted assessment compared to TA-assisted assessment in evaluating student posters in design thinking education. Two activities were conducted with 33 Ministry of Education (MOE) Singapore school teachers to (1) compare AI-generated scores with TA grading across three key dimensions: empathy and user understanding, identification of pain points and opportunities, and visual communication, and (2) examine teacher preferences for AI-assigned, TA-assigned, and hybrid scores. Results showed low statistical agreement between instructor and AI scores for empathy and pain points, with slightly higher alignment for visual communication. Teachers preferred TA-assigned scores in six of ten samples. Qualitative feedback highlighted the potential of AI for formative feedback, consistency, and student self-reflection, but raised concerns about its limitations in capturing contextual nuance and creative insight. The study underscores the need for hybrid assessment models that integrate computational efficiency with human insights. This research contributes to the evolving conversation on responsible AI adoption in creative disciplines, emphasizing the balance between automation and human judgment for scalable and pedagogically sound assessment.

Related papers

• Who Should Grade My Work? Student Perspectives on Transparent AI-Assisted Writing Assessment in Higher Education [0.0]
  The integration of GenAI tools into higher education assessment raises important questions about how students understand, interpret, and respond to AI-mediated evaluation.<n>This study reports findings from a qualitative pedagogical inquiry conducted in an undergraduate technical communication course for computing students at a Saudi public university.<n>Students were explicitly informed that ChatGPT had generated the score and feedback and were invited to reflect on the evaluation in writing.
  arXiv  Detail & Related papers  (2026-09-04T16:49:33Z)
• Framing, Judging, Steering: An Assessable Competency Model for Teach-ing Students to Reason With Generative AI [47.027290803102666]
  We propose CoRe-3 (Co-Reasoning), a competency model factoring productive AI use into three assessable skills.<n>We ground the skills in theory, state five testable propositions, and instantiate them in CoReasoningLab.<n>Human-rater agreement and outcomes are next; we release the instrument, data, and protocol.
  arXiv  Detail & Related papers  (2026-06-04T10:25:31Z)
• The Missing Evaluation Axis: What 10,000 Student Submissions Reveal About AI Tutor Effectiveness [9.482202294953526]
  We argue that AI tutor evaluation should be extended with a behavioral dimension grounded in student interaction data.<n>We propose an evaluation framework and apply it to 10,235 code submissions with corresponding AI tutor feedback.<n>Using this framework to compare two deployed AI tutors across different semesters in a large-scale introductory computer science course reveals substantial differences in student engagement patterns.
  arXiv  Detail & Related papers  (2026-05-07T03:58:04Z)
• How effective are VLMs in assisting humans in inferring the quality of mental models from Multimodal short answers? [38.304584579482054]
  Mental models can play a critical role in assessing students' conceptual understanding of a topic.<n> MMGrader infers the quality of students' mental models from their multimodal responses using concept graphs as an analytical framework.<n>With improved accuracy, these can be highly effective assistants to teachers in inferring the mental models of their entire classrooms.
  arXiv  Detail & Related papers  (2026-02-10T08:01:05Z)
• AI Literacy, Safety Awareness, and STEM Career Aspirations of Australian Secondary Students: Evaluating the Impact of Workshop Interventions [38.350232667249095]
  Deepfakes and other forms of synthetic media pose growing safety risks for adolescents.<n>This study evaluates the impact of Day of AI Australia's workshop-based intervention on Australian secondary students.
  arXiv  Detail & Related papers  (2026-01-30T02:55:53Z)
• Exposía: Academic Writing Assessment of Exposés and Peer Feedback [56.428320613219306]
  We present Exposa, the first public dataset that connects writing and feedback assessment in higher education.<n>We use Exposa to benchmark state-of-the-art open-source large language models (LLMs) for two tasks: automated scoring of (1) the proposals and (2) the student reviews.
  arXiv  Detail & Related papers  (2026-01-10T11:33:26Z)
• Writing With Machines and Peers: Designing for Critical Engagement with Generative AI [5.719812010814006]
  This study proposes a pedagogical design that integrates AI and peer feedback in a graduate-level academic writing activity.<n>Students developed literature review projects through multiple writing and revision stages, receiving feedback from both a custom-built AI reviewer and human peers.
  arXiv  Detail & Related papers  (2025-11-19T02:17:42Z)
• Assessment Twins: A Protocol for AI-Vulnerable Summative Assessment [0.0]
  We introduce assessment twins as an accessible approach for redesigning assessment tasks to enhance validity.<n>We use Messick's unified validity framework to systematically map the ways in which GenAI threaten content, structural, consequential, generalisability, and external validity.<n>We argue that the twin approach helps mitigate validity threats by triangulating evidence across complementary formats.
  arXiv  Detail & Related papers  (2025-10-03T12:05:34Z)
• AI-Educational Development Loop (AI-EDL): A Conceptual Framework to Bridge AI Capabilities with Classical Educational Theories [8.500617875591633]
  This study introduces the AI-Educational Development Loop (AI-EDL), a theory-driven framework that integrates classical learning theories with human-in-the-loop artificial intelligence (AI)<n>The framework emphasizes transparency, self-regulated learning, and pedagogical oversight.
  arXiv  Detail & Related papers  (2025-08-01T15:44:19Z)
• A Review of Generative AI in Computer Science Education: Challenges and Opportunities in Accuracy, Authenticity, and Assessment [2.1891582280781634]
  This paper surveys the use of Generative AI tools, such as ChatGPT and Claude, in computer science education.<n>Generative AI raises concerns such as AI hallucinations, error propagation, bias, and blurred lines between AI-assisted and student-authored content.
  arXiv  Detail & Related papers  (2025-06-17T19:20:58Z)
• Resurrecting Socrates in the Age of AI: A Study Protocol for Evaluating a Socratic Tutor to Support Research Question Development in Higher Education [0.0]
  This protocol lays out a study grounded in constructivist learning theory to evaluate a novel AI-based Socratic Tutor.<n>The tutor engages students through iterative, reflective questioning, aiming to promote System 2 thinking.<n>This study aims to advance the understanding of how generative AI can be pedagogically aligned to support, not replace, human cognition.
  arXiv  Detail & Related papers  (2025-04-05T00:49:20Z)
• Form-Substance Discrimination: Concept, Cognition, and Pedagogy [55.2480439325792]
  This paper examines form-substance discrimination as an essential learning outcome for curriculum development in higher education.<n>We propose practical strategies for fostering this ability through curriculum design, assessment practices, and explicit instruction.
  arXiv  Detail & Related papers  (2025-04-01T04:15:56Z)
• Beyond Detection: Designing AI-Resilient Assessments with Automated Feedback Tool to Foster Critical Thinking [0.0]
  This research proposes a proactive, AI-resilient solution based on assessment design rather than detection.<n>It introduces a web-based Python tool that integrates Bloom's taxonomy with advanced natural language processing techniques.<n>It helps educators determine whether a task targets lower-order thinking such as recall and summarization or higher-order skills such as analysis, evaluation, and creation.
  arXiv  Detail & Related papers  (2025-03-30T23:13:00Z)
• Human Bias in the Face of AI: Examining Human Judgment Against Text Labeled as AI Generated [48.70176791365903]
  This study explores how bias shapes the perception of AI versus human generated content.<n>We investigated how human raters respond to labeled and unlabeled content.
  arXiv  Detail & Related papers  (2024-09-29T04:31:45Z)
• Exploring User Perspectives on ChatGPT: Applications, Perceptions, and Implications for AI-Integrated Education [40.38809129759498]
  ChatGPT is most commonly used in the domains of higher education, K-12 education, and practical skills training. On one hand, some users view it as a transformative tool capable of amplifying student self-efficacy and learning motivation. On the other hand, there is a degree of apprehension among concerned users.
  arXiv  Detail & Related papers  (2023-05-22T15:13:14Z)
• AGI: Artificial General Intelligence for Education [41.45039606933712]
  This position paper reviews artificial general intelligence (AGI)'s key concepts, capabilities, scope, and potential within future education. It highlights that AGI can significantly improve intelligent tutoring systems, educational assessment, and evaluation procedures. The paper emphasizes that AGI's capabilities extend to understanding human emotions and social interactions.
  arXiv  Detail & Related papers  (2023-04-24T22:31:59Z)
• Personalized Education in the AI Era: What to Expect Next? [76.37000521334585]
  The objective of personalized learning is to design an effective knowledge acquisition track that matches the learner's strengths and bypasses her weaknesses to meet her desired goal. In recent years, the boost of artificial intelligence (AI) and machine learning (ML) has unfolded novel perspectives to enhance personalized education.
  arXiv  Detail & Related papers  (2021-01-19T12:23:32Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.