Evaluating Machine Expertise: How Graduate Students Develop Frameworks for Assessing GenAI Content

• URL: http://arxiv.org/abs/2504.17964v1
• Date: Thu, 24 Apr 2025 22:24:14 GMT
• Title: Evaluating Machine Expertise: How Graduate Students Develop Frameworks for Assessing GenAI Content
• Abstract summary: This paper examines how graduate students develop frameworks for evaluating machine-generated expertise in web-based interactions with large language models (LLMs)<n>Our findings reveal that students construct evaluation frameworks shaped by three main factors: professional identity, verification capabilities, and system navigation experience.
• Score: 1.967444231154626
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: This paper examines how graduate students develop frameworks for evaluating machine-generated expertise in web-based interactions with large language models (LLMs). Through a qualitative study combining surveys, LLM interaction transcripts, and in-depth interviews with 14 graduate students, we identify patterns in how these emerging professionals assess and engage with AI-generated content. Our findings reveal that students construct evaluation frameworks shaped by three main factors: professional identity, verification capabilities, and system navigation experience. Rather than uniformly accepting or rejecting LLM outputs, students protect domains central to their professional identities while delegating others--with managers preserving conceptual work, designers safeguarding creative processes, and programmers maintaining control over core technical expertise. These evaluation frameworks are further influenced by students' ability to verify different types of content and their experience navigating complex systems. This research contributes to web science by highlighting emerging human-genAI interaction patterns and suggesting how platforms might better support users in developing effective frameworks for evaluating machine-generated expertise signals in AI-mediated web environments.

Related papers

• Interpretable Difficulty-Aware Knowledge Tracing in Tutor-Student Dialogues [46.915280360954576]
  We propose an interpretable difficulty-aware conversational KT framework built upon large language models (LLMs)<n>The framework incorporates the original textual question and the next tutor-posed task to estimate the student's knowledge state and the difficulty of the upcoming turn.<n>We evaluate the framework on two tutor-student dialogue datasets.
  arXiv  Detail & Related papers  (2026-05-01T21:07:24Z)
• Developing a Multi-Agent System to Generate Next Generation Science Assessments with Evidence-Centered Design [9.558651359587358]
  The Next Generation Science Standards (NGSS) demand assessments to understand students' ability to use science knowledge to solve problems and design solutions.<n>To elicit such higher-order ability, educators need performance-based assessments, which are challenging to develop.<n>One solution that has been broadly adopted is Evidence-Centered Design (ECD), which emphasizes interconnected models of the learner, evidence, and tasks.
  arXiv  Detail & Related papers  (2026-02-03T04:37:24Z)
• Artificial Intelligence Agents in Music Analysis: An Integrative Perspective Based on Two Use Cases [0.0]
  This paper presents an integrative review and experimental validation of artificial intelligence (AI) agents applied to music analysis and education.<n>We synthesize the historical evolution from rule-based models to contemporary approaches involving deep learning, multi-agent architectures, and retrieval-augmented generation frameworks.<n> Experimental results demonstrate that AI agents effectively enhance musical pattern recognition, compositional parameterization, and educational feedback.
  arXiv  Detail & Related papers  (2025-11-17T23:46:47Z)
• "I Like That You Have to Poke Around": Instructors on How Experiential Approaches to AI Literacy Spark Inquiry and Critical Thinking [0.5872014229110213]
  This paper presents findings from a study of AI User, a modular, web-based curriculum that teaches core AI concepts through interactive, no-code projects grounded in real-world scenarios.<n>Fifteen community college instructors participated in structured focus groups, completing the projects as learners and providing feedback through individual reflection and group discussion.<n>Findings highlight instructors' appreciation for exploratory tasks, role-based simulations, and real-world relevance, while also surfacing design trade-offs around cognitive load, guidance, and adaptability for diverse learners.
  arXiv  Detail & Related papers  (2025-11-07T17:05:58Z)
• AI Literacy for Community Colleges: Instructors' Perspectives on Scenario-Based and Interactive Approaches to Teaching AI [0.500208619516796]
  This research category full paper investigates how community college instructors evaluate interactive, no-code AI literacy resources designed for non-STEM learners.<n>As artificial intelligence becomes increasingly integrated into everyday technologies, AI literacy has emerged as a critical skill across disciplines.<n>We developed AI User, an interactive online curriculum that introduces core AI concepts through scenario - based activities set in real - world contexts.
  arXiv  Detail & Related papers  (2025-11-07T15:51:53Z)
• Towards AI Agents for Course Instruction in Higher Education: Early Experiences from the Field [3.1941554288428193]
  This article presents early findings from designing, deploying and evaluating an AI-based educational agent deployed as the primary instructor in a graduate-level Cloud Computing course at IISc.<n>We detail the design of a Large Language Model (LLM)-driven Instructor Agent, and introduce a pedagogical framework that integrates the Instructor Agent into the course workflow.<n>We report early experiences on how students interact with the Agent to explore concepts, clarify doubts and sustain inquiry-driven dialogue during live classroom sessions.
  arXiv  Detail & Related papers  (2025-10-23T06:23:35Z)
• WebDevJudge: Evaluating (M)LLMs as Critiques for Web Development Quality [62.43165871914528]
  We introduce WebDevJudge, a systematic benchmark for assessing LLM-as-a-judge performance in web development.<n>WebDevJudge comprises human preference labels over paired web implementations, annotated with structured and query-grounded rubrics.<n>In-depth analysis indicates this gap stems from fundamental model limitations, including failures in recognizing functional equivalence, verifying task feasibility, and mitigating bias.
  arXiv  Detail & Related papers  (2025-10-21T12:16:04Z)
• Fundamentals of Building Autonomous LLM Agents [64.39018305018904]
  This paper reviews the architecture and implementation methods of agents powered by large language models (LLMs)<n>The research aims to explore patterns to develop "agentic" LLMs that can automate complex tasks and bridge the performance gap with human capabilities.
  arXiv  Detail & Related papers  (2025-10-10T10:32:39Z)
• Beyond Automation: Socratic AI, Epistemic Agency, and the Implications of the Emergence of Orchestrated Multi-Agent Learning Architectures [0.0]
  Generative AI is no longer a peripheral tool in higher education.<n>This paper presents findings from a controlled experiment evaluating a Socratic AI Tutor.<n>Students using the Tutor reported significantly greater support for critical, independent, and reflective thinking.
  arXiv  Detail & Related papers  (2025-08-07T07:49:03Z)
• Beyond Brainstorming: What Drives High-Quality Scientific Ideas? Lessons from Multi-Agent Collaboration [59.41889496960302]
  This paper investigates whether structured multi-agent discussions can surpass solitary ideation.<n>We propose a cooperative multi-agent framework for generating research proposals.<n>We employ a comprehensive protocol with agent-based scoring and human review across dimensions such as novelty, strategic vision, and integration depth.
  arXiv  Detail & Related papers  (2025-08-06T15:59:18Z)
• Teaching at Scale: Leveraging AI to Evaluate and Elevate Engineering Education [3.557803321422781]
  This article presents a scalable, AI-supported framework for qualitative student feedback using large language models.<n>The system employs hierarchical summarization, anonymization, and exception handling to extract actionable themes from open-ended comments.<n>We report on its successful deployment across a large college of engineering.
  arXiv  Detail & Related papers  (2025-08-01T20:27:40Z)
• Evolutionary Perspectives on the Evaluation of LLM-Based AI Agents: A Comprehensive Survey [45.485318955120924]
  The transition from traditional large language models (LLMs) to more advanced AI agents represents a pivotal evolutionary step.<n>Existing evaluation frameworks often blur the distinctions between LLM chatbots and AI agents, leading to confusion among researchers selecting appropriate benchmarks.<n>This paper introduces a systematic analysis of current evaluation approaches, grounded in an evolutionary perspective.
  arXiv  Detail & Related papers  (2025-06-06T17:52:18Z)
• Creativity in LLM-based Multi-Agent Systems: A Survey [56.25583236738877]
  Large language model (LLM)-driven multi-agent systems (MAS) are transforming how humans and AIs collaboratively generate ideas and artifacts.<n>This is the first survey dedicated to creativity in MAS.<n>We focus on text and image generation tasks, and present: (1) a taxonomy of agent proactivity and persona design; (2) an overview of generation techniques, including divergent exploration, iterative refinement, and collaborative synthesis, as well as relevant datasets and evaluation metrics; and (3) a discussion of key challenges, such as inconsistent evaluation standards, insufficient bias mitigation, coordination conflicts, and the lack of unified benchmarks.
  arXiv  Detail & Related papers  (2025-05-27T12:36:14Z)
• AITEE -- Agentic Tutor for Electrical Engineering [44.99833362998488]
  AITEE is an agent-based tutoring system for electrical engineering.<n>It supports both hand-drawn and digital circuits through an adapted circuit reconstruction process.<n>It implements a Socratic dialogue to foster learner autonomy through guided questioning.
  arXiv  Detail & Related papers  (2025-05-27T10:07:05Z)
• On the Evaluation of Engineering Artificial General Intelligence [5.802869598386355]
  We discuss the challenges and propose a framework for evaluating engineering artificial general intelligence (eAGI) agents.<n>We consider eAGI as a specialization of artificial general intelligence (AGI)<n>eAGI agents should possess a unique blend of background knowledge (recall and retrieve) of facts and methods.
  arXiv  Detail & Related papers  (2025-05-15T18:52:47Z)
• Enhancing tutoring systems by leveraging tailored promptings and domain knowledge with Large Language Models [2.5362697136900563]
  AI-driven tools like ChatGPT and Intelligent Tutoring Systems (ITS) have enhanced learning experiences through personalisation and flexibility.<n>ITSs can adapt to individual learning needs and provide customised feedback based on a student's performance, cognitive state, and learning path.<n>Our research aims to address these gaps by integrating skill-aligned feedback via Retrieval Augmented Generation (RAG) into prompt engineering for Large Language Models (LLMs)
  arXiv  Detail & Related papers  (2025-05-02T02:30:39Z)
• Beyond the Classroom: Bridging the Gap Between Academia and Industry with a Hands-on Learning Approach [2.8123958518740544]
  Self-adaptive software systems have emerged as a critical focus in software design and operation.<n>A survey among practitioners identified that the lack of knowledgeable individuals has hindered its adoption in the industry.<n>We present our experience teaching a course on self-adaptive software systems that integrates theoretical knowledge and hands-on learning with industry-relevant technologies.
  arXiv  Detail & Related papers  (2025-04-14T21:32:25Z)
• A Survey of Frontiers in LLM Reasoning: Inference Scaling, Learning to Reason, and Agentic Systems [93.8285345915925]
  Reasoning is a fundamental cognitive process that enables logical inference, problem-solving, and decision-making.<n>With the rapid advancement of large language models (LLMs), reasoning has emerged as a key capability that distinguishes advanced AI systems.<n>We categorize existing methods along two dimensions: (1) Regimes, which define the stage at which reasoning is achieved; and (2) Architectures, which determine the components involved in the reasoning process.
  arXiv  Detail & Related papers  (2025-04-12T01:27:49Z)
• Level Up Peer Review in Education: Investigating genAI-driven Gamification system and its influence on Peer Feedback Effectiveness [0.8087870525861938]
  This paper introduces Socratique, a gamified peer-assessment platform integrated with Generative AI (GenAI) assistance.<n>By incorporating game elements, Socratique aims to motivate students to provide more feedback.<n>Students in the treatment group provided significantly more voluntary feedback, with higher scores on clarity, relevance, and specificity.
  arXiv  Detail & Related papers  (2025-04-03T18:30:25Z)
• A Survey on (M)LLM-Based GUI Agents [62.57899977018417]
  Graphical User Interface (GUI) Agents have emerged as a transformative paradigm in human-computer interaction.<n>Recent advances in large language models and multimodal learning have revolutionized GUI automation across desktop, mobile, and web platforms.<n>This survey identifies key technical challenges, including accurate element localization, effective knowledge retrieval, long-horizon planning, and safety-aware execution control.
  arXiv  Detail & Related papers  (2025-03-27T17:58:31Z)
• LLM Agents for Education: Advances and Applications [49.3663528354802]
  Large Language Model (LLM) agents have demonstrated remarkable capabilities in automating tasks and driving innovation across diverse educational applications.<n>This survey aims to provide a comprehensive technological overview of LLM agents for education, fostering further research and collaboration to enhance their impact for the greater good of learners and educators alike.
  arXiv  Detail & Related papers  (2025-03-14T11:53:44Z)
• Human-Centered Design for AI-based Automatically Generated Assessment Reports: A Systematic Review [4.974197456441281]
  This study emphasizes the importance of reducing teachers' cognitive demands through user-centered and intuitive designs.<n>It highlights the potential of diverse information presentation formats such as text, visual aids, and plots and advanced functionalities such as live and interactive features to enhance usability.<n>The framework aims to address challenges in engaging teachers with technology-enhanced assessment results, facilitating data-driven decision-making, and providing personalized feedback to improve the teaching and learning process.
  arXiv  Detail & Related papers  (2024-12-30T16:20:07Z)
• GUI Agents: A Survey [129.94551809688377]
  Graphical User Interface (GUI) agents, powered by Large Foundation Models, have emerged as a transformative approach to automating human-computer interaction.<n>Motivated by the growing interest and fundamental importance of GUI agents, we provide a comprehensive survey that categorizes their benchmarks, evaluation metrics, architectures, and training methods.
  arXiv  Detail & Related papers  (2024-12-18T04:48:28Z)
• User-centric evaluation of explainability of AI with and for humans: a comprehensive empirical study [5.775094401949666]
  This study is located in the Human-Centered Artificial Intelligence (HCAI) It focuses on the results of a user-centered assessment of commonly used eXplainable Artificial Intelligence (XAI) algorithms.
  arXiv  Detail & Related papers  (2024-10-21T12:32:39Z)
• Data Analysis in the Era of Generative AI [56.44807642944589]
  This paper explores the potential of AI-powered tools to reshape data analysis, focusing on design considerations and challenges. We explore how the emergence of large language and multimodal models offers new opportunities to enhance various stages of data analysis workflow. We then examine human-centered design principles that facilitate intuitive interactions, build user trust, and streamline the AI-assisted analysis workflow across multiple apps.
  arXiv  Detail & Related papers  (2024-09-27T06:31:03Z)
• Constraining Participation: Affordances of Feedback Features in Interfaces to Large Language Models [49.74265453289855]
  Large language models (LLMs) are now accessible to anyone with a computer, a web browser, and an internet connection via browser-based interfaces. This paper examines the affordances of interactive feedback features in ChatGPT's interface, analysing how they shape user input and participation in iteration.
  arXiv  Detail & Related papers  (2024-08-27T13:50:37Z)
• Enhancing LLM-Based Feedback: Insights from Intelligent Tutoring Systems and the Learning Sciences [0.0]
  This work advocates careful and caring AIED research by going through previous research on feedback generation in ITS. The main contributions of this paper include: an avocation of applying more cautious, theoretically grounded methods in feedback generation in the era of generative AI.
  arXiv  Detail & Related papers  (2024-05-07T20:09:18Z)
• Prototyping with Prompts: Emerging Approaches and Challenges in Generative AI Design for Collaborative Software Teams [2.237039275844699]
  Generative AI models are increasingly being integrated into human task, enabling the production of expressive content.<n>Unlike traditional human-AI design methods, the new approach to designing generative capabilities focuses heavily on prompt engineering strategies.<n>Our findings highlight emerging practices and role shifts in AI system prototyping among multistakeholder teams.
  arXiv  Detail & Related papers  (2024-02-27T17:56:10Z)
• Modelling Assessment Rubrics through Bayesian Networks: a Pragmatic Approach [40.06500618820166]
  This paper presents an approach to deriving a learner model directly from an assessment rubric. We illustrate how the approach can be applied to automatize the human assessment of an activity developed for testing computational thinking skills.
  arXiv  Detail & Related papers  (2022-09-07T10:09:12Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.