AI Mentors for Student Projects: Spotting Early Issues in Computer Science Proposals

• URL: http://arxiv.org/abs/2503.05782v1
• Date: Wed, 26 Feb 2025 16:24:14 GMT
• Title: AI Mentors for Student Projects: Spotting Early Issues in Computer Science Proposals
• Abstract summary: We develop a software system that collects project proposals and information to support educators in determining whether a student is ready to engage with aptitude.<n>We find that users perceived the system as helpful for writing project proposals and identifying tools and technologies to learn more about.<n>While the prospect of using LLMs to rate the quality of students' project proposals is promising, its long-term effectiveness hinges on future efforts at characterizing indicators that reliably predict students' success and motivation to learn.
• Score: 35.92015790655676
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: When executed well, project-based learning (PBL) engages students' intrinsic motivation, encourages students to learn far beyond a course's limited curriculum, and prepares students to think critically and maturely about the skills and tools at their disposal. However, educators experience mixed results when using PBL in their classrooms: some students thrive with minimal guidance and others flounder. Early evaluation of project proposals could help educators determine which students need more support, yet evaluating project proposals and student aptitude is time-consuming and difficult to scale. In this work, we design, implement, and conduct an initial user study (n = 36) for a software system that collects project proposals and aptitude information to support educators in determining whether a student is ready to engage with PBL. We find that (1) users perceived the system as helpful for writing project proposals and identifying tools and technologies to learn more about, (2) educator ratings indicate that users with less technical experience in the project topic tend to write lower-quality project proposals, and (3) GPT-4o's ratings show agreement with educator ratings. While the prospect of using LLMs to rate the quality of students' project proposals is promising, its long-term effectiveness strongly hinges on future efforts at characterizing indicators that reliably predict students' success and motivation to learn.

Related papers

• Improving Capstone Team Outcomes through Dynamic Skill Matching and Preference Alignment [1.1807695418165365]
  Team-based projects are a cornerstone of engineering and computing courses, but unstructured team formation often leads to poor project outcomes.<n>This paper introduces a novel, three-stage methodology for creating effective student teams by integrating student preferences with project skill requirements.
  arXiv  Detail & Related papers  (2026-06-14T03:29:35Z)
• Human-AI Co-Mentorship in Project-Based Learning: A Case Study in Financial Forecasting [0.1864131501304829]
  This paper reflects on a AI research project carried out by a team of high-school and early-undergraduate students under the mentorship of graduate researchers.<n>We share our experience in how AI tools accelerated the process that enabled the high school students to focus on higher order problem formulation and solution.
  arXiv  Detail & Related papers  (2026-05-06T17:16:29Z)
• Stimulating Higher Order Thinking in Mechatronics by Comparing PID and Fuzzy Control [0.0]
  We develop a project in a semester-long mechatronics course in which students must evaluate two automatic control methodologies.<n>The project involves determining the superior control method for leader-follower behavior, where a ground vehicle autonomously follows a lead vehicle.<n> Laboratory exercises throughout the semester expose students to the skills required for the project.<n>In the final course project, students create their own evaluation criteria and experiments to make a design decision between PID and fuzzy control.
  arXiv  Detail & Related papers  (2026-01-10T19:16:58Z)
• Exposía: Academic Writing Assessment of Exposés and Peer Feedback [56.428320613219306]
  We present Exposa, the first public dataset that connects writing and feedback assessment in higher education.<n>We use Exposa to benchmark state-of-the-art open-source large language models (LLMs) for two tasks: automated scoring of (1) the proposals and (2) the student reviews.
  arXiv  Detail & Related papers  (2026-01-10T11:33:26Z)
• Automated Formative Feedback for Short-form Writing: An LLM-Driven Approach and Adoption Analysis [1.2234742322758418]
  This paper explores the development and adoption of AI-based formative feedback in an engineering program.<n>A tool was developed to provide students with personalized feedback on their draft reports, guiding them toward improved completeness and quality.
  arXiv  Detail & Related papers  (2025-09-25T15:17:51Z)
• Factors Impacting Faculty Adoption of Project-Based Learning in Computing Education: a Survey [4.956709222278242]
  Project-based learning (PjBL) has the potential to enhance student motivation, engagement, critical thinking, collaboration, and problem-solving skills.<n>Despite these benefits, faculty adoption remains inconsistent due to challenges such as insufficient institutional support, time constraints, limited training opportunities, designing or sourcing projects, and aligning them with course objectives.<n>This research explores these barriers and investigates the strategies and resources that facilitate a successful adoption.
  arXiv  Detail & Related papers  (2025-07-24T02:16:29Z)
• Improving Student-AI Interaction Through Pedagogical Prompting: An Example in Computer Science Education [1.1517315048749441]
  Large language model (LLM) applications have sparked both excitement and concern.<n>Recent studies consistently highlight students' (mis)use of LLMs can hinder learning outcomes.<n>This work aims to teach students how to effectively prompt LLMs to improve their learning.
  arXiv  Detail & Related papers  (2025-06-23T20:39:17Z)
• Alice: Proactive Learning with Teacher's Demonstrations for Weak-to-Strong Generalization [69.96794098855938]
  Weak-to-strong generalization (W2SG) offers a promising framework for supervising increasingly capable language models (LLMs) Traditional W2SG methods rely on passive learning, where a weak teacher provides noisy demonstrations to train a strong student. We introduce Alice, a framework that leverages complementary knowledge between teacher and student to enhance the learning process.
  arXiv  Detail & Related papers  (2025-04-09T22:33:06Z)
• Co-designing Large Language Model Tools for Project-Based Learning with K12 Educators [10.100127895043235]
  generative large language models (LLMs) have opened the door for student-based active learning methods.<n>Project design and management, assessment and guidance are challenges for student implementation.<n>We propose guidelines for future deployment of generative large language models in classrooms.
  arXiv  Detail & Related papers  (2025-02-13T22:23:08Z)
• Embracing AI in Education: Understanding the Surge in Large Language Model Use by Secondary Students [53.20318273452059]
  Large language models (LLMs) like OpenAI's ChatGPT have opened up new avenues in education.<n>Despite school restrictions, our survey of over 300 middle and high school students revealed that a remarkable 70% of students have utilized LLMs.<n>We propose a few ideas to address such issues, including subject-specific models, personalized learning, and AI classrooms.
  arXiv  Detail & Related papers  (2024-11-27T19:19:34Z)
• "Give me the code" -- Log Analysis of First-Year CS Students' Interactions With GPT [0.0]
  This paper analyzes the prompts used by 69 freshmen undergraduate students to solve a certain programming problem within a project assignment.<n>Despite using unsophisticated prompting techniques, our findings suggest that the majority of students successfully leveraged GPT.<n>Half of the students demonstrated the ability to exercise judgment in selecting from multiple GPT-generated solutions.
  arXiv  Detail & Related papers  (2024-11-26T20:11:46Z)
• Exploring Knowledge Tracing in Tutor-Student Dialogues using LLMs [49.18567856499736]
  We investigate whether large language models (LLMs) can be supportive of open-ended dialogue tutoring. We apply a range of knowledge tracing (KT) methods on the resulting labeled data to track student knowledge levels over an entire dialogue. We conduct experiments on two tutoring dialogue datasets, and show that a novel yet simple LLM-based method, LLMKT, significantly outperforms existing KT methods in predicting student response correctness in dialogues.
  arXiv  Detail & Related papers  (2024-09-24T22:31:39Z)
• Explainable Few-shot Knowledge Tracing [48.877979333221326]
  We propose a cognition-guided framework that can track the student knowledge from a few student records while providing natural language explanations. Experimental results from three widely used datasets show that LLMs can perform comparable or superior to competitive deep knowledge tracing methods.
  arXiv  Detail & Related papers  (2024-05-23T10:07:21Z)
• Evaluating and Optimizing Educational Content with Large Language Model Judgments [52.33701672559594]
  We use Language Models (LMs) as educational experts to assess the impact of various instructions on learning outcomes. We introduce an instruction optimization approach in which one LM generates instructional materials using the judgments of another LM as a reward function. Human teachers' evaluations of these LM-generated worksheets show a significant alignment between the LM judgments and human teacher preferences.
  arXiv  Detail & Related papers  (2024-03-05T09:09:15Z)
• Identifying Student Profiles Within Online Judge Systems Using Explainable Artificial Intelligence [6.638206014723678]
  Online Judge (OJ) systems are typically considered within programming-related courses as they yield fast and objective assessments of the code developed by the students. This work aims to tackle this limitation by considering the further exploitation of the information gathered by the OJ and automatically inferring feedback for both the student and the instructor.
  arXiv  Detail & Related papers  (2024-01-29T12:11:30Z)
• A Machine Learning system to monitor student progress in educational institutes [0.0]
  We propose a data driven approach that makes use of Machine Learning techniques to generate a classifier called credit score. The proposal to use credit score as progress indicator is well suited to be used in a Learning Management System.
  arXiv  Detail & Related papers  (2022-11-02T08:24:08Z)
• Experience of Teaching Data Visualization using Project-based Learning [0.3437656066916039]
  We show which input was provided when necessary for students to achieve their goals. We discuss and compare the tools we found useful for students to accomplish their goals.
  arXiv  Detail & Related papers  (2021-10-21T16:47:34Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.