TAMIGO: Empowering Teaching Assistants using LLM-assisted viva and code assessment in an Advanced Computing Class

• URL: http://arxiv.org/abs/2407.16805v1
• Date: Tue, 23 Jul 2024 19:12:13 GMT
• Title: TAMIGO: Empowering Teaching Assistants using LLM-assisted viva and code assessment in an Advanced Computing Class
• Authors: Anishka IIITD, Diksha Sethi, Nipun Gupta, Shikhar Sharma, Srishti Jain, Ujjwal Singhal, Dhruv Kumar,
• Abstract summary: This paper investigates the application of Large Language Models in assisting teaching assistants with viva and code assessments. We develop TAMIGO, an LLM-based system for TAs to evaluate programming assignments. We evaluate the quality of LLM-generated viva questions, model answers, feedback on viva answers, and feedback on student code submissions.
• Score: 3.3567738223900645
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Large Language Models (LLMs) have significantly transformed the educational landscape, offering new tools for students, instructors, and teaching assistants. This paper investigates the application of LLMs in assisting teaching assistants (TAs) with viva and code assessments in an advanced computing class on distributed systems in an Indian University. We develop TAMIGO, an LLM-based system for TAs to evaluate programming assignments. For viva assessment, the TAs generated questions using TAMIGO and circulated these questions to the students for answering. The TAs then used TAMIGO to generate feedback on student answers. For code assessment, the TAs selected specific code blocks from student code submissions and fed it to TAMIGO to generate feedback for these code blocks. The TAMIGO-generated feedback for student answers and code blocks was used by the TAs for further evaluation. We evaluate the quality of LLM-generated viva questions, model answers, feedback on viva answers, and feedback on student code submissions. Our results indicate that LLMs are highly effective at generating viva questions when provided with sufficient context and background information. However, the results for LLM-generated feedback on viva answers were mixed; instances of hallucination occasionally reduced the accuracy of feedback. Despite this, the feedback was consistent, constructive, comprehensive, balanced, and did not overwhelm the TAs. Similarly, for code submissions, the LLM-generated feedback was constructive, comprehensive and balanced, though there was room for improvement in aligning the feedback with the instructor-provided rubric for code evaluation. Our findings contribute to understanding the benefits and limitations of integrating LLMs into educational settings.

Related papers

• CIBench: Evaluating Your LLMs with a Code Interpreter Plugin [68.95137938214862]
  We propose an interactive evaluation framework, named CIBench, to comprehensively assess LLMs' ability to utilize code interpreters for data science tasks. The evaluation dataset is constructed using an LLM-human cooperative approach and simulates an authentic workflow by leveraging consecutive and interactive IPython sessions. We conduct extensive experiments to analyze the ability of 24 LLMs on CIBench and provide valuable insights for future LLMs in code interpreter utilization.
  arXiv  Detail & Related papers  (2024-07-15T07:43:55Z)
• Large Language Model as an Assignment Evaluator: Insights, Feedback, and Challenges in a 1000+ Student Course [49.296957552006226]
  It is unclear whether large language models (LLMs) for automatic evaluation can be applied in real-world classrooms. This report shares how we use GPT-4 as an automatic assignment evaluator in a university course with 1,028 students.
  arXiv  Detail & Related papers  (2024-07-07T00:17:24Z)
• Generating Feedback-Ladders for Logical Errors in Programming using Large Language Models [2.1485350418225244]
  Large language model (LLM)-based methods have shown great promise in feedback generation for programming assignments. This paper explores using LLMs to generate a "feedback-ladder", i.e., multiple levels of feedback for the same problem-submission pair. We evaluate the quality of the generated feedback-ladder via a user study with students, educators, and researchers.
  arXiv  Detail & Related papers  (2024-05-01T03:52:39Z)
• Improving the Validity of Automatically Generated Feedback via Reinforcement Learning [50.067342343957876]
  We propose a framework for feedback generation that optimize both correctness and alignment using reinforcement learning (RL) Specifically, we use GPT-4's annotations to create preferences over feedback pairs in an augmented dataset for training via direct preference optimization (DPO)
  arXiv  Detail & Related papers  (2024-03-02T20:25:50Z)
• Why and When LLM-Based Assistants Can Go Wrong: Investigating the Effectiveness of Prompt-Based Interactions for Software Help-Seeking [5.755004576310333]
  Large Language Model (LLM) assistants have emerged as potential alternatives to search methods for helping users navigate software. LLM assistants use vast training data from domain-specific texts, software manuals, and code repositories to mimic human-like interactions.
  arXiv  Detail & Related papers  (2024-02-12T19:49:58Z)
• Automated Assessment of Students' Code Comprehension using LLMs [0.3293989832773954]
  Large Language Models (LLMs) and encoder-based Semantic Textual Similarity (STS) models are assessed. Our findings indicate that LLMs, when prompted in few-shot and chain-of-thought setting, perform comparable to fine-tuned encoder-based models in evaluating students' short answers in programming domain.
  arXiv  Detail & Related papers  (2023-12-19T20:39:12Z)
• Students' Perceptions and Preferences of Generative Artificial Intelligence Feedback for Programming [15.372316943507506]
  We generated automated feedback using the ChatGPT API for four lab assignments in an introductory computer science class. Students perceived the feedback as aligning well with formative feedback guidelines established by Shute. Students generally expected specific and corrective feedback with sufficient code examples, but had diverged opinions on the tone of the feedback.
  arXiv  Detail & Related papers  (2023-12-17T22:26:53Z)
• Tuna: Instruction Tuning using Feedback from Large Language Models [74.04950416204551]
  We propose finetuning an instruction-tuned large language model using our novel textitprobabilistic ranking and textitcontextual ranking approaches. Probabilistic ranking enables the instruction-tuned model to inherit the relative rankings of high-quality and low-quality responses from the teacher LLM. On the other hand, learning with contextual ranking allows the model to refine its own response distribution using the contextual understanding ability of stronger LLMs.
  arXiv  Detail & Related papers  (2023-10-20T09:55:06Z)
• Calculating Originality of LLM Assisted Source Code [0.0]
  We propose a neural network-based tool to determine the original effort (and LLM's contribution) put by students in writing source codes. Our tool is motivated by minimum description length measures like Kolmogorov complexity.
  arXiv  Detail & Related papers  (2023-07-10T11:30:46Z)
• Check Your Facts and Try Again: Improving Large Language Models with External Knowledge and Automated Feedback [127.75419038610455]
  Large language models (LLMs) are able to generate human-like, fluent responses for many downstream tasks. This paper proposes a LLM-Augmenter system, which augments a black-box LLM with a set of plug-and-play modules.
  arXiv  Detail & Related papers  (2023-02-24T18:48:43Z)
• ProtoTransformer: A Meta-Learning Approach to Providing Student Feedback [54.142719510638614]
  In this paper, we frame the problem of providing feedback as few-shot classification. A meta-learner adapts to give feedback to student code on a new programming question from just a few examples by instructors. Our approach was successfully deployed to deliver feedback to 16,000 student exam-solutions in a programming course offered by a tier 1 university.
  arXiv  Detail & Related papers  (2021-07-23T22:41:28Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.