Comparing Code Explanations Created by Students and Large Language Models

• URL: http://arxiv.org/abs/2304.03938v1
• Date: Sat, 8 Apr 2023 06:52:54 GMT
• Title: Comparing Code Explanations Created by Students and Large Language Models
• Authors: Juho Leinonen, Paul Denny, Stephen MacNeil, Sami Sarsa, Seth Bernstein, Joanne Kim, Andrew Tran, Arto Hellas
• Abstract summary: Reasoning about code and explaining its purpose are fundamental skills for computer scientists. The ability to describe at a high-level of abstraction how code will behave over all possible inputs correlates strongly with code writing skills. Existing pedagogical approaches that scaffold the ability to explain code, such as producing code explanations on demand, do not currently scale well to large classrooms.
• Score: 4.526618922750769
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Reasoning about code and explaining its purpose are fundamental skills for computer scientists. There has been extensive research in the field of computing education on the relationship between a student's ability to explain code and other skills such as writing and tracing code. In particular, the ability to describe at a high-level of abstraction how code will behave over all possible inputs correlates strongly with code writing skills. However, developing the expertise to comprehend and explain code accurately and succinctly is a challenge for many students. Existing pedagogical approaches that scaffold the ability to explain code, such as producing exemplar code explanations on demand, do not currently scale well to large classrooms. The recent emergence of powerful large language models (LLMs) may offer a solution. In this paper, we explore the potential of LLMs in generating explanations that can serve as examples to scaffold students' ability to understand and explain code. To evaluate LLM-created explanations, we compare them with explanations created by students in a large course ($n \approx 1000$) with respect to accuracy, understandability and length. We find that LLM-created explanations, which can be produced automatically on demand, are rated as being significantly easier to understand and more accurate summaries of code than student-created explanations. We discuss the significance of this finding, and suggest how such models can be incorporated into introductory programming education.

Related papers

• Improving Complex Reasoning over Knowledge Graph with Logic-Aware Curriculum Tuning [89.89857766491475]
  We propose a complex reasoning schema over KG upon large language models (LLMs) We augment the arbitrary first-order logical queries via binary tree decomposition to stimulate the reasoning capability of LLMs. Experiments across widely used datasets demonstrate that LACT has substantial improvements(brings an average +5.5% MRR score) over advanced methods.
  arXiv  Detail & Related papers  (2024-05-02T18:12:08Z)
• How Far Have We Gone in Binary Code Understanding Using Large Language Models [51.527805834378974]
  We propose a benchmark to evaluate the effectiveness of Large Language Models (LLMs) in binary code understanding. Our evaluations reveal that existing LLMs can understand binary code to a certain extent, thereby improving the efficiency of binary code analysis.
  arXiv  Detail & Related papers  (2024-04-15T14:44:08Z)
• Code Prompting Elicits Conditional Reasoning Abilities in Text+Code LLMs [65.2379940117181]
  We introduce code prompting, a chain of prompts that transforms a natural language problem into code. We find that code prompting exhibits a high-performance boost for multiple LLMs. Our analysis of GPT 3.5 reveals that the code formatting of the input problem is essential for performance improvement.
  arXiv  Detail & Related papers  (2024-01-18T15:32:24Z)
• Explaining Code Examples in Introductory Programming Courses: LLM vs Humans [1.6431142588286851]
  We assess the feasibility of using LLMs to generate code explanations for passive and active example exploration systems. To achieve this goal, we compare the code explanations generated by chatGPT with the explanations generated by both experts and students.
  arXiv  Detail & Related papers  (2023-12-09T01:06:08Z)
• The Behavior of Large Language Models When Prompted to Generate Code Explanations [0.3293989832773954]
  This paper systematically investigates the generation of code explanations by Large Language Models (LLMs) Our findings reveal significant variations in the nature of code explanations produced by LLMs, influenced by factors such as the wording of the prompt. A consistent pattern emerges for Java and Python, where explanations exhibit a Flesch-Kincaid readability level of approximately 7-8 grade.
  arXiv  Detail & Related papers  (2023-11-02T17:14:38Z)
• When Do Program-of-Thoughts Work for Reasoning? [51.2699797837818]
  We propose complexity-impacted reasoning score (CIRS) to measure correlation between code and reasoning abilities. Specifically, we use the abstract syntax tree to encode the structural information and calculate logical complexity. Code will be integrated into the EasyInstruct framework at https://github.com/zjunlp/EasyInstruct.
  arXiv  Detail & Related papers  (2023-08-29T17:22:39Z)
• Large Language Models are Few-Shot Summarizers: Multi-Intent Comment Generation via In-Context Learning [34.006227676170504]
  This study investigates the feasibility of utilizing large language models (LLMs) to generate comments that can fulfill developers' diverse intents. Experiments on two large-scale datasets demonstrate the rationale of our insights.
  arXiv  Detail & Related papers  (2023-04-22T12:26:24Z)
• Complementary Explanations for Effective In-Context Learning [77.83124315634386]
  Large language models (LLMs) have exhibited remarkable capabilities in learning from explanations in prompts. This work aims to better understand the mechanisms by which explanations are used for in-context learning.
  arXiv  Detail & Related papers  (2022-11-25T04:40:47Z)
• Learning to Scaffold: Optimizing Model Explanations for Teaching [74.25464914078826]
  We train models on three natural language processing and computer vision tasks. We find that students trained with explanations extracted with our framework are able to simulate the teacher significantly more effectively than ones produced with previous methods.
  arXiv  Detail & Related papers  (2022-04-22T16:43:39Z)
• Explanation as a process: user-centric construction of multi-level and multi-modal explanations [0.34410212782758043]
  We present a process-based approach that combines multi-level and multi-modal explanations. We use Inductive Logic Programming, an interpretable machine learning approach, to learn a comprehensible model.
  arXiv  Detail & Related papers  (2021-10-07T19:26:21Z)
• Logic Explained Networks [27.800583434727805]
  We show how a mindful design of the networks leads to a family of interpretable deep learning models called Logic Explained Networks (LENs) LENs only require their inputs to be human-understandable predicates, and they provide explanations in terms of simple First-Order Logic (FOL) formulas. LENs may yield better classifications than established white-box models, such as decision trees and Bayesian rule lists.
  arXiv  Detail & Related papers  (2021-08-11T10:55:42Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.