Effects of a Prompt Engineering Intervention on Undergraduate Students' AI Self-Efficacy, AI Knowledge and Prompt Engineering Ability: A Mixed Methods Study

• URL: http://arxiv.org/abs/2408.07302v1
• Date: Tue, 30 Jul 2024 15:05:24 GMT
• Title: Effects of a Prompt Engineering Intervention on Undergraduate Students' AI Self-Efficacy, AI Knowledge and Prompt Engineering Ability: A Mixed Methods Study
• Authors: David James Woo, Deliang Wang, Tim Yung, Kai Guo,
• Abstract summary: This study designed and implemented a prompt engineering intervention at a university in Hong Kong. It examined students' AI self-efficacy, AI knowledge, and proficiency in creating effective prompts.
• Score: 36.48421439947282
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: Prompt engineering is critical for effective interaction with large language models (LLMs) such as ChatGPT. However, efforts to teach this skill to students have been limited. This study designed and implemented a prompt engineering intervention, examining its influence on undergraduate students' AI self-efficacy, AI knowledge, and proficiency in creating effective prompts. The intervention involved 27 students who participated in a 100-minute workshop conducted during their history course at a university in Hong Kong. During the workshop, students were introduced to prompt engineering strategies, which they applied to plan the course's final essay task. Multiple data sources were collected, including students' responses to pre- and post-workshop questionnaires, pre- and post-workshop prompt libraries, and written reflections. The study's findings revealed that students demonstrated a higher level of AI self-efficacy, an enhanced understanding of AI concepts, and improved prompt engineering skills because of the intervention. These findings have implications for AI literacy education, as they highlight the importance of prompt engineering training for specific higher education use cases. This is a significant shift from students haphazardly and intuitively learning to engineer prompts. Through prompt engineering education, educators can faciitate students' effective navigation and leverage of LLMs to support their coursework.

Related papers

• Exploring Knowledge Tracing in Tutor-Student Dialogues [53.52699766206808]
  We present a first attempt at performing knowledge tracing (KT) in tutor-student dialogues. We propose methods to identify the knowledge components/skills involved in each dialogue turn. We then apply a range of KT methods on the resulting labeled data to track student knowledge levels over an entire dialogue.
  arXiv  Detail & Related papers  (2024-09-24T22:31:39Z)
• BoilerTAI: A Platform for Enhancing Instruction Using Generative AI in Educational Forums [0.0]
  This paper describes a practical, scalable platform that seamlessly integrates Generative AI (GenAI) with online educational forums. The platform empowers instructional staff to efficiently manage, refine, and approve responses by facilitating interaction between student posts and a Large Language Model (LLM)
  arXiv  Detail & Related papers  (2024-09-20T04:00:30Z)
• Explainable Few-shot Knowledge Tracing [48.877979333221326]
  We propose a cognition-guided framework that can track the student knowledge from a few student records while providing natural language explanations. Experimental results from three widely used datasets show that LLMs can perform comparable or superior to competitive deep knowledge tracing methods.
  arXiv  Detail & Related papers  (2024-05-23T10:07:21Z)
• Efficient Prompting Methods for Large Language Models: A Survey [50.171011917404485]
  Prompting has become a mainstream paradigm for adapting large language models (LLMs) to specific natural language processing tasks. This approach brings the additional computational burden of model inference and human effort to guide and control the behavior of LLMs. We present the basic concepts of prompting, review the advances for efficient prompting, and highlight future research directions.
  arXiv  Detail & Related papers  (2024-04-01T12:19:08Z)
• I would love this to be like an assistant, not the teacher: a voice of the customer perspective of what distance learning students want from an Artificial Intelligence Digital Assistant [0.0]
  This study examined the perceptions of ten online and distance learning students regarding the design of a hypothetical AI Digital Assistant (AIDA) All participants agreed on the usefulness of such an AI tool while studying and reported benefits from using it for real-time assistance and query resolution, support for academic tasks, personalisation and accessibility, together with emotional and social support. Students' concerns related to the ethical and social implications of implementing AIDA, data privacy and data use, operational challenges, academic integrity and misuse, and the future of education.
  arXiv  Detail & Related papers  (2024-02-16T08:10:41Z)
• How Far Are We? The Triumphs and Trials of Generative AI in Learning Software Engineering [16.5141990552784]
  We evaluate the effectiveness of ChatGPT, a convo-genAI platform, in assisting students in Software Engineering tasks. Our study did not find statistical differences in participants' productivity or self-efficacy when using ChatGPT as compared to traditional resources. Our study also revealed 5 distinct faults arising from violations of Human-AI interaction guidelines, which led to 7 different (negative) consequences on participants.
  arXiv  Detail & Related papers  (2023-12-18T21:38:00Z)
• Understanding Teacher Perspectives and Experiences after Deployment of AI Literacy Curriculum in Middle-school Classrooms [12.35885897302579]
  We investigate the experiences of seven teachers following their implementation of modules from the MIT RAICA curriculum. Our analysis suggests that the AI modules expanded our teachers' knowledge in the field. Our teachers advocated their need for better external support when navigating technological resources.
  arXiv  Detail & Related papers  (2023-12-08T05:36:16Z)
• Mastering Robot Manipulation with Multimodal Prompts through Pretraining and Multi-task Fine-tuning [49.92517970237088]
  We tackle the problem of training a robot to understand multimodal prompts. This type of task poses a major challenge to robots' capability to understand the interconnection and complementarity between vision and language signals. We introduce an effective framework that learns a policy to perform robot manipulation with multimodal prompts.
  arXiv  Detail & Related papers  (2023-10-14T22:24:58Z)
• TecCoBot: Technology-aided support for self-regulated learning [52.77024349608834]
  Self-study activities can increase the degree of activity and the contribution of self-study activities to the achievement of learning outcomes. Especially in times of a global pandemic, self-study activities are increasingly executed at home, where students already use technology-enhanced materials, processes, and digital platforms.
  arXiv  Detail & Related papers  (2021-11-23T13:50:21Z)
• Teaching Tech to Talk: K-12 Conversational Artificial Intelligence Literacy Curriculum and Development Tools [9.797319790710711]
  We evaluate our Conversational Agent Interface for MIT App Inventor and workshop curriculum with respect to AI competencies. We found students struggled most with the concepts of AI ethics and learning, and recommend emphasizing these topics when teaching.
  arXiv  Detail & Related papers  (2020-09-11T20:52:46Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.