Exploring the Role of Security Experience and ChatGPT Usage Strategies on Secure Software Engineering Education

• URL: http://arxiv.org/abs/2609.11303v1
• Date: Thu, 10 Sep 2026 09:31:37 GMT
• Title: Exploring the Role of Security Experience and ChatGPT Usage Strategies on Secure Software Engineering Education
• Abstract summary: The rapid adoption of Large Language Models (LLMs) is reshaping software engineering education, but their role in secure software engineering education remains underexplored.<n>We report an exploratory empirical study of how 26 graduate students in a part-time MSc Cybersecurity programme used ChatGPT during a vulnerability-fixing assignment.
• Score: 3.6940298700319065
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: The rapid adoption of Large Language Models (LLMs) is reshaping software engineering education, but their role in secure software engineering education remains underexplored. We report an exploratory empirical study of how 26 graduate students in a part-time MSc Cybersecurity programme used ChatGPT during a vulnerability-fixing assignment. To characterise ChatGPT use, we analysed students' ChatGPT interaction logs using a structured double-coding procedure and examined whether usage patterns and prior cybersecurity expertise were associated with assignment performance. The results show that students with varying levels of cybersecurity expertise used broadly similar ChatGPT strategies. Individual usage patterns showed descriptive differences by grade, but none remained statistically significant after correcting for multiple comparisons. In contrast, diversity of ChatGPT usage, i.e., the number of distinct usage patterns adopted, was positively associated with performance, even after controlling for cybersecurity expertise. These exploratory findings suggest that the way students engage with ChatGPT may be more informative than whether they use it, and motivate future controlled studies to guide students toward effective LLM use in secure software engineering education.

Related papers

• Can Large Language Models Help Students Prove Software Correctness? An Experimental Study with Dafny [75.55915044740566]
  Students in computing education increasingly use large language models (LLMs) such as ChatGPT.<n>This paper investigates how students interact with an LLM when solving formal verification exercises in Dafny.
  arXiv  Detail & Related papers  (2025-06-27T16:34:13Z)
• A Qualitative Study on Using ChatGPT for Software Security: Perception vs. Practicality [1.7624347338410744]
  ChatGPT is a Large Language Model (LLM) that can perform a variety of tasks with remarkable semantic understanding and accuracy. This study aims to gain an understanding of the potential of ChatGPT as an emerging technology for supporting software security. It was determined that security practitioners view ChatGPT as beneficial for various software security tasks, including vulnerability detection, information retrieval, and penetration testing.
  arXiv  Detail & Related papers  (2024-08-01T10:14:05Z)
• Exploring the Impact of ChatGPT on Student Interactions in Computer-Supported Collaborative Learning [1.5961625979922607]
  This paper takes an initial step in exploring the applicability of ChatGPT in a computer-supported collaborative learning environment. Using statistical analysis, we validate the shifts in student interactions during an asynchronous group brainstorming session by introducing ChatGPT as an instantaneous question-answering agent.
  arXiv  Detail & Related papers  (2024-03-11T18:18:18Z)
• Exploring ChatGPT's Capabilities on Vulnerability Management [56.4403395100589]
  We explore ChatGPT's capabilities on 6 tasks involving the complete vulnerability management process with a large-scale dataset containing 70,346 samples. One notable example is ChatGPT's proficiency in tasks like generating titles for software bug reports. Our findings reveal the difficulties encountered by ChatGPT and shed light on promising future directions.
  arXiv  Detail & Related papers  (2023-11-11T11:01:13Z)
• Are Chatbots Reliable Text Annotators? Sometimes [0.0]
  ChatGPT is a closed-source product which has major drawbacks with regards to transparency, cost, and data protection.<n>Recent advances in open-source (OS) large language models (LLMs) offer an alternative without these drawbacks.
  arXiv  Detail & Related papers  (2023-11-09T22:28:14Z)
• ChatGPT as a Software Development Bot: A Project-based Study [5.518217604591736]
  This study examines the impact of generative AI tools, specifically ChatGPT, on the software development experiences of undergraduate students. Results showed that ChatGPT significantly addresses skill gaps in software development education, enhancing efficiency, accuracy, and collaboration.
  arXiv  Detail & Related papers  (2023-10-20T16:48:19Z)
• Evaluating the Impact of ChatGPT on Exercises of a Software Security Course [2.3017018980874617]
  ChatGPT can identify 20 of the 28 vulnerabilities we inserted in the web application in a white-box setting. ChatGPT makes nine satisfactory penetration testing and fixing recommendations for the ten vulnerabilities we want students to fix.
  arXiv  Detail & Related papers  (2023-09-18T18:53:43Z)
• Using ChatGPT as a Static Application Security Testing Tool [0.0]
  ChatGPT has caught a huge amount of attention with its remarkable performance. We study the feasibility of using ChatGPT for vulnerability detection in Python source code.
  arXiv  Detail & Related papers  (2023-08-28T09:21:37Z)
• Prompt-Enhanced Software Vulnerability Detection Using ChatGPT [9.35868869848051]
  Large language models (LLMs) like GPT have received considerable attention due to their stunning intelligence. This paper launches a study on the performance of software vulnerability detection using ChatGPT with different prompt designs.
  arXiv  Detail & Related papers  (2023-08-24T10:30:33Z)
• ChatGPT Beyond English: Towards a Comprehensive Evaluation of Large Language Models in Multilingual Learning [70.57126720079971]
  Large language models (LLMs) have emerged as the most important breakthroughs in natural language processing (NLP) This paper evaluates ChatGPT on 7 different tasks, covering 37 diverse languages with high, medium, low, and extremely low resources. Compared to the performance of previous models, our extensive experimental results demonstrate a worse performance of ChatGPT for different NLP tasks and languages.
  arXiv  Detail & Related papers  (2023-04-12T05:08:52Z)
• To ChatGPT, or not to ChatGPT: That is the question! [78.407861566006]
  This study provides a comprehensive and contemporary assessment of the most recent techniques in ChatGPT detection. We have curated a benchmark dataset consisting of prompts from ChatGPT and humans, including diverse questions from medical, open Q&A, and finance domains. Our evaluation results demonstrate that none of the existing methods can effectively detect ChatGPT-generated content.
  arXiv  Detail & Related papers  (2023-04-04T03:04:28Z)
• Safety Analysis in the Era of Large Language Models: A Case Study of STPA using ChatGPT [11.27440170845105]
  Using ChatGPT without human intervention may be inadequate due to reliability related issues, but with careful design, it may outperform human experts. No statistically significant differences are found when varying the semantic complexity or using common prompt guidelines.
  arXiv  Detail & Related papers  (2023-04-03T16:46:49Z)
• Consistency Analysis of ChatGPT [65.268245109828]
  This paper investigates the trustworthiness of ChatGPT and GPT-4 regarding logically consistent behaviour. Our findings suggest that while both models appear to show an enhanced language understanding and reasoning ability, they still frequently fall short of generating logically consistent predictions.
  arXiv  Detail & Related papers  (2023-03-11T01:19:01Z)
• A Categorical Archive of ChatGPT Failures [47.64219291655723]
  ChatGPT, developed by OpenAI, has been trained using massive amounts of data and simulates human conversation. It has garnered significant attention due to its ability to effectively answer a broad range of human inquiries. However, a comprehensive analysis of ChatGPT's failures is lacking, which is the focus of this study.
  arXiv  Detail & Related papers  (2023-02-06T04:21:59Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.