Generative AI Misuse Potential in Cyber Security Education: A Case Study of a UK Degree Program

• URL: http://arxiv.org/abs/2501.12883v3
• Date: Fri, 24 Jan 2025 15:27:44 GMT
• Title: Generative AI Misuse Potential in Cyber Security Education: A Case Study of a UK Degree Program
• Authors: Carlton Shepherd,
• Abstract summary: This paper investigates the susceptibility of a Master's-level cyber security degree program at a UK Russell Group university to LLM misuse. We identify a high exposure to misuse, particularly in independent project- and report-based assessments. To address these challenges, we discuss the adoption of LLM-resistant assessments, detection tools, and the importance of fostering an ethical learning environment.
• Score: 1.9217872171227137
• License:
• Abstract: Recent advances in generative artificial intelligence (AI), such as ChatGPT, Google Gemini, and other large language models (LLMs), pose significant challenges to upholding academic integrity in higher education. This paper investigates the susceptibility of a Master's-level cyber security degree program at a UK Russell Group university, accredited by a leading national body, to LLM misuse. Through the application and extension of a quantitative assessment framework, we identify a high exposure to misuse, particularly in independent project- and report-based assessments. Contributing factors, including block teaching and a predominantly international cohort, are highlighted as potential amplifiers of these vulnerabilities. To address these challenges, we discuss the adoption of LLM-resistant assessments, detection tools, and the importance of fostering an ethical learning environment. These approaches aim to uphold academic standards while preparing students for the complexities of real-world cyber security.

Related papers

• LLM Cyber Evaluations Don't Capture Real-World Risk [0.0]
  Large language models (LLMs) are demonstrating increasing prowess in cybersecurity applications. We argue that current efforts to evaluate risks posed by these capabilities are misaligned with the goal of understanding real-world impact.
  arXiv  Detail & Related papers  (2025-01-31T05:33:48Z)
• Open Problems in Machine Unlearning for AI Safety [61.43515658834902]
  Machine unlearning -- the ability to selectively forget or suppress specific types of knowledge -- has shown promise for privacy and data removal tasks. In this paper, we identify key limitations that prevent unlearning from serving as a comprehensive solution for AI safety.
  arXiv  Detail & Related papers  (2025-01-09T03:59:10Z)
• Ontology-Aware RAG for Improved Question-Answering in Cybersecurity Education [13.838970688067725]
  AI-driven question-answering (QA) systems can actively manage uncertainty in cybersecurity problem-solving. Large language models (LLMs) have gained prominence in AI-driven QA systems, offering advanced language understanding and user engagement. We propose CyberRAG, an ontology-aware retrieval-augmented generation (RAG) approach for developing a reliable and safe QA system in cybersecurity education.
  arXiv  Detail & Related papers  (2024-12-10T21:52:35Z)
• Global Challenge for Safe and Secure LLMs Track 1 [57.08717321907755]
  The Global Challenge for Safe and Secure Large Language Models (LLMs) is a pioneering initiative organized by AI Singapore (AISG) and the CyberSG R&D Programme Office (CRPO) This paper introduces the Global Challenge for Safe and Secure Large Language Models (LLMs), a pioneering initiative organized by AI Singapore (AISG) and the CyberSG R&D Programme Office (CRPO) to foster the development of advanced defense mechanisms against automated jailbreaking attacks.
  arXiv  Detail & Related papers  (2024-11-21T08:20:31Z)
• Exploring the Cybersecurity-Resilience Gap: An Analysis of Student Attitudes and Behaviors in Higher Education [0.0]
  This study addresses the gap using the Theory of Behavior as a theoretical framework. A modified Human Aspects of Information Security Questionnaire was employed to gather 266 valid responses from undergraduate and postgraduate students. Key dimensions of cybersecurity awareness and behavior, including password management, email usage, social media practices, and mobile device security, were assessed.
  arXiv  Detail & Related papers  (2024-11-05T16:09:37Z)
• EARBench: Towards Evaluating Physical Risk Awareness for Task Planning of Foundation Model-based Embodied AI Agents [53.717918131568936]
  Embodied artificial intelligence (EAI) integrates advanced AI models into physical entities for real-world interaction. Foundation models as the "brain" of EAI agents for high-level task planning have shown promising results. However, the deployment of these agents in physical environments presents significant safety challenges. This study introduces EARBench, a novel framework for automated physical risk assessment in EAI scenarios.
  arXiv  Detail & Related papers  (2024-08-08T13:19:37Z)
• Could ChatGPT get an Engineering Degree? Evaluating Higher Education Vulnerability to AI Assistants [176.39275404745098]
  We evaluate whether two AI assistants, GPT-3.5 and GPT-4, can adequately answer assessment questions. GPT-4 answers an average of 65.8% of questions correctly, and can even produce the correct answer across at least one prompting strategy for 85.1% of questions. Our results call for revising program-level assessment design in higher education in light of advances in generative AI.
  arXiv  Detail & Related papers  (2024-08-07T12:11:49Z)
• The WMDP Benchmark: Measuring and Reducing Malicious Use With Unlearning [87.1610740406279]
  White House Executive Order on Artificial Intelligence highlights the risks of large language models (LLMs) empowering malicious actors in developing biological, cyber, and chemical weapons. Current evaluations are private, preventing further research into mitigating risk. We publicly release the Weapons of Mass Destruction Proxy benchmark, a dataset of 3,668 multiple-choice questions.
  arXiv  Detail & Related papers  (2024-03-05T18:59:35Z)
• Prioritizing Safeguarding Over Autonomy: Risks of LLM Agents for Science [65.77763092833348]
  Intelligent agents powered by large language models (LLMs) have demonstrated substantial promise in autonomously conducting experiments and facilitating scientific discoveries across various disciplines. While their capabilities are promising, these agents also introduce novel vulnerabilities that demand careful consideration for safety. This paper conducts a thorough examination of vulnerabilities in LLM-based agents within scientific domains, shedding light on potential risks associated with their misuse and emphasizing the need for safety measures.
  arXiv  Detail & Related papers  (2024-02-06T18:54:07Z)
• Large Language Models in Cybersecurity: State-of-the-Art [4.990712773805833]
  The rise of Large Language Models (LLMs) has revolutionized our comprehension of intelligence bringing us closer to Artificial Intelligence. This study examines the existing literature, providing a thorough characterization of both defensive and adversarial applications of LLMs within the realm of cybersecurity.
  arXiv  Detail & Related papers  (2024-01-30T16:55:25Z)
• Building Resilient SMEs: Harnessing Large Language Models for Cyber Security in Australia [0.0]
  Small and medium-sized enterprises (SMEs) in Australia are experiencing increased vulnerability to cyber threats. Artificial Intelligence (AI), Machine Learning (ML) and Large Language Models (LLMs) can potentially strengthen cyber security policies for Australian SMEs. This study provides a comprehensive understanding of the potential role of LLMs in enhancing cyber security policies for Australian SMEs.
  arXiv  Detail & Related papers  (2023-06-05T06:01:00Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.