AI In Cybersecurity Education -- Scalable Agentic CTF Design Principles and Educational Outcomes

• URL: http://arxiv.org/abs/2603.21551v2
• Date: Tue, 31 Mar 2026 16:10:26 GMT
• Title: AI In Cybersecurity Education -- Scalable Agentic CTF Design Principles and Educational Outcomes
• Abstract summary: This paper presents a study of Capture-the-Flag competitions built on the Cyber Security Awareness Week competition system.<n>We formalize three autonomy levels: human-in-the-loop, autonomous agent frameworks, and hybrid.<n>Our results offer actionable guidance for designing LLM-assisted cybersecurity competitions as learning technologies.
• Score: 25.913186214869242
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Large language models are rapidly changing how learners acquire and demonstrate cybersecurity skills. However, when human--AI collaboration is allowed, educators still lack validated competition designs and evaluation practices that remain fair and evidence-based. This paper presents a cross-regional study of LLM-centered Capture-the-Flag competitions built on the Cyber Security Awareness Week competition system. To understand how autonomy levels and participants' knowledge backgrounds influence problem-solving performance and learning-related behaviors, we formalize three autonomy levels: human-in-the-loop, autonomous agent frameworks, and hybrid. To enable verification, we require traceable submissions including conversation logs, agent trajectories, and agent code. We analyze multi-region competition data covering an in-class track, a standard track, and a year-long expert track, each targeting participants with different knowledge backgrounds. Using data from the 2025 competition, we compare solve performance across autonomy levels and challenge categories, and observe that autonomous agent frameworks and hybrid achieve higher completion rates on challenges requiring iterative testing and tool interactions. In the in-class track, we classify participants' agent designs and find a preference for lightweight, tool-augmented prompting and reflection-based retries over complex multi-agent architectures. Our results offer actionable guidance for designing LLM-assisted cybersecurity competitions as learning technologies, including autonomy-specific scoring criteria, evidence requirements that support solution verification, and track structures that improve accessibility while preserving reliable evaluation and engagement.

Related papers

• When Agentic AI Meets Integrated Sensing and Communication [23.084474668808912]
  Agentic artificial intelligence (AI) is transforming Integrated Sensing and Communication (ISAC) into a goal-driven, closed-loop intelligent system.<n>This survey unifies the literature within a six-stage closed-loop framework.<n>It introduces five levels of agentic maturity, ranging from physical-layer primitives to fully closed-loop ISAC.
  arXiv  Detail & Related papers  (2026-08-06T09:26:49Z)
• The Disruptive Impact of Large Language Models on Capture the Flag Competitions and the Path Toward Fair Play [0.0]
  Capture the Flag competitions are among cybersecurity's most effective training grounds.<n>LLMs can solve a growing share of challenges with minimal human input.<n>This paper reports a mixed-methods study of LLM impact on modern CTFs.
  arXiv  Detail & Related papers  (2026-07-28T08:21:41Z)
• Sandpiper: Orchestrated AI-Annotation for Educational Discourse at Scale [3.0317183198617066]
  We present Sandpiper, a mixed-initiative system designed to serve as a bridge between high-volume conversational data and human qualitative expertise.<n>We propose a user study to evaluate the system's efficacy in improving research efficiency, inter-rater reliability, and researcher trust in AI-assisted qualitative analysis.
  arXiv  Detail & Related papers  (2026-03-09T14:01:42Z)
• Can AI Lower the Barrier to Cybersecurity? A Human-Centered Mixed-Methods Study of Novice CTF Learning [0.0]
  Agentic AI frameworks for cybersecurity promise to lower barriers by automating and coordinating penetration testing tasks.<n>We present a human-centered, mixed-methods case study examining how agentic AI frameworks mediates novice entry into CTF-based penetration testing.
  arXiv  Detail & Related papers  (2026-02-20T12:20:36Z)
• SelfAI: Building a Self-Training AI System with LLM Agents [79.10991818561907]
  SelfAI is a general multi-agent platform that combines a User Agent for translating high-level research objectives into standardized experimental configurations.<n>An Experiment Manager orchestrates parallel, fault-tolerant training across heterogeneous hardware while maintaining a structured knowledge base for continuous feedback.<n>Across regression, computer vision, scientific computing, medical imaging, and drug discovery benchmarks, SelfAI consistently achieves strong performance and reduces redundant trials.
  arXiv  Detail & Related papers  (2025-11-29T09:18:39Z)
• CATArena: Evaluation of LLM Agents through Iterative Tournament Competitions [49.02422075498554]
  Large Language Model (LLM) agents have evolved from basic text generation to autonomously completing complex tasks through interaction with external tools.<n>In this work, we emphasize the importance of learning ability, including both self-improvement and peer-learning, as a core driver for agent evolution toward human-level intelligence.<n>We propose an iterative, competitive peer-learning framework, which allows agents to refine and optimize their strategies through repeated interactions and feedback.
  arXiv  Detail & Related papers  (2025-10-30T15:22:53Z)
• A cybersecurity AI agent selection and decision support framework [0.0]
  This paper presents a novel, structured decision support framework that aligns AI agent architectures, reactive, cognitive, hybrid, and learning.<n>By integrating agent theory with industry guidelines, this framework provides a transparent and stepwise methodology for selecting and deploying AI solutions.
  arXiv  Detail & Related papers  (2025-10-02T07:38:21Z)
• UserRL: Training Interactive User-Centric Agent via Reinforcement Learning [104.63494870852894]
  Reinforcement learning (RL) has shown promise in training agentic models that engage in dynamic, multi-turn interactions.<n>We propose UserRL, a unified framework for training and evaluating user-centric abilities through standardized gym environments.
  arXiv  Detail & Related papers  (2025-09-24T03:33:20Z)
• Bridging LLMs and Symbolic Reasoning in Educational QA Systems: Insights from the XAI Challenge at IJCNN 2025 [1.0166217771455643]
  This paper presents a comprehensive analysis of the XAI Challenge 2025, a hackathon-style competition jointly organized by Ho Chi Minh City University of Technology (HCMUT) and the International Workshop on Trustworthiness and Reliability in Neurosymbolic AI (TRNS-AI)<n>The challenge tasked participants with building Question-Answering (QA) systems capable of answering student queries about university policies while generating clear, logic-based natural language explanations.
  arXiv  Detail & Related papers  (2025-08-02T08:46:06Z)
• Cognitive Kernel-Pro: A Framework for Deep Research Agents and Agent Foundation Models Training [67.895981259683]
  General AI Agents are increasingly recognized as foundational frameworks for the next generation of artificial intelligence.<n>Current agent systems are either closed-source or heavily reliant on a variety of paid APIs and proprietary tools.<n>We present Cognitive Kernel-Pro, a fully open-source and (to the maximum extent) free multi-module agent framework.
  arXiv  Detail & Related papers  (2025-08-01T08:11:31Z)
• Benchmarking Robustness and Generalization in Multi-Agent Systems: A Case Study on Neural MMO [50.58083807719749]
  We present the results of the second Neural MMO challenge, hosted at IJCAI 2022, which received 1600+ submissions. This competition targets robustness and generalization in multi-agent systems. We will open-source our benchmark including the environment wrapper, baselines, a visualization tool, and selected policies for further research.
  arXiv  Detail & Related papers  (2023-08-30T07:16:11Z)
• Federated Learning-Empowered AI-Generated Content in Wireless Networks [58.48381827268331]
  Federated learning (FL) can be leveraged to improve learning efficiency and achieve privacy protection for AIGC. We present FL-based techniques for empowering AIGC, and aim to enable users to generate diverse, personalized, and high-quality content.
  arXiv  Detail & Related papers  (2023-07-14T04:13:11Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.