The Disruptive Impact of Large Language Models on Capture the Flag Competitions and the Path Toward Fair Play

• URL: http://arxiv.org/abs/2607.25425v1
• Date: Tue, 28 Jul 2026 08:21:41 GMT
• Title: The Disruptive Impact of Large Language Models on Capture the Flag Competitions and the Path Toward Fair Play
• Abstract summary: Capture the Flag competitions are among cybersecurity's most effective training grounds.<n>LLMs can solve a growing share of challenges with minimal human input.<n>This paper reports a mixed-methods study of LLM impact on modern CTFs.
• Score: 0.0
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Capture the Flag (CTF) competitions are among cybersecurity's most effective training grounds, developing practical skill across cryptography, web exploitation, and binary exploitation. Large language models (LLMs) can now solve a growing share of challenges with minimal human input, raising urgent questions about fairness, the validity of rankings, and whether participation still delivers the learning that justifies the effort. This paper reports a mixed-methods study of LLM impact on modern CTFs, combining a synthesis of published benchmarks, including a recent government evaluation, case studies of live competition across three challenge categories, structured observation of the public channels where the community debates AI use, and semi-structured interviews with experienced players and organisers. We map the current human-machine capability boundary by category, showing that easy and intermediate challenges in cryptography, web, and binary exploitation are now reliably automated while narrower sub-categories continue to resist. We find that community disagreement about whether AI should be permitted is downstream of an undeclared prior question: what a competition is for. Against this backdrop we contribute a four-component safeguard framework, combining tiered competition divisions, LLM-resistant challenge design, telemetry used investigatively, and a draft community code of conduct, together with a decision tool that ties the combination of safeguards to a competition's declared purpose. The argument reaches beyond CTFs to any setting in cybersecurity where a demonstrated result is taken as evidence of an underlying ability.

Related papers

• AI In Cybersecurity Education -- Scalable Agentic CTF Design Principles and Educational Outcomes [25.913186214869242]
  This paper presents a study of Capture-the-Flag competitions built on the Cyber Security Awareness Week competition system.<n>We formalize three autonomy levels: human-in-the-loop, autonomous agent frameworks, and hybrid.<n>Our results offer actionable guidance for designing LLM-assisted cybersecurity competitions as learning technologies.
  arXiv  Detail & Related papers  (2026-03-23T04:05:59Z)
• Can AI Lower the Barrier to Cybersecurity? A Human-Centered Mixed-Methods Study of Novice CTF Learning [0.0]
  Agentic AI frameworks for cybersecurity promise to lower barriers by automating and coordinating penetration testing tasks.<n>We present a human-centered, mixed-methods case study examining how agentic AI frameworks mediates novice entry into CTF-based penetration testing.
  arXiv  Detail & Related papers  (2026-02-20T12:20:36Z)
• KryptoPilot: An Open-World Knowledge-Augmented LLM Agent for Automated Cryptographic Exploitation [16.43451504898208]
  KryptoPilot is an open-world knowledge-augmented LLM agent for automated cryptographic exploitation.<n>We show that KryptoPilot achieves a complete solve rate on InterCode-CTF, solves between 56 and 60 percent of cryptographic challenges on the NYU-CTF benchmark, and successfully solves 26 out of 33 cryptographic challenges in live competitions.
  arXiv  Detail & Related papers  (2026-01-14T04:02:40Z)
• The Hunger Game Debate: On the Emergence of Over-Competition in Multi-Agent Systems [90.96738882568224]
  This paper investigates the over-competition in multi-agent debate, where agents under extreme pressure exhibit unreliable, harmful behaviors.<n>To study this phenomenon, we propose HATE, a novel experimental framework that simulates debates under a zero-sum competition arena.
  arXiv  Detail & Related papers  (2025-09-30T11:44:47Z)
• ICPC-Eval: Probing the Frontiers of LLM Reasoning with Competitive Programming Contests [85.72404266850982]
  We propose textbfICPC-Eval, a top-level competitive coding benchmark designed to probing the frontiers of reasoning.<n>ICPC-Eval includes 118 carefully curated problems from 11 recent ICPC contests held in various regions of the world.<n>Results underscore the significant challenge in evaluating complex reasoning abilities.
  arXiv  Detail & Related papers  (2025-06-05T11:20:37Z)
• A Human Study of Cognitive Biases in Web Application Security [5.535195078929509]
  This paper investigates how cognitive biases could be used to improve Capture the Flag education and security.<n>We present an approach to control cognitive biases, specifically Satisfaction of Search and Loss Aversion.<n>Our study reveals that many participants exhibit the Satisfaction of Search bias and that this bias has a significant effect on their success.
  arXiv  Detail & Related papers  (2025-05-17T14:16:16Z)
• Towards Improving IDS Using CTF Events [1.812535004393714]
  This paper introduces a novel approach to evaluating Intrusion Detection Systems (IDS) through Capture the Flag (CTF) events.<n>Our research investigates the effectiveness of using tailored CTF challenges to identify weaknesses in IDS by integrating them into live CTF competitions.<n>We present a methodology that supports the development of IDS-specific challenges, a scoring system that fosters learning and engagement, and the insights of running such a challenge in a real Jeopardy-style CTF event.
  arXiv  Detail & Related papers  (2025-01-20T19:11:30Z)
• CompeteSMoE -- Effective Training of Sparse Mixture of Experts via Competition [52.2034494666179]
  Sparse mixture of experts (SMoE) offers an appealing solution to scale up the model complexity beyond the mean of increasing the network's depth or width. We propose a competition mechanism to address this fundamental challenge of representation collapse. By routing inputs only to experts with the highest neural response, we show that, under mild assumptions, competition enjoys the same convergence rate as the optimal estimator.
  arXiv  Detail & Related papers  (2024-02-04T15:17:09Z)
• Benchmarking Robustness and Generalization in Multi-Agent Systems: A Case Study on Neural MMO [50.58083807719749]
  We present the results of the second Neural MMO challenge, hosted at IJCAI 2022, which received 1600+ submissions. This competition targets robustness and generalization in multi-agent systems. We will open-source our benchmark including the environment wrapper, baselines, a visualization tool, and selected policies for further research.
  arXiv  Detail & Related papers  (2023-08-30T07:16:11Z)
• Combating Exacerbated Heterogeneity for Robust Models in Federated Learning [91.88122934924435]
  Combination of adversarial training and federated learning can lead to the undesired robustness deterioration. We propose a novel framework called Slack Federated Adversarial Training (SFAT) We verify the rationality and effectiveness of SFAT on various benchmarked and real-world datasets.
  arXiv  Detail & Related papers  (2023-03-01T06:16:15Z)
• Adversarial Attacks on ML Defense Models Competition [82.37504118766452]
  The TSAIL group at Tsinghua University and the Alibaba Security group organized this competition. The purpose of this competition is to motivate novel attack algorithms to evaluate adversarial robustness.
  arXiv  Detail & Related papers  (2021-10-15T12:12:41Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.