SoK: DARPA's AI Cyber Challenge (AIxCC): Competition Design, Architectures, and Lessons Learned

• URL: http://arxiv.org/abs/2602.07666v1
• Date: Sat, 07 Feb 2026 19:21:27 GMT
• Title: SoK: DARPA's AI Cyber Challenge (AIxCC): Competition Design, Architectures, and Lessons Learned
• Authors: Cen Zhang, Younggi Park, Fabian Fleischer, Yu-Fu Fu, Jiho Kim, Dongkwan Kim, Youngjoon Kim, Qingxiao Xu, Andrew Chin, Ze Sheng, Hanqing Zhao, Brian J. Lee, Joshua Wang, Michael Pelican, David J. Musliner, Jeff Huang, Jon Silliman, Mikel Mcdaniel, Jefferson Casavant, Isaac Goldthwaite, Nicholas Vidovich, Matthew Lehman, Taesoo Kim,
• Abstract summary: Drawing on design documents, source code, execution traces, and discussions with organizers and competing teams, we examine the competition's structure and key design decisions.<n>Our analysis reveals the factors that truly drove CRS performance, identifies genuine technical advances achieved by teams, and exposes limitations that remain open for future research.
• Score: 15.167322082236907
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: DARPA's AI Cyber Challenge (AIxCC, 2023--2025) is the largest competition to date for building fully autonomous cyber reasoning systems (CRSs) that leverage recent advances in AI -- particularly large language models (LLMs) -- to discover and remediate vulnerabilities in real-world open-source software. This paper presents the first systematic analysis of AIxCC. Drawing on design documents, source code, execution traces, and discussions with organizers and competing teams, we examine the competition's structure and key design decisions, characterize the architectural approaches of finalist CRSs, and analyze competition results beyond the final scoreboard. Our analysis reveals the factors that truly drove CRS performance, identifies genuine technical advances achieved by teams, and exposes limitations that remain open for future research. We conclude with lessons for organizing future competitions and broader insights toward deploying autonomous CRSs in practice.

Related papers

• Advances and Frontiers of LLM-based Issue Resolution in Software Engineering: A Comprehensive Survey [59.3507264893654]
  Issue resolution is a complex Software Engineering task integral to real-world development.<n> benchmarks like SWE-bench revealed this task as profoundly difficult for large language models.<n>This paper presents a systematic survey of this emerging domain.
  arXiv  Detail & Related papers  (2026-01-15T18:55:03Z)
• Let the Barbarians In: How AI Can Accelerate Systems Performance Research [80.43506848683633]
  We term this iterative cycle of generation, evaluation, and refinement AI-Driven Research for Systems.<n>We demonstrate that ADRS-generated solutions can match or even outperform human state-of-the-art designs.
  arXiv  Detail & Related papers  (2025-12-16T18:51:23Z)
• ATLANTIS: AI-driven Threat Localization, Analysis, and Triage Intelligence System [24.15037724921127]
  We present ATLANTIS, the cyber reasoning system developed by Team Atlanta that won 1st place in the Final Competition of DARPA's AI Cyber Challenge (AIxCC)<n>We detail the design philosophy, architectural decisions, and implementation strategies behind ATLANTIS.<n>We share lessons learned from pushing the boundaries of automated security when program analysis meets modern AI.
  arXiv  Detail & Related papers  (2025-09-18T03:46:18Z)
• Bridging LLMs and Symbolic Reasoning in Educational QA Systems: Insights from the XAI Challenge at IJCNN 2025 [1.0166217771455643]
  This paper presents a comprehensive analysis of the XAI Challenge 2025, a hackathon-style competition jointly organized by Ho Chi Minh City University of Technology (HCMUT) and the International Workshop on Trustworthiness and Reliability in Neurosymbolic AI (TRNS-AI)<n>The challenge tasked participants with building Question-Answering (QA) systems capable of answering student queries about university policies while generating clear, logic-based natural language explanations.
  arXiv  Detail & Related papers  (2025-08-02T08:46:06Z)
• A Framework for Evaluating Emerging Cyberattack Capabilities of AI [11.595840449117052]
  This work introduces a novel evaluation framework that addresses limitations by: (1) examining the end-to-end attack chain, (2) identifying gaps in AI threat evaluation, and (3) helping defenders prioritize targeted mitigations.<n>We analyzed over 12,000 real-world instances of AI involvement in cyber incidents, catalogued by Google's Threat Intelligence Group, to curate seven representative attack chain archetypes.<n>We report on AI's potential to amplify offensive capabilities across specific attack stages, and offer recommendations for prioritizing defenses.
  arXiv  Detail & Related papers  (2025-03-14T23:05:02Z)
• Cyber Deception: State of the art, Trends and Open challenges [0.5735035463793009]
  This article develops a comprehensive classification taxonomy for Cyber Deception (CYDEC) The different frameworks used to generate CYDEC are reviewed, presenting a more comprehensive one. Existing solutions in the literature using CYDEC, both without Artificial Intelligence (AI) and with AI, are studied and compared.
  arXiv  Detail & Related papers  (2024-09-11T11:31:34Z)
• Combining AI Control Systems and Human Decision Support via Robustness and Criticality [53.10194953873209]
  We extend a methodology for adversarial explanations (AE) to state-of-the-art reinforcement learning frameworks. We show that the learned AI control system demonstrates robustness against adversarial tampering. In a training / learning framework, this technology can improve both the AI's decisions and explanations through human interaction.
  arXiv  Detail & Related papers  (2024-07-03T15:38:57Z)
• Coordinated Flaw Disclosure for AI: Beyond Security Vulnerabilities [1.3225694028747144]
  We propose a Coordinated Flaw Disclosure framework tailored to the complexities of machine learning (ML) issues. Our framework introduces innovations such as extended model cards, dynamic scope expansion, an independent adjudication panel, and an automated verification process. We argue that CFD could significantly enhance public trust in AI systems.
  arXiv  Detail & Related papers  (2024-02-10T20:39:04Z)
• Benchmarking Robustness and Generalization in Multi-Agent Systems: A Case Study on Neural MMO [50.58083807719749]
  We present the results of the second Neural MMO challenge, hosted at IJCAI 2022, which received 1600+ submissions. This competition targets robustness and generalization in multi-agent systems. We will open-source our benchmark including the environment wrapper, baselines, a visualization tool, and selected policies for further research.
  arXiv  Detail & Related papers  (2023-08-30T07:16:11Z)
• Artificial Intelligence Security Competition (AISC) [52.20676747225118]
  The Artificial Intelligence Security Competition (AISC) was organized by the Zhongguancun Laboratory, China Industrial Control Systems Cyber Emergency Response Team, Institute for Artificial Intelligence, Tsinghua University, and RealAI. The competition consists of three tracks, including Deepfake Security Competition, Autonomous Driving Security Competition, and Face Recognition Security Competition. This report will introduce the competition rules of these three tracks and the solutions of top-ranking teams in each track.
  arXiv  Detail & Related papers  (2022-12-07T02:45:27Z)
• NeurIPS 2020 EfficientQA Competition: Systems, Analyses and Lessons Learned [122.429985063391]
  We describe the motivation and organization of the EfficientQA competition from NeurIPS 2020. The competition focused on open-domain question answering (QA), where systems take natural language questions as input and return natural language answers.
  arXiv  Detail & Related papers  (2021-01-01T01:24:34Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.