CVE-Factory: Scaling Expert-Level Agentic Tasks for Code Security Vulnerability

• URL: http://arxiv.org/abs/2602.03012v1
• Date: Tue, 03 Feb 2026 02:27:16 GMT
• Title: CVE-Factory: Scaling Expert-Level Agentic Tasks for Code Security Vulnerability
• Abstract summary: We present CVE-Factory, the first multiagent framework to achieve expert-level quality in automatically transforming vulnerability tasks.<n>It is also evaluated on the latest realistic vulnerabilities and achieves a 66.2% verified success.<n>We synthesize over 1,000 executable training environments, the first large-scale scaling of agentic tasks in code security.
• Score: 50.57373283154859
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Evaluating and improving the security capabilities of code agents requires high-quality, executable vulnerability tasks. However, existing works rely on costly, unscalable manual reproduction and suffer from outdated data distributions. To address these, we present CVE-Factory, the first multi-agent framework to achieve expert-level quality in automatically transforming sparse CVE metadata into fully executable agentic tasks. Cross-validation against human expert reproductions shows that CVE-Factory achieves 95\% solution correctness and 96\% environment fidelity, confirming its expert-level quality. It is also evaluated on the latest realistic vulnerabilities and achieves a 66.2\% verified success. This automation enables two downstream contributions. First, we construct LiveCVEBench, a continuously updated benchmark of 190 tasks spanning 14 languages and 153 repositories that captures emerging threats including AI-tooling vulnerabilities. Second, we synthesize over 1,000 executable training environments, the first large-scale scaling of agentic tasks in code security. Fine-tuned Qwen3-32B improves from 5.3\% to 35.8\% on LiveCVEBench, surpassing Claude 4.5 Sonnet, with gains generalizing to Terminal Bench (12.5\% to 31.3\%). We open-source CVE-Factory, LiveCVEBench, Abacus-cve (fine-tuned model), training dataset, and leaderboard. All resources are available at https://github.com/livecvebench/CVE-Factory .

Related papers

• CyberFactory: Scaling Cyber Security Capabilities with Instances from the Wild [49.44069582581181]
  CyberFactory transforms public vulnerability artifacts, including CVEs from the wild, into executable and verifiable task instances.<n>It uses a reusable vulnerability-analysis skill to guide the teacher through source inspection, problem solving with domain prior, and evidence-based validation.<n>On CyberGym, modelname reaches 52.4% Pass@1 under a one-hour budget, improving over its Qwen3.5 base model by +22.8 points.
  arXiv  Detail & Related papers  (2026-08-24T12:30:10Z)
• VICBench: A Multi-Language Benchmark for Code Vulnerability Detection [12.72412925135797]
  Existing vulnerability datasets suffer from limited programming language coverage, restricted patch complexity, and narrow project scope.<n>We create a benchmark - VICBench - of 100 verified VICs for 100 CVEs across 88 projects in Python, Java, and C++, covering 48 CWE types.<n> VICBench features complex real-world vulnerability fixes averaging 38.6 lines and corresponding VICs of 252.5 lines - significantly larger than prior work.
  arXiv  Detail & Related papers  (2026-08-12T16:45:49Z)
• CyberForge: Verified Vulnerability Injection at Repository Level for Cybersecurity Agent Training [42.087410274052836]
  CyberForge is a framework that synthesizes executable, repository-level security training data by injecting vulnerabilities into real C/C++ projects.<n>The resulting corpus holds 1034 validated vulnerabilities across 80 projects and 63 weakness categories.<n>Fine-tuning on trajectories collected over this corpus improves SEC-bench patch repair by +3.3 to +14.7 points.
  arXiv  Detail & Related papers  (2026-08-06T18:03:27Z)
• KAT-Coder-V2.5 Technical Report [61.907486544595336]
  KAT-Coder-V2.5 is a coding-focused agentic model trained to act autonomously inside real, executable repositories.<n>It reconstructs multilingual repositories into sandboxed environments with fail-to-pass and pass-to-pass verification at scale.<n>We further scale reinforcement learning with harness randomization, a reliability-hardened sandbox, an asymmetric actor--critic PPO with hindsight-augmented value estimation, and a harness-oriented reward framework.
  arXiv  Detail & Related papers  (2026-07-06T08:14:02Z)
• SEC-bench Pro: Can Language Models Solve Long-Horizon Software Security Tasks? [26.62307092688903]
  SEC-bench Pro is a benchmark for measuring agent bug hunting on critical, high-complexity software systems.<n>We instantiate SEC-bench Pro with 183 validated vulnerabilities across V8 and SpiderMonkey.<n>Our evaluation shows that coding agents with frontier models remain below 40% success on both evaluated engines.
  arXiv  Detail & Related papers  (2026-05-26T04:59:49Z)
• FuzzingBrain V2: A Multi-Agent LLM System for Automated Vulnerability Discovery and Reproduction [7.802658978010212]
  Software vulnerabilities pose critical security threats, with nearly 50,000 CVEs reported in 2025.<n>While Large Language Models (LLMs) show promise for automated vulnerability detection, three key challenges remain.<n>We present FuzzingBrain V2, a multi-agent system that addresses these gaps through four key contributions.
  arXiv  Detail & Related papers  (2026-05-20T22:17:14Z)
• Orchard: An Open-Source Agentic Modeling Framework [124.68499958175111]
  We present Orchard, an open-source framework for scalable agentic modeling.<n>At its core is Orchard Env, a lightweight environment service providing reusable primitives for sandbox lifecycle management.<n>On top of Orchard Env, we build three agentic modeling recipes.
  arXiv  Detail & Related papers  (2026-05-14T16:35:12Z)
• Your Agent, Their Asset: A Real-World Safety Analysis of OpenClaw [87.97230960702274]
  We present the first real-world safety evaluation of OpenClaw.<n>We introduce the CIK taxonomy, which unifies an agent's persistent state into three dimensions.<n>Our evaluations cover 12 attack scenarios on a live OpenClaw instance.
  arXiv  Detail & Related papers  (2026-04-06T15:27:05Z)
• From Leaderboard to Deployment: Code Quality Challenges in AV Perception Repositories [4.603321798937855]
  This study systematically analyzed 178 unique models from the KITTI and NuScenes 3D Object Detection leaderboards.<n>Our findings revealed that only 7.3% of the studied repositories meet basic production-readiness criteria.<n>The adoption of Continuous Integration/Continuous Deployment pipelines was correlated with better code maintainability.
  arXiv  Detail & Related papers  (2026-03-02T18:54:28Z)
• Outrunning LLM Cutoffs: A Live Kernel Crash Resolution Benchmark for All [57.23434868678603]
  Live-kBench is an evaluation framework for self-evolving benchmarks that scrapes and evaluates agents on freshly discovered kernel bugs.<n> kEnv is an agent-agnostic crash-resolution environment for kernel compilation, execution, and feedback.<n>Using kEnv, we benchmark three state-of-the-art agents, showing that they resolve 74% of crashes on the first attempt.
  arXiv  Detail & Related papers  (2026-02-02T19:06:15Z)
• SWE-Universe: Scale Real-World Verifiable Environments to Millions [84.63665266236963]
  SWE-Universe is a framework for automatically constructing real-world software engineering (SWE) verifiable environments from GitHub pull requests (PRs)<n>We propose a building agent powered by an efficient custom-trained model to overcome the prevalent challenges of automatic building.<n>We demonstrate the profound value of our environments through large-scale agentic mid-training and reinforcement learning.
  arXiv  Detail & Related papers  (2026-02-02T17:20:30Z)
• RepoGenesis: Benchmarking End-to-End Microservice Generation from Readme to Repository [52.98970048197381]
  RepoGenesis is the first multilingual benchmark for repository-level end-to-end web microservice generation.<n>It consists of 106 repositories (60 Python, 46 Java) across 18 domains and 11 frameworks, with 1,258 API endpoints and 2,335 test cases verified.<n>Results reveal that despite high AC (up to 73.91%) and DSR (up to 100%), the best-performing system achieves only 23.67% Pass@1 on Python and 21.45% on Java.
  arXiv  Detail & Related papers  (2026-01-20T13:19:20Z)
• From CVE Entries to Verifiable Exploits: An Automated Multi-Agent Framework for Reproducing CVEs [23.210122086674048]
  CVE-GENIE is an automated framework designed to reproduce real-world vulnerabilities.<n>It reproduces 51% (428 of 841) CVEs published in 2024-2025, complete with their verifiable exploits, at an average cost of $2.77 per CVE.<n>Our pipeline offers a robust method to generate reproducible CVE benchmarks, valuable for diverse applications.
  arXiv  Detail & Related papers  (2025-09-01T23:37:44Z)
• Deployability-Centric Infrastructure-as-Code Generation: An LLM-based Iterative Framework [19.710477636179426]
  Infrastructure-as-Code (IaC) generation holds significant promise for automating cloud infrastructure provisioning.<n>Recent advances in Large Language Models (LLMs) present a promising opportunity to democratize IaC development.<n>Recent evaluation focuses on syntactic correctness while ignoring deployability, the fatal measure of IaC template utility.<n>We address this gap through two contributions: (1) IaCGen, an LLM-based deployability-centric framework that uses iterative feedback mechanism to generate IaC templates, and (2) DPIaC-Eval, a deployability-centric IaC template benchmark.
  arXiv  Detail & Related papers  (2025-06-05T22:53:12Z)
• BountyBench: Dollar Impact of AI Agent Attackers and Defenders on Real-World Cybersecurity Systems [62.17474934536671]
  We introduce the first framework to capture offensive and defensive cyber-capabilities in evolving real-world systems.<n>To capture the vulnerability lifecycle, we define three task types: Detect (detecting a new vulnerability), Exploit (exploiting a specific vulnerability), and Patch (patching a specific vulnerability)<n>We evaluate 8 agents: Claude Code, OpenAI Codex CLI with o3-high and o4-mini, and custom agents with o3-high, GPT-4.1, Gemini 2.5 Pro Preview, Claude 3.7 Sonnet Thinking, and DeepSeek-R1.
  arXiv  Detail & Related papers  (2025-05-21T07:44:52Z)
• SeCodePLT: A Unified Platform for Evaluating the Security of Code GenAI [58.29510889419971]
  Existing benchmarks for evaluating the security risks and capabilities of code-generating large language models (LLMs) face several key limitations.<n>We introduce a general and scalable benchmark construction framework that begins with manually validated, high-quality seed examples and expands them via targeted mutations.<n>Applying this framework to Python, C/C++, and Java, we build SeCodePLT, a dataset of more than 5.9k samples spanning 44 CWE-based risk categories and three security capabilities.
  arXiv  Detail & Related papers  (2024-10-14T21:17:22Z)
• Cybench: A Framework for Evaluating Cybersecurity Capabilities and Risks of Language Models [33.1538965735133]
  Cybench is a framework for specifying cybersecurity tasks and evaluating agents on those tasks.<n>We include 40 professional-level Capture the Flag (CTF) tasks from 4 distinct CTF competitions.<n>We construct a cybersecurity agent and evaluate 8 models: GPT-4o, OpenAI o1-preview, Claude 3 Opus, Claude 3.5 Sonnet, Mixtral 8x22b Instruct, Gemini 1.5 Pro, Llama 3 70B Chat, and Llama 3.1 405B Instruct.
  arXiv  Detail & Related papers  (2024-08-15T17:23:10Z)
• Cybersecurity Defenses: Exploration of CVE Types through Attack Descriptions [1.0474508494260908]
  VULDAT is a classification tool using a sentence transformer MPNET to identify system vulnerabilities from attack descriptions. Our model was applied to 100 attack techniques from the ATT&CK repository and 685 issues from the CVE repository. Our findings indicate that our model achieves the best performance with F1 score of 0.85, Precision of 0.86, and Recall of 0.83.
  arXiv  Detail & Related papers  (2024-07-09T11:08:35Z)
• VGX: Large-Scale Sample Generation for Boosting Learning-Based Software Vulnerability Analyses [30.65722096096949]
  This paper proposes VGX, a new technique aimed for large-scale generation of high-quality vulnerability datasets. VGX materializes vulnerability-injection code editing in identified contexts using patterns of such edits. For in-the-wild sample production, VGX generated 150,392 vulnerable samples, from which we randomly chose 10% to assess how much these samples help vulnerability detection, localization, and repair.
  arXiv  Detail & Related papers  (2023-10-24T01:05:00Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.