HoF-Bench: Rediscovering Real AI-Discovered CVEs Without Frontier Models
- URL: http://arxiv.org/abs/2607.27030v1
- Date: Wed, 29 Jul 2026 15:25:07 GMT
- Title: HoF-Bench: Rediscovering Real AI-Discovered CVEs Without Frontier Models
- Abstract summary: HoF-Bench is a benchmark built from 95 public AI-discovered CVEs across eight repositories pinned at vulnerable commits.<n>A deliberately minimal LLM-based analyzer rediscovers up to 65 of the 95 CVEs (68%) under this strict protocol.<n>HoF-Bench provides a compact test bed for comparing scanners, their vulnerability reliability across repeated runs, and the candidate volume they create.
- Score: 4.552360153162494
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: LLM-based analyzers have begun finding real vulnerabilities in mature open-source projects: AISLE's analyzer is credited with more than 280 CVEs across 78 projects, including OpenSSL, curl, and GnuTLS. We introduce HoF-Bench (named after AISLE's public Hall of Fame), a benchmark built from 95 of these public AI-discovered CVEs across eight repositories pinned at vulnerable commits. Analyzers receive source and target-file scope but not CVE identifiers, descriptions, fixes, or expected mechanisms; a detector-blinded frontier-model judge credits only findings that identify the same code path, root cause, attack condition, and impact. A deliberately minimal LLM-based analyzer rediscovers up to 65 of the 95 CVEs (68%) under this strict protocol. No frontier model performs detection anywhere in the study. The ten detector backbones are five open-weight models (21B--284B total parameters, 3--13B active) and five proprietary small or "flash"-tier models. All of them run in the fixed scaffold with four repeated passes, an optional generated-context stage, and a replayable multi-round triage stage (7,600 model--CVE pass records). Difficulty is strongly structured by language; the CVEs missed by every model concentrate in C infrastructure code. HoF-Bench provides a compact test bed for comparing vulnerability scanners, their reliability across repeated runs, and the candidate volume they create. The dataset is available at https://huggingface.co/datasets/aisleinc/HoF-Bench.
Related papers
- FuzzingBrain-Bench V1: Evaluating Open-Ended Bug Discovery by LLMs [46.58695049126903]
We present FuzzingBrain-Bench, a benchmark for assessing AI models' ability to discover bugs in open-source software.<n>A model's performance on each challenge is scored based on the number of distinct crash signatures it produces.<n>Claude Opus 4.8 performs best, triggering crashes in 60 of 77 challenges and achieving a score of 196 out of 579.
arXiv Detail & Related papers (2026-08-25T21:09:20Z) - CyberForge: Verified Vulnerability Injection at Repository Level for Cybersecurity Agent Training [42.087410274052836]
CyberForge is a framework that synthesizes executable, repository-level security training data by injecting vulnerabilities into real C/C++ projects.<n>The resulting corpus holds 1034 validated vulnerabilities across 80 projects and 63 weakness categories.<n>Fine-tuning on trajectories collected over this corpus improves SEC-bench patch repair by +3.3 to +14.7 points.
arXiv Detail & Related papers (2026-08-06T18:03:27Z) - Cross-Cutting Security Analysis of LLM-Generated Code via Metamorphic Testing and Association Rule Mining [0.7844254062143549]
Large language models (LLMs) frequently generate code with security vulnerabilities, yet these weaknesses are rarely isolated.<n>We present a framework that combines security-oriented Metamorphic Relations (MRs) with Association Rule (AR) mining to detect vulnerabilities in LLM-generated code.
arXiv Detail & Related papers (2026-07-13T19:10:10Z) - Correct Code, Vulnerable Dependencies: A Large Scale Measurement Study of LLM-Specified Library Versions [52.50730821321986]
We present the first large-scale measurement study of version-level risk in large language models (LLMs)<n>We evaluate 10 LLMs on PinTrace, a curated benchmark of 1,000 Stack Overflow programming tasks.<n>Our findings reveal LLM version selection as a first-class, previously overlooked risk surface in LLM-based development.
arXiv Detail & Related papers (2026-05-07T13:52:59Z) - Delta-Based Neural Architecture Search: LLM Fine-Tuning via Code Diffs [48.83701310501069]
Large language models (LLMs) show strong potential for neural architecture generation.<n>Existing approaches produce complete model implementations from scratch.<n>We propose Delta-Code Generation, where fine-tuned LLMs generate compact unified diffs.
arXiv Detail & Related papers (2026-05-06T13:32:05Z) - Seclens: Role-specific Evaluation of LLM's for security vulnerablity detection [0.0]
We introduce SecLens-R, a multi-stakeholder evaluation framework structured around 35 shared dimensions grouped into 7 measurement categories.<n>We apply SecLens-R to evaluate 12 frontier models on a dataset of 406 tasks derived from 93 open-source projects.<n>Results show substantial variation across stakeholder perspectives, with Decision Scores differing by as much as 31 points for the same model.
arXiv Detail & Related papers (2026-04-02T05:25:50Z) - What Do They Fix? LLM-Aided Categorization of Security Patches for Critical Memory Bugs [46.325755802511026]
We developLM, a dual-method pipeline that integrates two approaches based on a Large Language Model (LLM) and a fine-tuned small language model.<n>LM successfully identified 111 of 5,140 recent Linux kernel patches addressing OOB or UAF vulnerabilities, with 90 true positives confirmed by manual verification.
arXiv Detail & Related papers (2025-09-26T18:06:36Z) - DyePack: Provably Flagging Test Set Contamination in LLMs Using Backdoors [52.85182605005619]
We introduce DyePack, a framework that leverages backdoor attacks to identify models that used benchmark test sets during training.<n>Like how banks mix dye packs with their money to mark robbers, DyePack mixes backdoor samples with the test data to flag models that trained on it.<n>We evaluate DyePack on five models across three datasets, covering both multiple-choice and open-ended generation tasks.
arXiv Detail & Related papers (2025-05-29T02:22:14Z) - AutoPatch: Multi-Agent Framework for Patching Real-World CVE Vulnerabilities [7.812032134834162]
Large Language Models (LLMs) have emerged as promising tools in software development.<n>Their knowledge is limited to a fixed cutoff date, making them prone to generating code vulnerable to newly disclosed CVEs.<n>We propose AutoPatch, a framework designed to patch vulnerable LLM-generated code.
arXiv Detail & Related papers (2025-05-07T07:49:05Z) - IRIS: LLM-Assisted Static Analysis for Detecting Security Vulnerabilities [14.188864624736938]
Large language models (or LLMs) have shown impressive code generation capabilities but they cannot do complex reasoning over code to detect such vulnerabilities.<n>We propose IRIS, a neuro-symbolic approach that systematically combines LLMs with static analysis to perform whole-repository reasoning for security vulnerability detection.
arXiv Detail & Related papers (2024-05-27T14:53:35Z) - How secure is AI-generated Code: A Large-Scale Comparison of Large Language Models [3.4887856546295333]
This study compares state-of-the-art Large Language Models (LLMs) on their tendency to generate vulnerabilities when writing C programs.<n>At least 62.07% of the generated programs are vulnerable.
arXiv Detail & Related papers (2024-04-29T01:24:14Z) - On the Security Blind Spots of Software Composition Analysis [46.1389163921338]
We present a novel approach to detect vulnerable clones in the Maven repository.
We retrieve over 53k potential vulnerable clones from Maven Central.
We detect 727 confirmed vulnerable clones and synthesize a testable proof-of-vulnerability project for each of those.
arXiv Detail & Related papers (2023-06-08T20:14:46Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.