FuzzingBrain-Bench V1: Evaluating Open-Ended Bug Discovery by LLMs

• URL: http://arxiv.org/abs/2608.25158v1
• Date: Tue, 25 Aug 2026 21:09:20 GMT
• Title: FuzzingBrain-Bench V1: Evaluating Open-Ended Bug Discovery by LLMs
• Abstract summary: We present FuzzingBrain-Bench, a benchmark for assessing AI models' ability to discover bugs in open-source software.<n>A model's performance on each challenge is scored based on the number of distinct crash signatures it produces.<n>Claude Opus 4.8 performs best, triggering crashes in 60 of 77 challenges and achieving a score of 196 out of 579.
• Score: 46.58695049126903
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Evaluating the ability of large language models (LLMs) to discover software bugs is increasingly important. Existing benchmarks typically evaluate this capability by asking the model to generate a proof-of-concept input that triggers a predefined target vulnerability. However, this setup may overlook valid crashes discovered by the model when they do not match the predefined target. As a result, the evaluation may not reflect the model's real capability. We present FuzzingBrain-Bench, a benchmark for assessing AI models' ability to discover bugs in open-source software. Models are given an open-source project and a sanitizer-instrumented harness in a self-contained Docker image. Their goal is to generate inputs that trigger as many distinct crashes as possible through the harness. A model's performance on each challenge is scored based on the number of distinct crash signatures it produces, capped at a predefined maximum and weighted by a difficulty coefficient. FuzzingBrain-Bench V1 consists of 77 challenges drawn from 43 open-source projects, with 36 C, 32 C++, and 9 Java/JVM challenges. We evaluate Claude Haiku 4.5, Claude Sonnet 4.6, and Claude Opus 4.8 on the full benchmark. Claude Opus 4.8 performs best, triggering crashes in 60 of 77 challenges and achieving a score of 196 out of 579. None of the three models triggers a crash in 13 challenges. The FuzzingBrain-Bench corpus and harnesses are publicly available at https://github.com/fuzzingbrain/FuzzingBrain-Bench.

Related papers

• LLM-Assisted Dynamic Threat Analysis for Attacker-Reachable Software Weaknesses in Autonomous Vehicles [6.008950554571275]
  We investigate whether large language models (LLMs) can automate the static analysis of an autonomous-driving stack.<n>We perform compiler-precise static analysis across 185 packages, identifying 1,375 decision rules, 2,274 validation checks, and 482 input-to-safety-output flows.<n>Two local open-weight LLMs, a no-static-context ablation, and a naive-template baseline generate 3,700 artifact sets.<n>The main result is a build-integration failure taxonomy showing that 80% of first-shot compilation failures arise from dependency wiring rather than program logic.
  arXiv  Detail & Related papers  (2026-08-13T16:33:44Z)
• HoF-Bench: Rediscovering Real AI-Discovered CVEs Without Frontier Models [4.552360153162494]
  HoF-Bench is a benchmark built from 95 public AI-discovered CVEs across eight repositories pinned at vulnerable commits.<n>A deliberately minimal LLM-based analyzer rediscovers up to 65 of the 95 CVEs (68%) under this strict protocol.<n>HoF-Bench provides a compact test bed for comparing scanners, their vulnerability reliability across repeated runs, and the candidate volume they create.
  arXiv  Detail & Related papers  (2026-07-29T15:25:07Z)
• Quality-Assured Fuzz Harness Generation via the Four Principles Framework [9.484544174862325]
  We present QuartetFuzz, an autonomous harness-generation system that improves correctness throughout the generation process.<n>At its core is the Four Principles framework -- Logic Correctness (P1), API Protocol Compliance (P2), Security Boundary Respect (P3), and Entry Point Adequacy (P4)
  arXiv  Detail & Related papers  (2026-05-20T23:48:26Z)
• ExploitBench: A Capability Ladder Benchmark for LLM Cybersecurity Agents [7.0226553698460465]
  We present ExploitBench, a capability-graded benchmark that decomposes exploitation into 16 measurable flags.<n>We instantiate ExploitBench on 41 V8 bugs because V8 is both widely deployed and exploitation-hardened.<n>Our results show a sharp capability split between publicly deployed frontier models and the private frontier.
  arXiv  Detail & Related papers  (2026-05-13T22:08:05Z)
• Outrunning LLM Cutoffs: A Live Kernel Crash Resolution Benchmark for All [57.23434868678603]
  Live-kBench is an evaluation framework for self-evolving benchmarks that scrapes and evaluates agents on freshly discovered kernel bugs.<n> kEnv is an agent-agnostic crash-resolution environment for kernel compilation, execution, and feedback.<n>Using kEnv, we benchmark three state-of-the-art agents, showing that they resolve 74% of crashes on the first attempt.
  arXiv  Detail & Related papers  (2026-02-02T19:06:15Z)
• Goedel-Prover-V2: Scaling Formal Theorem Proving with Scaffolded Data Synthesis and Self-Correction [95.91743732150233]
  Goedel-Prover-V2, a series of open-source language models, set a new state-of-the-art in automated theorem proving.<n>We generate synthetic tasks of increasing difficulty to train the model to master increasingly complex theorems.<n>Goedel-Prover-V2-32B achieves 88.1% on MiniF2F at pass@32 in standard mode and 90.4% in self-correction mode.
  arXiv  Detail & Related papers  (2025-08-05T16:28:22Z)
• DyePack: Provably Flagging Test Set Contamination in LLMs Using Backdoors [52.85182605005619]
  We introduce DyePack, a framework that leverages backdoor attacks to identify models that used benchmark test sets during training.<n>Like how banks mix dye packs with their money to mark robbers, DyePack mixes backdoor samples with the test data to flag models that trained on it.<n>We evaluate DyePack on five models across three datasets, covering both multiple-choice and open-ended generation tasks.
  arXiv  Detail & Related papers  (2025-05-29T02:22:14Z)
• CKGFuzzer: LLM-Based Fuzz Driver Generation Enhanced By Code Knowledge Graph [29.490817477791357]
  We propose an automated fuzz testing method driven by a code knowledge graph and powered by an intelligent agent system.<n>The code knowledge graph is constructed through interprocedural program analysis, where each node in the graph represents a code entity.<n> CKGFuzzer achieved an average improvement of 8.73% in code coverage compared to state-of-the-art techniques.
  arXiv  Detail & Related papers  (2024-11-18T12:41:16Z)
• DebugBench: Evaluating Debugging Capability of Large Language Models [80.73121177868357]
  DebugBench is a benchmark for Large Language Models (LLMs) It covers four major bug categories and 18 minor types in C++, Java, and Python. We evaluate two commercial and four open-source models in a zero-shot scenario.
  arXiv  Detail & Related papers  (2024-01-09T15:46:38Z)
• RobustBench: a standardized adversarial robustness benchmark [84.50044645539305]
  Key challenge in benchmarking robustness is that its evaluation is often error-prone leading to robustness overestimation. We evaluate adversarial robustness with AutoAttack, an ensemble of white- and black-box attacks. We analyze the impact of robustness on the performance on distribution shifts, calibration, out-of-distribution detection, fairness, privacy leakage, smoothness, and transferability.
  arXiv  Detail & Related papers  (2020-10-19T17:06:18Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.