An Empirical Study of Fuzz Harness Degradation

• URL: http://arxiv.org/abs/2505.06177v1
• Date: Fri, 09 May 2025 16:39:20 GMT
• Title: An Empirical Study of Fuzz Harness Degradation
• Authors: Philipp Görz, Joschua Schilling, Thorsten Holz, Marcel Böhme,
• Abstract summary: We study Google's OSS-Fuzz continuous fuzzing platform containing harnesses for 510 open-source C/C++ projects.<n>A harness is the glue code between the fuzzer and the project, so it needs to adapt to changes in the project.<n>Our analysis shows a consistent overall fuzzer coverage percentage for projects in OSS-Fuzz and a surprising longevity of the bug-finding capability of harnesses even without explicit updates.
• Score: 24.989253174000922
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: The purpose of continuous fuzzing platforms is to enable fuzzing for software projects via \emph{fuzz harnesses} -- but as the projects continue to evolve, are these harnesses updated in lockstep, or do they run out of date? If these harnesses remain unmaintained, will they \emph{degrade} over time in terms of coverage achieved or number of bugs found? This is the subject of our study. We study Google's OSS-Fuzz continuous fuzzing platform containing harnesses for 510 open-source C/C++ projects, many of which are security-critical. A harness is the glue code between the fuzzer and the project, so it needs to adapt to changes in the project. It is often added by a project maintainer or as part of a, sometimes short-lived, testing effort. Our analysis shows a consistent overall fuzzer coverage percentage for projects in OSS-Fuzz and a surprising longevity of the bug-finding capability of harnesses even without explicit updates, as long as they still build. However, we also identify and manually examine individual cases of harness coverage degradation and categorize their root causes. Furthermore, we contribute to OSS-Fuzz and Fuzz Introspector to support metrics to detect harness degradation in OSS-Fuzz projects guided by this research.

Related papers

• LLAMA: Multi-Feedback Smart Contract Fuzzing Framework with LLM-Guided Seed Generation [56.84049855266145]
  We propose a Multi-feedback Smart Contract Fuzzing framework (LLAMA) that integrates evolutionary mutation strategies, and hybrid testing techniques.<n>LLAMA achieves 91% instruction coverage and 90% branch coverage, while detecting 132 out of 148 known vulnerabilities.<n>These results highlight LLAMA's effectiveness, adaptability, and practicality in real-world smart contract security testing scenarios.
  arXiv  Detail & Related papers  (2025-07-16T09:46:58Z)
• In the Magma chamber: Update and challenges in ground-truth vulnerabilities revival for automatic input generator comparison [42.95491588006701]
  Magma introduced the notion of forward-porting to reintroduce vulnerable code in current software releases.<n>While their results are promising, the state-of-the-art lacks an update on the maintainability of this approach over time.<n>We characterise the challenges with forward-porting by reassessing the portability of Magma's CVEs four years after its release.
  arXiv  Detail & Related papers  (2025-03-25T17:59:27Z)
• Your Fix Is My Exploit: Enabling Comprehensive DL Library API Fuzzing with Large Language Models [49.214291813478695]
  Deep learning (DL) libraries, widely used in AI applications, often contain vulnerabilities like overflows and use buffer-free errors.<n>Traditional fuzzing struggles with the complexity and API diversity of DL libraries.<n>We propose DFUZZ, an LLM-driven fuzzing approach for DL libraries.
  arXiv  Detail & Related papers  (2025-01-08T07:07:22Z)
• FuzzWiz -- Fuzzing Framework for Efficient Hardware Coverage [2.1626093085892144]
  We create an automated hardware fuzzing framework called FuzzWiz. It includes parsing the RTL design module, converting it into C/C++ models, creating generic testbench with assertions, linking, and fuzzing. Our benchmarking results show that we could achieve around 90% of the coverage 10 times faster than traditional simulation regression based approach.
  arXiv  Detail & Related papers  (2024-10-23T10:06:08Z)
• FuzzCoder: Byte-level Fuzzing Test via Large Language Model [46.18191648883695]
  We propose to adopt fine-tuned large language models (FuzzCoder) to learn patterns in the input files from successful attacks. FuzzCoder can predict mutation locations and strategies locations in input files to trigger abnormal behaviors of the program.
  arXiv  Detail & Related papers  (2024-09-03T14:40:31Z)
• VersiCode: Towards Version-controllable Code Generation [58.82709231906735]
  Large Language Models (LLMs) have made tremendous strides in code generation, but existing research fails to account for the dynamic nature of software development. We propose two novel tasks aimed at bridging this gap: version-specific code completion (VSCC) and version-aware code migration (VACM) We conduct an extensive evaluation on VersiCode, which reveals that version-controllable code generation is indeed a significant challenge.
  arXiv  Detail & Related papers  (2024-06-11T16:15:06Z)
• Prompt Fuzzing for Fuzz Driver Generation [6.238058387665971]
  We propose PromptFuzz, a coverage-guided fuzzer for prompt fuzzing. It iteratively generates fuzz drivers to explore undiscovered library code. PromptFuzz achieved 1.61 and 1.63 times higher branch coverage than OSS-Fuzz and Hopper, respectively.
  arXiv  Detail & Related papers  (2023-12-29T16:43:51Z)
• Vulnerability Detection Through an Adversarial Fuzzing Algorithm [2.074079789045646]
  This project aims to increase the efficiency of existing fuzzers by allowing fuzzers to explore more paths and find more bugs in shorter amounts of time. adversarial methods are built on top of current evolutionary algorithms to generate test cases for further and more efficient fuzzing.
  arXiv  Detail & Related papers  (2023-07-21T21:46:28Z)
• On the Security Blind Spots of Software Composition Analysis [46.1389163921338]
  We present a novel approach to detect vulnerable clones in the Maven repository. We retrieve over 53k potential vulnerable clones from Maven Central. We detect 727 confirmed vulnerable clones and synthesize a testable proof-of-vulnerability project for each of those.
  arXiv  Detail & Related papers  (2023-06-08T20:14:46Z)
• What Happens When We Fuzz? Investigating OSS-Fuzz Bug History [0.9772968596463595]
  We analyzed 44,102 reported issues made public by OSS-Fuzz prior to March 12, 2022. We identified the bug-contributing commits to estimate when the bug containing code was introduced, and measure the timeline from introduction to detection to fix.
  arXiv  Detail & Related papers  (2023-05-19T05:15:36Z)
• D2A: A Dataset Built for AI-Based Vulnerability Detection Methods Using Differential Analysis [55.15995704119158]
  We propose D2A, a differential analysis based approach to label issues reported by static analysis tools. We use D2A to generate a large labeled dataset to train models for vulnerability identification.
  arXiv  Detail & Related papers  (2021-02-16T07:46:53Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.