An Empirical Study of Fuzz Harness Degradation

• URL: http://arxiv.org/abs/2505.06177v1
• Date: Fri, 09 May 2025 16:39:20 GMT
• Title: An Empirical Study of Fuzz Harness Degradation
• Abstract summary: We study Google's OSS-Fuzz continuous fuzzing platform containing harnesses for 510 open-source C/C++ projects.<n>A harness is the glue code between the fuzzer and the project, so it needs to adapt to changes in the project.<n>Our analysis shows a consistent overall fuzzer coverage percentage for projects in OSS-Fuzz and a surprising longevity of the bug-finding capability of harnesses even without explicit updates.
• Score: 24.989253174000922
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: The purpose of continuous fuzzing platforms is to enable fuzzing for software projects via \emph{fuzz harnesses} -- but as the projects continue to evolve, are these harnesses updated in lockstep, or do they run out of date? If these harnesses remain unmaintained, will they \emph{degrade} over time in terms of coverage achieved or number of bugs found? This is the subject of our study. We study Google's OSS-Fuzz continuous fuzzing platform containing harnesses for 510 open-source C/C++ projects, many of which are security-critical. A harness is the glue code between the fuzzer and the project, so it needs to adapt to changes in the project. It is often added by a project maintainer or as part of a, sometimes short-lived, testing effort. Our analysis shows a consistent overall fuzzer coverage percentage for projects in OSS-Fuzz and a surprising longevity of the bug-finding capability of harnesses even without explicit updates, as long as they still build. However, we also identify and manually examine individual cases of harness coverage degradation and categorize their root causes. Furthermore, we contribute to OSS-Fuzz and Fuzz Introspector to support metrics to detect harness degradation in OSS-Fuzz projects guided by this research.

Related papers

• HarnessDev: Can LLMs Create and Evolve Their Own Agent Harness? [61.02699867449589]
  We introduce HarnessDev, a benchmark that shifts the unit of evaluation from task outputs to run infrastructure.<n>In Creation, the agent starts from a minimal seed and a small number of cases, then builds a complete execution system.<n>In Evolution, it starts from its own created harness and iteratively revises it using downstream execution feedback, with the goal of improving benchmark performance.
  arXiv  Detail & Related papers  (2026-09-01T15:45:33Z)
• Beyond Fail-to-Pass: Iterative Hardening of Co-Generated Bug Reproduction Tests and Fixes [55.114939648705395]
  Large language models (LLMs) have made automated program repair increasingly practical for real-world bugs.<n>Bug reproduction tests (BRTs) help close this gap by turning a bug report into an executable, bug-specific signal.<n>We propose CoHarden, a co-generation framework that uses the Lax signal as an in-loop convergence criterion.
  arXiv  Detail & Related papers  (2026-07-22T07:30:07Z)
• FuzzAgent: Multi-Agent System for Evolutionary Library Fuzzing [8.054484953700365]
  Library fuzzing is essential for hardening the software supply chain, but adopting it at scale remains expensive.<n>We present FuzzAgent, a multi-agent system that turns library fuzzing into an evolutionary process.<n>We evaluate FuzzAgent on 20 real-world C/C++ libraries against four state-of-the-art baselines.
  arXiv  Detail & Related papers  (2026-05-14T06:21:33Z)
• Enhancing Fuzz Testing Efficiency through Automated Fuzz Target Generation [0.0]
  We introduce an approach to improving fuzz target generation through static analysis of library source code.<n>Our findings are demonstrated through the application of this approach to the generation of fuzz targets for C/C++ libraries.
  arXiv  Detail & Related papers  (2026-01-17T09:08:11Z)
• Large-Scale Empirical Analysis of Continuous Fuzzing: Insights from 1 Million Fuzzing Sessions [0.07183613290627339]
  This study aims to elucidate the role of continuous fuzzing in vulnerability detection.<n>We collect issue reports, coverage reports, and fuzzing logs from OSS-Fuzz, an online service provided by Google.<n>We reveal that a substantial number of fuzzing bugs exist prior to the integration of continuous fuzzing.
  arXiv  Detail & Related papers  (2025-10-18T10:13:19Z)
• What Do They Fix? LLM-Aided Categorization of Security Patches for Critical Memory Bugs [46.325755802511026]
  We developLM, a dual-method pipeline that integrates two approaches based on a Large Language Model (LLM) and a fine-tuned small language model.<n>LM successfully identified 111 of 5,140 recent Linux kernel patches addressing OOB or UAF vulnerabilities, with 90 true positives confirmed by manual verification.
  arXiv  Detail & Related papers  (2025-09-26T18:06:36Z)
• LLAMA: Multi-Feedback Smart Contract Fuzzing Framework with LLM-Guided Seed Generation [56.84049855266145]
  We propose a Multi-feedback Smart Contract Fuzzing framework (LLAMA) that integrates evolutionary mutation strategies, and hybrid testing techniques.<n>LLAMA achieves 91% instruction coverage and 90% branch coverage, while detecting 132 out of 148 known vulnerabilities.<n>These results highlight LLAMA's effectiveness, adaptability, and practicality in real-world smart contract security testing scenarios.
  arXiv  Detail & Related papers  (2025-07-16T09:46:58Z)
• In the Magma chamber: Update and challenges in ground-truth vulnerabilities revival for automatic input generator comparison [42.95491588006701]
  Magma introduced the notion of forward-porting to reintroduce vulnerable code in current software releases.<n>While their results are promising, the state-of-the-art lacks an update on the maintainability of this approach over time.<n>We characterise the challenges with forward-porting by reassessing the portability of Magma's CVEs four years after its release.
  arXiv  Detail & Related papers  (2025-03-25T17:59:27Z)
• Your Fix Is My Exploit: Enabling Comprehensive DL Library API Fuzzing with Large Language Models [49.214291813478695]
  Deep learning (DL) libraries, widely used in AI applications, often contain vulnerabilities like overflows and use buffer-free errors.<n>Traditional fuzzing struggles with the complexity and API diversity of DL libraries.<n>We propose DFUZZ, an LLM-driven fuzzing approach for DL libraries.
  arXiv  Detail & Related papers  (2025-01-08T07:07:22Z)
• FuzzWiz -- Fuzzing Framework for Efficient Hardware Coverage [2.1626093085892144]
  We create an automated hardware fuzzing framework called FuzzWiz. It includes parsing the RTL design module, converting it into C/C++ models, creating generic testbench with assertions, linking, and fuzzing. Our benchmarking results show that we could achieve around 90% of the coverage 10 times faster than traditional simulation regression based approach.
  arXiv  Detail & Related papers  (2024-10-23T10:06:08Z)
• FuzzCoder: Byte-level Fuzzing Test via Large Language Model [46.18191648883695]
  We propose to adopt fine-tuned large language models (FuzzCoder) to learn patterns in the input files from successful attacks. FuzzCoder can predict mutation locations and strategies locations in input files to trigger abnormal behaviors of the program.
  arXiv  Detail & Related papers  (2024-09-03T14:40:31Z)
• VersiCode: Towards Version-controllable Code Generation [58.82709231906735]
  Large Language Models (LLMs) have made tremendous strides in code generation, but existing research fails to account for the dynamic nature of software development. We propose two novel tasks aimed at bridging this gap: version-specific code completion (VSCC) and version-aware code migration (VACM) We conduct an extensive evaluation on VersiCode, which reveals that version-controllable code generation is indeed a significant challenge.
  arXiv  Detail & Related papers  (2024-06-11T16:15:06Z)
• Prompt Fuzzing for Fuzz Driver Generation [6.238058387665971]
  We propose PromptFuzz, a coverage-guided fuzzer for prompt fuzzing. It iteratively generates fuzz drivers to explore undiscovered library code. PromptFuzz achieved 1.61 and 1.63 times higher branch coverage than OSS-Fuzz and Hopper, respectively.
  arXiv  Detail & Related papers  (2023-12-29T16:43:51Z)
• Vulnerability Detection Through an Adversarial Fuzzing Algorithm [2.074079789045646]
  This project aims to increase the efficiency of existing fuzzers by allowing fuzzers to explore more paths and find more bugs in shorter amounts of time. adversarial methods are built on top of current evolutionary algorithms to generate test cases for further and more efficient fuzzing.
  arXiv  Detail & Related papers  (2023-07-21T21:46:28Z)
• On the Security Blind Spots of Software Composition Analysis [46.1389163921338]
  We present a novel approach to detect vulnerable clones in the Maven repository. We retrieve over 53k potential vulnerable clones from Maven Central. We detect 727 confirmed vulnerable clones and synthesize a testable proof-of-vulnerability project for each of those.
  arXiv  Detail & Related papers  (2023-06-08T20:14:46Z)
• What Happens When We Fuzz? Investigating OSS-Fuzz Bug History [0.9772968596463595]
  We analyzed 44,102 reported issues made public by OSS-Fuzz prior to March 12, 2022. We identified the bug-contributing commits to estimate when the bug containing code was introduced, and measure the timeline from introduction to detection to fix.
  arXiv  Detail & Related papers  (2023-05-19T05:15:36Z)
• D2A: A Dataset Built for AI-Based Vulnerability Detection Methods Using Differential Analysis [55.15995704119158]
  We propose D2A, a differential analysis based approach to label issues reported by static analysis tools. We use D2A to generate a large labeled dataset to train models for vulnerability identification.
  arXiv  Detail & Related papers  (2021-02-16T07:46:53Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.