Comment on Revisiting Neural Program Smoothing for Fuzzing

• URL: http://arxiv.org/abs/2409.04504v1
• Date: Fri, 6 Sep 2024 16:07:22 GMT
• Title: Comment on Revisiting Neural Program Smoothing for Fuzzing
• Authors: Dongdong She, Kexin Pei, Junfeng Yang, Baishakhi Ray, Suman Jana,
• Abstract summary: MLFuzz, a work accepted at ACM FSE 2023, revisits the performance of a machine learning-based fuzzer, NEUZZ. We demonstrate that its main conclusion is entirely wrong due to several fatal bugs in the implementation and wrong evaluation setups.
• Score: 34.32355705821806
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: MLFuzz, a work accepted at ACM FSE 2023, revisits the performance of a machine learning-based fuzzer, NEUZZ. We demonstrate that its main conclusion is entirely wrong due to several fatal bugs in the implementation and wrong evaluation setups, including an initialization bug in persistent mode, a program crash, an error in training dataset collection, and a mistake in fuzzing result collection. Additionally, MLFuzz uses noisy training datasets without sufficient data cleaning and preprocessing, which contributes to a drastic performance drop in NEUZZ. We address these issues and provide a corrected implementation and evaluation setup, showing that NEUZZ consistently performs well over AFL on the FuzzBench dataset. Finally, we reflect on the evaluation methods used in MLFuzz and offer practical advice on fair and scientific fuzzing evaluations.

Related papers

• FuzzCoder: Byte-level Fuzzing Test via Large Language Model [46.18191648883695]
  We propose to adopt fine-tuned large language models (FuzzCoder) to learn patterns in the input files from successful attacks. FuzzCoder can predict mutation locations and strategies locations in input files to trigger abnormal behaviors of the program.
  arXiv  Detail & Related papers  (2024-09-03T14:40:31Z)
• HuntFUZZ: Enhancing Error Handling Testing through Clustering Based Fuzzing [19.31537246674011]
  This paper introduces HuntFUZZ, a novel SFI-based fuzzing framework that addresses the issue of redundant testing of error points with correlated paths. We evaluate HuntFUZZ on a diverse set of 42 applications, and HuntFUZZ successfully reveals 162 known bugs, with 62 of them being related to error handling.
  arXiv  Detail & Related papers  (2024-07-05T06:58:30Z)
• LLAMAFUZZ: Large Language Model Enhanced Greybox Fuzzing [6.042114639413868]
  Specialized fuzzers can handle complex structured data, but require additional efforts in grammar and suffer from low throughput. In this paper, we explore the potential of utilizing the Large Language Model to enhance greybox fuzzing for structured data. Our LLM-based fuzzer, LLAMAFUZZ, integrates the power of LLM to understand and mutate structured data to fuzzing.
  arXiv  Detail & Related papers  (2024-06-11T20:48:28Z)
• SoK: Prudent Evaluation Practices for Fuzzing [21.113311952857778]
  We systematically analyze the evaluation of 150 fuzzing papers published between 2018 and 2023. We study how existing guidelines are implemented and observe potential shortcomings and pitfalls. For example, when investigating reported bugs, we find a surprising disregard of the existing guidelines regarding statistical tests and systematic errors in fuzzing evaluations.
  arXiv  Detail & Related papers  (2024-05-16T16:10:41Z)
• Learning with Noisy Foundation Models [95.50968225050012]
  This paper is the first work to comprehensively understand and analyze the nature of noise in pre-training datasets. We propose a tuning method (NMTune) to affine the feature space to mitigate the malignant effect of noise and improve generalization.
  arXiv  Detail & Related papers  (2024-03-11T16:22:41Z)
• Noisy Pair Corrector for Dense Retrieval [59.312376423104055]
  We propose a novel approach called Noisy Pair Corrector (NPC) NPC consists of a detection module and a correction module. We conduct experiments on text-retrieval benchmarks Natural Question and TriviaQA, code-search benchmarks StaQC and SO-DS.
  arXiv  Detail & Related papers  (2023-11-07T08:27:14Z)
• Understanding and Mitigating the Label Noise in Pre-training on Downstream Tasks [91.15120211190519]
  This paper aims to understand the nature of noise in pre-training datasets and to mitigate its impact on downstream tasks. We propose a light-weight black-box tuning method (NMTune) to affine the feature space to mitigate the malignant effect of noise.
  arXiv  Detail & Related papers  (2023-09-29T06:18:15Z)
• Revisiting Neural Program Smoothing for Fuzzing [8.861172379630899]
  This paper presents the most extensive evaluation of NPS fuzzers against standard gray-box fuzzers. We implement Neuzz++, which shows that addressing the practical limitations of NPS fuzzers improves performance. We present MLFuzz, a platform with GPU access for easy and reproducible evaluation of ML-based fuzzers.
  arXiv  Detail & Related papers  (2023-09-28T17:17:11Z)
• Deep F-measure Maximization for End-to-End Speech Understanding [52.36496114728355]
  We propose a differentiable approximation to the F-measure and train the network with this objective using standard backpropagation. We perform experiments on two standard fairness datasets, Adult, Communities and Crime, and also on speech-to-intent detection on the ATIS dataset and speech-to-image concept classification on the Speech-COCO dataset. In all four of these tasks, F-measure results in improved micro-F1 scores, with absolute improvements of up to 8% absolute, as compared to models trained with the cross-entropy loss function.
  arXiv  Detail & Related papers  (2020-08-08T03:02:27Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.