A Comparative Quality Metric for Untargeted Fuzzing with Logic State Coverage

• URL: http://arxiv.org/abs/2409.14987v1
• Date: Mon, 23 Sep 2024 13:08:17 GMT
• Title: A Comparative Quality Metric for Untargeted Fuzzing with Logic State Coverage
• Authors: Gwangmu Lee,
• Abstract summary: We propose logic state coverage as a proxy metric to count observed interesting behaviors. A logic state distinguishes less repetitive (i.e., more interesting) behaviors in a finer granularity, making the amount of logic state coverage reliably proportional to the number of observed interesting behaviors.
• Score: 2.9914612342004503
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: While fuzzing is widely accepted as an efficient program testing technique, it is still unclear how to measure the comparative quality of different fuzzers. The current de facto quality metrics are edge coverage and the number of discovered bugs, but they are frequently discredited by inconclusive, exaggerated, or even counter-intuitive results. To establish a more reliable quality metric, we first note that fuzzing aims to reduce the number of unknown abnormal behaviors by observing more interesting (i.e., relating to unknown abnormal) behaviors. The more interesting behaviors a fuzzer has observed, the stronger guarantee it can provide about the absence of unknown abnormal behaviors. This suggests that the number of observed interesting behaviors must directly indicate the fuzzing quality. In this work, we propose logic state coverage as a proxy metric to count observed interesting behaviors. A logic state is a set of satisfied branches during one execution, where its coverage is the count of individual observed logic states during a fuzzing campaign. A logic state distinguishes less repetitive (i.e., more interesting) behaviors in a finer granularity, making the amount of logic state coverage reliably proportional to the number of observed interesting behaviors. We implemented logic state coverage using a bloom filter and performed a preliminary evaluation with AFL++ and XMLLint.

Related papers

• SoK: Prudent Evaluation Practices for Fuzzing [21.113311952857778]
  We systematically analyze the evaluation of 150 fuzzing papers published between 2018 and 2023. We study how existing guidelines are implemented and observe potential shortcomings and pitfalls. For example, when investigating reported bugs, we find a surprising disregard of the existing guidelines regarding statistical tests and systematic errors in fuzzing evaluations.
  arXiv  Detail & Related papers  (2024-05-16T16:10:41Z)
• Don't Miss Out on Novelty: Importance of Novel Features for Deep Anomaly Detection [64.21963650519312]
  Anomaly Detection (AD) is a critical task that involves identifying observations that do not conform to a learned model of normality. We propose a novel approach to AD using explainability to capture such novel features as unexplained observations in the input space. Our approach establishes a new state-of-the-art across multiple benchmarks, handling diverse anomaly types.
  arXiv  Detail & Related papers  (2023-10-01T21:24:05Z)
• Initial State Interventions for Deconfounded Imitation Learning [11.605936648692543]
  We consider the problem of masking observed confounders in a disentangled representation of the observation space. Our novel masking algorithm leverages the usual ability to intervene in the initial system state. Under certain assumptions, we theoretically prove that this algorithm is conservative in the sense that it does not incorrectly mask observations that causally influence the expert.
  arXiv  Detail & Related papers  (2023-07-29T13:02:45Z)
• Fuzzing with Quantitative and Adaptive Hot-Bytes Identification [6.442499249981947]
  American fuzzy lop, a leading fuzzing tool, has demonstrated its powerful bug finding ability through a vast number of reported CVEs. We propose an approach called toolwhich is designed based on the following principles. Our evaluation results on 10 real-world programs and LAVA-M dataset show that toolachieves sustained increases in branch coverage and discovers more bugs than other fuzzers.
  arXiv  Detail & Related papers  (2023-07-05T13:41:35Z)
• Shortcomings of Top-Down Randomization-Based Sanity Checks for Evaluations of Deep Neural Network Explanations [67.40641255908443]
  We identify limitations of model-randomization-based sanity checks for the purpose of evaluating explanations. Top-down model randomization preserves scales of forward pass activations with high probability.
  arXiv  Detail & Related papers  (2022-11-22T18:52:38Z)
• SLA$^2$P: Self-supervised Anomaly Detection with Adversarial Perturbation [77.71161225100927]
  Anomaly detection is a fundamental yet challenging problem in machine learning. We propose a novel and powerful framework, dubbed as SLA$2$P, for unsupervised anomaly detection.
  arXiv  Detail & Related papers  (2021-11-25T03:53:43Z)
• Nested Counterfactual Identification from Arbitrary Surrogate Experiments [95.48089725859298]
  We study the identification of nested counterfactuals from an arbitrary combination of observations and experiments. Specifically, we prove the counterfactual unnesting theorem (CUT), which allows one to map arbitrary nested counterfactuals to unnested ones.
  arXiv  Detail & Related papers  (2021-07-07T12:51:04Z)
• Deconfounded Score Method: Scoring DAGs with Dense Unobserved Confounding [101.35070661471124]
  We show that unobserved confounding leaves a characteristic footprint in the observed data distribution that allows for disentangling spurious and causal effects. We propose an adjusted score-based causal discovery algorithm that may be implemented with general-purpose solvers and scales to high-dimensional problems.
  arXiv  Detail & Related papers  (2021-03-28T11:07:59Z)
• A New Bandit Setting Balancing Information from State Evolution and Corrupted Context [52.67844649650687]
  We propose a new sequential decision-making setting combining key aspects of two established online learning problems with bandit feedback. The optimal action to play at any given moment is contingent on an underlying changing state which is not directly observable by the agent. We present an algorithm that uses a referee to dynamically combine the policies of a contextual bandit and a multi-armed bandit.
  arXiv  Detail & Related papers  (2020-11-16T14:35:37Z)
• High-recall causal discovery for autocorrelated time series with latent confounders [12.995632804090198]
  We show that existing causal discovery methods such as FCI and variants suffer from low recall in the autocorrelated time series case. We provide Python code for all methods involved in the simulation studies.
  arXiv  Detail & Related papers  (2020-07-03T18:01:04Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.