Large Language Models Based Fuzzing Techniques: A Survey

• URL: http://arxiv.org/abs/2402.00350v2
• Date: Wed, 7 Feb 2024 06:03:15 GMT
• Title: Large Language Models Based Fuzzing Techniques: A Survey
• Authors: Linghan Huang, Peizhou Zhao, Huaming Chen, Lei Ma
• Abstract summary: fuzzing test, as an efficient software testing method, are widely used in various domains. The rapid development of Large Language Models (LLMs) has facilitated their application in the field of software testing. There is a growing trend towards employing fuzzing test generated based on large language models.
• Score: 4.155653485098873
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: In the modern era where software plays a pivotal role, software security and vulnerability analysis have become essential for software development. Fuzzing test, as an efficient software testing method, are widely used in various domains. Moreover, the rapid development of Large Language Models (LLMs) has facilitated their application in the field of software testing, demonstrating remarkable performance. Considering that existing fuzzing test techniques are not entirely automated and software vulnerabilities continue to evolve, there is a growing trend towards employing fuzzing test generated based on large language models. This survey provides a systematic overview of the approaches that fuse LLMs and fuzzing tests for software testing. In this paper, a statistical analysis and discussion of the literature in three areas, namely LLMs, fuzzing test, and fuzzing test generated based on LLMs, are conducted by summarising the state-of-the-art methods up until 2024. Our survey also investigates the potential for widespread deployment and application of fuzzing test techniques generated by LLMs in the future.

Related papers

• The Potential of LLMs in Automating Software Testing: From Generation to Reporting [0.0]
  Manual testing, while effective, can be time consuming and costly, leading to an increased demand for automated methods. Recent advancements in Large Language Models (LLMs) have significantly influenced software engineering. This paper explores an agent-oriented approach to automated software testing, using LLMs to reduce human intervention and enhance testing efficiency.
  arXiv  Detail & Related papers  (2024-12-31T02:06:46Z)
• Automated Robustness Testing for LLM-based NLP Software [6.986328098563149]
  There are no known automated robustness testing methods specifically designed for LLM-based NLP software. Existing testing methods can be applied to LLM-based software by AORTA, but their effectiveness is limited. We propose a novel testing method for LLM-based software within AORTA called Adaptive Beam Search.
  arXiv  Detail & Related papers  (2024-12-30T15:33:34Z)
• Large-scale, Independent and Comprehensive study of the power of LLMs for test case generation [11.056044348209483]
  Unit testing, crucial for identifying bugs in code modules like classes and methods, is often neglected by developers due to time constraints. Large Language Models (LLMs), like GPT and Mistral, show promise in software engineering, including in test generation.
  arXiv  Detail & Related papers  (2024-06-28T20:38:41Z)
• Benchmarking Uncertainty Quantification Methods for Large Language Models with LM-Polygraph [83.90988015005934]
  Uncertainty quantification is a key element of machine learning applications. We introduce a novel benchmark that implements a collection of state-of-the-art UQ baselines. We conduct a large-scale empirical investigation of UQ and normalization techniques across eleven tasks, identifying the most effective approaches.
  arXiv  Detail & Related papers  (2024-06-21T20:06:31Z)
• Automatic benchmarking of large multimodal models via iterative experiment programming [71.78089106671581]
  We present APEx, the first framework for automatic benchmarking of LMMs. Given a research question expressed in natural language, APEx leverages a large language model (LLM) and a library of pre-specified tools to generate a set of experiments for the model at hand. The report drives the testing procedure: based on the current status of the investigation, APEx chooses which experiments to perform and whether the results are sufficient to draw conclusions.
  arXiv  Detail & Related papers  (2024-06-18T06:43:46Z)
• Test Oracle Automation in the era of LLMs [52.69509240442899]
  Large Language Models (LLMs) have demonstrated remarkable proficiency in tackling diverse software testing tasks. This paper aims to enable discussions on the potential of using LLMs for test oracle automation, along with the challenges that may emerge during the generation of various types of oracles.
  arXiv  Detail & Related papers  (2024-05-21T13:19:10Z)
• Prompting Large Language Models to Tackle the Full Software Development Lifecycle: A Case Study [72.24266814625685]
  We explore the performance of large language models (LLMs) across the entire software development lifecycle with DevEval. DevEval features four programming languages, multiple domains, high-quality data collection, and carefully designed and verified metrics for each task. Empirical studies show that current LLMs, including GPT-4, fail to solve the challenges presented within DevEval.
  arXiv  Detail & Related papers  (2024-03-13T15:13:44Z)
• Are We Testing or Being Tested? Exploring the Practical Applications of Large Language Models in Software Testing [0.0]
  A Large Language Model (LLM) represents a cutting-edge artificial intelligence model that generates coherent content. LLM can play a pivotal role in software development, including software testing. This study explores the practical application of LLMs in software testing within an industrial setting.
  arXiv  Detail & Related papers  (2023-12-08T06:30:37Z)
• LM-Polygraph: Uncertainty Estimation for Language Models [71.21409522341482]
  Uncertainty estimation (UE) methods are one path to safer, more responsible, and more effective use of large language models (LLMs) We introduce LM-Polygraph, a framework with implementations of a battery of state-of-the-art UE methods for LLMs in text generation tasks, with unified program interfaces in Python. It introduces an extendable benchmark for consistent evaluation of UE techniques by researchers, and a demo web application that enriches the standard chat dialog with confidence scores.
  arXiv  Detail & Related papers  (2023-11-13T15:08:59Z)
• Software Testing with Large Language Models: Survey, Landscape, and Vision [32.34617250991638]
  Pre-trained large language models (LLMs) have emerged as a breakthrough technology in natural language processing and artificial intelligence. This paper provides a comprehensive review of the utilization of LLMs in software testing.
  arXiv  Detail & Related papers  (2023-07-14T08:26:12Z)
• Exploring Software Naturalness through Neural Language Models [56.1315223210742]
  The Software Naturalness hypothesis argues that programming languages can be understood through the same techniques used in natural language processing. We explore this hypothesis through the use of a pre-trained transformer-based language model to perform code analysis tasks.
  arXiv  Detail & Related papers  (2020-06-22T21:56:14Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.