Large Language Models Based JSON Parser Fuzzing for Bug Discovery and Behavioral Analysis

• URL: http://arxiv.org/abs/2410.21806v2
• Date: Wed, 30 Oct 2024 02:28:41 GMT
• Title: Large Language Models Based JSON Parser Fuzzing for Bug Discovery and Behavioral Analysis
• Authors: Zhiyuan Zhong, Zhezhen Cao, Zhanwei Zhang,
• Abstract summary: This research project focuses on leveraging Large Language Models (LLMs) to enhance testing. The primary objectives are to generate test cases and mutants using LLMs for the discovery of potential bugs in open-sources. We aim to uncover underlying bugs, plus discovering (and overcoming) behavioral diversities.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Fuzzing has been incredibly successful in uncovering bugs and vulnerabilities across diverse software systems. JSON parsers play a vital role in modern software development, and ensuring their reliability is of great importance. This research project focuses on leveraging Large Language Models (LLMs) to enhance JSON parser testing. The primary objectives are to generate test cases and mutants using LLMs for the discovery of potential bugs in open-source JSON parsers and the identification of behavioral diversities among them. We aim to uncover underlying bugs, plus discovering (and overcoming) behavioral diversities.

Related papers

• Detecting Functional Bugs in Smart Contracts through LLM-Powered and Bug-Oriented Composite Analysis [34.8337182669106]
  We design and implement PROMFUZZ, an automated and scalable system to detect functional bugs in smart contracts. We first propose a novel Large Language Model (LLM)-driven analysis framework, which leverages a dual-agent prompt engineering strategy. Finally, we design a bug-oriented fuzzing engine, which maps the logical information from the high-level business model to the low-level smart contract implementations.
  arXiv  Detail & Related papers  (2025-03-31T04:39:51Z)
• LLPut: Investigating Large Language Models for Bug Report-Based Input Generation [0.0]
  Failure-inducing inputs play a crucial role in diagnosing and analyzing software bugs. Prior research has leveraged various Natural Language Processing (NLP) techniques for automated input extraction. With the advent of Large Language Models (LLMs), an important research question arises: how effectively can generative LLMs extract failure-inducing inputs from bug reports?
  arXiv  Detail & Related papers  (2025-03-26T14:25:01Z)
• Learning to Generate Structured Output with Schema Reinforcement Learning [83.09230124049667]
  This study investigates the structured generation capabilities of large language models (LLMs) We find that the latest LLMs are still struggling to generate a valid string. Our models demonstrate significant improvement in both generating outputs and downstream tasks.
  arXiv  Detail & Related papers  (2025-02-26T06:45:29Z)
• Design choices made by LLM-based test generators prevent them from finding bugs [0.850206009406913]
  This paper critically examines whether recent LLM-based test generation tools, such as Codium CoverAgent and CoverUp, can effectively find bugs or unintentionally validate faulty code. Using real human-written buggy code as input, we evaluate these tools, showing how LLM-generated tests can fail to detect bugs and, more alarmingly, how their design can worsen the situation by validating bugs in the generated test suite and rejecting bug-revealing tests.
  arXiv  Detail & Related papers  (2024-12-18T18:33:26Z)
• Are Large Language Models Memorizing Bug Benchmarks? [6.640077652362016]
  Large Language Models (LLMs) have become integral to various software engineering tasks, including code generation, bug detection, and repair. A growing concern within the software engineering community is that benchmarks may not reliably reflect true LLM performance due to the risk of data leakage. We systematically evaluate popular LLMs to assess their susceptibility to data leakage from widely used bug benchmarks.
  arXiv  Detail & Related papers  (2024-11-20T13:46:04Z)
• Advancing Bug Detection in Fastjson2 with Large Language Models Driven Unit Test Generation [8.977049061406325]
  Unit test generation techniques are widely adopted to identify bugs in various libraries. There is limited systematic testing effort specifically for exposing oracle bugs within libraries in industrial practice. WithTestGen, we found 34 real bugs in fast2, 30 of which have already been fixed, including 12 non-crashing bugs.
  arXiv  Detail & Related papers  (2024-10-12T07:46:05Z)
• MEGen: Generative Backdoor in Large Language Models via Model Editing [56.46183024683885]
  Large language models (LLMs) have demonstrated remarkable capabilities. Their powerful generative abilities enable flexible responses based on various queries or instructions. This paper proposes an editing-based generative backdoor, named MEGen, aiming to create a customized backdoor for NLP tasks with the least side effects.
  arXiv  Detail & Related papers  (2024-08-20T10:44:29Z)
• Leveraging Large Language Models for Efficient Failure Analysis in Game Development [47.618236610219554]
  This paper proposes a new approach to automatically identify which change in the code caused a test to fail. The method leverages Large Language Models (LLMs) to associate error messages with the corresponding code changes causing the failure. Our approach reaches an accuracy of 71% in our newly created dataset, which comprises issues reported by developers at EA over a period of one year.
  arXiv  Detail & Related papers  (2024-06-11T09:21:50Z)
• DebugBench: Evaluating Debugging Capability of Large Language Models [80.73121177868357]
  DebugBench is a benchmark for Large Language Models (LLMs) It covers four major bug categories and 18 minor types in C++, Java, and Python. We evaluate two commercial and four open-source models in a zero-shot scenario.
  arXiv  Detail & Related papers  (2024-01-09T15:46:38Z)
• Generative Judge for Evaluating Alignment [84.09815387884753]
  We propose a generative judge with 13B parameters, Auto-J, designed to address these challenges. Our model is trained on user queries and LLM-generated responses under massive real-world scenarios. Experimentally, Auto-J outperforms a series of strong competitors, including both open-source and closed-source models.
  arXiv  Detail & Related papers  (2023-10-09T07:27:15Z)
• What Happens When We Fuzz? Investigating OSS-Fuzz Bug History [0.9772968596463595]
  We analyzed 44,102 reported issues made public by OSS-Fuzz prior to March 12, 2022. We identified the bug-contributing commits to estimate when the bug containing code was introduced, and measure the timeline from introduction to detection to fix.
  arXiv  Detail & Related papers  (2023-05-19T05:15:36Z)
• Large Language Models are Few-shot Testers: Exploring LLM-based General Bug Reproduction [14.444294152595429]
  The number of tests added in open source repositories due to issues was about 28% of the corresponding project test suite size. We propose LIBRO, a framework that uses Large Language Models (LLMs), which have been shown to be capable of performing code-related tasks. Our evaluation of LIBRO shows that, on the widely studied Defects4J benchmark, LIBRO can generate failure reproducing test cases for 33% of all studied cases.
  arXiv  Detail & Related papers  (2022-09-23T10:50:47Z)
• Adversarial GLUE: A Multi-Task Benchmark for Robustness Evaluation of Language Models [86.02610674750345]
  Adversarial GLUE (AdvGLUE) is a new multi-task benchmark to explore and evaluate the vulnerabilities of modern large-scale language models under various types of adversarial attacks. We apply 14 adversarial attack methods to GLUE tasks to construct AdvGLUE, which is further validated by humans for reliable annotations. All the language models and robust training methods we tested perform poorly on AdvGLUE, with scores lagging far behind the benign accuracy.
  arXiv  Detail & Related papers  (2021-11-04T12:59:55Z)
• On the Robustness of Language Encoders against Grammatical Errors [66.05648604987479]
  We collect real grammatical errors from non-native speakers and conduct adversarial attacks to simulate these errors on clean text data. Results confirm that the performance of all tested models is affected but the degree of impact varies.
  arXiv  Detail & Related papers  (2020-05-12T11:01:44Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.