Harnessing LLMs for Document-Guided Fuzzing of Python Libraries

• URL: http://arxiv.org/abs/2608.11744v1
• Date: Wed, 12 Aug 2026 07:34:29 GMT
• Title: Harnessing LLMs for Document-Guided Fuzzing of Python Libraries
• Abstract summary: Python libraries underpin deep learning, scientific computing, data analysis, and computer vision.<n>This paper introduces VistaFuzz, a document-guided fuzzing technique.<n>We evaluate VistaFuzz on 7,718 APIs across twelve Python libraries.
• Score: 13.459103316463006
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Python libraries underpin deep learning, scientific computing, data analysis, and computer vision, making their reliability critical to downstream applications. Testing their APIs requires inputs that satisfy both per-parameter constraints and dependencies among parameters. Existing approaches either leave such constraints implicit in generated programs or rely on library-specific parsing rules. This paper introduces VistaFuzz, a document-guided fuzzing technique that uses a locally served open-sourced LLM to extract parameter specifications from API documents and generate inputs that satisfy both parameter constraints and inter-parameter dependencies. We evaluate VistaFuzz on 7,718 APIs across twelve Python libraries. Inter-parameter relationships occur in 40.1\% of tested APIs, and disabling their resolution reduces the valid generation rate on those APIs from above 95\% to 31.6\%--52.8\%. VistaFuzz reports 74 issues, of which 43 have been confirmed by developers and 29 have been fixed.

Related papers

• Knowledge Boundary Probing and Demand-Guided Intervention for LLM-Based Power System Code Generation [8.136015158683694]
  We show that first-pass failures in power-system code generation are dominated by structured API-knowledge boundary: hallucinated function names, misused parameters, and mishandled result tables.<n>We introduce PowerCodeBench, an execution-validated benchmark generator that pairs natural-language operator queries with pandapower code and numerical ground truth.<n>We also introduce an L0-L3 documentation-driven probing procedure that measures per-model API knowledge profiles.
  arXiv  Detail & Related papers  (2026-05-29T16:06:34Z)
• Correct Code, Vulnerable Dependencies: A Large Scale Measurement Study of LLM-Specified Library Versions [52.50730821321986]
  We present the first large-scale measurement study of version-level risk in large language models (LLMs)<n>We evaluate 10 LLMs on PinTrace, a curated benchmark of 1,000 Stack Overflow programming tasks.<n>Our findings reveal LLM version selection as a first-class, previously overlooked risk surface in LLM-based development.
  arXiv  Detail & Related papers  (2026-05-07T13:52:59Z)
• Babbling Suppression: Making LLMs Greener One Token at a Time [46.879983975894135]
  Large Language Models (LLMs) are increasingly used in modern software development.<n>LLMs produce extraneous output, referred to as "babbling", which incurs additional cognitive, economic, and energy costs.<n>This work proposes a practical, model-agnostic approach to reduce unnecessary output without compromising solution accuracy.
  arXiv  Detail & Related papers  (2026-04-08T07:21:02Z)
• Reinforcement Learning for Long-Horizon Interactive LLM Agents [56.9860859585028]
  Interactive digital agents (IDAs) leverage APIs of stateful digital environments to perform tasks in response to user requests.<n>We present a reinforcement learning (RL) approach that trains IDAs directly in their target environments.<n>We derive LOOP, a data- and memory-efficient variant of proximal policy optimization.
  arXiv  Detail & Related papers  (2025-02-03T18:35:42Z)
• Your Fix Is My Exploit: Enabling Comprehensive DL Library API Fuzzing with Large Language Models [49.214291813478695]
  Deep learning (DL) libraries, widely used in AI applications, often contain vulnerabilities like overflows and use buffer-free errors.<n>Traditional fuzzing struggles with the complexity and API diversity of DL libraries.<n>We propose DFUZZ, an LLM-driven fuzzing approach for DL libraries.
  arXiv  Detail & Related papers  (2025-01-08T07:07:22Z)
• LLM Based Input Space Partitioning Testing for Library APIs [13.070272424794744]
  We present an LLM-based input space partitioning testing approach, LISP, for library API testing.<n>We evaluate LISP on more than 2,205 library API methods taken from 10 popular open-source Java libraries.<n>On average, LISP achieves 67.82% branch coverage, surpassing EvoSuite by 1.21 times.
  arXiv  Detail & Related papers  (2024-12-15T17:50:50Z)
• ExploraCoder: Advancing code generation for multiple unseen APIs via planning and chained exploration [70.26807758443675]
  ExploraCoder is a training-free framework that empowers large language models to invoke unseen APIs in code solution.<n> Experimental results demonstrate that ExploraCoder significantly improves performance for models lacking prior API knowledge.
  arXiv  Detail & Related papers  (2024-12-06T19:00:15Z)
• Detecting Multi-Parameter Constraint Inconsistencies in Python Data Science Libraries [21.662640566736098]
  We propose MPDetector, for detecting inconsistencies between code and documentation. MPDetector identifies these constraints at the code level by exploring execution paths through symbolic execution. We propose a customized fuzzy constraint logic to reconcile the unpredictability of LLM outputs.
  arXiv  Detail & Related papers  (2024-11-18T09:30:14Z)
• A Multi-Agent Approach for REST API Testing with Semantic Graphs and LLM-Driven Inputs [46.65963514391019]
  We present AutoRestTest, the first black-box tool to adopt a dependency-embedded multi-agent approach for REST API testing.<n>Our approach treats REST API testing as a separable problem, where four agents collaborate to optimize API exploration.<n>Our evaluation of AutoRestTest on 12 real-world REST services shows that it outperforms the four leading black-box REST API testing tools.
  arXiv  Detail & Related papers  (2024-11-11T16:20:27Z)
• Enhancing Differential Testing With LLMs For Testing Deep Learning Libraries [8.779035160734523]
  This paper introduces an LLM-enhanced differential testing technique for DL libraries.<n>It addresses the challenges of finding alternative implementations for a given API and generating diverse test inputs.<n>It synthesizes counterparts for 1.84 times as many APIs as those found by state-of-the-art techniques.
  arXiv  Detail & Related papers  (2024-06-12T07:06:38Z)
• pyvene: A Library for Understanding and Improving PyTorch Models via Interventions [79.72930339711478]
  $textbfpyvene$ is an open-source library that supports customizable interventions on a range of different PyTorch modules. We show how $textbfpyvene$ provides a unified framework for performing interventions on neural models and sharing the intervened upon models with others.
  arXiv  Detail & Related papers  (2024-03-12T16:46:54Z)
• DocTer: Documentation Guided Fuzzing for Testing Deep Learning API Functions [16.62942039883249]
  We use DocTer to analyze API documentation to extract input constraints for API functions of deep learning (DL) libraries. DocTer features a novel algorithm that automatically constructs rules to extract API parameter constraints from syntactic patterns in the form of dependency parse trees of API descriptions. Our evaluation on three popular DL libraries shows that the precision of DocTer in extracting input constraints is 85.4%.
  arXiv  Detail & Related papers  (2021-09-02T14:57:36Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.