Testing Deep Learning Library APIs via Cross-Framework Differential Fuzzing

• URL: http://arxiv.org/abs/2608.11886v1
• Date: Wed, 12 Aug 2026 10:11:20 GMT
• Title: Testing Deep Learning Library APIs via Cross-Framework Differential Fuzzing
• Abstract summary: Xamt is a cross-framework differential fuzzing approach for deep learning library APIs.<n>It constructs and tests execution-validated groups of APIs intended to implement equivalent operations across seven libraries.<n>Xamt identifies 72 independently reproduced discrepancy cases, including 4 crash cases and 68 output inconsistencies.
• Score: 7.717302467822687
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Deep learning libraries underpin many safety- and reliability-critical applications, yet existing API-level testing techniques often rely on intra-library properties or CPU--GPU differential oracles and may miss defects that behave consistently across hardware backends. We present Xamt, a cross-framework differential fuzzing approach for deep learning library APIs. Xamt constructs and tests execution-validated groups of APIs intended to implement equivalent operations across seven libraries. It uses explicit API aliases and parameter-role normalization to construct candidate correspondences and validates them through pairwise execution and a group-level behavioral check on canonical ordinary inputs. The resulting groups are explored using variance-guided differential fuzzing with ordinary, boundary, and non-finite inputs. Crash and inconsistency oracles flag executions exhibiting abnormal termination or inconsistent outputs for subsequent reproduction and analysis. Across the seven libraries, Xamt constructs 676 execution-validated groups containing 2,563 matched APIs. Among these, Xamt identifies 72 independently reproduced discrepancy cases, including 4 crash cases and 68 output inconsistencies. Among the 72 developer reports, 25 have been confirmed, including 23 that have been fixed.

Related papers

• Apodex Discovery: Reality Benchmarks and Environments for Evaluating and Building Discoverative Artificial Intelligence [86.61561123749011]
  We introduce Apodex Discovery, a framework for building and evaluating discoverative AI through the heavy-duty solver.<n>First, a problem-scouting process surveyed 561 industries across 16 sectors, assembled 423 high-value real-world problems, and selected 20 for the initial release.<n>Second, a common environment-task-episode abstraction provides data, tools, constraints, feedback, trajectory recording, and verification of intermediate artifacts and final submissions.
  arXiv  Detail & Related papers  (2026-08-11T18:48:40Z)
• TRAJDEBUG: Tracing Error Lifecycle to Identify Critical Failures in Long-Horizon Agent Trajectories [48.638067300108276]
  Critical error detection aims to locate the earliest error step in a failed trajectory that is responsible for the final failure.<n>Long trajectories make it difficult to identify individual errors, since the evidence for judging a step may be scattered across distant instructions, observations, and prior context.<n>We propose Traj Debug, an error-lifecycle tracing framework that addresses long-trajectory error discovery.
  arXiv  Detail & Related papers  (2026-08-06T17:51:20Z)
• ATTAIN: Automated Exploit Failure Analysis through Trace-Driven Diff Analysis [9.619949003948392]
  ATTAIN is a trace-driven diff analysis framework to assess vulnerability presence across evolving library versions.<n>ATTAIN achieves an F1-score of 93.24%, outperforming the commit-based methods V-SZZ and LLM4SZZ by 116.28% and 33.30%, respectively.
  arXiv  Detail & Related papers  (2026-06-08T05:49:50Z)
• Dynamic analysis enhances issue resolution [53.50448142467294]
  DAIRA (Dynamic Analysis-enhanced Issue Resolution Agent) is an automated repair framework that embeds dynamic analysis into the agent's reasoning cycle.<n> Driven by a Test Tracing-Driven methodology, DAIRA utilizes lightweight monitors to extract critical runtime data.<n>When powered by Gemini 3 Flash Preview, DAIRA establishes a new state-of-the-art (SOTA) performance, achieving a 79.4% resolution rate on the SWE-bench Verified dataset.
  arXiv  Detail & Related papers  (2026-03-23T14:48:54Z)
• Outrunning LLM Cutoffs: A Live Kernel Crash Resolution Benchmark for All [57.23434868678603]
  Live-kBench is an evaluation framework for self-evolving benchmarks that scrapes and evaluates agents on freshly discovered kernel bugs.<n> kEnv is an agent-agnostic crash-resolution environment for kernel compilation, execution, and feedback.<n>Using kEnv, we benchmark three state-of-the-art agents, showing that they resolve 74% of crashes on the first attempt.
  arXiv  Detail & Related papers  (2026-02-02T19:06:15Z)
• XAMT: Cross-Framework API Matching for Testing Deep Learning Libraries [6.00258665435786]
  We propose XAMT, a cross-framework fuzzing method for testing deep learning libraries.<n>XAMT matches APIs using similarity-based rules based on names, descriptions, and parameter structures.<n>It then aligns inputs and applies variance-guided testing to detect bugs.
  arXiv  Detail & Related papers  (2025-08-18T00:48:50Z)
• Are You Getting What You Pay For? Auditing Model Substitution in LLM APIs [71.7892165868749]
  Commercial Large Language Model (LLM) APIs create a fundamental trust problem.<n>Users pay for specific models but have no guarantee that providers deliver them faithfully.<n>We formalize this model substitution problem and evaluate detection methods under realistic adversarial conditions.<n>We propose and evaluate the use of Trusted Execution Environments (TEEs) as one practical and robust solution.
  arXiv  Detail & Related papers  (2025-04-07T03:57:41Z)
• Your Fix Is My Exploit: Enabling Comprehensive DL Library API Fuzzing with Large Language Models [49.214291813478695]
  Deep learning (DL) libraries, widely used in AI applications, often contain vulnerabilities like overflows and use buffer-free errors.<n>Traditional fuzzing struggles with the complexity and API diversity of DL libraries.<n>We propose DFUZZ, an LLM-driven fuzzing approach for DL libraries.
  arXiv  Detail & Related papers  (2025-01-08T07:07:22Z)
• Subgraph-Oriented Testing for Deep Learning Libraries [9.78188667672054]
  We propose SORT (Subgraph-Oriented Realistic Testing) to test Deep Learning (DL) libraries on different hardware platforms.<n>SORT takes popular API interaction patterns, represented as frequent subgraphs of model graphs, as test subjects.<n>SORT achieves a 100% valid input generation rate, detects more precision bugs than existing methods, and reveals interaction-related bugs missed by single-API testing.
  arXiv  Detail & Related papers  (2024-12-09T12:10:48Z)
• Commit0: Library Generation from Scratch [77.38414688148006]
  Commit0 is a benchmark that challenges AI agents to write libraries from scratch.<n>Agents are provided with a specification document outlining the library's API as well as a suite of interactive unit tests.<n> Commit0 also offers an interactive environment where models receive static analysis and execution feedback on the code they generate.
  arXiv  Detail & Related papers  (2024-12-02T18:11:30Z)
• Detecting Multi-Parameter Constraint Inconsistencies in Python Data Science Libraries [21.662640566736098]
  We propose MPDetector, for detecting inconsistencies between code and documentation. MPDetector identifies these constraints at the code level by exploring execution paths through symbolic execution. We propose a customized fuzzy constraint logic to reconcile the unpredictability of LLM outputs.
  arXiv  Detail & Related papers  (2024-11-18T09:30:14Z)
• Enhancing Differential Testing With LLMs For Testing Deep Learning Libraries [8.779035160734523]
  This paper introduces an LLM-enhanced differential testing technique for DL libraries.<n>It addresses the challenges of finding alternative implementations for a given API and generating diverse test inputs.<n>It synthesizes counterparts for 1.84 times as many APIs as those found by state-of-the-art techniques.
  arXiv  Detail & Related papers  (2024-06-12T07:06:38Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.