Client--Library Compatibility Testing with API Interaction Snapshots

• URL: http://arxiv.org/abs/2507.20814v1
• Date: Mon, 28 Jul 2025 13:22:19 GMT
• Title: Client--Library Compatibility Testing with API Interaction Snapshots
• Authors: Gustave Monce, Thomas Degueule, Jean-Rémy Falleri, Romain Robbes,
• Abstract summary: behavioral breaking changes (BBCs) can silently break client applications without being detected at compile time.<n>Traditional regression tests on the client side often fail to detect such BBCs.<n>We propose a novel approach to client--library compatibility testing that leverages existing client tests in a novel way.
• Score: 6.2272693929051295
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Modern software development heavily relies on third-party libraries to speed up development and enhance quality. As libraries evolve, they may break the tacit contract established with their clients by introducing behavioral breaking changes (BBCs) that alter run-time behavior and silently break client applications without being detected at compile time. Traditional regression tests on the client side often fail to detect such BBCs, either due to limited library coverage or weak assertions that do not sufficiently exercise the library's expected behavior. To address this issue, we propose a novel approach to client--library compatibility testing that leverages existing client tests in a novel way. Instead of relying on developer-written assertions, we propose recording the actual interactions at the API boundary during the execution of client tests (protocol, input and output values, exceptions, etc.). These sequences of API interactions are stored as snapshots which capture the exact contract expected by a client at a specific point in time. As the library evolves, we compare the original and new snapshots to identify perturbations in the contract, flag potential BBCs, and notify clients. We implement this technique in our prototype tool Gilesi, a Java framework that automatically instruments library APIs, records snapshots, and compares them. Through a preliminary case study on several client--library pairs with artificially seeded BBCs, we show that Gilesi reliably detects BBCs missed by client test suites.

Related papers

• Latte: Collaborative Test-Time Adaptation of Vision-Language Models in Federated Learning [66.35456997311742]
  Test-time adaptation with pre-trained vision-language models has gained increasing attention for addressing distribution shifts during testing.<n>Existing test-time adaptation methods are typically designed for a single domain with abundant data.<n>We propose Latte, a novel framework where each client maintains a local memory to store embeddings from its own historical test data.
  arXiv  Detail & Related papers  (2025-07-29T04:27:29Z)
• Understanding API Usage and Testing: An Empirical Study of C Libraries [0.2532202013576546]
  This study is the first to compare API usage and API testing at scale for the C/C++ ecosystem.<n>For our empirical study, we have developed LibProbe, a framework that can be used to analyse a large corpus of clients for a given library.
  arXiv  Detail & Related papers  (2025-06-13T09:07:16Z)
• LlamaRestTest: Effective REST API Testing with Small Language Models [50.058600784556816]
  We present LlamaRestTest, a novel approach that employs two custom Large Language Models (LLMs) to generate realistic test inputs.<n>We evaluate it against several state-of-the-art REST API testing tools, including RESTGPT, a GPT-powered specification-enhancement tool.<n>Our study shows that small language models can perform as well as, or better than, large language models in REST API testing.
  arXiv  Detail & Related papers  (2025-01-15T05:51:20Z)
• Commit0: Library Generation from Scratch [77.38414688148006]
  Commit0 is a benchmark that challenges AI agents to write libraries from scratch.<n>Agents are provided with a specification document outlining the library's API as well as a suite of interactive unit tests.<n> Commit0 also offers an interactive environment where models receive static analysis and execution feedback on the code they generate.
  arXiv  Detail & Related papers  (2024-12-02T18:11:30Z)
• Model Equality Testing: Which Model Is This API Serving? [59.005869726179455]
  API providers may quantize, watermark, or finetune the underlying model, changing the output distribution.<n>We formalize detecting such distortions by Model Equality Testing, a two-sample testing problem.<n>A test built on a simple string kernel achieves a median of 77.4% power against a range of distortions.
  arXiv  Detail & Related papers  (2024-10-26T18:34:53Z)
• KAT: Dependency-aware Automated API Testing with Large Language Models [1.7264233311359707]
  KAT (Katalon API Testing) is a novel AI-driven approach that autonomously generates test cases to validate APIs. Our evaluation of KAT using 12 real-world services shows that it can improve validation coverage, detect more undocumented status codes, and reduce false positives in these services.
  arXiv  Detail & Related papers  (2024-07-14T14:48:18Z)
• Lightweight Syntactic API Usage Analysis with UCov [0.0]
  We present a novel conceptual framework designed to assist library maintainers in understanding the interactions allowed by their APIs. These customizable models enable library maintainers to improve their design ahead of release, reducing friction during evolution. We implement these models for Java libraries in a new tool UCov and demonstrate its capabilities on three libraries exhibiting diverse styles of interaction.
  arXiv  Detail & Related papers  (2024-02-19T10:33:41Z)
• CompSuite: A Dataset of Java Library Upgrade Incompatibility Issues [25.189328666070107]
  We introduce CompSuite, a dataset that includes 123 real-world Java client-library pairs where upgrading the library causes an incompatibility issue. Each incompatibility issue in CompSuite is associated with a test case authored by the developers, which can be used to reproduce the issue.
  arXiv  Detail & Related papers  (2023-05-15T14:26:14Z)
• SequeL: A Continual Learning Library in PyTorch and JAX [50.33956216274694]
  SequeL is a library for Continual Learning that supports both PyTorch and JAX frameworks. It provides a unified interface for a wide range of Continual Learning algorithms, including regularization-based approaches, replay-based approaches, and hybrid approaches. We release SequeL as an open-source library, enabling researchers and developers to easily experiment and extend the library for their own purposes.
  arXiv  Detail & Related papers  (2023-04-21T10:00:22Z)
• REaaS: Enabling Adversarially Robust Downstream Classifiers via Robust Encoder as a Service [67.0982378001551]
  We show how a service provider pre-trains an encoder and then deploys it as a cloud service API. A client queries the cloud service API to obtain feature vectors for its training/testing inputs. We show that the cloud service only needs to provide two APIs to enable a client to certify the robustness of its downstream classifier.
  arXiv  Detail & Related papers  (2023-01-07T17:40:11Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.