Protocol Testing with I/O Grammars

• URL: http://arxiv.org/abs/2509.20308v1
• Date: Wed, 24 Sep 2025 16:41:04 GMT
• Title: Protocol Testing with I/O Grammars
• Authors: Alexander Liggesmeyer, José Antonio Zamudio Amaya, Andreas Zeller,
• Abstract summary: We propose a novel approach to protocol testing that combines input generation and output checking in a single framework.<n>We demonstrate that I/O grammars can specify advanced protocol features correctly and completely, while also enabling output validation of the programs under test.
• Score: 45.68497486907946
• License: http://creativecommons.org/licenses/by-sa/4.0/
• Abstract: Generating software tests faces two fundamental problems. First, one needs to _generate inputs_ that are syntactically and semantically correct, yet sufficiently diverse to cover behavior. Second, one needs an _oracle_ to _check outputs_ whether a test case is correct or not. Both problems become apparent in _protocol testing_, where inputs are messages exchanged between parties, and outputs are the responses of these parties. In this paper, we propose a novel approach to protocol testing that combines input generation and output checking in a single framework. We introduce _I/O grammars_ as the first means to _completely_ specify the syntax and semantics of protocols, including messages, states, and interactions. Our implementation, based on the FANDANGO framework, takes a single I/O grammar, and can act as a _test generator_, as a _mock object_, and as an _oracle_ for a _client_, a _server_, or both (or actually any number of parties), a versatility not found in any existing tool or formalism. User-defined _constraints}_can have the generator focus on arbitrary protocol features; $k$-path guidance systematically covers states, messages, responses, and value alternatives in a unified fashion. We evaluate the effectiveness of our approach by applying it to several protocols, including DNS, FTP, and SMTP. We demonstrate that I/O grammars can specify advanced protocol features correctly and completely, while also enabling output validation of the programs under test. In its evaluation, we find that systematic coverage of the I/O grammar results in much quicker coverage of the input and response spaces (and thus functionality) compared to the random-based state-of-the-art approaches.

Related papers

• BRIDGE: Building Representations In Domain Guided Program Verification [67.36686119518441]
  BRIDGE decomposes verification into three interconnected domains: Code, Specifications, and Proofs.<n>We show that this approach substantially improves both accuracy and efficiency beyond standard error feedback methods.
  arXiv  Detail & Related papers  (2025-11-26T06:39:19Z)
• Synthesizing Precise Protocol Specs from Natural Language for Effective Test Generation [42.582977261473324]
  AutoSPEC recovers on average 92.8% of client and 80.2% of server message types, and 81.5% message acceptance across diverse real-world systems.<n>Prototype successfully demonstrated the feasibility of our approach on five widely used.<n>internet-based protocols.
  arXiv  Detail & Related papers  (2025-11-22T08:39:52Z)
• Rethinking Testing for LLM Applications: Characteristics, Challenges, and a Lightweight Interaction Protocol [83.83217247686402]
  Large Language Models (LLMs) have evolved from simple text generators into complex software systems that integrate retrieval augmentation, tool invocation, and multi-turn interactions.<n>Their inherent non-determinism, dynamism, and context dependence pose fundamental challenges for quality assurance.<n>This paper decomposes LLM applications into a three-layer architecture: textbftextitSystem Shell Layer, textbftextitPrompt Orchestration Layer, and textbftextitLLM Inference Core.
  arXiv  Detail & Related papers  (2025-08-28T13:00:28Z)
• iPanda: An LLM-based Agent for Automated Conformance Testing of Communication Protocols [11.749977502129898]
  Large Language Models (LLMs) have demonstrated impressive text comprehension and code generation abilities.<n>We propose iPanda, the first framework that leverages LLMs to automate protocol conformance testing.<n>Experiments on various protocols show that iPanda significantly outperforms pure LLM-based approaches.
  arXiv  Detail & Related papers  (2025-07-01T02:27:44Z)
• Validating Network Protocol Parsers with Traceable RFC Document Interpretation [11.081773172066766]
  oracle and traceability problems determine when a protocol implementation can be considered buggy.<n>This work considers both and provides an effective solution using recent advances in large language models (LLMs)<n>We have extensively evaluated our approach using nine network protocols and their implementations written in C, Python, and Go.
  arXiv  Detail & Related papers  (2025-04-25T03:39:19Z)
• Large Language Models for Validating Network Protocol Parsers [8.007994733372675]
  Protocol standards are typically written in natural language, whereas implementations are in source code.<n>We propose PARVAL, a framework built on large language models (LLMs)<n>It transforms both protocol standards and their implementations into a unified intermediate representation, referred to as format specifications.<n>It successfully identifies inconsistencies between the implementation and its RFC standard, achieving a low false positive rate of 5.6%.
  arXiv  Detail & Related papers  (2025-04-18T07:09:56Z)
• DT-SIM: Property-Based Testing for MPC Security [2.0308771704846245]
  Property-based testing is effective for detecting security bugs in secure protocols. We specifically target Secure Multi-Party Computation (MPC) We devise a test that can detect various flaws in a bit-level implementation of an MPC protocol.
  arXiv  Detail & Related papers  (2024-03-08T02:02:24Z)
• Semi-DETR: Semi-Supervised Object Detection with Detection Transformers [105.45018934087076]
  We analyze the DETR-based framework on semi-supervised object detection (SSOD) We present Semi-DETR, the first transformer-based end-to-end semi-supervised object detector. Our method outperforms all state-of-the-art methods by clear margins.
  arXiv  Detail & Related papers  (2023-07-16T16:32:14Z)
• APANet: Adaptive Prototypes Alignment Network for Few-Shot Semantic Segmentation [56.387647750094466]
  Few-shot semantic segmentation aims to segment novel-class objects in a given query image with only a few labeled support images. Most advanced solutions exploit a metric learning framework that performs segmentation through matching each query feature to a learned class-specific prototype. We present an adaptive prototype representation by introducing class-specific and class-agnostic prototypes.
  arXiv  Detail & Related papers  (2021-11-24T04:38:37Z)
• Query and Extract: Refining Event Extraction as Type-oriented Binary Decoding [51.57864297948228]
  We propose a novel event extraction framework that takes event types and argument roles as natural language queries. Our framework benefits from the attention mechanisms to better capture the semantic correlation between the event types or argument roles and the input text.
  arXiv  Detail & Related papers  (2021-10-14T15:49:40Z)
• VL-NMS: Breaking Proposal Bottlenecks in Two-Stage Visual-Language Matching [75.71523183166799]
  The prevailing framework for matching multimodal inputs is based on a two-stage process. We argue that these methods overlook an obvious emphmismatch between the roles of proposals in the two stages. We propose VL-NMS, which is the first method to yield query-aware proposals at the first stage.
  arXiv  Detail & Related papers  (2021-05-12T13:05:25Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.