Integrating High-Level Requirements to Low-Level Tests with Machine-Readable V&V Specifications

• URL: http://arxiv.org/abs/2607.17686v1
• Date: Mon, 20 Jul 2026 08:36:54 GMT
• Title: Integrating High-Level Requirements to Low-Level Tests with Machine-Readable V&V Specifications
• Abstract summary: In practice, the link between users' high-level requirements and the low-level tests that machines actually run is maintained by hand, if at all.<n>We introduce VNVSpec, an open-source framework that makes V&V specifications machine-readable and executable.<n>We evaluate the framework by self-application, in which it is continuously assessed in CI against its own specification of 36 requirements verified by 449 tests.
• Score: 0.7874708385247353
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Modern software teams have mature tools for low-level testing, such as pytest, JUnit, and Jest, which make it inexpensive to write unit tests and run them on every commit. Systems engineering, in parallel, has developed rigorous principles for design verification and validation (V&V), which has worked very well across engineering discipline to align user expecations and requirements with developers' deliverables. In practice, however, the two rarely connect, and the link between users' high-level requirements and the low-level tests that machines actually run is maintained by hand, if at all. This gap is increasingly costly for AI-enabled and cyber-physical systems, for which regulators now ask for traceable evidence that high-level requirements are met, while raw test results provide little of the structure such evidence requires. We introduce VNVSpec, an open-source framework that makes V&V specifications machine-readable and executable. With this framework, users state high-level requirements directly or import them from catalogs derived from published standards. Then, the framework checks requirement quality, supports decomposition into module-level requirements with explicit metrics and acceptance criteria, links these requirements to test results through a traceability graph, and compiles the collected evidence into verdicts and audit-ready reports. We evaluate the framework by self-application, in which it is continuously assessed in CI against its own specification of 36 requirements verified by 449 tests, completed within limited time which scales linearly and thus can handle up to 10,000 requirements. We also discuss how the framework extends to testing black-box AI models and AI coding agents. The framework, its full test suite, the catalogs, and the benchmark scripts are available at https://github.com/ai-vnv/vnvspec.

Related papers

• WebTestBench: Evaluating Computer-Use Agents towards End-to-End Automated Web Testing [57.7131457251794]
  We introduce WebTestBench, a benchmark for evaluating end-to-end automated web testing.<n>We decompose the testing process into two cascaded sub-tasks, checklist generation and defect detection, and propose WebTester.<n>Our findings expose a substantial gap between current computer-use agent capabilities and industrial-grade deployment demands.
  arXiv  Detail & Related papers  (2026-03-26T09:27:29Z)
• ARC: Compiling Hundreds of Requirement Scenarios into A Runnable Web System [10.49539144799249]
  Agentic Requirement Compilation (ARC) is a technique that moves beyond simple code generation to requirement compilation.<n>ARC generates not only source code but also modular designs for UI, API, and database layers.<n>A user study with 21 participants showed that novice users can successfully write DSL documents for complex systems.
  arXiv  Detail & Related papers  (2026-02-14T11:07:58Z)
• RAL-Bench: Benchmarking for Application-Level Functional Correctness and Non-Functional Quality Attributes [12.202503919149118]
  RAL-Bench is a benchmark and evaluation framework for application-level code generation.<n>For each task, we distill a concise natural-language requirement from a high-quality reference project.<n>Black-box system tests cover functional and non-functional attributes, and keep only tests that pass on the reference repository.
  arXiv  Detail & Related papers  (2026-02-03T12:35:09Z)
• Automatically Benchmarking LLM Code Agents through Agent-Driven Annotation and Evaluation [47.85891728056131]
  PRDBench is a novel benchmark comprising 50 real-world Python projects across 20 domains, each with structured Product Requirement Document (PRD) requirements, comprehensive evaluation criteria, and reference implementations.<n>We employ an Agent-as-a-Judge paradigm to score agent outputs, enabling the evaluation of various test types beyond unit tests.
  arXiv  Detail & Related papers  (2025-10-28T12:26:45Z)
• Rethinking Testing for LLM Applications: Characteristics, Challenges, and a Lightweight Interaction Protocol [83.83217247686402]
  Large Language Models (LLMs) have evolved from simple text generators into complex software systems that integrate retrieval augmentation, tool invocation, and multi-turn interactions.<n>Their inherent non-determinism, dynamism, and context dependence pose fundamental challenges for quality assurance.<n>This paper decomposes LLM applications into a three-layer architecture: textbftextitSystem Shell Layer, textbftextitPrompt Orchestration Layer, and textbftextitLLM Inference Core.
  arXiv  Detail & Related papers  (2025-08-28T13:00:28Z)
• VERINA: Benchmarking Verifiable Code Generation [46.582574591358735]
  Large language models (LLMs) are increasingly integrated in software development.<n> Ensuring correctness in LLM-generated code remains challenging.<n>Verifiable code generation offers a promising path to address this limitation.
  arXiv  Detail & Related papers  (2025-05-29T06:12:52Z)
• Requirements-Driven Automated Software Testing: A Systematic Review [12.953746641112518]
  This systematic literature review critically examines the current state of requirements input formats, transformation techniques, generated test artifacts, evaluation methods, and prevailing limitations.<n>Our findings highlight the predominance of functional requirements, model-based specifications, and natural language formats.<n>Most existing frameworks are sequential and dependent on singular intermediate representations, and while test cases, structured textual formats, and requirements coverage are common, full automation remains rare.
  arXiv  Detail & Related papers  (2025-02-25T23:13:09Z)
• CLOVER: A Test Case Generation Benchmark with Coverage, Long-Context, and Verification [71.34070740261072]
  This paper presents a benchmark, CLOVER, to evaluate models' capabilities in generating and completing test cases.<n>The benchmark is containerized for code execution across tasks, and we will release the code, data, and construction methodologies.
  arXiv  Detail & Related papers  (2025-02-12T21:42:56Z)
• Commit0: Library Generation from Scratch [77.38414688148006]
  Commit0 is a benchmark that challenges AI agents to write libraries from scratch.<n>Agents are provided with a specification document outlining the library's API as well as a suite of interactive unit tests.<n> Commit0 also offers an interactive environment where models receive static analysis and execution feedback on the code they generate.
  arXiv  Detail & Related papers  (2024-12-02T18:11:30Z)
• Harnessing the Power of LLMs: Automating Unit Test Generation for High-Performance Computing [7.3166218350585135]
  Unit testing is crucial in software engineering for ensuring quality. It's not widely used in parallel and high-performance computing software, particularly scientific applications. We propose an automated method for generating unit tests for such software.
  arXiv  Detail & Related papers  (2024-07-06T22:45:55Z)
• A General Framework for Verification and Control of Dynamical Models via Certificate Synthesis [54.959571890098786]
  We provide a framework to encode system specifications and define corresponding certificates. We present an automated approach to formally synthesise controllers and certificates. Our approach contributes to the broad field of safe learning for control, exploiting the flexibility of neural networks.
  arXiv  Detail & Related papers  (2023-09-12T09:37:26Z)
• Complete Agent-driven Model-based System Testing for Autonomous Systems [0.0]
  A novel approach to testing complex autonomous transportation systems is described. It is intended to mitigate some of the most critical problems regarding verification and validation.
  arXiv  Detail & Related papers  (2021-10-25T01:55:24Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.