FlyCatcher: Neural Inference of Runtime Checkers from Tests

• URL: http://arxiv.org/abs/2604.22028v1
• Date: Thu, 23 Apr 2026 19:43:53 GMT
• Title: FlyCatcher: Neural Inference of Runtime Checkers from Tests
• Abstract summary: FlyCatcher is an automated approach to derive runtime checkers from existing tests.<n>It infers a checker that monitors specific method calls and asserts properties that should hold when they are called.<n>Compared with a state-of-the-art approach, our approach infers 2.6x more correct checkers, which enables it to detect 5.2x more errors.
• Score: 15.772540292298151
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Complex software systems often suffer from silent failures, i.e., violations of the intended semantics that do not cause explicit errors. A promising approach to detect such errors is to use system-specific runtime checkers that monitor the execution of a system and check for violations of the intended semantics. However, writing such checkers for a given software system is challenging and time-consuming, and hence, rarely done in practice. This work presents FlyCatcher, an automated approach to derive runtime checkers from existing tests, i.e., from a resource available for most software systems. The critical challenge of such an approach is to generalize the behavioral properties encoded in a test case to arbitrary executions of a system. FlyCatcher addresses this challenge through a combination of LLM-based synthesis, static analysis, and dynamic validation, which infers a checker that monitors specific method calls and asserts properties that should hold when they are called. The inferred checkers are stateful, i.e., they reason about the system's behavior by maintaining a shadow state that abstracts the actual system state as needed by the checker. Our evaluation applies FlyCatcher to 400 tests from four widely used, complex software systems. The approach infers 334 checkers, out of which 300 are found to be correct via cross-validation. Compared with a state-of-the-art approach, our approach infers 2.6x more correct checkers, which enables it to detect 5.2x more errors. By contributing to the automated inference of runtime checkers from tests, this work enables the broader adoption of runtime checking as a practical approach to detect silent failures in complex software systems.

Related papers

• Fantastic Adaptive Taxonomies and How to Use Them [100.20614173157708]
  AdaMAST converts execution traces into a compact, evidence-grounded failure taxonomy.<n>No code is hand-authored, no trace human-annotated.<n>In agent-system search, taxonomy-coded diagnoses of failed candidates outperform free-form.
  arXiv  Detail & Related papers  (2026-07-17T17:41:16Z)
• In Perfect Harmony: Orchestrating Causality in Actor-Based Systems [0.0]
  Actor-based systems present significant challenges for runtime verification.<n>ACTORCHESTRA is a runtime verification framework for Erlang that automatically tracks causality across multi-actor interactions.<n>Three case studies demonstrate ACTORCHESTRA's effectiveness in detecting complex behavioral violations in real-world actor systems.
  arXiv  Detail & Related papers  (2026-03-18T16:47:25Z)
• Automated Self-Testing as a Quality Gate: Evidence-Driven Release Management for LLM Applications [51.56484100374058]
  We present an automated self-testing framework that introduces quality gates with evidence-based release decisions.<n>We evaluate the framework through a longitudinal case study of an internally deployed multi-agent conversational AI system.
  arXiv  Detail & Related papers  (2026-03-13T20:44:15Z)
• Detecting Object Tracking Failure via Sequential Hypothesis Testing [80.7891291021747]
  Real-time online object tracking in videos constitutes a core task in computer vision.<n>We propose interpreting object tracking as a sequential hypothesis test, wherein evidence for or against tracking failures is gradually accumulated over time.<n>We propose both supervised and unsupervised variants by leveraging either ground-truth or solely internal tracking information.
  arXiv  Detail & Related papers  (2026-02-13T14:57:15Z)
• Reactive Bottom-Up Testing [15.280664862119565]
  We introduce a new paradigm that we call Reactive Bottom-Up Testing.<n>Our insight is that function-level testing is necessary but not sufficient for the validation of vulnerabilities in functions.<n>We develop a three-stage bottom-up testing scheme that identifies likely-vulnerable functions and generates type- and context-aware harnesses.
  arXiv  Detail & Related papers  (2025-09-03T20:54:43Z)
• An Automated Blackbox Noncompliance Checker for QUIC Server Implementations [2.9248916859490173]
  QUICtester is an automated approach for uncovering non-compliant behaviors in the ratified QUIC protocol implementations (RFC 9000/).<n>We used QUICtester to analyze 186 learned models from 19 QUIC implementations under the five security settings and discovered 55 implementation errors.
  arXiv  Detail & Related papers  (2025-05-19T04:28:49Z)
• A Case Study on Model Checking and Runtime Verification for Awkernel [0.7199733380797578]
  It is difficult for humans to manually review concurrent behaviors or to write test cases covering all possible executions.<n>This paper proposes a development method for concurrent software, such as schedulers.
  arXiv  Detail & Related papers  (2025-03-12T11:27:45Z)
• Write Your Own CodeChecker: An Automated Test-Driven Checker Development Approach with LLMs [9.551021559603349]
  AutoChecker is an innovative approach that can write code checkers automatically based on only a rule description and a test suite.<n>Test results show that AutoChecker significantly outperforms others across all effectiveness metrics, with an average test pass rate of 82.28%.
  arXiv  Detail & Related papers  (2024-11-11T08:50:24Z)
• Bisimulation Learning [55.859538562698496]
  We compute finite bisimulations of state transition systems with large, possibly infinite state space. Our technique yields faster verification results than alternative state-of-the-art tools in practice.
  arXiv  Detail & Related papers  (2024-05-24T17:11:27Z)
• PULL: Reactive Log Anomaly Detection Based On Iterative PU Learning [58.85063149619348]
  We propose PULL, an iterative log analysis method for reactive anomaly detection based on estimated failure time windows. Our evaluation shows that PULL consistently outperforms ten benchmark baselines across three different datasets.
  arXiv  Detail & Related papers  (2023-01-25T16:34:43Z)
• Beyond Accuracy: Behavioral Testing of NLP models with CheckList [66.42971817954806]
  CheckList is a task-agnostic methodology for testing NLP models. CheckList includes a matrix of general linguistic capabilities and test types that facilitate comprehensive test ideation. In a user study, NLP practitioners with CheckList created twice as many tests, and found almost three times as many bugs as users without it.
  arXiv  Detail & Related papers  (2020-05-08T15:48:31Z)
• Claim Check-Worthiness Detection as Positive Unlabelled Learning [53.24606510691877]
  Claim check-worthiness detection is a critical component of fact checking systems. We illuminate a central challenge in claim check-worthiness detection underlying all of these tasks. Our best performing method is a unified approach which automatically corrects for this using a variant of positive unlabelled learning.
  arXiv  Detail & Related papers  (2020-03-05T16:06:07Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.