Automatically Write Code Checker: An LLM-based Approach with Logic-guided API Retrieval and Case by Case Iteration

• URL: http://arxiv.org/abs/2411.06796v1
• Date: Mon, 11 Nov 2024 08:50:24 GMT
• Title: Automatically Write Code Checker: An LLM-based Approach with Logic-guided API Retrieval and Case by Case Iteration
• Abstract summary: AutoChecker is an innovative approach to write code checkers based on only a rule description and a test suite. It incrementally updates the checker with the rule and one single test case each time. The checkers generated by AutoChecker are successfully applied to real-world projects, matching the performance of official checkers.
• Score: 9.551021559603349
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: With the rising demand for code quality assurance, developers are not only utilizing existing static code checkers but also seeking custom checkers to satisfy their specific needs. Nowadays, various code-checking frameworks provide extensive checker customization interfaces to meet this need. However, both the abstract checking logic as well as the complex API usage of large-scale frameworks make this task challenging. To this end, automated code checker generation is anticipated to ease the burden of checker development. In this paper, we explore the feasibility of automated checker generation and propose AutoChecker, an innovative LLM-powered approach that can write code checkers automatically based on only a rule description and a test suite. Instead of generating the checker at once, AutoChecker incrementally updates the checker with the rule and one single test case each time, i.e., it iteratively generates the checker case by case. During each iteration, AutoChecker first decomposes the whole logic into a series of sub-operations and then uses the logic-guided API-context retrieval strategy to search related API-contexts from all the framework APIs. To evaluate the effectiveness of AutoChecker, we apply AutoChecker and two LLM-based baseline approaches to automatically generate checkers for 20 built-in PMD rules, including easy rules and hard rules. Experimental results demonstrate that AutoChecker significantly outperforms baseline approaches across all effectiveness metrics, where its average test pass rate improved over 4.2 times. Moreover, the checkers generated by AutoChecker are successfully applied to real-world projects, matching the performance of official checkers.

Related papers

• MTAC-IFBench: Benchmarking Instruction-Following in Multi-Turn Agentic Coding [83.0805548642958]
  MTAC-IFBench is a comprehensive benchmark for instruction-following in software development.<n>It features multi-turn progressive software development instructions with diverse constraints spanning 6 primary and 18 secondary categories.<n>MTAC-IFBench identifies significant deficiencies in existing code agents in multi-turn instruction-following, with their performance degrading rapidly as the interaction session grows longer.
  arXiv  Detail & Related papers  (2026-09-14T03:57:37Z)
• AutoScientist-Quant: Self-Evolving Coding Agents for Automatic Research in Quantitative Investment [28.36446889018906]
  We present AutoScientist-Quant, a self evolving search process that regards quantitative research as one budgeted search problem.<n>A single controller conditions every decision on the remaining budget, choosing at each round whether to improve, combine, pivot, or stop, which node to expand, how many alphas to generate, and how to retrieve past trajectories from the shared memory.<n>The framework attains the best value of nearly every metric in every setting, and these conclusions hold across several backbones and markets.
  arXiv  Detail & Related papers  (2026-08-05T23:34:46Z)
• Self-Improving AI Coding Agents Through Accumulated Behavioral Rules: A Closed-Loop Framework [0.3867363075280543]
  We present a closed-loop framework in which every accepted review comment is codified as a persistent behavioral rule.<n>The framework combines an accumulating rule set in a version-controlled instruction file, a self-review checklist, and automated validation.<n>We show that our framework achieves persistent cross-session learning without weight updates, operates on productions rather than synthetic benchmarks, and addresses an experiential dimension (behavioral consistency over time) that existing benchmarks do not measure.
  arXiv  Detail & Related papers  (2026-07-13T21:13:46Z)
• FineVerify: Scaling Test-Time Compute with Fine-Grained Self-Verification for Agentic Search [88.16262636915975]
  FineVerify is a fine-grained self-verification framework for agentic search.<n>It decomposes each question into checkable sub-questions, verifies sampled candidates, and selects the candidate with the highest aggregated score.<n>FineVerify consistently outperforms standard scaling baselines.
  arXiv  Detail & Related papers  (2026-05-30T10:21:20Z)
• FlyCatcher: Neural Inference of Runtime Checkers from Tests [15.772540292298151]
  FlyCatcher is an automated approach to derive runtime checkers from existing tests.<n>It infers a checker that monitors specific method calls and asserts properties that should hold when they are called.<n>Compared with a state-of-the-art approach, our approach infers 2.6x more correct checkers, which enables it to detect 5.2x more errors.
  arXiv  Detail & Related papers  (2026-04-23T19:43:53Z)
• AutoChecklist: Composable Pipelines for Checklist Generation and Scoring with LLM-as-a-Judge [16.729112949512807]
  AutoChecklist is an open-source library that unifies checklist-based evaluation into composable pipelines.<n>A modular Generator $rightarrow$ Refiner $rightarrow$ Scorer pipeline connects any generator with a unified scorer.<n>Beyond the Python API, the library includes a CLI for off-the-shelf evaluation and a web interface for interactive exploration.
  arXiv  Detail & Related papers  (2026-03-07T03:36:17Z)
• CompassVerifier: A Unified and Robust Verifier for LLMs Evaluation and Outcome Reward [50.97588334916863]
  We develop CompassVerifier, an accurate and robust lightweight verifier model for evaluation and outcome reward.<n>It demonstrates multi-domain competency spanning math, knowledge, and diverse reasoning tasks, with the capability to process various answer types.<n>We introduce VerifierBench benchmark comprising model outputs collected from multiple data sources, augmented through manual analysis of metaerror patterns to enhance CompassVerifier.
  arXiv  Detail & Related papers  (2025-08-05T17:55:24Z)
• Repeton: Structured Bug Repair with ReAct-Guided Patch-and-Test Cycles [1.387448620257867]
  Large Language Models (LLMs) have shown strong capabilities in code generation and comprehension, yet their application to complex software engineering tasks often suffers from low precision and limited interpretability.<n>We present Repeton, a fully open-source framework that leverages LLMs for precise and automated code manipulation in real-world Git.
  arXiv  Detail & Related papers  (2025-06-09T19:36:40Z)
• Effective Instruction Parsing Plugin for Complex Logical Query Answering on Knowledge Graphs [51.33342412699939]
  Knowledge Graph Query Embedding (KGQE) aims to embed First-Order Logic (FOL) queries in a low-dimensional KG space for complex reasoning over incomplete KGs. Recent studies integrate various external information (such as entity types and relation context) to better capture the logical semantics of FOL queries. We propose an effective Query Instruction Parsing (QIPP) that captures latent query patterns from code-like query instructions.
  arXiv  Detail & Related papers  (2024-10-27T03:18:52Z)
• Software Verification with CPAchecker 3.0: Tutorial and User Guide (Extended Version) [5.7013421104492075]
  This tutorial presents basic use cases for CPAchecker in formal software verification. The envisioned readers are assumed to possess a background in automatic formal verification and program analysis. This tutorial and user guide is based on CPAchecker in version 3.0.
  arXiv  Detail & Related papers  (2024-09-03T17:50:50Z)
• KAT: Dependency-aware Automated API Testing with Large Language Models [1.7264233311359707]
  KAT (Katalon API Testing) is a novel AI-driven approach that autonomously generates test cases to validate APIs. Our evaluation of KAT using 12 real-world services shows that it can improve validation coverage, detect more undocumented status codes, and reduce false positives in these services.
  arXiv  Detail & Related papers  (2024-07-14T14:48:18Z)
• Self-play with Execution Feedback: Improving Instruction-following Capabilities of Large Language Models [54.14602121129874]
  We introduce AutoIF, the first scalable and reliable method for automatically generating instruction-following training data. AutoIF transforms the validation of instruction-following data quality into code verification.
  arXiv  Detail & Related papers  (2024-06-19T13:29:53Z)
• SWT-Bench: Testing and Validating Real-World Bug-Fixes with Code Agents [10.730852617039451]
  We investigate the capability of LLM-based Code Agents to formalize user issues into test cases. We propose a novel benchmark based on popular GitHub repositories, containing real-world issues, ground-truth bug-fixes, and golden tests. We find that LLMs generally perform surprisingly well at generating relevant test cases, with Code Agents designed for code repair exceeding the performance of systems designed for test generation.
  arXiv  Detail & Related papers  (2024-06-18T14:54:37Z)
• CoSQA+: Pioneering the Multi-Choice Code Search Benchmark with Test-Driven Agents [25.861575256100153]
  Existing code search datasets face limitations. They rely on human annotators who assess code primarily through semantic understanding. This paper introduces CoSQA+, pairing high-quality queries from CoSQA with multiple suitable codes.
  arXiv  Detail & Related papers  (2024-06-17T14:34:14Z)
• OpenFactCheck: Building, Benchmarking Customized Fact-Checking Systems and Evaluating the Factuality of Claims and LLMs [59.836774258359945]
  OpenFactCheck is a framework for building customized automatic fact-checking systems.<n>It allows users to easily customize an automatic fact-checker and verify the factual correctness of documents and claims.<n>CheckerEVAL is a solution for gauging the reliability of automatic fact-checkers' verification results using human-annotated datasets.
  arXiv  Detail & Related papers  (2024-05-09T07:15:19Z)
• Automating REST API Postman Test Cases Using LLM [0.0]
  This research paper is dedicated to the exploration and implementation of an automated approach to generate test cases using Large Language Models. The methodology integrates the use of Open AI to enhance the efficiency and effectiveness of test case generation. The model that is developed during the research is trained using manually collected postman test cases or instances for various Rest APIs.
  arXiv  Detail & Related papers  (2024-04-16T15:53:41Z)
• From Instructions to Constraints: Language Model Alignment with Automatic Constraint Verification [70.08146540745877]
  We investigate common constraints in NLP tasks, categorize them into three classes based on the types of their arguments. We propose a unified framework, ACT (Aligning to ConsTraints), to automatically produce supervision signals for user alignment with constraints.
  arXiv  Detail & Related papers  (2024-03-10T22:14:54Z)
• Factcheck-Bench: Fine-Grained Evaluation Benchmark for Automatic Fact-checkers [121.53749383203792]
  We present a holistic end-to-end solution for annotating the factuality of large language models (LLMs)-generated responses. We construct an open-domain document-level factuality benchmark in three-level granularity: claim, sentence and document. Preliminary experiments show that FacTool, FactScore and Perplexity are struggling to identify false claims.
  arXiv  Detail & Related papers  (2023-11-15T14:41:57Z)
• Automata Learning from Preference and Equivalence Queries [17.33092604696224]
  We propose a novel variant of the active automata learning problem: actively learn finite automata using preference queries. ReMAP is guaranteed to correctly infer the minimal complexity with query complexity under exact equivalence queries. Our empirical evaluations indicate REMAP scales to large automata is effective at learning correct automata from consistent teachers.
  arXiv  Detail & Related papers  (2023-08-18T04:49:45Z)
• SelfCheck: Using LLMs to Zero-Shot Check Their Own Step-by-Step Reasoning [55.76083560152823]
  SelfCheck is a general-purpose zero-shot verification schema for recognizing errors in step-by-step reasoning. We test SelfCheck on three datasets (GSM8K, MathQA, and MATH) and find that it successfully recognizes errors and, in turn, increases final answer accuracies.
  arXiv  Detail & Related papers  (2023-08-01T10:31:36Z)
• ALGO: Synthesizing Algorithmic Programs with LLM-Generated Oracle Verifiers [60.6418431624873]
  Large language models (LLMs) excel at implementing code from functionality descriptions but struggle with algorithmic problems. We propose ALGO, a framework that synthesizes Algorithmic programs with LLM-Generated Oracles to guide the generation and verify their correctness. Experiments show that when equipped with ALGO, we achieve an 8x better one-submission pass rate over the Codex model and a 2.6x better one-submission pass rate over CodeT.
  arXiv  Detail & Related papers  (2023-05-24T00:10:15Z)
• Applying Machine Learning for Duplicate Detection, Throttling and Prioritization of Equipment Commissioning Audits at Fulfillment Network [1.933681537640272]
  VQ (Vendor Qualification) and IOQ (Installation and Operation Qualification) audits are implemented in warehouses to ensure all equipment being turned over meets the quality standards. In this work, Natural Language Processing and Machine Learning are applied to trim a large checklist dataset for a network of warehouses.
  arXiv  Detail & Related papers  (2022-09-28T20:40:32Z)
• Induction and Exploitation of Subgoal Automata for Reinforcement Learning [75.55324974788475]
  We present ISA, an approach for learning and exploiting subgoals in episodic reinforcement learning (RL) tasks. ISA interleaves reinforcement learning with the induction of a subgoal automaton, an automaton whose edges are labeled by the task's subgoals. A subgoal automaton also consists of two special states: a state indicating the successful completion of the task, and a state indicating that the task has finished without succeeding.
  arXiv  Detail & Related papers  (2020-09-08T16:42:55Z)
• GAMA: a General Automated Machine learning Assistant [4.035753155957698]
  The General Automated Machine learning Assistant (GAMA) is a modular AutoML system developed to empower users to track and control how AutoML algorithms search for optimal machine learning pipelines. GAMA allows users to plug in different AutoML and post-processing techniques, logs and visualizes the search process, and supports easy benchmarking.
  arXiv  Detail & Related papers  (2020-07-09T16:16:25Z)
• Beyond Accuracy: Behavioral Testing of NLP models with CheckList [66.42971817954806]
  CheckList is a task-agnostic methodology for testing NLP models. CheckList includes a matrix of general linguistic capabilities and test types that facilitate comprehensive test ideation. In a user study, NLP practitioners with CheckList created twice as many tests, and found almost three times as many bugs as users without it.
  arXiv  Detail & Related papers  (2020-05-08T15:48:31Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.