METAMON: Finding Inconsistencies between Program Documentation and Behavior using Metamorphic LLM Queries

• URL: http://arxiv.org/abs/2502.02794v1
• Date: Wed, 05 Feb 2025 00:42:50 GMT
• Title: METAMON: Finding Inconsistencies between Program Documentation and Behavior using Metamorphic LLM Queries
• Abstract summary: This paper proposes METAMON, which uses an existing search-based test generation technique to capture the current program behavior in the form of test cases.<n> METAMON is supported in this task by metamorphic testing and self-consistency.<n>An empirical evaluation against 9,482 pairs of code documentation and code snippets, generated using five open-source projects from Defects4J v2.0.1, shows that METAMON can classify the code-and-documentation inconsistencies with a precision of 0.72 and a recall of 0.48.
• Score: 10.9334354663311
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Code documentation can, if written precisely, help developers better understand the code they accompany. However, unlike code, code documentation cannot be automatically verified via execution, potentially leading to inconsistencies between documentation and the actual behavior. While such inconsistencies can be harmful for the developer's understanding of the code, checking and finding them remains a costly task due to the involvement of human engineers. This paper proposes METAMON, which uses an existing search-based test generation technique to capture the current program behavior in the form of test cases, and subsequently uses LLM-based code reasoning to identify the generated regression test oracles that are not consistent with the program specifications in the documentation. METAMON is supported in this task by metamorphic testing and self-consistency. An empirical evaluation against 9,482 pairs of code documentation and code snippets, generated using five open-source projects from Defects4J v2.0.1, shows that METAMON can classify the code-and-documentation inconsistencies with a precision of 0.72 and a recall of 0.48.

Related papers

• LM2Alloy: Investigating LLM-Generated Formal Specifications for Automated Test Derivation in Production Software [0.9023847175654603]
  We present an exploratory study on using Large Language Models (LLMs) to generate Alloy formal specifications.<n>We evaluate on two real open-source Python libraries: Flipper, a feature flag management system, and Cerberus, a data validation library.
  arXiv  Detail & Related papers  (2026-07-20T22:41:55Z)
• Guiding Human Validation of LLM-Generated Code via Verifiable Literate Programming [18.004246438683147]
  Vibe coding democratizes software development by allowing users to generate code via natural-language (NL) interaction with large language models (LLMs)<n>This paper presents verifiable literate programming, a human-in-the-loop framework designed to make the review/validation process accessible to users at all programming levels.<n>The documentation demonstrates concrete program semantics and enables users to provide feedback on potential intent-code mismatches.
  arXiv  Detail & Related papers  (2026-07-02T15:37:54Z)
• Natural Language-Focused Software Engineering via Code-Documentation Equivalence [17.47344940875897]
  We present a novel approach, called Documentary, to automatically generate equivalent documentation for a given code snippet.<n>Our results show that documentation-to-code equivalence allows an LLM to predict the output of a function with 12.8--24.5% higher accuracy.<n>Human developers consider documentation generated by Documentary to be more useful for understanding and editing code than the original human-written documentation.
  arXiv  Detail & Related papers  (2026-06-20T22:19:00Z)
• Code-QA-Bench: Separating Code Reasoning from Documentation Memorization in Repository-Level QA [7.478327602641997]
  Code-QA-Bench is a framework for synthesizing repository-level code understanding benchmarks.<n>We generate 528 code-derivable and 100 doc-dependent tasks across 10 Python repositories.<n>The framework is open-source and applicable to any well-documented Python repository.
  arXiv  Detail & Related papers  (2026-05-28T02:52:58Z)
• CASCADE: Detecting Inconsistencies between Code and Documentation with Automatic Test Generation [0.7611870296994722]
  Maintaining consistency between code and documentation is a crucial yet frequently overlooked aspect of software development.<n>We introduce CASCADE, a novel tool for detecting inconsistencies with a strong emphasis on reducing false positives.<n>We evaluated CASCADE on a novel dataset of 71 inconsistent and 814 consistent code-documentation pairs drawn from open-source Java projects.
  arXiv  Detail & Related papers  (2026-04-21T12:26:54Z)
• Detecting and Correcting Hallucinations in LLM-Generated Code via Deterministic AST Analysis [11.687400527666476]
  This paper investigates whether a deterministic, static-analysis framework can reliably detect textitand auto-correct KCHs.<n>We propose a post-processing framework that parses generated code into an Abstract Syntax Tree (AST) and validates it against a dynamically-generated Knowledge Base (KB)<n>This non-executing approach uses deterministic rules to find and fix both API and identifier-level conflicts.
  arXiv  Detail & Related papers  (2026-01-27T02:16:37Z)
• DocPrism: Local Categorization and External Filtering to Identify Relevant Code-Documentation Inconsistencies [5.693844702145728]
  This paper introduces DocPrism, a code-documentation inconsistency detection tool.<n> DocPrism uses a standard large language model (LLM) to analyze and explain inconsistencies.<n>On a broad evaluation across Python, TypeScript, C++, and Java, DocPrism maintains a low flag rate of 15%, and achieves a precision of 0.62 without performing any fine-tuning.
  arXiv  Detail & Related papers  (2025-10-31T19:22:54Z)
• IFEvalCode: Controlled Code Generation [69.28317223249358]
  The paper introduces forward and backward constraints generation to improve the instruction-following capabilities of Code LLMs.<n>The authors present IFEvalCode, a multilingual benchmark comprising 1.6K test samples across seven programming languages.
  arXiv  Detail & Related papers  (2025-07-30T08:08:48Z)
• Codetations: Intelligent, Persistent Notes and UIs for Programs and Other Documents [0.85830154886823]
  We present Codetations, a system that helps developers contextualize documents with rich notes and tools. Unlike previous approaches, notes in Codetations stay outside the document to prevent code clutter, attaching to spans in the document using a hybrid edit-tracking/LLM-based method. Their content is dynamic, interactive, and synchronized with code changes.
  arXiv  Detail & Related papers  (2025-04-25T21:33:25Z)
• ExeCoder: Empowering Large Language Models with Executability Representation for Code Translation [57.604506522287814]
  Existing large language models (LLMs) only learn the contextual semantics of code during pre-training.<n>We propose ExeCoder to utilize executability representations such as functional semantics, syntax structures, and variable dependencies.<n>We show that ExeCoder achieves state-of-the-art performance in code translation, surpassing existing open-source code LLMs by over 10.88% to 38.78% and over 27.44% to 42.97% on two metrics.
  arXiv  Detail & Related papers  (2025-01-30T16:18:52Z)
• Commit0: Library Generation from Scratch [77.38414688148006]
  Commit0 is a benchmark that challenges AI agents to write libraries from scratch.<n>Agents are provided with a specification document outlining the library's API as well as a suite of interactive unit tests.<n> Commit0 also offers an interactive environment where models receive static analysis and execution feedback on the code they generate.
  arXiv  Detail & Related papers  (2024-12-02T18:11:30Z)
• A Deep Dive Into Large Language Model Code Generation Mistakes: What and Why? [9.246899995643918]
  Large Language Models can still generate defective code that deviates from the specification. Seven categories of non-syntactic mistakes were identified through extensive manual analyses. Our evaluation demonstrated that GPT-4 with the ReAct prompting technique can achieve an F1 score of up to 0.65 when identifying reasons for LLM's mistakes.
  arXiv  Detail & Related papers  (2024-11-03T02:47:03Z)
• Chain of Targeted Verification Questions to Improve the Reliability of Code Generated by LLMs [10.510325069289324]
  We propose a self-refinement method aimed at improving the reliability of code generated by LLMs. Our approach is based on targeted Verification Questions (VQs) to identify potential bugs within the initial code. Our method attempts to repair these potential bugs by re-prompting the LLM with the targeted VQs and the initial code.
  arXiv  Detail & Related papers  (2024-05-22T19:02:50Z)
• Iterative Refinement of Project-Level Code Context for Precise Code Generation with Compiler Feedback [29.136378191436396]
  We present CoCoGen, a new code generation approach that uses compiler feedback to improve the LLM-generated code. CoCoGen first leverages static analysis to identify mismatches between the generated code and the project's context. It then iteratively aligns and fixes the identified errors using information extracted from the code repository.
  arXiv  Detail & Related papers  (2024-03-25T14:07:27Z)
• Test-Driven Development for Code Generation [0.850206009406913]
  Large Language Models (LLMs) have demonstrated significant capabilities in generating code snippets directly from problem statements. This paper investigates if and how Test-Driven Development (TDD) can be incorporated into AI-assisted code-generation processes.
  arXiv  Detail & Related papers  (2024-02-21T04:10:12Z)
• GenAudit: Fixing Factual Errors in Language Model Outputs with Evidence [64.95492752484171]
  We present GenAudit -- a tool intended to assist fact-checking LLM responses for document-grounded tasks. GenAudit suggests edits to the LLM response by revising or removing claims that are not supported by the reference document, and also presents evidence from the reference for facts that do appear to have support. Comprehensive evaluation by human raters shows that GenAudit can detect errors in 8 different LLM outputs when summarizing documents from diverse domains.
  arXiv  Detail & Related papers  (2024-02-19T21:45:55Z)
• ML-Bench: Evaluating Large Language Models and Agents for Machine Learning Tasks on Repository-Level Code [76.84199699772903]
  ML-Bench is a benchmark rooted in real-world programming applications that leverage existing code repositories to perform tasks. To evaluate both Large Language Models (LLMs) and AI agents, two setups are employed: ML-LLM-Bench for assessing LLMs' text-to-code conversion within a predefined deployment environment, and ML-Agent-Bench for testing autonomous agents in an end-to-end task execution within a Linux sandbox environment.
  arXiv  Detail & Related papers  (2023-11-16T12:03:21Z)
• FLAG: Finding Line Anomalies (in code) with Generative AI [18.612900041820875]
  FLAG is based on the lexical capabilities of generative AI, specifically, Large Language Models (LLMs) We use 121 benchmarks across C, Python and Verilog; with each benchmark containing a known security or functional weakness. FLAG can identify 101 of the defects and helps reduce the search space to 12-17% of source code.
  arXiv  Detail & Related papers  (2023-06-22T03:04:56Z)
• Execution-based Evaluation for Data Science Code Generation Models [97.96608263010913]
  We introduce ExeDS, an evaluation dataset for execution evaluation for data science code generation tasks. ExeDS contains a set of 534 problems from Jupyter Notebooks, each consisting of code context, task description, reference program, and desired execution output. We evaluate the execution performance of five state-of-the-art code generation models that have achieved high surface-form evaluation scores.
  arXiv  Detail & Related papers  (2022-11-17T07:04:11Z)
• Interactive Code Generation via Test-Driven User-Intent Formalization [60.90035204567797]
  Large language models (LLMs) produce code from informal natural language (NL) intent. It is hard to define a notion of correctness since natural language can be ambiguous and lacks a formal semantics. We describe a language-agnostic abstract algorithm and a concrete implementation TiCoder.
  arXiv  Detail & Related papers  (2022-08-11T17:41:08Z)
• ReACC: A Retrieval-Augmented Code Completion Framework [53.49707123661763]
  We propose a retrieval-augmented code completion framework, leveraging both lexical copying and referring to code with similar semantics by retrieval. We evaluate our approach in the code completion task in Python and Java programming languages, achieving a state-of-the-art performance on CodeXGLUE benchmark.
  arXiv  Detail & Related papers  (2022-03-15T08:25:08Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.