Code-QA-Bench: Separating Code Reasoning from Documentation Memorization in Repository-Level QA

• URL: http://arxiv.org/abs/2605.29277v1
• Date: Thu, 28 May 2026 02:52:58 GMT
• Title: Code-QA-Bench: Separating Code Reasoning from Documentation Memorization in Repository-Level QA
• Abstract summary: Code-QA-Bench is a framework for synthesizing repository-level code understanding benchmarks.<n>We generate 528 code-derivable and 100 doc-dependent tasks across 10 Python repositories.<n>The framework is open-source and applicable to any well-documented Python repository.
• Score: 7.478327602641997
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: We present Code-QA-Bench, a fully automated framework for synthesizing repository-level code understanding benchmarks that separates genuine code comprehension from documentation recall and pretraining memorization. The framework makes two methodological contributions: (1) an answer-first generation pipeline where a tool-equipped agent explores source code to produce verified gold answers before deriving questions, ensuring every task is grounded in real code structure; and (2) a three-condition experimental design evaluating agents under closed-book (no repository), code-only (documentation removed), and documented (full repository) conditions, with deltas directly quantifying documentation utility and memorization. We generate 528 code-derivable and 100 doc-dependent tasks across 10 Python repositories from SWE-Bench, scored by an LLM judge on accuracy, completeness, and specificity. Experiments on four frontier models reveal that code access is the dominant factor (+0.23 mean gain over closed-book), documentation provides modest additional benefit (+0.071 on doc-dependent tasks), and code-only $\approx$ documented on code-derivable tasks, validating the design. The framework is open-source and applicable to any well-documented Python repository.

Related papers

• Tencent WorkBuddy Bench: A Multi-Domain Coding-Agent Benchmark with Contamination-Resistant Task Construction [57.138342889101345]
  Tencent WorkBuddy Bench is a multi-domain evaluation suite for coding agents.<n>This report documents its construction methodology, scoring protocol, and a cross-model leaderboard.
  arXiv  Detail & Related papers  (2026-07-23T04:34:06Z)
• Natural Language-Focused Software Engineering via Code-Documentation Equivalence [17.47344940875897]
  We present a novel approach, called Documentary, to automatically generate equivalent documentation for a given code snippet.<n>Our results show that documentation-to-code equivalence allows an LLM to predict the output of a function with 12.8--24.5% higher accuracy.<n>Human developers consider documentation generated by Documentary to be more useful for understanding and editing code than the original human-written documentation.
  arXiv  Detail & Related papers  (2026-06-20T22:19:00Z)
• Evaluating Repository-level Software Documentation via Question Answering and Feature-Driven Development [11.174059895410357]
  We introduce SWD-Bench, a novel benchmark for evaluating repository-level software documentation.<n>inspired by documentation-driven development, our strategy evaluates documentation quality by assessing an LLM's ability to understand and implement functionalities.<n>We construct the benchmark, containing 4,170 entries, by mining high-quality Pull Requests and enriching them with repository-level context.
  arXiv  Detail & Related papers  (2026-04-08T07:58:18Z)
• AlignCoder: Aligning Retrieval with Target Intent for Repository-Level Code Completion [55.21541958868449]
  We propose AlignCoder, a repository-level code completion framework.<n>Our framework generates an enhanced query that bridges the semantic gap between the initial query and the target code.<n>We employ reinforcement learning to train an AlignRetriever that learns to leverage inference information in the enhanced query for more accurate retrieval.
  arXiv  Detail & Related papers  (2026-01-27T15:23:14Z)
• Gistify! Codebase-Level Understanding via Runtime Execution [97.63528391679807]
  We propose Gistify, a task where a coding LLM must create a single, minimal, self-contained file that can reproduce a specific functionality.<n>We show that current state-of-the-art models struggle to reliably solve Gistify tasks, especially ones with long executions.
  arXiv  Detail & Related papers  (2025-10-30T17:58:26Z)
• RepoSummary: Feature-Oriented Summarization and Documentation Generation for Code Repositories [7.744086870383438]
  RepoSummary is a feature-oriented code repository summarization approach.<n>It simultaneously generates repository documentation automatically.<n>It establishes more accurate traceability links from functional features to the corresponding code elements.
  arXiv  Detail & Related papers  (2025-10-13T06:16:44Z)
• SolBench: A Dataset and Benchmark for Evaluating Functional Correctness in Solidity Code Completion and Repair [51.0686873716938]
  We introduce SolBench, a benchmark for evaluating the functional correctness of Solidity smart contracts generated by code completion models.<n>We propose a Retrieval-Augmented Code Repair framework to verify functional correctness of smart contracts.<n>Results show that code repair and retrieval techniques effectively enhance the correctness of smart contract completion while reducing computational costs.
  arXiv  Detail & Related papers  (2025-03-03T01:55:20Z)
• Repository-level Code Search with Neural Retrieval Methods [25.222964965449286]
  We define the task of repository-level code search as retrieving the set of files from the current state of a code repository that are most relevant to addressing a user's question or bug.<n>The proposed approach combines BM25-based retrieval over commit messages with neural reranking using CodeBERT to identify the most pertinent files.<n> Experiments on a new dataset created from 7 popular open-source repositories demonstrate substantial improvements of up to 80% in MAP, MRR and P@1 over the BM25 baseline.
  arXiv  Detail & Related papers  (2025-02-10T21:59:01Z)
• RepoCoder: Repository-Level Code Completion Through Iterative Retrieval and Generation [96.75695811963242]
  RepoCoder is a framework to streamline the repository-level code completion process. It incorporates a similarity-based retriever and a pre-trained code language model. It consistently outperforms the vanilla retrieval-augmented code completion approach.
  arXiv  Detail & Related papers  (2023-03-22T13:54:46Z)
• Generation-Augmented Query Expansion For Code Retrieval [51.20943646688115]
  We propose a generation-augmented query expansion framework. Inspired by the human retrieval process - sketching an answer before searching. We achieve new state-of-the-art results on the CodeSearchNet benchmark.
  arXiv  Detail & Related papers  (2022-12-20T23:49:37Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.