Evaluating Repository-level Software Documentation via Question Answering and Feature-Driven Development

• URL: http://arxiv.org/abs/2604.06793v1
• Date: Wed, 08 Apr 2026 07:58:18 GMT
• Title: Evaluating Repository-level Software Documentation via Question Answering and Feature-Driven Development
• Abstract summary: We introduce SWD-Bench, a novel benchmark for evaluating repository-level software documentation.<n>inspired by documentation-driven development, our strategy evaluates documentation quality by assessing an LLM's ability to understand and implement functionalities.<n>We construct the benchmark, containing 4,170 entries, by mining high-quality Pull Requests and enriching them with repository-level context.
• Score: 11.174059895410357
• License: http://creativecommons.org/licenses/by-sa/4.0/
• Abstract: Software documentation is crucial for repository comprehension. While Large Language Models (LLMs) advance documentation generation from code snippets to entire repositories, existing benchmarks have two key limitations: (1) they lack a holistic, repository-level assessment, and (2) they rely on unreliable evaluation strategies, such as LLM-as-a-judge, which suffers from vague criteria and limited repository-level knowledge. To address these issues, we introduce SWD-Bench, a novel benchmark for evaluating repository-level software documentation. Inspired by documentation-driven development, our strategy evaluates documentation quality by assessing an LLM's ability to understand and implement functionalities using the documentation, rather than by directly scoring it. This is measured through function-driven Question Answering (QA) tasks. SWD-Bench comprises three interconnected QA tasks: (1) Functionality Detection, to determine if a functionality is described; (2) Functionality Localization, to evaluate the accuracy of locating related files; and (3) Functionality Completion, to measure the comprehensiveness of implementation details. We construct the benchmark, containing 4,170 entries, by mining high-quality Pull Requests and enriching them with repository-level context. Experiments reveal limitations in current documentation generation methods and show that source code provides complementary value. Notably, documentation from the best-performing method improves the issue-solving rate of SWE-Agent by 20.00%, which demonstrates the practical value of high-quality documentation in supporting documentation-driven development.

Related papers

• Adaptive Critical Token-Aware Retrieval for Repository-Level Code Generation [53.85136813868056]
  ACToR is an adaptive critical token-aware retrieval framework for repository-level code generation.<n>We show that ACToR consistently outperforms state-of-the-art methods on repository-level benchmarks.
  arXiv  Detail & Related papers  (2026-09-01T17:59:39Z)
• DocClaw: A Unified Agentic System for Intelligent Document Processing [70.82545096015238]
  We introduce DocClaw, a unified agentic system that formulates diverse intelligent document processing tasks as a shared process of interaction between an agent and a document.<n>Under this formulation, task-specific requirements are captured by the agent's interpretation of the query objective and the corresponding document skill.<n>Experiments across multiple intelligent document processing benchmarks demonstrate that DocClaw effectively handles diverse tasks within a single agentic framework.
  arXiv  Detail & Related papers  (2026-08-19T08:34:48Z)
• Code-QA-Bench: Separating Code Reasoning from Documentation Memorization in Repository-Level QA [7.478327602641997]
  Code-QA-Bench is a framework for synthesizing repository-level code understanding benchmarks.<n>We generate 528 code-derivable and 100 doc-dependent tasks across 10 Python repositories.<n>The framework is open-source and applicable to any well-documented Python repository.
  arXiv  Detail & Related papers  (2026-05-28T02:52:58Z)
• Remember Your Trace: Memory-Guided Long-Horizon Agentic Framework for Consistent and Hierarchical Repository-Level Code Documentation [27.40384564559557]
  We propose MemDocAgent, a framework that generates documentation within a single, integrated context spanning the entire repository.<n>MemDocAgent achieves the best performance over both open and closed-source baselines.
  arXiv  Detail & Related papers  (2026-05-14T08:35:20Z)
• RAL-Bench: Benchmarking for Application-Level Functional Correctness and Non-Functional Quality Attributes [12.202503919149118]
  RAL-Bench is a benchmark and evaluation framework for application-level code generation.<n>For each task, we distill a concise natural-language requirement from a high-quality reference project.<n>Black-box system tests cover functional and non-functional attributes, and keep only tests that pass on the reference repository.
  arXiv  Detail & Related papers  (2026-02-03T12:35:09Z)
• Benchmarking LLMs for Fine-Grained Code Review with Enriched Context in Practice [18.222990693059756]
  ContextCRBench is a benchmark for fine-grained LLM evaluation in code review.<n>It collects 153.7K issues and pull requests from top-tier repositories.<n>It supports three evaluation scenarios aligned with the review workflow.
  arXiv  Detail & Related papers  (2025-11-10T12:06:35Z)
• RepoSummary: Feature-Oriented Summarization and Documentation Generation for Code Repositories [7.744086870383438]
  RepoSummary is a feature-oriented code repository summarization approach.<n>It simultaneously generates repository documentation automatically.<n>It establishes more accurate traceability links from functional features to the corresponding code elements.
  arXiv  Detail & Related papers  (2025-10-13T06:16:44Z)
• Document Attribution: Examining Citation Relationships using Large Language Models [62.46146670035751]
  We propose a zero-shot approach that frames attribution as a straightforward textual entailment task.<n>We also explore the role of the attention mechanism in enhancing the attribution process.
  arXiv  Detail & Related papers  (2025-05-09T04:40:11Z)
• M-DocSum: Do LVLMs Genuinely Comprehend Interleaved Image-Text in Document Summarization? [49.53982792497275]
  We investigate whether Large Vision-Language Models (LVLMs) genuinely comprehend interleaved image-text in the document.<n>Existing document understanding benchmarks often assess LVLMs using question-answer formats.<n>We introduce a novel and challenging Multimodal Document Summarization Benchmark (M-DocSum-Bench)<n>M-DocSum-Bench comprises 500 high-quality arXiv papers, along with interleaved multimodal summaries aligned with human preferences.
  arXiv  Detail & Related papers  (2025-03-27T07:28:32Z)
• Benchmarking Large Language Models for Conversational Question Answering in Multi-instructional Documents [61.41316121093604]
  We present InsCoQA, a novel benchmark for evaluating large language models (LLMs) in the context of conversational question answering (CQA) Sourced from extensive, encyclopedia-style instructional content, InsCoQA assesses models on their ability to retrieve, interpret, and accurately summarize procedural guidance from multiple documents. We also propose InsEval, an LLM-assisted evaluator that measures the integrity and accuracy of generated responses and procedural instructions.
  arXiv  Detail & Related papers  (2024-10-01T09:10:00Z)
• RAGEval: Scenario Specific RAG Evaluation Dataset Generation Framework [66.93260816493553]
  This paper introduces RAGEval, a framework designed to assess RAG systems across diverse scenarios.<n>With a focus on factual accuracy, we propose three novel metrics: Completeness, Hallucination, and Irrelevance.<n> Experimental results show that RAGEval outperforms zero-shot and one-shot methods in terms of clarity, safety, conformity, and richness of generated samples.
  arXiv  Detail & Related papers  (2024-08-02T13:35:11Z)
• DOCBENCH: A Benchmark for Evaluating LLM-based Document Reading Systems [99.17123445211115]
  We introduce DocBench, a benchmark to evaluate large language model (LLM)-based document reading systems. Our benchmark involves the recruitment of human annotators and the generation of synthetic questions. It includes 229 real documents and 1,102 questions, spanning across five different domains and four major types of questions.
  arXiv  Detail & Related papers  (2024-07-15T13:17:42Z)
• On the Impacts of Contexts on Repository-Level Code Generation [5.641402231731082]
  We present RepoExec, a novel benchmark designed to evaluate repository-level code generation.<n>We focus on three key aspects: executability, functional correctness through comprehensive test case generation, and accurate utilization of cross-file contexts.
  arXiv  Detail & Related papers  (2024-06-17T10:45:22Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.