RepoProbe: Benchmarking Architecture-Aware Repository Comprehension with Checklists

• URL: http://arxiv.org/abs/2608.04783v2
• Date: Thu, 06 Aug 2026 09:16:21 GMT
• Title: RepoProbe: Benchmarking Architecture-Aware Repository Comprehension with Checklists
• Abstract summary: RepoProbe is a novel benchmark for evaluating repository-level code understanding.<n>We propose a Checklist-Based Verification Protocol that decomposes answers into atomic, verifiable facts.
• Score: 12.810164698494281
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: The integration of Large Language Models (LLMs) into software engineering has shifted the focus from function-level generation to repository-scale assistance. However, existing benchmarks largely rely on bug reports from GitHub Issues, which often allow models to bypass genuine understanding via pattern matching on error logs. This misalignment under-measures Edit Bias, which refers to premature generation, where models prematurely propose code modifications instead of understanding the existing repository architecture. Furthermore, current LLM-as-a-Judge scalar scoring suffers from high variance and low interpretability. This work introduces RepoProbe, a novel benchmark for evaluating repository-level code understanding through open-ended Q&A using GitHub Discussions, which focuses on open-ended architectural inquiries rather than defect reporting. To ensure rigorous evaluation, we propose a Checklist-Based Verification Protocol that decomposes answers into atomic, verifiable facts, thereby replacing subjective ratings with objective verification. Our evaluation of state-of-the-art (SOTA) LLMs reveals a persistent gap between high clarity and evidencegrounded technical correctness. It also quantitatively confirms the prevalence of edit bias, in which models prioritize code generation instead of architectural analysis. Finally, we demonstrate that our verification protocol significantly improves evaluation reliability compared to traditional evaluations with scalar scoring.

Related papers

• Adaptive Critical Token-Aware Retrieval for Repository-Level Code Generation [53.85136813868056]
  ACToR is an adaptive critical token-aware retrieval framework for repository-level code generation.<n>We show that ACToR consistently outperforms state-of-the-art methods on repository-level benchmarks.
  arXiv  Detail & Related papers  (2026-09-01T17:59:39Z)
• RepoReasoner: Evaluating Repository-Level Code Reasoning Ability of Long-Context Language Models [54.53236295237077]
  We introduce RepoReasoner, a benchmark for evaluating repository-level code reasoning.<n> Output Prediction measures fine-grained, stateful execution reasoning across files.<n>Call Chain Prediction evaluates high-level architectural dependency understanding under noisy context.
  arXiv  Detail & Related papers  (2026-07-28T17:12:41Z)
• RepoZero: Can LLMs Generate a Code Repository from Scratch? [13.87780777614509]
  RepoZero is the first benchmark that enables fully automated, execution-based verification of repository-level generation from scratch.<n>Our results establish RepoZero as a challenging, scalable, and reliable testbed for end-to-end code generation.
  arXiv  Detail & Related papers  (2026-05-08T01:56:02Z)
• DocSync: Agentic Documentation Maintenance via Critic-Guided Reflexion [0.0]
  DocSync is an agentic workflow that frames documentation maintenance as a structurally grounded, iterative generation task.<n>We empirically evaluate a resource-constrained implementation of DocSync-using a LoRA-adapted small language model.<n>Our findings demonstrate that this AST-aware agentic approach substantially outperforms standard encoder-decoder baselines.
  arXiv  Detail & Related papers  (2026-05-04T02:41:33Z)
• LiveFMBench: Unveiling the Power and Limits of Agentic Workflows in Specification Generation [75.05397479715576]
  Large language models (LLMs) and agents have shown promising progress, but their true capabilities and failure modes remain unclear.<n>We present the first systematic and contamination-aware study of LLM- and agent-based formal specification generation for C programs.
  arXiv  Detail & Related papers  (2026-05-02T11:31:33Z)
• RepoReviewer: A Local-First Multi-Agent Architecture for Repository-Level Code Review [3.272585676511349]
  RepoReviewer is a local-first multi-agent system for automated GitHub repository review.<n>We present RepoReviewer with a Python CLI, FastAPI API, LangGraph orchestration layer, and Next.js user interface.
  arXiv  Detail & Related papers  (2026-03-17T04:17:51Z)
• GREPO: A Benchmark for Graph Neural Networks on Repository-Level Bug Localization [50.009407518866965]
  Repository-level bug localization is a critical software engineering challenge.<n>GNNs offer a promising alternative due to their ability to model complex, repository-wide dependencies.<n>We introduce GREPO, the first GNN benchmark for repository-scale bug localization tasks.
  arXiv  Detail & Related papers  (2026-02-14T23:22:15Z)
• EdiVal-Agent: An Object-Centric Framework for Automated, Fine-Grained Evaluation of Multi-Turn Editing [170.71134330650796]
  EdiVal-Agent is an object-centric evaluation framework for instruction-based image editing.<n>It is designed to assess not only standard single-turn but also multi-turn instruction-based editing with precision.<n>We build EdiVal-Bench, a benchmark covering 9 instruction types and 13 state-of-the-art editing models spanning in-context, flow-matching, and diffusion paradigms.
  arXiv  Detail & Related papers  (2025-09-16T17:45:39Z)
• Probing Pre-trained Language Models on Code Changes: Insights from ReDef, a High-Confidence Just-in-Time Defect Prediction Dataset [0.0]
  We present ReDef, a high-confidence benchmark of function-level modifications curated from 22 large-scale C/C++ projects.<n>Defective cases are anchored by revert commits, while clean cases are validated through post-hoc history checks.<n>This pipeline yields 3,164 defective and 10,268 clean modifications, offering substantially more reliable labels than prior existing resources.
  arXiv  Detail & Related papers  (2025-09-11T07:07:11Z)
• CompassVerifier: A Unified and Robust Verifier for LLMs Evaluation and Outcome Reward [50.97588334916863]
  We develop CompassVerifier, an accurate and robust lightweight verifier model for evaluation and outcome reward.<n>It demonstrates multi-domain competency spanning math, knowledge, and diverse reasoning tasks, with the capability to process various answer types.<n>We introduce VerifierBench benchmark comprising model outputs collected from multiple data sources, augmented through manual analysis of metaerror patterns to enhance CompassVerifier.
  arXiv  Detail & Related papers  (2025-08-05T17:55:24Z)
• End-to-End Object Detection with Transformers [88.06357745922716]
  We present a new method that views object detection as a direct set prediction problem. Our approach streamlines the detection pipeline, effectively removing the need for many hand-designed components. The main ingredients of the new framework, called DEtection TRansformer or DETR, are a set-based global loss.
  arXiv  Detail & Related papers  (2020-05-26T17:06:38Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.