When Retrieval Hurts Code Completion: A Diagnostic Study of Stale Repository Context

• URL: http://arxiv.org/abs/2605.14478v1
• Date: Thu, 14 May 2026 07:18:30 GMT
• Title: When Retrieval Hurts Code Completion: A Diagnostic Study of Stale Repository Context
• Abstract summary: Retrieval-augmented code generation relies on cross-file repository context, but retrieved snippets may come from obsolete project states.<n>We study whether temporally stale repository snippets act as harmless noise or actively induce current-state-incompatible code.
• Score: 4.235809074981841
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Context: Retrieval-augmented code generation relies on cross-file repository context, but retrieved snippets may come from obsolete project states. Objectives: We study whether temporally stale repository snippets act as harmless noise or actively induce current-state-incompatible code. Methods: We conduct a controlled diagnostic study on a curated 17-sample set of production-helper signature changes from five Python repositories. For each sample, we compare current-only, stale-only, no-retrieval, and mixed current/stale retrieval conditions under prompts that hide commit freshness and expected current signatures. Results: Under neutralized prompts, stale-only retrieval induces stale helper references on 15/17 Qwen2.5-Coder-7B-Instruct samples and 13/17 gpt-4.1-mini samples, corresponding to 88.2 and 76.5 percentage-point increases over current-only retrieval. No retrieval produces zero stale references but only 1/17 passing completions. The two models share 75.0% Jaccard overlap among stale-triggering samples, and mixed conditions show that adding valid current evidence largely rescues stale-only failures. Conclusion: Temporal validity of retrieved repository context is a distinct diagnostic variable for Code RAG robustness: stale context can actively bias models toward obsolete repository state rather than merely removing useful evidence.

Related papers

• Adaptive Critical Token-Aware Retrieval for Repository-Level Code Generation [53.85136813868056]
  ACToR is an adaptive critical token-aware retrieval framework for repository-level code generation.<n>We show that ACToR consistently outperforms state-of-the-art methods on repository-level benchmarks.
  arXiv  Detail & Related papers  (2026-09-01T17:59:39Z)
• Agent Retrieval Bench: Evaluating Repository Context Retrieval for Coding Agents [8.81342573156581]
  Agent Retrieval Bench is a file-level benchmark for this upstream retrieval problem.<n>The benchmark covers four positive-retrieval tasks: code2test, comment2context, trace2code, and edit2ripple.<n>The corpus includes 308 base-commit snapshots, 392,000 files, and 7.9 million chunks.
  arXiv  Detail & Related papers  (2026-07-27T09:39:09Z)
• Agent-Safety Evaluations as Load-Bearing Evidence: A Vendor-Neutral, Cross-Harness Reconstructability Metric [0.0]
  No vendor-neutral, runnable instrument scores reconstructability as an evaluation-validity metric.<n>This paper introduces a property-level reconstructability metric over eight decision-property classes.
  arXiv  Detail & Related papers  (2026-07-14T07:54:37Z)
• Diagnosing Evidence Utilization in Long-Context and Retrieval-Augmented Language Models under Matched Evidence Conditions [0.0]
  A model may answer from parametric priors, fail to use evidence that is present, or cite relevant text without converting it into the final answer.<n>This paper introduces a four-condition diagnostic protocol for evidence-utilization evaluation.
  arXiv  Detail & Related papers  (2026-06-04T22:44:57Z)
• Code-Centric Detection of Vulnerability-Fixing Commits: A Unified Benchmark and Empirical Study [4.512751676075442]
  We present a comprehensive evaluation of code language model based VFC detection through a unified framework.<n>We find no evidence that models acquire transferable security-relevant code understanding from code changes alone.<n>Group-stratified evaluation exposes approximately 17% performance drops compared to random splits.
  arXiv  Detail & Related papers  (2026-05-13T08:05:14Z)
• BUILD-AND-FIND: An Effort-Aware Protocol for Evaluating Agent-Managed Codebases [0.0]
  We introduce BUILD-AND-FIND, a protocol for evaluating whether downstream agents can recover intended choices from generated repositories.<n>For each task, a builder sees a hidden repository specification and creates a specification-traced multiple-choice question bank.<n>The protocol separates behavioral correctness from artifact-side recovery and reports recovery accuracy, repeatability, implementation coverage, and inspection effort.
  arXiv  Detail & Related papers  (2026-05-07T12:35:27Z)
• DataDignity: Training Data Attribution for Large Language Models [8.195274857647782]
  We introduce FakeWiki, a benchmark of 3,537 fabricated Wikipedia-style articles.<n>FakeWiki includes QA probes, source-preserving paraphrases, retro-generated variants, hard anti- documents that remain topically similar while removing answer-critical facts.<n>We evaluate seven retrieval baselines, a training-free activation-steering retrieval-fusion method, SteerFuse, and a supervised contrastive provenance ranker, ScoringModel.
  arXiv  Detail & Related papers  (2026-05-07T05:27:45Z)
• FRESCO: Benchmarking and Optimizing Re-rankers for Evolving Semantic Conflict in Retrieval-Augmented Generation [73.22935457705057]
  We introduce FRESCO, a benchmark for evaluating re-rankers in temporally dynamic contexts.<n>By pairing recency-seeking queries with historical Wikipedia revisions, FRESCO tests whether re-rankers can prioritize factually recent evidence while maintaining semantic relevance.<n>Our evaluation reveals a consistent failure mode across existing re-rankers: a strong bias toward older, semantically rich documents, even when they are factually obsolete.
  arXiv  Detail & Related papers  (2026-04-14T17:04:25Z)
• Verifier-Bound Communication for LLM Agents: Certified Bounds on Covert Signaling [0.0]
  Colluding language-model agents can hide coordination in messages that remain policy-compliant at the surface level.<n>We present CLBC, a protocol where generation and admission are separated.<n>We show how this protocol yields an upper bound on transcript leakage in terms of latent leakage plus explicit residual channels.
  arXiv  Detail & Related papers  (2026-02-27T23:42:37Z)
• SpatialBench-UC: Uncertainty-Aware Evaluation of Spatial Prompt Following in Text-to-Image Generation [0.0]
  SpatialBench-UC is a small, reproducible benchmark for pairwise spatial relations.<n>We release a benchmark package, versioned prompts, pinned configs, per-sample checker outputs, and report tables.<n>We evaluate three baselines, Stable Diffusion 1.5, SD 1.5 BoxDiff, and SD 1.4 GLIGEN.
  arXiv  Detail & Related papers  (2026-01-19T23:37:10Z)
• ImpliRet: Benchmarking the Implicit Fact Retrieval Challenge [50.93758649363798]
  Impliret is a benchmark that shifts the reasoning challenge to document-side processing.<n>We evaluate a range of sparse and dense retrievers, all of which struggle in this setting.
  arXiv  Detail & Related papers  (2025-06-17T11:08:29Z)
• Faithfulness-Aware Uncertainty Quantification for Fact-Checking the Output of Retrieval Augmented Generation [108.13261761812517]
  We introduce FRANQ (Faithfulness-based Retrieval Augmented UNcertainty Quantification), a novel method for hallucination detection in RAG outputs.<n>We present a new long-form Question Answering (QA) dataset annotated for both factuality and faithfulness.
  arXiv  Detail & Related papers  (2025-05-27T11:56:59Z)
• Collapse of Dense Retrievers: Short, Early, and Literal Biases Outranking Factual Evidence [56.09494651178128]
  Retrieval models are commonly used in Information Retrieval (IR) applications, such as Retrieval-Augmented Generation (RAG)<n>We quantify the impact of biases, such as a preference for shorter documents, on retrievers like Dragon+ and Contriever.<n>We uncover major vulnerabilities, showing retrievers favor shorter documents, early positions, repeated entities, and literal matches, all while ignoring the answer's presence!
  arXiv  Detail & Related papers  (2025-03-06T23:23:13Z)
• Does Recommend-Revise Produce Reliable Annotations? An Analysis on Missing Instances in DocRED [60.39125850987604]
  We show that a textit-revise scheme results in false negative samples and an obvious bias towards popular entities and relations. The relabeled dataset is released to serve as a more reliable test set of document RE models.
  arXiv  Detail & Related papers  (2022-04-17T11:29:01Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.