The Recall Trap: A Recall-Maximizing Retriever Configuration Reduces Issue Resolution in Fixed-Budget Code Context

• URL: http://arxiv.org/abs/2608.14838v1
• Date: Fri, 14 Aug 2026 19:22:50 GMT
• Title: The Recall Trap: A Recall-Maximizing Retriever Configuration Reduces Issue Resolution in Fixed-Budget Code Context
• Abstract summary: We report a controlled case study in code repair, not a new phenomenon.<n>We toggle one flag (one-chunk-perfile deduplication) on an otherwise identical stack.<n>We map where it holds: it reverses on a lexical BM25 (3.2pp, significant cross-paradigm interaction)<n>We show that at a tight fixed budget, do not hard-deduplicate by file, and A/B packing policies against the task, not the metric flag was tuned.
• Score: 45.88028371034407
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Retrieval components for code assistants are tuned against retrieval metrics: a configuration that raises recall@k is adopted, and downstream task success is assumed to follow. We report a controlled case study in code repair, not a new phenomenon but a deployed-flag, execution-graded instance of the known relevance-diversity and objective-mismatch tradeoff (Levy et al., 2025). On SWE-bench Verified we inject a retriever's hits as a fixed 12-slot context pack with no search tools and toggle one flag (one-chunk-per-file deduplication) on an otherwise identical stack. The flag is the higher-recall configuration (gold file present in 0.878 of served packs against 0.806 disabled), yet disabling it, trading file breadth for within-file depth, raises the single-shot resolve rate: gpt-5.6-sol +7.6pp (39.2% to 46.8%, n=500, McNemar exact p=0.0003), and a pre-registered open-weights replication any reviewer can re-run (Qwen3.6-27B, +3.6pp, n=499, p=0.0133); both survive repository-clustered inference. The gain tracks within-file anchor dose, and a random-chunk control refutes an argmax-selection artifact. We map where it holds: it reverses on a lexical BM25 retriever (-3.2pp, significant cross-paradigm interaction), is not detected under unrestricted-Read agents (a powered null), and across four languages (SWE-PolyBench, N=617) is positive but not significant (+2.6pp, p=0.056), a mapped boundary rather than a confirmed extension. Operationally, at a tight fixed budget: do not hard-deduplicate by file, and A/B packing policies against the task, not the metric the flag was tuned to.

Related papers

• ReDraft, Don't Just Distill: Reference-Driven Revision for Continual VLLM Post-Training [66.72708893922605]
  Post-training of large multimodal models should add new capabilities while preserving those from pre-training.<n>SFT gives explicit target supervision that learns a task from near-zero accuracy, but its off-policy targets move the model far enough to cause forgetting.<n>We introduce ReDraft, which obtains both from the model's own failures.
  arXiv  Detail & Related papers  (2026-09-15T04:59:03Z)
• NovaFabric: Tamper-Evident, Replayable Evidence for Autonomous AI Agent Runs [0.0]
  We present NovaFabric, producing audit-grade execution evidence.<n>It records an agent run, without modifying agent logic, into a portable Run Capsule.<n>Sealed runs are re-executable under a four-mode replay protocol.<n>We evaluate eight research questions at measured scope.
  arXiv  Detail & Related papers  (2026-09-11T08:35:43Z)
• Hindsight Memory-PRM: Supervising Memory Management with Auditable Hindsight Credit [48.811961095386]
  Hindsight Memory-PRM exploits this audit trail twice.<n> offline to train an operation-conditioned memory-utility critic.<n>Online, where retrievals, citations, and one controlled deletion-and-reanswer per probe settle an intervention-calibrated entry-level presence credit.
  arXiv  Detail & Related papers  (2026-08-30T06:53:13Z)
• Navigation-Informed Embeddings: Dense-Retriever Adaptation from Agent Search Traces [0.0]
  We introduce Navigation-Informed Embeddings (NIE), a family of trace-derived objectives.<n>NIE-Stop turns the stopping document into a soft positive; NIE-Path additionally uses preceding path documents as hard comparisons.<n>NIE-Path raises long-path performance to 55.4, compared with 46.7 for the unadapted encoder.
  arXiv  Detail & Related papers  (2026-08-16T23:07:44Z)
• AgentDebugX: An Open-Source Toolkit for Failure Observability, Attribution, and Recovery in LLM Agents [83.37969790806069]
  LLM agent failures are difficult to debug because the step where an error surfaces is often not the one that caused it.<n>We present Agent DebuggerX, an open-source framework that organizes debug as a closed loop of Detect, Attribute, Recover, and Rerun.<n>At its core, Deep Debugger performs multi-turn root-cause diagnosis through global trajectory understanding.
  arXiv  Detail & Related papers  (2026-07-21T06:21:13Z)
• Operational Proto-Introspection in Looped Language Models: Process-Quality Taps, Executable Branching, and the Readout-Control Boundary [0.0]
  Can a language model read the quality of its ongoing computation?<n>Can an external intervention turn that readout into better outcomes?<n>We test both questions in a frozen 2.6B looped transformer, Ouro-RLTT.
  arXiv  Detail & Related papers  (2026-07-20T22:40:36Z)
• Fantastic Adaptive Taxonomies and How to Use Them [100.20614173157708]
  AdaMAST converts execution traces into a compact, evidence-grounded failure taxonomy.<n>No code is hand-authored, no trace human-annotated.<n>In agent-system search, taxonomy-coded diagnoses of failed candidates outperform free-form.
  arXiv  Detail & Related papers  (2026-07-17T17:41:16Z)
• Efficient Visual Pointing for Embodied AI:Agent-Driven Data Synthesis, Cross-Block Attention, and Iterative Correction [55.11480729304395]
  We describe our PointArena 2026 solution, which achieves 77.2% overall accuracy and ranks second on the benchmark.<n>The ap proach targets three failure modes. First, agent-driven syn thesis builds large semantic and anchor-relative candidate pools.<n>Second, a determinis tic steerable-data pipeline creates a verified 10,000-sample main set, plus reserve samples, using masks, templates, and path verification.
  arXiv  Detail & Related papers  (2026-06-29T06:39:03Z)
• SCAR: Semantic Continuity-Aware Retrieval for Efficient Context Expansion in RAG [0.0]
  SCAR (Semantic Continuity-Aware Retrieval) is an adaptive retrieval policy that expands neighboring chunks by weighing query-neighbor relevance against a structural continuity penalty.<n>Across four diverse corpora (RFC, 10-K report, and a Merger agreement; N=320 queries; 160 boundary-fragmented), SCAR achieves 9 recall on boundary-fragmented queries with only 7.84 chunks, a 22.9% reduction compared to static windowing (10.16 chunks)
  arXiv  Detail & Related papers  (2026-06-15T12:55:49Z)
• Control-Plane Placement Shapes Forgetting: An Architectural Study of Agent Memory Across Thirteen System Configurations [1.2763567932588586]
  ForgetEval is a 1000-case templated suite plus a 385-case adversarial layer (132 hand-crafted + 253 LLM-drafted oracle-validated)<n>We observe three placement regimes with partly complementary coverage: deterministic primitives suffice for lexical/temporal categories but fail canonicalization.<n>We expose the trade-off via ForgetEval, a 1000-case templated suite plus a 385-case adversarial layer.
  arXiv  Detail & Related papers  (2026-06-14T16:32:15Z)
• Decision-Aware Memory Cards: Counterfactual-Inspired Context Selection and Compression for Tool-Using LLM Agents [3.964533007623828]
  Modern large language model (LLM) agents need decision-relevant evidence at the moment of action.<n>We present the Counterfactual-Inspired Context Layer (CICL), which builds an instance context graph, estimates decision-oriented utility for candidate units, and compresses selected evidence into typed memory cards.<n>CICL provides a practical layer for measuring, ranking, and compressing decision-critical context for tool-using agents.
  arXiv  Detail & Related papers  (2026-06-06T13:02:28Z)
• Strained Coherence: A Pre-Failure Signal in Coding Agent Execution Trajectories [0.0]
  LLM-based coding agents sometimes acknowledge a problem in their own reasoning and then proceed anyway.<n>We build a Claude Sonnet 4.6 judge that reads full trajectories and flags spans where the pattern occurs.<n>We evaluate it on 44 Terminal-bench-2 trajectories using a Qwen3.5-35B-A3B backbone.
  arXiv  Detail & Related papers  (2026-06-05T22:52:16Z)
• Overeager Coding Agents: Measuring Out-of-Scope Actions on Benign Tasks [40.270213696031625]
  OverEager-Gen is a benchmark dedicated to overeager behavior on benign tasks.<n>On Claude Code, stripping the consent declaration alone raises the overeager rate from 0.0% to 17.1%.<n>OverEager-Gen therefore certifies each scenario's discriminative power before admission.
  arXiv  Detail & Related papers  (2026-05-18T16:00:41Z)
• Distributional Energy-Based Models for Uncertainty-Aware Structured LLM Reasoning [40.342912574072024]
  Large Language Models produce structured outputs such as travel plans or code solutions.<n>Individual reasoning steps may appear correct while the output as a whole violates budgets, fails test cases, or contradicts earlier deductions.<n>We propose a learned quality scorer with deterministic analytical constraint penalties for verifying structured LLM outputs.
  arXiv  Detail & Related papers  (2026-05-15T17:08:27Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.