Does Fault Localization Beat a Fresh Attempt? A Placebo-Controlled Study of Test-Guided Code Repair

• URL: http://arxiv.org/abs/2609.00854v1
• Date: Tue, 01 Sep 2026 07:48:51 GMT
• Title: Does Fault Localization Beat a Fresh Attempt? A Placebo-Controlled Study of Test-Guided Code Repair
• Abstract summary: Fault localization can focus a code model's repair on the statements a failing test implicates, but a targeted edit may succeed merely because it is small.<n>We separate these explanations with three arms applied to the same failed candidate.<n>Across three frozen 26-32B models, three benchmarks and 488 failing candidates, three results follow.
• Score: 0.0
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Fault localization can focus a code model's repair on the statements a failing test implicates, but a targeted edit may succeed merely because it is small, and a second model call may succeed without using the failure at all. We separate these explanations with three arms applied to the same failed candidate: blind whole-solution resampling, spectrum-based localization followed by suspect-span infilling, and same-length infilling at a disjoint random code span. Across three frozen 26-32B models, three benchmarks and 488 failing candidates, plus a separately declared 24B fourth model from a third family, three results follow. First, localization is rarely available: only 9.0% of failing candidates expose a failing public test with a usable spectrum. Second, among the 177 candidates localizable from a strong suite, localized infilling loses decisively to blind resampling at a matched attempt count (3:40, p = 3.0 x 10^-9), opposite to our hypothesis; the loss replicates in a third family at -11.3 points (95% CI [-16.6, -6.8]), and widening the edit does not rescue it. Third, against the random-span placebo localized infilling leads pooled (11:1, Holm-adjusted p = .019), but that lead resolves in no individual model under the analysis our shipped plan designates primary (best Holm p = .087), so we report the location effect as suggestive rather than established. Re-pricing attempts as tokens narrows but does not overturn this: a span attempt spends 21.7 generated tokens against 371.1, yet 16 localized attempts reach 6.8% while one blind attempt already reaches 10.1%. Infilling reproduces the removed span verbatim in 48.9% of attempts, which is why more budget does not help. We restrict every localization conclusion to the 24-32B models tested.

Related papers

• Target-Checked Reliability Score Refinement for Video Question Answering [3.8271803328378677]
  We study whether answer-level reliability scores can be improved under target shift without retraining the models or changing their answers.<n>We collect option-probability lists from three fixed video-language models under four deterministic video samplings.<n>We compare the original score, defined as the probability assigned to the chosen answer, with a histogram-based gradient-boosting (HGB) score trained on the development datasets.<n>A candidate replaces the original score only when repeated video-level checks indicate a positive, stable improvement.
  arXiv  Detail & Related papers  (2026-09-09T08:29:00Z)
• Marginal Fidelity Does Not Establish User Simulation in Demographic Synthetic Survey Panels: Response Contracts, Support Collapse and Conditioning Failure [51.736723807086385]
  Population-marginal agreement is evidence about an elicitation contract and an estimand obtainable without simulated respondents, not evidence of individual simulation.<n>On nine aligned model-battery pairs, a no-persona population-prevalence query averages 6.27 MAE versus 12.39 for committed panels and wins all nine comparisons.
  arXiv  Detail & Related papers  (2026-09-07T10:22:22Z)
• When Self-Consistency Backfires: Majority Vote Hurts the Majority of Hard Science Problems for Small LLMs [0.0]
  Self-consistency via majority vote reduces per-problem accuracy on most GPQA Diamond problems for small instruction-tuned models.<n>This version reports that the most natural repair also fails, and separates three signal failures that v1 treated as one.
  arXiv  Detail & Related papers  (2026-08-11T20:08:51Z)
• Accurate Ensembles, Fragile Narratives: Multi-Scale Stacking and a Fidelity Audit of LLM-Generated Explanations for Credit Risk [0.0]
  Credit scoring increasingly relies on models whose decision logic cannot be read off their parameters.<n>A common proposal closes that gap with a language model: compute feature attributions, hand them to an LLM, and let it write the rationale.<n>We build such a system end to end and test whether the second half of the promise holds.
  arXiv  Detail & Related papers  (2026-08-08T13:22:14Z)
• Does Splitting a Triage Decision Across Agents Hide Bias or Help Catch It? A Multi-Agent Simulation Study of LLM-Based Resource Allocation Under Audit Capacity Constraints [0.0]
  We study what happens to bias when the same decision is distributed across a role-differentiated multi-agent pipeline.<n>We find no measurable difference in how often biased outcomes occur between the two conditions.<n>We do find a large and significant effect of audit capacity on whether bias is caught.
  arXiv  Detail & Related papers  (2026-08-07T08:25:53Z)
• Passing Coarse Marginal Checks Can Be Cheap: Persona Mixtures and Imprecise Treatment-Response Estimates in an LLM Persona Panel [0.0]
  We study a fixed panel of sixteen lightweight persona-conditioned GPT-4.1 configurations in repeated strategic games.<n>Variation was strongly prompt-indexed, but its share depended on uncertainty assumptions.<n>Results concern one fixed model-prompt panel and do not establish human substitutability.
  arXiv  Detail & Related papers  (2026-08-02T04:10:11Z)
• Catching One in Five: LLM-as-Judge Blind Spots in Production Multi-Turn Transaction Agents [45.148328075418156]
  We study a deployed multi-turn food-and-beverage ordering agent and measure how many genuine quality problems it catches.<n>Our blind-spot taxonomy shows the failure is structured, not random.<n>For production multi-turn agents, automated judging is a regression floor, not a substitute for human review.
  arXiv  Detail & Related papers  (2026-06-09T02:11:01Z)
• Automated Proving of Shannon-Type Entropy Inequalities via Fine-Tuned Language Models and Guided Tree Search [50.16356451328644]
  Proving Shannon-type entropy inequalities is a fundamental task in information theory.<n>We investigate whether small-scale large language models, fine-tuned on atomic proof steps, can automate this process.<n>We show that GPT-5.5 solves 1.7% samples under zero-shot prompting while Psitip solves 33.3% samples.
  arXiv  Detail & Related papers  (2026-06-04T05:43:12Z)
• Distributional Energy-Based Models for Uncertainty-Aware Structured LLM Reasoning [40.342912574072024]
  Large Language Models produce structured outputs such as travel plans or code solutions.<n>Individual reasoning steps may appear correct while the output as a whole violates budgets, fails test cases, or contradicts earlier deductions.<n>We propose a learned quality scorer with deterministic analytical constraint penalties for verifying structured LLM outputs.
  arXiv  Detail & Related papers  (2026-05-15T17:08:27Z)
• Below-Chance Blindness: Prompted Underperformance in Small LLMs Produces Positional Bias Rather than Answer Avoidance [0.0]
  We tested whether symptom validity testing logic could identify sandbagging through below-chance performance (BCB) on forced-choice items.<n>We found zero of 12 model-domain cells showed significant below-chance performance under sandbagging instruction.<n>We propose that positional-distribution shift may be a more effective behavioural signature than below-chance accuracy for detecting prompted underperformance at this model scale.
  arXiv  Detail & Related papers  (2026-04-28T05:57:23Z)
• Causal Understanding by LLMs: The Role of Uncertainty [43.87879175532034]
  Recent papers show LLMs achieve near-random accuracy in causal relation classification.<n>We investigate whether pretraining exposure to causal examples improves causal understanding.
  arXiv  Detail & Related papers  (2025-09-24T13:06:35Z)
• Exploring Response Uncertainty in MLLMs: An Empirical Evaluation under Misleading Scenarios [49.53589774730807]
  Multimodal large language models (MLLMs) have recently achieved state-of-the-art performance on tasks ranging from visual question answering to video understanding.<n>We reveal a response uncertainty phenomenon: twelve state-of-the-art open-source MLLMs overturn a previously correct answer in 65% of cases after receiving a single deceptive cue.
  arXiv  Detail & Related papers  (2024-11-05T01:11:28Z)
• SBEST: Spectrum-Based Fault Localization Without Fault-Triggering Tests [17.90798133817018]
  This study investigates the feasibility of using stack traces from crash reports as proxies for fault-triggering tests in Spectrum-Based Fault localization.<n>We propose SBEST, a novel approach that integrates stack trace information with test coverage data to perform fault localization when fault-triggering tests are missing.
  arXiv  Detail & Related papers  (2024-05-01T15:15:52Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.