Reachability Is Not Realization: Tracing the Sources of LLM Benchmark Gains

• URL: http://arxiv.org/abs/2608.03219v1
• Date: Tue, 04 Aug 2026 06:52:27 GMT
• Title: Reachability Is Not Realization: Tracing the Sources of LLM Benchmark Gains
• Abstract summary: A model may reach new answers, or produce answers that were already within reach.<n>We establish a question-level audit under fixed budgets, temperatures, and answer formats.<n>We first test whether inference-time layer routing can expand reachability.
• Score: 15.723674503598948
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Benchmark gains are often treated as evidence of greater LLM capability. Yet the same gain can reflect different changes in model behavior. A model may reach new answers, or produce answers that were already within reach. Aggregate scores do not distinguish these changes question by question. We establish a question-level audit under fixed budgets, temperatures, and answer formats. A question is realized when the default deployment procedure produces the correct answer. A question is reachable when a specified probe finds that answer within a fixed budget. We first test whether inference-time layer routing can expand reachability. Under a matched budget, random routes match or exceed structured search in all 43 model and task settings. Answer-blind procedures retain almost none of this gain, which instead requires access to the correct answer. We then ask why reachable answers sometimes fail to appear. Across six cases spanning 0.5B to 31B, silencing one identified MLP block repairs 68 to 92 percent of a predefined failure set. We next test whether training closes the gap by expanding reachability. In five of six matched evaluations, deployed performance rises while the reachable ceiling remains flat or falls. For DAPO, the deployed score rises by 14.7 points while the reachable ceiling falls by 13.3 points. Across the settings we audit, realization and reachability therefore do not always change together. Claims of capability expansion should report both realized performance and reachability under matched evaluation conditions. Code is available at https://github.com/LiZaiyuan0619/reachability-not-realization

Related papers

• TRACE: A Self-Evolving Skill Bank for Consistent, Limit-Aware LLM Agents [41.82249291285459]
  TRACE iteratively improves a skill-based agent's behavioral knowledge without modifying model weights.<n>On GPT-5.5, TRACE improves consistency (Pass3) by 34.6 points, from 59.9% to 94.5%, while shrinking the gap between potential and reliable performance to just 4.0 points.<n>These results show that TRACE converts high model potential into stable, consistent performance gain.
  arXiv  Detail & Related papers  (2026-08-24T04:36:03Z)
• Business Truth, not SQL Accuracy: A Rule-Gated 7B Analytics Agent Outperforms a Direct-Prompted 32B Baseline [0.0]
  This paper introduces WarehouseBench, 400 frozen tasks over two synthetic warehouses in which roughly half the correct responses are a clarification, an abstention or a refusal.<n>QueryProof, a 7B agent, uses rules derived from a semantic layer and physical catalog to determine its behaviour.
  arXiv  Detail & Related papers  (2026-08-10T08:14:24Z)
• Rethinking Self-Evolving Agent Skills: Feedback Dynamics over Multiple Rounds [43.5891705224868]
  Self-evolving skill systems promise to improve agents by turning execution feedback into persistent skill updates without changing the underlying model.<n>We present a controlled evaluation framework across five benchmarks and three models.<n>Overall, persistent skill self-evolution is better understood as sparse, validation-filtered search with model- and benchmark-dependent returns.
  arXiv  Detail & Related papers  (2026-07-31T04:02:51Z)
• Closing the Loop on Latent Reasoning via Test-Time Reconstruction [45.08180971427891]
  Recent work moves intermediate reasoning from natural-language traces into latent or cache-level representations.<n>We propose ReLAT, a self-supervised test-time training method that closes this loop using the query itself as the reference.<n>We show that ReLAT consistently improves over single-model inference, text-based collaboration, open-loop latent collaboration, and alternative test-time training objectives.
  arXiv  Detail & Related papers  (2026-06-04T14:54:40Z)
• Beyond Inference-Time Search: Reinforcement Learning Synthesizes Reusable Solvers [0.0]
  Large language models (LLMs) typically approach optimization as an inference-time procedure, solving each instance separately.<n>We study this question on Synergistic Dependency Selection (SDS), a controlled variant of constrained Quadratic Knapsack.<n>We fine-tune Qwen2.5-Coder-14B-Instruct with Group Relative Policy Optimization using a feasibility-gated reward and light structural scaffolding.
  arXiv  Detail & Related papers  (2026-05-18T13:21:40Z)
• Grounded Continuation: A Linear-Time Runtime Verifier for LLM Conversations [15.537674351419234]
  Context-manipulation attacks against deployed agents now actively exploit this gap.<n>We close it with a runtime verifier that maintains an explicit dependency graph.<n>Checking whether a continuation is supported reduces to a graph walk; retraction propagates through the same graph to flag exactly the conclusions that lose support.
  arXiv  Detail & Related papers  (2026-05-13T22:54:16Z)
• When Reasoning Traces Become Performative: Step-Level Evidence that Chain-of-Thought Is an Imperfect Oversight Channel [5.106950500256654]
  Chain-of-thought (CoT) traces are increasingly used to improve language model capability and to audit model behavior.<n>We test this assumption with a step-level Detect-Classify-Compare framework built around an answer-commitment proxy.<n>Across nine models and seven reasoning benchmarks, latent commitment and explicit answer arrival align on only 61.9% of steps on average.
  arXiv  Detail & Related papers  (2026-05-12T08:24:47Z)
• HiL-Bench (Human-in-Loop Benchmark): Do Agents Know When to Ask for Help? [32.54022440678003]
  coding agents solve complex tasks when given complete context but collapse when specifications are incomplete or ambiguous.<n>Current benchmarks are blind to this failure mode.<n>We present HiL-Bench to measure this selective escalation skill.
  arXiv  Detail & Related papers  (2026-04-10T15:21:44Z)
• AdaFuse: Adaptive Ensemble Decoding with Test-Time Scaling for LLMs [46.52320938421707]
  Inference-time ensembling provides a practical way to combine large language model capabilities without retraining.<n>We propose AdaFuse, an adaptive ensemble decoding framework that dynamically selects semantically appropriate fusion units during generation.<n>AdaFuse consistently outperforms strong ensemble baselines, achieving an average relative improvement of 6.88%.
  arXiv  Detail & Related papers  (2026-01-09T18:58:22Z)
• SABER: Small Actions, Big Errors -- Safeguarding Mutating Steps in LLM Agents [52.20768003832476]
  We analyze execution traces on $$-Bench (Airline/Retail) and SWE-Bench Verified.<n>We formalize emphdecisive deviations, earliest action, level divergences that flip success to failure.<n>We introduce cm, a model-agnostic, gradient-free, test-time safeguard.
  arXiv  Detail & Related papers  (2025-11-26T01:28:22Z)
• Eigen-1: Adaptive Multi-Agent Refinement with Monitor-Based RAG for Scientific Reasoning [53.45095336430027]
  We develop a unified framework that combines implicit retrieval and structured collaboration.<n>On Humanity's Last Exam (HLE) Bio/Chem Gold, our framework achieves 48.3% accuracy.<n>Results on SuperGPQA and TRQA confirm robustness across domains.
  arXiv  Detail & Related papers  (2025-09-25T14:05:55Z)
• RankAlign: A Ranking View of the Generator-Validator Gap in Large Language Models [43.89751891729739]
  We consider the discrepancy between a model's generated answer and their own verification of that answer, the generator-validator gap.<n>We show that according to this measure, a large gap exists in various settings, including question answering, lexical semantics tasks, and next-word prediction.<n>We then propose RankAlign, a ranking-based training method, and show that it significantly closes the gap, surpassing all baseline methods.
  arXiv  Detail & Related papers  (2025-04-15T16:53:31Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.