Rehearse: Stepping Back from the Confidence Cliff in Self-Improving Autoresearch

• URL: http://arxiv.org/abs/2607.27687v1
• Date: Thu, 30 Jul 2026 05:08:15 GMT
• Title: Rehearse: Stepping Back from the Confidence Cliff in Self-Improving Autoresearch
• Abstract summary: Autoresearch improves machine-learning code by proposing changes, running full training jobs, and keeping changes that improve the metric.<n>We study how the reliability of this pre-execution judgment changes over the course of an autoresearch trajectory.
• Score: 44.05969616406614
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Autoresearch improves machine-learning code by proposing changes, running full training jobs, and keeping changes that improve the metric. The efficiency of this loop depends not only on generating ideas, but also on the agent's ability to decide, before spending a training run, whether a proposed modification is likely to work. We study how the reliability of this pre-execution judgment changes over the course of an autoresearch trajectory. In public AutoSOTA logs (Li et al., 2026; Tsinghua FIB Lab, 2026), the fraction of helpful modifications falls from 70% in the first two iterations to 43% by iteration 6+. On 296 same-baseline modification pairs from 39 paper-derived AutoSOTA tasks, each containing one modification that improved the metric and one that did not, with measured outcomes hidden, an LLM judge given candidate rationales but no prior-attempt history reaches 79.5% accuracy on the pairs where strict consensus returns a verdict. On the full 366-pair benchmark, however, this ability weakens substantially late in the loop. As successful changes accumulate, selective accuracy - accuracy conditioned on a strict-consensus verdict - falls from 82.8% to 56.9%, while the judge remains willing to decide. We call this operational pattern the confidence cliff. Rehearse implements the loop change as a lightweight skill for autoresearch loops: propose several ideas, compare them before execution, run the most promising, and judge with a focused memory of similar past attempts and outcomes. This focused outcome memory raises late selective accuracy to 83.5%. Across 4,000 budgeted training runs over three loops, Rehearse improves the endpoint under the same training-run budget on nanochat, image classification, and time-series forecasting.

Related papers

• CODS: Iterative Bellman-Residual Data Selection for Reusable Offline Reinforcement Learning [5.894992157122175]
  We introduce CODS, a critic-guided selector that alternates between fitting an algorithm-matched critic and acquiring high-residual transitions.<n>At a 10% budget, CODS retains 96.6% of eligible-pool performance across 20 valid D4RL task--algorithm cells.
  arXiv  Detail & Related papers  (2026-08-07T19:11:54Z)
• Share the Judge, Learn the Deferral: Where Specialization Helps LLM Evaluation [0.1269104766024433]
  We study 99,952 public, rubric-conditioned examples.<n>Supplying the correct rubric improves locked-test accuracy by 2.11 points over a response-only control.<n>On RewardBench 2, learned correctness heads route examples through a 0.6B-4B-8B cascade without changing any reward score.
  arXiv  Detail & Related papers  (2026-07-30T10:29:22Z)
• Do Agent Optimizers Compound? A Continual-Learning Evaluation on Terminal-Bench 2.0 [53.71882250666667]
  Most reported gains from agent-optimization methods are one-shot.<n>This does not test the setting that matters for deployed agents.<n>We compare three approaches to agent-harness optimization.
  arXiv  Detail & Related papers  (2026-07-15T16:36:04Z)
• TASR: Training-Free Adaptive Stopping for Iterative Retrieval [12.177557521540082]
  Iterative retrieval-augmented generation agents commonly overspend by continuing to retrieve after the model has converged on an answer.<n>We propose TASR (Training-Free Adaptive Stopping Rule), a one-line predicate that fires when the model repeats its previous-round normalized answer.
  arXiv  Detail & Related papers  (2026-06-11T18:35:14Z)
• Converted, Not Equivalent: Benchmarking Codebase Conversion via Observational Equivalence [56.25095230687242]
  Coding agents often over-trust their own local validation routines and declare success on artifacts that satisfy surface checks.<n>This problem is especially acute in conversion, where prior evaluation is outcome-driven.<n>Across blind conversion attempts, the best system reaches 26.7--28.9% overall pass rate despite Spec pass rates up to 91.1%.<n>This suggests that failures stem more from contract-misaligned self-validation than from limited budget or backbone strength.
  arXiv  Detail & Related papers  (2026-05-27T19:57:15Z)
• Step-wise Rubric Rewards for LLM Reasoning [72.17879367869503]
  Reinforcement Learning with Verifiable Rewards (RLVR) is widely used to improve reasoning in large language models.<n>We find 18.2% of steps in correct responses are wrong yet positively rewarded.<n>Across six mathematical reasoning benchmarks, SRaR improves average accuracy over RaR by 3.57 points.
  arXiv  Detail & Related papers  (2026-05-17T07:08:14Z)
• Relational Preference Encoding in Looped Transformer Internal States [0.0]
  We investigate how looped transformers encode human preference in their internal iteration states.<n>Using Ouro-2.6B-Thinking, a 2.6B- parameter looped transformer with iterative refinement, we extract hidden states from each loop iteration.<n>We train lightweight evaluator heads to predict human preference on the Anthropic HH-RLHF dataset.<n>Our pairwise evaluator achieves 95.2% test accuracy on 8,552 unseen examples, surpassing a full-batch L-BFGS probe (84.5%) while the base model remains completely frozen.
  arXiv  Detail & Related papers  (2026-04-10T20:00:49Z)
• Tracing LLM Reasoning Processes with Strategic Games: A Framework for Planning, Revision, and Resource-Constrained Decision Making [38.75183725659772]
  Large language models (LLMs) are increasingly used for tasks that require complex reasoning.<n>We argue that measuring internal processes is essential for understanding model behavior and improving reliability.<n>We introduce a framework that evaluates LLMs along three core dimensions: planning, revision, and resource-constrained decision making.
  arXiv  Detail & Related papers  (2025-06-13T17:59:10Z)
• Iterative Reasoning Preference Optimization [84.15992372132507]
  We develop an iterative approach to optimize the preference between generated Chain-of-Thought (CoT) candidates. We show reasoning improves across repeated iterations of this scheme. For example, we see a large improvement from 55.6% to 81.6% on GSM8K and an accuracy of 88.7% with majority voting out of 32 samples.
  arXiv  Detail & Related papers  (2024-04-30T17:28:05Z)
• Improving Auto-Augment via Augmentation-Wise Weight Sharing [123.71986174280741]
  A key component of automatic augmentation search is the evaluation process for a particular augmentation policy. In this paper, we dive into the dynamics of augmented training of the model. We design a powerful and efficient proxy task based on the Augmentation-Wise Weight Sharing (AWS) to form a fast yet accurate evaluation process.
  arXiv  Detail & Related papers  (2020-09-30T15:23:12Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.