AI4AI-Bench: Benchmarking LLM Agents in Algorithmic Design for Recursive Self-Improvement

• URL: http://arxiv.org/abs/2608.20318v1
• Date: Thu, 20 Aug 2026 17:56:59 GMT
• Title: AI4AI-Bench: Benchmarking LLM Agents in Algorithmic Design for Recursive Self-Improvement
• Abstract summary: Recursive self-improvement (RSI) asks whether an AI system can improve the process that produces AI systems.<n>We present AI4AImbox-Bench, 10 frozen research repositories spanning 10 training algorithm families.
• Score: 4.962568775070268
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Recursive self-improvement (RSI) asks whether an AI system can improve the process that produces AI systems, so that the next system inherits the improvement. That process is the training algorithm: a better objective or update rule improves the compute\mbox{-}capability exchange rate for every subsequent run, including the one that produces the next agent. Whether RSI is feasible therefore turns on whether an agent can design training algorithms. No benchmark isolates that ability: existing suites are won by collecting data or by tuning hyperparameters, and none tells a change to how a run is executed apart from a change to how the model learns. We present AI4AI\mbox{-}Bench, 10 frozen research repositories spanning 10 training algorithm families. In each task, an agent has 4 hours on one B300 to rewrite the training algorithm; its code is then rerun from scratch for up to 12 hours and scored by a fixed evaluator hidden from the agent, against the repository's original algorithm under the same procedure. Because the 10 metrics are incommensurable, every task is mapped onto one scale on which $0$ is an uninformative model, $0.1$ is the algorithm the repository ships, and $1.0$ is the task optimum. Across 29 configurations of 6 systems on all 10 tasks the mean score is $0.166$, and the best system reaches $0.250$: even the strongest closes under a fifth of the distance between the algorithm that was already there and the optimum. The submissions show where that distance went: most never change how the model learns at all, and the minority that do average $0.226$ against $0.126$ for the rest. More reasoning effort mostly buys the willingness to go there, taking that minority from $8\%$ of submissions to $64\%$ and the mean score from $0.094$ to $0.196$. We release the task suite, the evaluators and every scored submission, so that the measurement can be repeated as these systems change.

Related papers

• AI Research Preference Models [87.14303218008381]
  Research agents can propose, implement, and evaluate their own machine learning experiments.<n> progress on frontier tasks is throttled by cost.<n>We introduce AI Research Preference Models ( RPMs) that predict which of multiple candidate solutions are most worth executing.
  arXiv  Detail & Related papers  (2026-08-14T04:20:37Z)
• Provably Optimal Learning Algorithms for Assistance Games [60.843021846414956]
  We provide the first provably efficient learning algorithms for repeated assistance games.<n>We present decentralized algorithms for both an informed human and the assistant.<n>We prove that achieving a regret approximation factor better than $(1-1/e)$ is computationally intractable.
  arXiv  Detail & Related papers  (2026-07-09T00:38:42Z)
• Toward Compiler World Models: Learning Latent Dynamics for Efficient Tensor Program Search [9.85275171877854]
  We propose an evaluator that schedules evaluation as action-conditioned latent dynamics over program states.<n>Our method improves representative-subgraph latency over Ansor by 1.37$times$ on GPU and 1.54$times$ on CPU under the same 64-trial budget.<n>It also accelerates full-model inference over PyTorch/PyTorch-opt(cuDNN) by 4.61$times$/3.67$times$ geometric mean.
  arXiv  Detail & Related papers  (2026-06-08T10:17:27Z)
• Do Post-Training Algorithms Actually Differ? A Controlled Study Across Model Scales Uncovers Scale-Dependent Ranking Inversions [1.6498361958317636]
  We present a unified framework implementing 51 post-training algorithms with identical infrastructure.<n>Our study spans 8 algorithms across 4 model scales (0.5B--7B), 3 evaluation domains, and a 20-variant DPO taxonomy.<n>None of 20 DPO variants significantly outperform vanilla DPO after Bonferroni correction; the sole significant outlier, SimPO, is worse.
  arXiv  Detail & Related papers  (2026-03-19T04:10:38Z)
• Efficiently Learning Branching Networks for Multitask Algorithmic Reasoning [28.332657856837788]
  AutoBRANE is a principled architecture for multitask algorithmic reasoning.<n>It clusters tasks using gradient-based affinity scores and can be used on top of any base model.<n>It achieves a 28% accuracy gain over existing multitask and branching architectures.
  arXiv  Detail & Related papers  (2025-11-30T22:19:55Z)
• Online Fair Division for Personalized $2$-Value Instances [51.278096593080456]
  We study an online fair division setting, where goods arrive one at a time and there is a fixed set of $n$ agents.<n>Once a good appears, the value each agent has for it is revealed and it must be allocated immediately and irrevocably to one of the agents.<n>We show how to obtain worst case guarantees with respect to well-known fairness notions.
  arXiv  Detail & Related papers  (2025-05-28T09:48:16Z)
• From Continual Learning to SGD and Back: Better Rates for Continual Linear Models [50.11453013647086]
  We analyze the forgetting, i.e., loss on previously seen tasks, after $k$ iterations.<n>We develop novel last-iterate upper bounds in the realizable least squares setup.<n>We prove for the first time that randomization alone, with no task repetition, can prevent catastrophic in sufficiently long task sequences.
  arXiv  Detail & Related papers  (2025-04-06T18:39:45Z)
• Algorithm Design for Continual Learning in IoT Networks [16.35495567193046]
  Continual learning (CL) is a new online learning technique over sequentially generated streaming data from different tasks.<n>In practical IoT networks, an autonomous vehicle to sample data and learn different tasks can route and alter the order of task pattern.
  arXiv  Detail & Related papers  (2024-12-22T02:36:09Z)
• On the Sample Complexity of Representation Learning in Multi-task Bandits with Global and Local structure [77.60508571062958]
  We investigate the sample complexity of learning the optimal arm for multi-task bandit problems. Arms consist of two components: one that is shared across tasks (that we call representation) and one that is task-specific (that we call predictor) We devise an algorithm OSRL-SC whose sample complexity approaches the lower bound, and scales at most as $H(Glog(delta_G)+ Xlog(delta_H))$, with $X,G,H$ being, respectively, the number of tasks, representations and predictors.
  arXiv  Detail & Related papers  (2022-11-28T08:40:12Z)
• Policy Optimization for Markov Games: Unified Framework and Faster Convergence [81.3266426402464]
  We show that the state-wise average policy of this algorithm converges to an approximate Nash equilibrium (NE) of the game. We extend this algorithm to multi-player general Markov Games and show a $mathcalwidetildeO(T-1/2)$ convergence rate to Correlated Equilibria (CCE)
  arXiv  Detail & Related papers  (2022-06-06T14:23:13Z)
• Fast Learning for Renewal Optimization in Online Task Scheduling [18.935043109084543]
  This paper considers online optimization of a renewal-reward system. The probability distribution for the task type vector is unknown. The proposed algorithm uses an auxiliary variable that is updated according to a classic Robbins-Monro iteration.
  arXiv  Detail & Related papers  (2020-07-18T22:44:13Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.