RSI Arena: Towards Recursive Self-Improvement with Human Feedback
Abstract Overview
RSI Arena investigates whether frontier language models can autonomously post-train and iteratively improve a shared base model under realistic resource constraints. Ten training agents independently selected data, implemented training, and nominated checkpoints while being evaluated across 20 benchmarks and a blind human arena. In Stage 1, all ten agents produced working assistants, with the strongest model closing a median of 76% of the base-to-Instruct gap across 12 capability benchmarks and matching NVIDIA Instruct in style-adjusted human ratings. In Stage 2, agents received battle records involving their own models and trained for three additional days, with eight of ten narrowing their style-adjusted gap to the reference model, though benchmark gains remained modest and safety metrics showed regressions in some cases.
Novelty
The study introduces an auditable, two-stage evaluation arena where multiple frontier models act as autonomous post-training agents on a shared base model and refine them using human comparison data from real users. Unlike narrower benchmarks targeting single metrics, this framework combines realistic compute constraints, public execution trajectories, comprehensive benchmark suites, and blind human preference feedback in a multi-stage improvement loop.
Results
All ten agents produced functioning assistants, with the GPT-6 Astra-trained model achieving the highest Stage 1 performance by matching NVIDIA Instruct in style-controlled human ratings and reaching 76% median benchmark progress toward it. Following Stage 2 training on human feedback, eight models improved their style-controlled standing relative to NVIDIA Instruct, led by the DeepSeek-trained model's rating increase and a top benchmark progress of 87%. However, changes in human preferences were weakly correlated with benchmark score changes, and some models exhibited increased harmful compliance on StrongREJECT.
Key Points
- RSI Arena delegates end-to-end post-training of a 30B-parameter base model to ten frontier-model agents under matched budgets, shared cluster infrastructure, and logged trajectories.
- Human feedback in Stage 2 improved the style-controlled preference ratings for eight out of ten agents relative to NVIDIA Instruct, but these improvements did not correspond to large or consistent benchmark gains.
- Safety did not improve automatically alongside human preference, as certain models regressed on harmful-compliance metrics when safety was not explicitly targeted.