Open LLM Post-Training and Evaluation

Over the next three years, open post-training research could produce a small set of repeatable specialist workflows rather than broadly autonomous agents. TÜLU 3's documented training sequence and DR Tulu's reward experiments provide a starting point, but stronger training scores still require separate tests of practical performance. In the first year, teams would reproduce individual training stages and compare reward designs while recording how each model was produced. A meaningful early signal would be at least two independent groups reproducing stage-level gains under separate downstream evaluations, rather than merely publishing additional recipes.

If that evidence accumulates, the second year would extend the recipes into longer conversations and tool use, with other languages testing the limits of transfer. Teams would also check whether changing scoring rubrics continue to distinguish good answers from poor ones as models adapt to those rubrics. Practical work could settle around a few tested templates for specialist tasks, reducing the need to rebuild each training and evaluation pipeline. The mechanism resembles process validation: documented training stages make development inspectable, while a separate acceptance test checks whether the resulting model serves its intended purpose. More selective testing could lower evaluation costs and support additional experiments, provided it does not achieve savings by avoiding difficult tasks.

By year three, teams could add specialist skills individually, test whether earlier abilities remain intact, and retain the previous version when regressions are unacceptable. Reusing earlier training examples or attaching small specialist modules could help preserve capabilities, although neither method is established here as a guaranteed solution. Researchers would then test whether these controlled additions remain useful after accounting for the full cost of training and verification, including answer generation. The supplied weekly reports remain starting points for replication, not independently verified evidence that this transition has already occurred. Gains that disappear under a different evaluator would weaken the scenario, because documentation alone cannot establish reliability with unfamiliar users or tools.

This scenario assumes that rewards improve enough to support repeated training stages, while checking everything those stages might damage remains costly. The resulting three-year shift would make preservation of earlier capabilities a central constraint on dependable model development, rather than replacing reward quality as a concern. During the first year, researchers would measure skills after each stage and compare methods that reuse earlier examples or separate specialist functions. Engineering teams would keep a stage record showing what changed, what improved, and what regressed before accepting a new model version.

Independent reproduction of both gains and retained skills within comparable budgets would be a strong early monitoring signal that these preservation methods work. In the second year, longer interactions and unfamiliar tools would expand the abilities needing protection, while multilingual testing would reveal further gaps in coverage. The mechanism resembles a biological ratchet: each stage can improve its immediate objective while introducing a loss elsewhere that subsequent work fails to reverse. Reusing earlier examples or routing tasks to specialist components could interrupt that accumulation, but researchers would need to measure the associated repair costs. The danger emerges when new stages create hidden losses faster than available checks can detect them, and successive repairs add still more complexity.

By year three, the main evaluation question would be how to keep the full set of required skills observable within a fixed testing budget. Selective testing would need to preserve coverage of uncommon capabilities, not simply reduce repeated trials on familiar tasks while leaving whole skill categories unchecked. If preservation controls succeed, teams could maintain multi-skill models through reversible specialist additions, supported by coverage records and a reliable return to earlier versions. If they fail, a stable general model paired with separate specialist modules could become more practical than an increasingly complicated sequence of training stages. The analogy remains limited because checkpoints can be restored; cheap detection of forgotten skills or consistent recovery without special preservation methods would weaken this scenario.

Over three years, shared measurement tools could become a trusted basis for deciding which post-training results deserve reuse, rather than remaining optional reporting aids. The starting concern is that better training rewards do not consistently predict better downstream performance, even when teams openly document their training recipes. In the first year, researchers would independently reproduce stage-level gains and test whether conclusions survive changes in evaluators and computing budgets. A shared evaluation harness, meaning software that runs comparable checks and records results, could make these comparisons easier to repeat across different teams.

The key early signal would combine recurring reproduction failures with a maintained shared harness that at least one team uses to accept or reject delivery. If that threshold is crossed, second-year testing would expand into longer conversations and unfamiliar tools, while other languages would expose limits in recipe transfer. Engineering teams would introduce release checks that block a new training stage when it erases earlier abilities required for the intended task. The reinforcing mechanism is straightforward: comparable tests expose weak recipes, teams favor stronger external evidence, and shared checks gain influence over subsequent research. Like common reporting definitions in public health, these instruments could make scattered observations comparable without implying that any single test establishes reliability everywhere.

By year three, continued reuse could favor a small number of compatible evaluation systems that let teams adapt established workflows without reconstructing every check. Their maintainers would increasingly help define acceptable evaluation uncertainty and capability loss, while making the resources needed to demonstrate performance more visible. Researchers would still need to test whether these instruments measure the intended abilities, including source support in research reports and successful completion of unfamiliar tool interactions. A major caveat is that familiar checks can become optimization targets, allowing models to satisfy scoring rules without genuinely improving the underlying task. Independent evaluators and refreshed tasks could help, but unmaintained repositories or continued preference for inexpensive custom checks would weaken the proposed shift toward shared verification.