Beyond Solvability: Task Learnability as a Static Prior for LLM RL Post-Training

• URL: http://arxiv.org/abs/2608.09217v1
• Date: Mon, 10 Aug 2026 07:43:05 GMT
• Title: Beyond Solvability: Task Learnability as a Static Prior for LLM RL Post-Training
• Abstract summary: Reinforcement learning (RL) has become a central post-training paradigm for eliciting reasoning capabilities in large language models.<n>We study this residual axis as task learnability: a regime-conditional measure of expected positive response to continued training.<n>We propose TrajVal, a lightweight probe-based estimator that approximates per-task learnability from a short probe run and two endpoint evaluations.
• Score: 52.78248352851176
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Reinforcement learning (RL) has become a central post-training paradigm for eliciting reasoning capabilities in large language models, yet uniform task sampling allocates compute without regard to differences in how tasks respond to optimization. Existing task-valuation methods mostly rely on snapshot-based signals such as current pass rate or reward, which estimate how solvable a task is under the current policy. However, tasks with similar current solvability can still differ substantially in how positively they respond to further training. We study this residual axis as task learnability: a regime-conditional measure of expected positive response to continued training under a fixed RL post-training regime. By analyzing per-task reward trajectories, we find that learnability is reproducible across independently sampled training contexts and predictive of downstream utility. To make this signal practical before training begins, we propose TrajVal, a lightweight probe-based estimator that approximates per-task learnability from a short probe run and two endpoint evaluations. TrajVal can be used either as a standalone static prior for task sampling or as a multiplicative prior for existing online schedulers. Experiments on mathematical and logical reasoning benchmarks across multiple model scales show that TrajVal improves data efficiency over uniform sampling and provides complementary gains when combined with online scheduling methods.

Related papers

• Test-time RL alignment exposes task familiarity artifacts in LLM benchmarks [20.988724384459392]
  The train-before-test approach controls for task familiarity by giving each model task-relevant training before evaluation.<n>We propose a two-stage test-time reinforcement learning (RL) alignment method for train-before-test.
  arXiv  Detail & Related papers  (2026-03-13T10:24:19Z)
• Zero-Shot Off-Policy Learning [9.729890516322781]
  Off-policy learning methods seek to derive an optimal policy directly from a fixed dataset of prior interactions.<n>In this work, we address the off-policy problem in a zero-shot setting by discovering a theoretical connection of successor measures to stationary density ratios.<n>Our algorithm can infer optimal importance sampling ratios, effectively performing a stationary distribution correction with an optimal policy for any task on the fly.
  arXiv  Detail & Related papers  (2026-02-02T11:06:31Z)
• BOTS: A Unified Framework for Bayesian Online Task Selection in LLM Reinforcement Finetuning [82.925106913459]
  Reinforcement finetuning (RFT) is a key technique for aligning Large Language Models (LLMs) with human preferences and enhancing reasoning.<n>We introduce BOTS, a unified framework for Bayesian Online Task Selection in RFT reinforcement finetuning.
  arXiv  Detail & Related papers  (2025-10-30T11:15:23Z)
• Investigating task-specific prompts and sparse autoencoders for activation monitoring [0.0]
  Internal activations of language models encode additional information that could be useful for this.<n>Recent work has proposed several approaches which may improve on naive linear probing.<n>We develop and test novel refinements of these methods and compare them against each other.
  arXiv  Detail & Related papers  (2025-04-28T21:28:17Z)
• Exploring RL-based LLM Training for Formal Language Tasks with Programmed Rewards [49.7719149179179]
  This paper investigates the feasibility of using PPO for reinforcement learning (RL) from explicitly programmed reward signals. We focus on tasks expressed through formal languages, such as programming, where explicit reward functions can be programmed to automatically assess quality of generated outputs. Our results show that pure RL-based training for the two formal language tasks is challenging, with success being limited even for the simple arithmetic task.
  arXiv  Detail & Related papers  (2024-10-22T15:59:58Z)
• Model-Based Transfer Learning for Contextual Reinforcement Learning [5.5597941107270215]
  We introduce Model-Based Transfer Learning to solve contextual RL problems.<n>We show theoretically that the method exhibits sublinear regret in the number of training tasks.<n>We experimentally validate our methods using urban traffic and standard continuous control benchmarks.
  arXiv  Detail & Related papers  (2024-08-08T14:46:01Z)
• How Many Pretraining Tasks Are Needed for In-Context Learning of Linear Regression? [92.90857135952231]
  Transformers pretrained on diverse tasks exhibit remarkable in-context learning (ICL) capabilities. We study ICL in one of its simplest setups: pretraining a linearly parameterized single-layer linear attention model for linear regression.
  arXiv  Detail & Related papers  (2023-10-12T15:01:43Z)
• Supervised Pretraining Can Learn In-Context Reinforcement Learning [96.62869749926415]
  In this paper, we study the in-context learning capabilities of transformers in decision-making problems. We introduce and study Decision-Pretrained Transformer (DPT), a supervised pretraining method where the transformer predicts an optimal action. We find that the pretrained transformer can be used to solve a range of RL problems in-context, exhibiting both exploration online and conservatism offline.
  arXiv  Detail & Related papers  (2023-06-26T17:58:50Z)
• Task-Customized Self-Supervised Pre-training with Scalable Dynamic Routing [76.78772372631623]
  A common practice for self-supervised pre-training is to use as much data as possible. For a specific downstream task, however, involving irrelevant data in pre-training may degenerate the downstream performance. It is burdensome and infeasible to use different downstream-task-customized datasets in pre-training for different tasks.
  arXiv  Detail & Related papers  (2022-05-26T10:49:43Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.