Learning from Less: Measuring the Effectiveness of RLVR in Low Data and Compute Regimes

• URL: http://arxiv.org/abs/2604.18381v1
• Date: Mon, 20 Apr 2026 15:04:57 GMT
• Title: Learning from Less: Measuring the Effectiveness of RLVR in Low Data and Compute Regimes
• Abstract summary: Fine-tuning Large Language Models (LLMs) typically relies on large quantities of high-quality annotated data, or questions with well-defined ground truth answers.<n>Previous work has explored the benefits to model reasoning capabilities by scaling both data and compute used for Reinforcement Learning with Verifiable Rewards (RLVR)<n>We present a comprehensive empirical study of open-source Small Language Model (SLM) performance after RLVR in low data regimes.
• Score: 18.00712219143378
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Fine-tuning Large Language Models (LLMs) typically relies on large quantities of high-quality annotated data, or questions with well-defined ground truth answers in the case of Reinforcement Learning with Verifiable Rewards (RLVR). While previous work has explored the benefits to model reasoning capabilities by scaling both data and compute used for RLVR, these results lack applicability in many real-world settings where annotated data and accessible compute may be scarce. In this work, we present a comprehensive empirical study of open-source Small Language Model (SLM) performance after RLVR in low data regimes. Across three novel datasets covering number counting problems, graph reasoning, and spatial reasoning, we characterize how model performance scales with dataset size, diversity, and complexity. We demonstrate that (1) procedural datasets allow for fine-grained evaluation and training dataset development with controllable properties (size, diversity, and complexity), (2) under RLVR, models trained on lower complexity tasks can generalize to higher complexity tasks, and (3) training on mixed complexity datasets is associated with the greatest benefits in low data regimes, providing up to 5x sample efficiency versus training on easy tasks. These findings inspire future work on the development of data scaling laws for RLVR and the use of procedural data generators to further understand effective data development for efficient LLM fine-tuning.

Related papers

• Guiding LLM Post-training Data Engineering with Model Internals from Sparse Autoencoders [49.87696007629591]
  Post-training data engineering largely relies on external signals and ignores rich intrinsic signals lying in model internals.<n>We propose SAERL, a data engineering framework for reinforcement learning (RL)<n>It models three intrinsic data properties: diversity, difficulty, and quality, using model internals extracted with Sparse Autoencoder (SAE), an advanced mechanistic interpretability tool.
  arXiv  Detail & Related papers  (2026-05-26T17:55:59Z)
• Efficient Reinforcement Learning for Large Language Models with Intrinsic Exploration [33.02780998281276]
  Reinforcement learning with verifiable rewards (RLVR) has improved the reasoning ability of large language models.<n>This study investigates how simply leveraging intrinsic data properties, almost free benefit during training, can improve data efficiency for RLVR.
  arXiv  Detail & Related papers  (2025-11-02T04:16:47Z)
• Efficient Long-Tail Learning in Latent Space by sampling Synthetic Data [1.9290392443571385]
  Imbalanced classification datasets pose significant challenges in machine learning.<n>We propose a novel framework that leverages the rich semantic latent space of Vision Foundation Models to generate synthetic data and train a simple linear classifier.<n>Our method sets a new state-of-the-art for the CIFAR-100-LT benchmark and demonstrates strong performance on the Places-LT benchmark.
  arXiv  Detail & Related papers  (2025-09-19T10:52:31Z)
• SPaRFT: Self-Paced Reinforcement Fine-Tuning for Large Language Models [51.74498855100541]
  Large language models (LLMs) have shown strong reasoning capabilities when fine-tuned with reinforcement learning (RL)<n>We propose textbfSPaRFT, a self-paced learning framework that enables efficient learning based on the capability of the model being trained.
  arXiv  Detail & Related papers  (2025-08-07T03:50:48Z)
• Large Language Models and Synthetic Data for Monitoring Dataset Mentions in Research Papers [0.0]
  This paper presents a machine learning framework that automates dataset mention detection across research domains.<n>We employ zero-shot extraction from research papers, an LLM-as-a-Judge for quality assessment, and a reasoning agent for refinement to generate a weakly supervised synthetic dataset.<n>At inference, a ModernBERT-based classifier efficiently filters dataset mentions, reducing computational overhead while maintaining high recall.
  arXiv  Detail & Related papers  (2025-02-14T16:16:02Z)
• Learning with Less: Knowledge Distillation from Large Language Models via Unlabeled Data [54.934578742209716]
  In real-world NLP applications, Large Language Models (LLMs) offer promising solutions due to their extensive training on vast datasets.<n>LLKD is an adaptive sample selection method that incorporates signals from both the teacher and student.<n>Our comprehensive experiments show that LLKD achieves superior performance across various datasets with higher data efficiency.
  arXiv  Detail & Related papers  (2024-11-12T18:57:59Z)
• Forewarned is Forearmed: Leveraging LLMs for Data Synthesis through Failure-Inducing Exploration [90.41908331897639]
  Large language models (LLMs) have significantly benefited from training on diverse, high-quality task-specific data. We present a novel approach, ReverseGen, designed to automatically generate effective training samples.
  arXiv  Detail & Related papers  (2024-10-22T06:43:28Z)
• Generative Expansion of Small Datasets: An Expansive Graph Approach [13.053285552524052]
  We introduce an Expansive Synthesis model generating large-scale, information-rich datasets from minimal samples. An autoencoder with self-attention layers and optimal transport refines distributional consistency. Results show comparable performance, demonstrating the model's potential to augment training data effectively.
  arXiv  Detail & Related papers  (2024-06-25T02:59:02Z)
• AvaTaR: Optimizing LLM Agents for Tool Usage via Contrastive Reasoning [93.96463520716759]
  Large language model (LLM) agents have demonstrated impressive capabilities in utilizing external tools and knowledge to boost accuracy and hallucinations. Here, we introduce AvaTaR, a novel and automated framework that optimize an LLM agent to effectively leverage provided tools, improving performance on a given task.
  arXiv  Detail & Related papers  (2024-06-17T04:20:02Z)
• SHED: Shapley-Based Automated Dataset Refinement for Instruction Fine-Tuning [16.307467144690683]
  Large Language Models can achieve desirable performance with only a small amount of high-quality data. Identifying high-quality data from vast datasets to curate small yet effective datasets has emerged as a critical challenge. We introduce SHED, an automated dataset refinement framework based on Shapley value for instruction fine-tuning.
  arXiv  Detail & Related papers  (2024-04-23T04:56:48Z)
• Curated LLM: Synergy of LLMs and Data Curation for tabular augmentation in low-data regimes [57.62036621319563]
  We introduce CLLM, which leverages the prior knowledge of Large Language Models (LLMs) for data augmentation in the low-data regime. We demonstrate the superior performance of CLLM in the low-data regime compared to conventional generators.
  arXiv  Detail & Related papers  (2023-12-19T12:34:46Z)
• Dynamics of Instruction Fine-Tuning for Chinese Large Language Models [19.832906541004114]
  We systematically investigate the effects of data quantity, model size, and data construction methods on instruction tuning for Chinese Large Language Models.<n>Our experiments, conducted on models ranging from 7b to 33b parameters, yield three key findings.
  arXiv  Detail & Related papers  (2023-10-30T15:37:10Z)
• STAR: Boosting Low-Resource Information Extraction by Structure-to-Text Data Generation with Large Language Models [56.27786433792638]
  STAR is a data generation method that leverages Large Language Models (LLMs) to synthesize data instances. We design fine-grained step-by-step instructions to obtain the initial data instances. Our experiments show that the data generated by STAR significantly improve the performance of low-resource event extraction and relation extraction tasks.
  arXiv  Detail & Related papers  (2023-05-24T12:15:19Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.