Guiding LLM Post-training Data Engineering with Model Internals from Sparse Autoencoders

• URL: http://arxiv.org/abs/2605.27354v1
• Date: Tue, 26 May 2026 17:55:59 GMT
• Title: Guiding LLM Post-training Data Engineering with Model Internals from Sparse Autoencoders
• Abstract summary: Post-training data engineering largely relies on external signals and ignores rich intrinsic signals lying in model internals.<n>We propose SAERL, a data engineering framework for reinforcement learning (RL)<n>It models three intrinsic data properties: diversity, difficulty, and quality, using model internals extracted with Sparse Autoencoder (SAE), an advanced mechanistic interpretability tool.
• Score: 49.87696007629591
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Model internals encode rich information about how a large language model (LLM) processes its training data; however, post-training data engineering largely relies on external signals and ignores rich intrinsic signals lying in model internals. We propose SAERL, a data engineering framework for LLM reinforcement learning (RL). It models three intrinsic data properties: diversity, difficulty, and quality, using model internals extracted with Sparse Autoencoder (SAE), an advanced mechanistic interpretability tool. Each property grounds a concrete data engineering operation: SAE-space clustering with moderate batch mixing for batch diversity control, a difficulty proxy for easy-to-hard curriculum ordering, and a quality probe for data filtering. SAERL improves average accuracy by 3.00% over vanilla GRPO and reaches target accuracy with 20% fewer training steps on Qwen2.5-Math-1.5B, with consistent gains across model scales and RL algorithms. Experiments show that SAE transfers effectively across model families and scales, serving as a lightweight and reusable data engineering tool. These results demonstrate that model internals are a powerful and practical source of signals for post-training data engineering.

Related papers

• Exploring Autonomous Agentic Data Engineering for Model Specialization [37.35310287793494]
  Large Language Models (LLMs) have demonstrated strong performance on general tasks, while often struggling to adapt to specialized domains.<n>We formalize textbfAutonomous Agentic Data Engineering, a novel task designed to evaluate LLMs as autonomous data engineers.<n> Experiments show that autonomous LLM data engineers yield substantial gains, as GPT-5.2 constructs a training curriculum that improves a student model by textbf57.29%.
  arXiv  Detail & Related papers  (2026-05-28T17:50:10Z)
• Learning from Less: Measuring the Effectiveness of RLVR in Low Data and Compute Regimes [18.00712219143378]
  Fine-tuning Large Language Models (LLMs) typically relies on large quantities of high-quality annotated data, or questions with well-defined ground truth answers.<n>Previous work has explored the benefits to model reasoning capabilities by scaling both data and compute used for Reinforcement Learning with Verifiable Rewards (RLVR)<n>We present a comprehensive empirical study of open-source Small Language Model (SLM) performance after RLVR in low data regimes.
  arXiv  Detail & Related papers  (2026-04-20T15:04:57Z)
• DataFlex: A Unified Framework for Data-Centric Dynamic Training of Large Language Models [51.48564522455171]
  We present DataFlex, a unified data-centric dynamic training framework built upon LLaMA-Factory.<n>DataFlex supports three major paradigms of dynamic data optimization: sample selection, domain mixture adjustment adjustment, and data reweighting.<n>It provides trainer abstractions and modular components, enabling a drop-in replacement for standard LLM training.
  arXiv  Detail & Related papers  (2026-03-27T08:28:02Z)
• LoopTool: Closing the Data-Training Loop for Robust LLM Tool Calls [46.34510189812439]
  LoopTool is a fully automated, model-aware data evolution framework.<n>It iteratively refines both the data and the model through three synergistic modules.<n> Experiments show that our 8B model trained with LoopTool significantly surpasses its 32B data generator.
  arXiv  Detail & Related papers  (2025-11-12T09:34:39Z)
• Angles Don't Lie: Unlocking Training-Efficient RL Through the Model's Own Signals [49.17123504516502]
  CurrentReinforcement Fine-tuning (RFT) paradigms for Large Language Models (LLMs) suffer from inefficiency due to redundant exposure of identical queries under uniform data sampling.<n>We propose a Gradient-driven Angle-Informed Navigated RL framework.<n>By leveraging the model's intrinsic angle concentration signal, GAIN-RL dynamically selects training data in each epoch, ensuring consistently impactful gradient updates.
  arXiv  Detail & Related papers  (2025-06-02T21:40:38Z)
• Evaluating Language Models as Synthetic Data Generators [99.16334775127875]
  AgoraBench is a benchmark that provides standardized settings and metrics to evaluate LMs' data generation abilities.<n>Through synthesizing 1.26 million training instances using 6 LMs and training 99 student models, we uncover key insights about LMs' data generation capabilities.
  arXiv  Detail & Related papers  (2024-12-04T19:20:32Z)
• Forewarned is Forearmed: Leveraging LLMs for Data Synthesis through Failure-Inducing Exploration [90.41908331897639]
  Large language models (LLMs) have significantly benefited from training on diverse, high-quality task-specific data. We present a novel approach, ReverseGen, designed to automatically generate effective training samples.
  arXiv  Detail & Related papers  (2024-10-22T06:43:28Z)
• When More Data Hurts: Optimizing Data Coverage While Mitigating Diversity Induced Underfitting in an Ultra-Fast Machine-Learned Potential [0.0]
  This study investigates how training data diversity affects the performance of machine-learned interatomic potentials (MLIPs) We employ expert and autonomously generated data to create the training data and fit four force-field variants to subsets of the data. Our findings reveal a critical balance in training data diversity: insufficient diversity hinders generalization, while excessive diversity can exceed the MLIP's learning capacity.
  arXiv  Detail & Related papers  (2024-09-11T20:45:44Z)
• Scaling Retrieval-Based Language Models with a Trillion-Token Datastore [85.4310806466002]
  We find that increasing the size of the datastore used by a retrieval-based LM monotonically improves language modeling and several downstream tasks without obvious saturation. By plotting compute-optimal scaling curves with varied datastore, model, and pretraining data sizes, we show that using larger datastores can significantly improve model performance for the same training compute budget.
  arXiv  Detail & Related papers  (2024-07-09T08:27:27Z)
• From Quantity to Quality: Boosting LLM Performance with Self-Guided Data Selection for Instruction Tuning [52.257422715393574]
  We introduce a self-guided methodology for Large Language Models (LLMs) to autonomously discern and select cherry samples from open-source datasets. Our key innovation, the Instruction-Following Difficulty (IFD) metric, emerges as a pivotal metric to identify discrepancies between a model's expected responses and its intrinsic generation capability.
  arXiv  Detail & Related papers  (2023-08-23T09:45:29Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.