Introspective X Training: Feedback Conditioning Improves Scaling Across all LLM Training Stages

• URL: http://arxiv.org/abs/2605.20285v1
• Date: Tue, 19 May 2026 06:04:08 GMT
• Title: Introspective X Training: Feedback Conditioning Improves Scaling Across all LLM Training Stages
• Abstract summary: We tackle the question of how to scale more efficiently across the many, ever-growing stages of current LLM training pipelines.<n>We propose Introspective Training (or IXT), inspired by offline reward-conditioned reinforcement learning.<n>IXT uses a thinking reward model to annotate data with natural language critique based feedback, enabling quality aware training from the earliest stages of the pipeline.
• Score: 51.86479668480496
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: We tackle the question of how to scale more efficiently across the many, ever-growing stages of current LLM training pipelines. Our guiding intuition stems from the fact that the dynamics of later stages of the pipeline, e.g. post-training, can be used to inform earlier stages such as pre-training. To this end, we propose Introspective Training (or IXT), inspired by offline reward-conditioned reinforcement learning and applicable to any stage of training. IXT uses a thinking reward model to annotate data with natural language critique based feedback, enabling quality aware training from the earliest stages of the pipeline. Models are then trained by prefix-conditioning the data with the generated feedback -- ensuring that not all tokens are treated equally starting much earlier in training than usual. Comprehensive experiments on 7.5-12B transformer-based dense LLMs trained from scratch all the way up to 18 Trillion tokens seen show that our method: bends scaling curves resulting in up to 2.8x more compute efficiency generally; and reaches performance levels unachievable for models trained otherwise in domains such as math and code.

Related papers

• Post-training is (Massive) Supervised Learning [35.992105178197185]
  The prevailing paradigm for training LLMs has evolved to rely on a massive post-training phase consisting of SFT and RL.<n>We argue that this methodology effectively marks a reversion to the pre-train then fine-tune'' approach of the BERT era.
  arXiv  Detail & Related papers  (2026-04-20T08:52:09Z)
• ReMiT: RL-Guided Mid-Training for Iterative LLM Evolution [49.496216822640974]
  We analyze training dynamics and identify the mid-training phase as a critical turning point for model capabilities.<n>We introduce ReMiT (Reinforcement Learning-Guided Mid-Training), which prioritizes tokens during the mid-training phase, prioritizing those pivotal for reasoning.
  arXiv  Detail & Related papers  (2026-02-03T04:04:41Z)
• EvoLM: In Search of Lost Language Model Training Dynamics [97.69616550374579]
  EvoLM is a model suite that enables systematic and transparent analysis of LMs' training dynamics across pre-training, continued pre-training, supervised fine-tuning, and reinforcement learning.<n>By training over 100 LMs with 1B and 4B parameters from scratch, we rigorously evaluate both upstream (language modeling) and downstream (problem-solving) reasoning capabilities.
  arXiv  Detail & Related papers  (2025-06-19T04:58:47Z)
• Training Dynamics of a 1.7B LLaMa Model: A Data-Efficient Approach [10.39475177812483]
  We share insights gained from training DMaS-LLaMa-Lite on approximately 20 billion tokens of data.<n>We chronicle the full training trajectory, documenting how evolving validation loss levels and downstream benchmarks reflect transitions from incoherent text to fluent, contextually grounded output.<n>By detailing these experiences and offering training logs, checkpoints, and sample outputs, we aim to guide future researchers and practitioners in refining their pretraining strategies.
  arXiv  Detail & Related papers  (2024-12-17T21:15:52Z)
• Beyond Next Token Prediction: Patch-Level Training for Large Language Models [69.67438563485887]
  We introduce patch-level training for Large Language Models (LLMs)<n>During patch-level training, we feed the language model shorter sequences of patches and train it to predict the next patch.<n>We show that patch-level training can reduce the overall training costs to 0.5$times$, without compromising the model performance.
  arXiv  Detail & Related papers  (2024-07-17T15:48:39Z)
• An Emulator for Fine-Tuning Large Language Models using Small Language Models [91.02498576056057]
  We introduce emulated fine-tuning (EFT), a principled and practical method for sampling from a distribution that approximates the result of pre-training and fine-tuning at different scales. We show that EFT enables test-time adjustment of competing behavioral traits like helpfulness and harmlessness without additional training. Finally, a special case of emulated fine-tuning, which we call LM up-scaling, avoids resource-intensive fine-tuning of large pre-trained models by ensembling them with small fine-tuned models.
  arXiv  Detail & Related papers  (2023-10-19T17:57:16Z)
• Staged Training for Transformer Language Models [47.99321376123886]
  We consider a staged training setup that begins with a small model and incrementally increases the amount of compute used for training. By initializing each stage with the output of the previous one, the training process effectively re-uses the compute. We empirically validate our growth operators and staged training for autoregressive language models, showing up to 22% compute savings.
  arXiv  Detail & Related papers  (2022-03-11T19:05:42Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.