LoopRPT: Reinforcement Pre-Training for Looped Language Models

• URL: http://arxiv.org/abs/2603.19714v1
• Date: Fri, 20 Mar 2026 07:35:38 GMT
• Title: LoopRPT: Reinforcement Pre-Training for Looped Language Models
• Abstract summary: LoopRPT is a reinforcement pre-training framework tailored for LoopLMs.<n>We instantiate LoopRPT on the Ouro architecture across multiple model scales.<n>Results demonstrate that LoopRPT consistently improves per-step representation quality.
• Score: 40.95348011001987
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Looped language models (LoopLMs) perform iterative latent computation to refine internal representations, offering a promising alternative to explicit chain-of-thought (CoT) reasoning. However, existing reinforcement learning (RL) paradigms primarily target output tokens, creating a structural mismatch with looped architectures whose reasoning unfolds implicitly. In this work, we propose LoopRPT, a reinforcement pre-training framework tailored for LoopLMs. By reframing next-token prediction as a next-token reasoning task, LoopRPT assigns reinforcement signals directly to latent steps using an EMA teacher reference and noisy latent rollouts. This formulation enables RL to directly shape intermediate representations, compressing effective reasoning into fewer iterations. We instantiate LoopRPT on the Ouro architecture across multiple model scales. Results demonstrate that LoopRPT consistently improves per-step representation quality, achieving Pareto dominance in accuracy-computation trade-offs. Notably, significant gains on hard tokens indicate that LoopRPT enhances early-stage reasoning rather than merely encouraging premature exits. Our findings highlight reinforcement pre-training as a principled paradigm for learning efficient latent reasoning in LoopLMs.

Related papers

• Think Deep, Speak Once: Relit, A Recursive Latent Implicit Transformer Framework [2.007262412327553]
  We introduce ReLIT, a hybrid framework for deep, semantically grounded implicit reasoning.<n>We show that ReLIT can be scaled efficiently through recurrent depth rather than parameter width.<n>These results demonstrate that reasoning capability can be scaled efficiently through recurrent depth rather than parameter width.
  arXiv  Detail & Related papers  (2026-08-08T12:50:51Z)
• To Reason or to Fabricate: Reasoning Without Shortcuts via Hint-Anchored Pairwise Aggregation [20.92880098291496]
  HIPPO is a novel RL framework that integrates hint-injected aggregation with a tailored pairwise reward model.<n>It provides highly discriminable preference signals, enabling a lightweight judge model to reliably distinguish genuine reasoning deduction.
  arXiv  Detail & Related papers  (2026-06-28T16:21:04Z)
• LoopUS: Recasting Pretrained LLMs into Looped Latent Refinement Models [35.78769938627291]
  We introduce textbfLooped Depth Up-Scaling (LoopUS), a post-training framework that converts a standard pretrained LLM into a looped architecture.<n>LoopUS recasts the pretrained LLM into an encoder, a looped reasoning block, and a decoder.<n>These mechanisms transform a standard non-looped model into a looped form while stabilizing it against both computational bottlenecks and representation collapse.
  arXiv  Detail & Related papers  (2026-05-10T11:05:20Z)
• DiffuReason: Bridging Latent Reasoning and Generative Refinement for Sequential Recommendation [20.756497463882763]
  We propose DiffuReason, a unified "Think-then-Diffuse" framework for sequential recommendation.<n>It integrates multi-step Thinking Tokens for latent reasoning, diffusion-based refinement for denoising intermediate representations, and end-to-end Group Relative Policy Optimization.<n>Experiments on four benchmarks demonstrate that DiffuReason consistently improves diverse backbone architectures.
  arXiv  Detail & Related papers  (2026-02-10T12:55:30Z)
• Well Begun, Half Done: Reinforcement Learning with Prefix Optimization for LLM Reasoning [18.477056077256233]
  Reinforcement Learning with Verifiable Rewards (RLVR) significantly enhances the reasoning capability of Large Language Models (LLMs)<n>Current RLVR approaches typically conduct training across all generated tokens, but neglect to explore which tokens (e.g., prefix tokens) actually contribute to reasoning.<n>We propose a novel RLVR approach called Progressive Prefix-token Policy Optimization (PPPO), which highlights the significance of the prefix segment of generated outputs.
  arXiv  Detail & Related papers  (2025-12-17T10:26:11Z)
• Principled RL for Diffusion LLMs Emerges from a Sequence-Level Perspective [85.06838178922791]
  Reinforcement Learning (RL) has proven highly effective for autoregressive language models.<n>But adapting these methods to diffusion large language models (dLLMs) presents fundamental challenges.<n>We propose a principled RL framework that treats entire sequence generation as a single action and uses the ELBO as a tractable sequence-level likelihood proxy.
  arXiv  Detail & Related papers  (2025-12-03T13:05:32Z)
• MRO: Enhancing Reasoning in Diffusion Language Models via Multi-Reward Optimization [66.82303841930752]
  diffusion language models (DLMs) have presented a promising alternative to traditional autoregressive large language models (LLMs)<n>DLMs still lag behind LLMs in reasoning performance, especially as the number of denoising steps decreases.<n>We propose a Multi-Reward Optimization (MRO) approach, which encourages DLMs to consider the token correlation during the denoising process.
  arXiv  Detail & Related papers  (2025-10-24T13:57:59Z)
• LaSeR: Reinforcement Learning with Last-Token Self-Rewarding [54.72617309922891]
  Reinforcement Learning with Verifiable Rewards (RLVR) has emerged as a core paradigm for enhancing the reasoning capabilities of Large Language Models (LLMs)<n>Previous practice requires the LLM to sequentially generate solutions and self-verifications using two separate prompt templates, which significantly reduces efficiency.<n>We propose LaSeR (Reinforcement Learning with Last-Token Self-Rewarding), an algorithm that simply augments the original RLVR loss with a MSE loss.
  arXiv  Detail & Related papers  (2025-10-16T17:55:11Z)
• Revisiting LLM Reasoning via Information Bottleneck [57.519119962528166]
  Large language models (LLMs) have recently demonstrated remarkable progress in reasoning capabilities through reinforcement learning with verifiable rewards (RLVR)<n>We present a theoretical characterization of LLM reasoning grounded in information bottleneck (IB) principle.<n>We propose IB-aware reasoning optimization (IBRO), a framework that encourages reasoning trajectories to be both informative about the final correct answer and generalizable.
  arXiv  Detail & Related papers  (2025-07-24T13:14:25Z)
• SCOUT: Teaching Pre-trained Language Models to Enhance Reasoning via Flow Chain-of-Thought [37.53215651690168]
  Chain of Thought (CoT) prompting improves the reasoning performance of large language models (LLMs) by encouraging step by step thinking.<n>While promising, CoT-based approaches often require costly pretraining and lack a principled framework for how reasoning should evolve.<n>We propose SCOUT, a lightweight fine tuning framework that enables Flow CoT style reasoning without the need for pretraining.
  arXiv  Detail & Related papers  (2025-05-30T03:43:24Z)
• Walk Before You Run! Concise LLM Reasoning via Reinforcement Learning [10.255235456427037]
  We propose a simple yet effective two-stage reinforcement learning framework for achieving concise reasoning in Large Language Models (LLMs)<n>The first stage, using more training steps, aims to incentivize the model's reasoning capabilities via Group Relative Policy Optimization.<n>The second stage, using fewer training steps, explicitly enforces conciseness and improves efficiency via Length-aware Group Relative Policy Optimization.
  arXiv  Detail & Related papers  (2025-05-27T13:29:51Z)
• Reinforced Latent Reasoning for LLM-based Recommendation [92.56166822197919]
  Large Language Models (LLMs) have demonstrated impressive reasoning capabilities in complex problem-solving tasks.<n>Existing methods typically rely on fine-tuning with explicit chain-of-thought (CoT) data.<n>In this work, we explore an alternative approach that shifts from explicit CoT reasoning to compact, information-dense latent reasoning.
  arXiv  Detail & Related papers  (2025-05-25T11:03:45Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.