Low-rank Optimization Trajectories Modeling for LLM RLVR Acceleration

• URL: http://arxiv.org/abs/2604.11446v1
• Date: Mon, 13 Apr 2026 13:28:12 GMT
• Title: Low-rank Optimization Trajectories Modeling for LLM RLVR Acceleration
• Abstract summary: scaling reinforcement learning with verifiable rewards (RLVR) for large language models (LLMs) has emerged as an effective training paradigm.<n>We propose the textbfNonlinear textbfExtrapolation of low-rank trajectories (textbfNExt)<n>Our method reduces computational overhead by approximately 37.5% while remaining compatible with a wide range of RLVR algorithms and tasks.
• Score: 88.7332439088792
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Recently, scaling reinforcement learning with verifiable rewards (RLVR) for large language models (LLMs) has emerged as an effective training paradigm for significantly improving model capabilities, which requires guiding the model to perform extensive exploration and learning, leading to substantial computational overhead and becoming a key challenge. To reduce the number of training steps, Prior work performs linear extrapolation of model parameters. However, the dynamics of model parameter updates during RLVR training remain insufficiently understood. To further investigate the evolution of LLMs during RLVR training, we conduct empirical experiments and find that the rank-1 subspace of the model does not evolve linearly, and its dominance over the original parameters is further amplified during LoRA training. Based on the above insights, we propose the \textbf{N}onlinear \textbf{Ext}rapolation of low-rank trajectories (\textbf{NExt}), a novel framework that models and extrapolates low-rank parameter trajectories in a nonlinear manner. Concretely, we first train the model using LoRA and extract the rank-1 subspace of parameter differences at multiple training steps, which is then used for the subsequent nonlinear extrapolation. Afterward, we utilized the extracted rank-1 subspace to train a predictor, which can model the trajectory of parameter updates during RLVR, and then perform the predict-extend process to extrapolate model parameters, achieving the acceleration of RLVR. To further study and understand NExt, we conduct comprehensive experiments that demonstrate the effectiveness and robustness of the method. Our method reduces computational overhead by approximately 37.5\% while remaining compatible with a wide range of RLVR algorithms and tasks. We release our code in https://github.com/RUCAIBox/NExt.

Related papers

• You Only Need Minimal RLVR Training: Extrapolating LLMs via Rank-1 Trajectories [23.542887618146988]
  We show thatReinforcement learning with verifiable rewards (RLVR) is extremely low-rank and highly predictable.<n>We propose a simple and compute-efficient method RELEX, which estimates the rank-1 subspace from a short observation window.<n>Remarkably, RELEX is able to extrapolate far beyond the observation window at no training cost.
  arXiv  Detail & Related papers  (2026-05-20T17:53:20Z)
• On the Implicit Reward Overfitting and the Low-rank Dynamics in RLVR [51.935533482549545]
  We show that RLVR may exhibit implicit reward overfitting to the training dataset.<n>The model can achieve satisfactory performance on the test set even when its rewards remain relatively low during the training process.
  arXiv  Detail & Related papers  (2026-05-07T16:30:28Z)
• Not All Steps are Informative: On the Linearity of LLMs' RLVR Training [14.59942263367421]
  Reinforcement learning with verifiable rewards (RLVR) has become a central component of large language model (LLM) post-training.<n>We investigate whether future model states can be predicted from intermediate checkpoints via extrapolation, avoiding continued expensive training.<n>We show that Weight Extrapolation produces models with performance comparable to standard RL training while requiring significantly less computation.
  arXiv  Detail & Related papers  (2026-01-08T03:06:18Z)
• Recover-to-Forget: Gradient Reconstruction from LoRA for Efficient LLM Unlearning [17.898277374771254]
  We introduce Recover-to-Forget (R2F), a novel framework for efficient unlearning in large foundation models.<n>R2F reconstructs full-model gradient directions from low-rank LoRA adapter updates.<n>We show that R2F offers a scalable and lightweight alternative for unlearning in pretrained LLMs without requiring full retraining or access to internal parameters.
  arXiv  Detail & Related papers  (2025-12-08T10:10:12Z)
• The Path Not Taken: RLVR Provably Learns Off the Principals [85.41043469428365]
  We show that sparsity is a surface artifact of a model-conditioned optimization bias.<n>We mechanistically explain these dynamics with a Three-Gate Theory.<n>We provide a parameter-level characterization of RLVR's learning dynamics.
  arXiv  Detail & Related papers  (2025-11-11T18:49:45Z)
• On Predictability of Reinforcement Learning Dynamics for Large Language Models [20.320268628019047]
  This work identifies two fundamental properties of RL-induced parameter updates in large language models.<n>We propose AlphaRL, a plug-in acceleration framework that extrapolates the final parameter update using a short early training window.
  arXiv  Detail & Related papers  (2025-10-01T06:13:50Z)
• SimpleVLA-RL: Scaling VLA Training via Reinforcement Learning [81.7764584515496]
  Vision-Language-Action (VLA) models have emerged as a powerful paradigm for robotic manipulation.<n>These models face two fundamental challenges: scarcity and high cost of large-scale human-operated robotic trajectories.<n>We introduce SimpleVLA-RL, an efficient reinforcement learning framework tailored for VLA models.
  arXiv  Detail & Related papers  (2025-09-11T17:59:17Z)
• Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model? [66.61292196146016]
  Reinforcement Learning with Verifiable Rewards (RLVR) has recently demonstrated notable success in enhancing the reasoning performance of large language models (LLMs)<n>This study critically examines the current state of RLVR.<n>We find that the current training setup does not elicit fundamentally new reasoning patterns.
  arXiv  Detail & Related papers  (2025-04-18T17:59:56Z)
• SaRA: High-Efficient Diffusion Model Fine-tuning with Progressive Sparse Low-Rank Adaptation [52.6922833948127]
  In this work, we investigate the importance of parameters in pre-trained diffusion models.<n>We propose a novel model fine-tuning method to make full use of these ineffective parameters.<n>Our method enhances the generative capabilities of pre-trained models in downstream applications.
  arXiv  Detail & Related papers  (2024-09-10T16:44:47Z)
• POAR: Efficient Policy Optimization via Online Abstract State Representation Learning [6.171331561029968]
  State Representation Learning (SRL) is proposed to specifically learn to encode task-relevant features from complex sensory data into low-dimensional states. We introduce a new SRL prior called domain resemblance to leverage expert demonstration to improve SRL interpretations. We empirically verify POAR to efficiently handle tasks in high dimensions and facilitate training real-life robots directly from scratch.
  arXiv  Detail & Related papers  (2021-09-17T16:52:03Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.