Efficient Post-training of LLMs for Code Generation With Offline Reinforcement Learning

• URL: http://arxiv.org/abs/2605.28409v1
• Date: Wed, 27 May 2026 12:43:14 GMT
• Title: Efficient Post-training of LLMs for Code Generation With Offline Reinforcement Learning
• Abstract summary: Post-training using online reinforcement learning (RL) is an important training step for LLMs, including code-generating models.<n>Online RL for code generation involves inference and verification of the generated output, which can take considerable time and resources.<n>In this paper, we explore the application of offline RL to code-generating models by leveraging existing code datasets.
• Score: 5.1981024469718315
• License: http://creativecommons.org/licenses/by-sa/4.0/
• Abstract: Post-training using online reinforcement learning (RL) is an important training step for LLMs, including code-generating models. However, online RL for code generation involves LLM inference and verification of the generated output, which can take considerable time and resources. In this paper, we explore the application of offline RL to code-generating models by leveraging existing code datasets. Our experiments demonstrate that offline RL is an effective training strategy for improving LLM performance. We show that offline RL can be especially beneficial for small LLMs and challenging coding problems.

Related papers

• Memory-Based Advantage Shaping for LLM-Guided Reinforcement Learning [18.215893951726166]
  In environments with sparse or delayed rewards, reinforcement learning incurs high sample complexity.<n>This limitation has motivated the use of large language models (LLMs) for subgoal discovery and trajectory guidance.<n>We address these challenges by constructing a memory graph that encodes subgoals and trajectories from both LLM guidance and the agent's own successful rollouts.
  arXiv  Detail & Related papers  (2026-02-20T01:44:35Z)
• Bridging Online and Offline RL: Contextual Bandit Learning for Multi-Turn Code Generation [60.14439536069839]
  Multi-turn code generation can be formulated as a one-step recoverable Markov decision process.<n>Cobalt is a new method that combines the benefits of online and offline RL.<n>Our results demonstrate Cobalt as a promising solution for iterative decision-making tasks like code generation.
  arXiv  Detail & Related papers  (2026-02-03T18:08:41Z)
• Reinforcement Learning Meets Large Language Models: A Survey of Advancements and Applications Across the LLM Lifecycle [66.80133103857703]
  Reinforcement Learning (RL) has markedly enhanced the reasoning and alignment performance of Large Language Models (LLMs)<n>This survey aims to present researchers and practitioners with the latest developments and frontier trends at the intersection of RL and LLMs.
  arXiv  Detail & Related papers  (2025-09-20T13:11:28Z)
• Large Language Model-enhanced Reinforcement Learning for Low-Altitude Economy Networking [71.83640290222928]
  Low-Altitude Economic Networking (LAENet) aims to support diverse flying applications below 1,000 meters.<n>Complex decision-making, resource constraints, and environmental uncertainty pose significant challenges to the development of the LAENet.
  arXiv  Detail & Related papers  (2025-05-27T11:25:42Z)
• SeRL: Self-Play Reinforcement Learning for Large Language Models with Limited Data [65.56911325914582]
  We propose Self-play Reinforcement Learning (SeRL) to bootstrap Large Language Models (LLMs) training with limited initial data.<n>The proposed SeRL yields results superior to its counterparts and achieves performance on par with those obtained by high-quality data with verifiable rewards.
  arXiv  Detail & Related papers  (2025-05-25T13:28:04Z)
• Toward Efficient Exploration by Large Language Model Agents [14.712532175418884]
  Large language models (LLMs) can be used to explicitly implement an existing reinforcement learning algorithm.<n>We show how our LLM-based implementation of a known, data-efficient RL algorithm can be considerably more effective in natural language tasks.
  arXiv  Detail & Related papers  (2025-04-29T17:59:48Z)
• Robust RL with LLM-Driven Data Synthesis and Policy Adaptation for Autonomous Driving [41.87011820577736]
  This paper introduces RAPID, a novel framework for training mix-of-policy Reinforcement Learning agents. It trains specialized mix-of-policy RL agents using data synthesized by an LLM-based driving agent and online adaptation. It reduces the robustness of LLM knowledge while maintaining adaptability to different tasks.
  arXiv  Detail & Related papers  (2024-10-16T13:43:00Z)
• When Linear Attention Meets Autoregressive Decoding: Towards More Effective and Efficient Linearized Large Language Models [13.061946833851605]
  We conduct the first comprehensive study on the efficacy of existing linear attention methods for autoregressive LLMs. We introduce an augmentation technique for linear attention that ensures compatibility with speculative decoding. Our approach achieves up to a 6.67 reduction in perplexity on the LLaMA model and up to a 2$times$ speedup during generation compared to prior linear attention methods.
  arXiv  Detail & Related papers  (2024-06-11T15:34:43Z)
• ArCHer: Training Language Model Agents via Hierarchical Multi-Turn RL [80.10358123795946]
  We develop a framework for building multi-turn RL algorithms for fine-tuning large language models. Our framework adopts a hierarchical RL approach and runs two RL algorithms in parallel. Empirically, we find that ArCHer significantly improves efficiency and performance on agent tasks.
  arXiv  Detail & Related papers  (2024-02-29T18:45:56Z)
• Reflect-RL: Two-Player Online RL Fine-Tuning for LMs [38.5495318990769]
  We propose Reflect-RL, a system to fine-tune language models (LMs) using online reinforcement learning (RL) and supervised fine-tuning (SFT) Test results indicate GPT-2 XL 1.56B fine-tuned with Reflect-RL outperforms larger open-source LMs, such as Mistral 7B.
  arXiv  Detail & Related papers  (2024-02-20T01:04:21Z)
• StepCoder: Improve Code Generation with Reinforcement Learning from Compiler Feedback [58.20547418182074]
  We introduce StepCoder, a novel framework for code generation, consisting of two main components. CCCS addresses the exploration challenge by breaking the long sequences code generation task into a Curriculum of Code Completion Subtasks. FGO only optimize the model by masking the unexecuted code segments to provide Fine-Grained Optimization. Our method improves the ability to explore the output space and outperforms state-of-the-art approaches in corresponding benchmarks.
  arXiv  Detail & Related papers  (2024-02-02T13:14:31Z)
• Learning to Generate Better Than Your LLM [16.74454360961681]
  Reinforcement learning has emerged as a powerful paradigm for fine-tuning Large Language Models. We extend RL algorithms to allow them to interact with a dynamic black-box guide LLM. We show that our RL algorithms achieve higher performance than supervised learning.
  arXiv  Detail & Related papers  (2023-06-20T18:19:17Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.