MLREF: Efficient Module Reuse for Reward Design in Reinforcement Learning via Large Language Models

• URL: http://arxiv.org/abs/2608.18827v1
• Date: Wed, 19 Aug 2026 11:36:55 GMT
• Title: MLREF: Efficient Module Reuse for Reward Design in Reinforcement Learning via Large Language Models
• Abstract summary: Reward function design remains a bottleneck in reinforcement learning.<n>We propose Module Level Reward Evolution Framework (MLREF)<n> MLREF treats the module pool as the primary optimization object.<n>We show that MLREF outperforms strong baselines by 25.2% in locomotion and 6.6% in manipulation.
• Score: 52.31020768687143
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Reward function design remains a bottleneck in reinforcement learning. While large language models (LLMs) have enabled automated reward generation, existing methods generate and revise reward functions as monolithic programs, making it difficult to reliably preserve and reuse effective components discovered in earlier iterations, leading to unstable performance across iterations. To address this, we propose Module Level Reward Evolution Framework (MLREF). At the core of MLREF is a module pool, a persistent repository of reusable reward components. MLREF treats the module pool as the primary optimization object: the pool evolves across iterations by accumulating successful modules, refining underperforming ones, and reusing proven components; while reward functions are constructed as linear combinations of modules drawn from this pool. To drive this evolution, MLREF integrates three mechanisms: reflection-based refinement, hybrid credit assignment, and a merge strategy with rollback, which together improve the effectiveness and robustness of reward optimization. Experiments on 17 tasks show that MLREF outperforms strong baselines by 25.2% in locomotion and 6.6% in manipulation, with more stable optimization dynamics.

Related papers

• CoRe: Combined Rewards with Vision-Language Model Feedback for Preference-Aligned Reinforcement Learning [6.874780979291594]
  CoRe is a framework that integrates Formal Rewards (FR) and Residual Rewards (RR) with vision-language models (VLMs) feedback to achieve preference-aligned policies.<n>Our contributions are twofold: (1) We propose a Formal Reward Module (FRM) that leverages VLMs to iteratively design FR based on task knowledge and preference feedback, enabling the continual improvement of policy during training; (2) We introduce a Residual Reward Module (RRM) that learns RR from video-level preference by employing VLMs to generate preference labels and capture nuanced rewards that complement FR, ensuring alignment with human intent.
  arXiv  Detail & Related papers  (2026-07-02T05:20:47Z)
• ROSUM-MCTS: Monte Carlo Tree Search-Inspired HDL Code Summarization with Structural Rewards [8.443067463539068]
  Large language models (LLMs) have shown promise in code summarization, yet their effectiveness for Hardware Description Languages (HDLs) like VHDL and Verilog remains underexplored.<n>We propose ROSUM-MCTS, an LLM-guided approach inspired by Monte Carlo Tree Search (MCTS) that refines summaries through structured exploration and reinforcement-driven optimization.
  arXiv  Detail & Related papers  (2026-06-06T01:27:20Z)
• R2IF: Aligning Reasoning with Decisions via Composite Rewards for Interpretable LLM Function Calling [7.385881045816043]
  We propose R2IF, a reasoning-aware RL framework for interpretable function calling.<n> Experiments on BFCL/ACEBench show R2IF outperforms baselines by up to 34.62%.
  arXiv  Detail & Related papers  (2026-04-22T08:13:24Z)
• VeriAgent: A Tool-Integrated Multi-Agent System with Evolving Memory for PPA-Aware RTL Code Generation [52.43796056267479]
  We propose a PPA-aware, tool-integrated multi-agent framework for high-quality verilog code generation.<n>Our framework explicitly incorporates EDA tools into a closed-loop workflow composed of a textitProgrammer Agent, a textitCorrectness Agent, and a textitPPA Agent<n>To support continuous improvement without model retraining, we introduce an textitEvolved Memory Mechanism that externalizes optimization experience into structured memory nodes.
  arXiv  Detail & Related papers  (2026-03-18T11:25:40Z)
• ReVEL: Multi-Turn Reflective LLM-Guided Heuristic Evolution via Structured Performance Feedback [0.0]
  We propose ReVEL: Multi-Turn Reflective LLM-Heuristic Evolution via Structured Performance Feedback.<n>The core of ReVEL lies in two mechanisms: (i) performance-profile grouping and (ii) multi-turn feedback-driven reflection.<n>Our results highlight multi-turn reasoning with structured grouping as a principled paradigm for automated design.
  arXiv  Detail & Related papers  (2026-03-05T04:52:11Z)
• Small Reward Models via Backward Inference [100.59075794599768]
  FLIP (FLipped Inference for Prompt Reconstruction) is a reference-free and rubric-free reward modeling approach.<n>It reformulates reward modeling through backward inference: inferring the instruction that would most plausibly produce a given response.
  arXiv  Detail & Related papers  (2026-02-14T01:55:39Z)
• FunPRM: Function-as-Step Process Reward Model with Meta Reward Correction for Code Generation [28.329829729546002]
  Code generation is a core application of large language models (LLMs)<n>We propose FunPRM to encourage code generation organized into functions.<n>FunPRM produces code that is more readable and reusable for developers.
  arXiv  Detail & Related papers  (2026-01-29T19:14:59Z)
• Round-trip Reinforcement Learning: Self-Consistent Training for Better Chemical LLMs [51.29260537017623]
  Large Language Models (LLMs) are emerging as versatile foundation models for computational chemistry.<n>These models often lack round-trip consistency.<n>We introduce Round-Trip Reinforcement Learning (RTRL), a novel framework that trains a model to improve its consistency.
  arXiv  Detail & Related papers  (2025-10-01T23:58:58Z)
• A Simple "Motivation" Can Enhance Reinforcement Finetuning of Large Reasoning Models [103.88578274567784]
  Motivation-enhanced Reinforcement Finetuning (MeRF) is an intuitive yet effective method enhancing reinforcement finetuning of Large Reasoning Models.<n>MeRF directly injects the reward specification into the prompt, which serves as an in-context motivation for the model to be aware of the optimization objective.<n>MeRF achieves substantial performance gains over RLVR baseline.
  arXiv  Detail & Related papers  (2025-06-23T10:37:57Z)
• Direct Retrieval-augmented Optimization: Synergizing Knowledge Selection and Language Models [83.8639566087953]
  We propose a direct retrieval-augmented optimization framework, named DRO, that enables end-to-end training of two key components.<n>DRO alternates between two phases: (i) document permutation estimation and (ii) re-weighted, progressively improving RAG components.<n>Our theoretical analysis reveals that DRO is analogous to policy-gradient methods in reinforcement learning.
  arXiv  Detail & Related papers  (2025-05-05T23:54:53Z)
• Can SAM Boost Video Super-Resolution? [78.29033914169025]
  We propose a simple yet effective module -- SAM-guidEd refinEment Module (SEEM) This light-weight plug-in module is specifically designed to leverage the attention mechanism for the generation of semantic-aware feature. We apply our SEEM to two representative methods, EDVR and BasicVSR, resulting in consistently improved performance with minimal implementation effort.
  arXiv  Detail & Related papers  (2023-05-11T02:02:53Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.