Proxy Exploration and Reusable Guidance: A Modular LLM Post-Training Paradigm via Proxy-Guided Update Signals

• URL: http://arxiv.org/abs/2607.11505v1
• Date: Mon, 13 Jul 2026 12:56:21 GMT
• Title: Proxy Exploration and Reusable Guidance: A Modular LLM Post-Training Paradigm via Proxy-Guided Update Signals
• Abstract summary: Proxy-guided Update Signal Transfer (PUST) is a novel post-training framework that fundamentally decouples update-signal exploration from distribution alignment.<n>PUST significantly reduces computational overhead and enables optimization signals to be asynchronously generated, cached, and reused.
• Score: 48.04226595566376
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Post-training is essential for refining the domain-specific capabilities of large language models (LLMs), yet existing reward optimization and distribution matching methods tightly couple policy exploration with distribution alignment. This coupling forces expensive exploration directly on the policy model and severely hinders the asynchronous generation, reuse, and cross-model transfer of optimization signals. In this paper, we propose Proxy-guided Update Signal Transfer (PUST), a novel post-training framework that fundamentally decouples update-signal exploration from distribution alignment. Instead of utilizing the primary model for costly exploration, PUST employs a lightweight proxy model as an efficient testbed to discover high-reward behaviors. We extract the relative improvement signal between the proxy's initial and optimized states, transferring this directional update to the primary model to guide its policy alignment. This decoupled pipeline, comprising proxy exploration, update-signal extraction, and signal transfer, significantly reduces computational overhead and enables optimization signals to be asynchronously generated, cached, and reused. Crucially, by transferring relative improvements rather than absolute policy distributions, PUST naturally supports weak-to-strong improvement and seamless cross-model transfer. Systematic evaluations on Qwen3-family models across math and code domains demonstrate that update signals extracted from substantially weaker proxies can robustly and adjustably enhance stronger primary models. Ultimately, PUST transforms post-training from a monolithic online optimization process into a highly modular, reusable, and cost-efficient paradigm.

Related papers

• Step Back to Move Forward: Reflection-Aware Preference Optimization for Visual Generation [12.58348084349179]
  Reflection-Aware GRPO is a new RL-based preference alignment framework for diffusion generative models.<n>We introduce Diffusion Reflection, which rectifies intermediate sampling trajectories by inverting the diffusion process with a weak estimator.<n>We also introduce Counterfactual Path Synthesis to implicitly distill these rectified trajectories into the policy, enabling the model to internalize the benefits of search-based exploration.
  arXiv  Detail & Related papers  (2026-09-03T04:37:19Z)
• Scaling Reinforcement Learning for Diffusion Models via Velocity Matching [66.34541945134916]
  RewardBench fine-tuning is an important tool for adapting diffusion models to human preferences and task-specific objectives.<n>Unlike autoregressive models, diffusion models do not provide tractable likelihoods for generated samples.<n>We propose reward-based velocity matching (RVM), a simple trajectory-free update that acts directly on the velocity field.
  arXiv  Detail & Related papers  (2026-08-24T17:13:43Z)
• GIRL-DETR: Gradient-Isolated Reinforcement Learning for Video Moment Retrieval [10.034595422596562]
  Video Moment Retrieval (VMR) task requires accurately localizing temporal boundaries aligned with natural language queries.<n>GIRL-DETR introduces RL post-training into a lightweight temporal localization framework for the first time.
  arXiv  Detail & Related papers  (2026-05-30T15:40:00Z)
• Dance Across Shifts: Forward-Facilitation Continual Test-Time Adaptation through Dynamic Style Bridging [65.11234199720428]
  Continual Test-Time Adaptation aims to empower perception systems to handle dynamic distribution shifts encountered after deployment.<n>We introduce a novel forward-facilitation paradigm through a method termed Dynamic Style Bridging.<n>During test time, to mitigate inherent generative bias and adapt these proxies to incoming data, we propose a multi-level bridging mechanism.
  arXiv  Detail & Related papers  (2026-05-18T16:18:17Z)
• PARM: Pipeline-Adapted Reward Model [60.769414637325326]
  Reward models (RMs) are central to aligning large language models (LLMs) with human preferences, powering advanced decoding strategies.<n>While most prior work focuses on single-step generation, real-world applications increasingly adopt multi-stage pipelines.<n>We investigate this through code generation for optimization, constructing a pipeline that integrates reward models into both formulation and solution stages.
  arXiv  Detail & Related papers  (2026-04-20T14:29:08Z)
• RecNet: Self-Evolving Preference Propagation for Agentic Recommender Systems [109.9061591263748]
  RecNet is a self-evolving preference propagation framework for recommender systems.<n>It proactively propagates real-time preference updates across related users and items.<n>In the backward phase, the feedback-driven propagation optimization mechanism simulates a multi-agent reinforcement learning framework.
  arXiv  Detail & Related papers  (2026-01-29T12:14:31Z)
• Network-Level Vehicle Delay Estimation at Heterogeneous Signalized Intersections [4.534054317956599]
  This study introduces a domain adaptation (DA) framework for estimating vehicle delays across diverse intersections.<n>A novel DA model, Gradient Boosting with Balanced Weighting (GBBW), reweights source data based on similarity to the target domain.<n>Performance is evaluated against eight state-of-the-art ML regression models and seven instance-based DA methods.
  arXiv  Detail & Related papers  (2025-10-01T05:19:50Z)
• Leveraging Importance Sampling to Detach Alignment Modules from Large Language Models [48.15777554876988]
  Traditional alignment methods often require retraining large pretrained models.<n>We propose a novel textitResidual Alignment Model (textitRAM) that formalizes the alignment process as a type of importance sampling.<n>We develop a resampling algorithm with iterative token-level decoding to address the common first-token latency issue in comparable methods.
  arXiv  Detail & Related papers  (2025-05-26T08:53:02Z)
• Efficient Safety Alignment of Large Language Models via Preference Re-ranking and Representation-based Reward Modeling [84.00480999255628]
  Reinforcement Learning algorithms for safety alignment of Large Language Models (LLMs) encounter the challenge of distribution shift.<n>Current approaches typically address this issue through online sampling from the target policy.<n>We propose a new framework that leverages the model's intrinsic safety judgment capability to extract reward signals.
  arXiv  Detail & Related papers  (2025-03-13T06:40:34Z)
• The Gradient Convergence Bound of Federated Multi-Agent Reinforcement Learning with Efficient Communication [20.891460617583302]
  The paper considers independent reinforcement learning (IRL) for collaborative decision-making in the paradigm of federated learning (FL) FL generates excessive communication overheads between agents and a remote central server. This paper proposes two advanced optimization schemes to improve the system's utility value.
  arXiv  Detail & Related papers  (2021-03-24T07:21:43Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.