Lever: Inference-Time Policy Reuse under Support Constraints

• URL: http://arxiv.org/abs/2604.20174v2
• Date: Tue, 28 Apr 2026 08:10:22 GMT
• Title: Lever: Inference-Time Policy Reuse under Support Constraints
• Abstract summary: Reinforcement learning (RL) policies are typically trained for fixed objectives, making reuse difficult when task requirements change.<n>We study inference-time policy reuse: given a library of pre-trained policies and a new composite objective, can a high-quality policy be constructed entirely offline, without additional environment interaction?<n>We introduce lever, an end-to-end framework that retrieves relevant policies, evaluates them using behavioral embeddings, and composes new policies via offline Q-value composition.
• Score: 8.884389224195912
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Reinforcement learning (RL) policies are typically trained for fixed objectives, making reuse difficult when task requirements change. We study inference-time policy reuse: given a library of pre-trained policies and a new composite objective, can a high-quality policy be constructed entirely offline, without additional environment interaction? We introduce lever (Leveraging Efficient Vector Embeddings for Reusable policies), an end-to-end framework that retrieves relevant policies, evaluates them using behavioral embeddings, and composes new policies via offline Q-value composition. We focus on the support-limited regime, where no value propagation is possible, and show that the effectiveness of reuse depends critically on the coverage of available transitions. To balance performance and computational cost, lever proposes composition strategies that control the exploration of candidate policies. Experiments in deterministic GridWorld environments show that inference-time composition can match, and in some cases exceed, training-from-scratch performance while providing substantial speedups. At the same time, performance degrades when long-horizon dependencies require value propagation, highlighting a fundamental limitation of offline reuse.

Related papers

• Inference-Time Policy Alignment for Fair Reinforcement Learning [1.9342617041096457]
  Policy of deep reinforcement learning agents is often rigid and costly to adapt to new performance criteria.<n>We propose a multiplicative policy shaping framework that adjusts action probabilities using action-dependent welfare scores.<n>Our framework is general and compatible with any deep RL agent.
  arXiv  Detail & Related papers  (2026-07-31T18:00:50Z)
• Conservative Query and Adaptive Regularization for Offline RL Under Uncertainty Estimation [30.50829492427299]
  Action preference queries leverage expert feedback without additional environment interaction, enabling policy improvement during offline training.<n>Current approaches rely only on the distance between policy actions and dataset actions for query selection.<n>We propose Conservative Query and Adaptive Regularization under Uncertainty Estimation, a lightweight framework that jointly improves preference querying and preference exploitation.
  arXiv  Detail & Related papers  (2026-07-21T15:34:06Z)
• Beyond Single-Model Optimization: Preserving Plasticity in Continual Reinforcement Learning [1.618563064839635]
  We introduce textscTeLAPA (Transfer-Enabled Latent-Aligned Policy Archives), a continual RL framework that organizes behaviorally diverse policy neighborhoods into per-task archives.<n>In our MiniGrid CL setting, textscTeLAPA learns more tasks successfully, recovers competence faster on revisited tasks after interference, and retains higher performance across a sequence of tasks.
  arXiv  Detail & Related papers  (2026-04-16T17:06:54Z)
• Relative Entropy Pathwise Policy Optimization [66.03329137921949]
  We present an on-policy algorithm that trains Q-value models purely from on-policy trajectories.<n>We show how to combine policies for exploration with constrained updates for stable training, and evaluate important architectural components that stabilize value function learning.
  arXiv  Detail & Related papers  (2025-07-15T06:24:07Z)
• Policy Constraint by Only Support Constraint for Offline Reinforcement Learning [11.006709826558465]
  We present Only Support Constraint (OSC) which is derived from maximizing the total probability of learned policy in the support of behavior policy.<n>OSC significantly enhances performance, alleviating the challenges associated with distributional shifts and mitigating conservatism of policy constraints.
  arXiv  Detail & Related papers  (2025-03-07T07:55:51Z)
• Offline Hierarchical Reinforcement Learning via Inverse Optimization [23.664330010602708]
  OHIO is a framework for offline reinforcement learning of hierarchical policies.<n>We show it substantially outperforms end-to-end RL methods and improves robustness.
  arXiv  Detail & Related papers  (2024-10-10T14:00:21Z)
• Projected Off-Policy Q-Learning (POP-QL) for Stabilizing Offline Reinforcement Learning [57.83919813698673]
  Projected Off-Policy Q-Learning (POP-QL) is a novel actor-critic algorithm that simultaneously reweights off-policy samples and constrains the policy to prevent divergence and reduce value-approximation error. In our experiments, POP-QL not only shows competitive performance on standard benchmarks, but also out-performs competing methods in tasks where the data-collection policy is significantly sub-optimal.
  arXiv  Detail & Related papers  (2023-11-25T00:30:58Z)
• Co-learning Planning and Control Policies Constrained by Differentiable Logic Specifications [4.12484724941528]
  This paper presents a novel reinforcement learning approach to solving high-dimensional robot navigation tasks. We train high-quality policies with much fewer samples compared to existing reinforcement learning algorithms. Our approach also demonstrates capabilities for high-dimensional control and avoiding suboptimal policies via policy alignment.
  arXiv  Detail & Related papers  (2023-03-02T15:24:24Z)
• COptiDICE: Offline Constrained Reinforcement Learning via Stationary Distribution Correction Estimation [73.17078343706909]
  offline constrained reinforcement learning (RL) problem, in which the agent aims to compute a policy that maximizes expected return while satisfying given cost constraints, learning only from a pre-collected dataset. We present an offline constrained RL algorithm that optimize the policy in the space of the stationary distribution. Our algorithm, COptiDICE, directly estimates the stationary distribution corrections of the optimal policy with respect to returns, while constraining the cost upper bound, with the goal of yielding a cost-conservative policy for actual constraint satisfaction.
  arXiv  Detail & Related papers  (2022-04-19T15:55:47Z)
• Latent-Variable Advantage-Weighted Policy Optimization for Offline RL [70.01851346635637]
  offline reinforcement learning methods hold the promise of learning policies from pre-collected datasets without the need to query the environment for new transitions. In practice, offline datasets are often heterogeneous, i.e., collected in a variety of scenarios. We propose to leverage latent-variable policies that can represent a broader class of policy distributions. Our method improves the average performance of the next best-performing offline reinforcement learning methods by 49% on heterogeneous datasets.
  arXiv  Detail & Related papers  (2022-03-16T21:17:03Z)
• Constructing a Good Behavior Basis for Transfer using Generalized Policy Updates [63.58053355357644]
  We study the problem of learning a good set of policies, so that when combined together, they can solve a wide variety of unseen reinforcement learning tasks. We show theoretically that having access to a specific set of diverse policies, which we call a set of independent policies, can allow for instantaneously achieving high-level performance.
  arXiv  Detail & Related papers  (2021-12-30T12:20:46Z)
• Goal-Conditioned Reinforcement Learning with Imagined Subgoals [89.67840168694259]
  We propose to incorporate imagined subgoals into policy learning to facilitate learning of complex tasks. Imagined subgoals are predicted by a separate high-level policy, which is trained simultaneously with the policy and its critic. We evaluate our approach on complex robotic navigation and manipulation tasks and show that it outperforms existing methods by a large margin.
  arXiv  Detail & Related papers  (2021-07-01T15:30:59Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.