Discussion of Kallus (2020) and Mo, Qi, and Liu (2020): New Objectives for Policy Learning

• URL: http://arxiv.org/abs/2010.04805v1
• Date: Fri, 9 Oct 2020 21:05:01 GMT
• Title: Discussion of Kallus (2020) and Mo, Qi, and Liu (2020): New Objectives for Policy Learning
• Abstract summary: We show that it is important to take the value function into account when working within the framework. We describe more efficient approaches for leveraging data when learning distributionally robust policies.
• Score: 2.578242050187029
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: We discuss the thought-provoking new objective functions for policy learning that were proposed in "More efficient policy learning via optimal retargeting" by Nathan Kallus and "Learning optimal distributionally robust individualized treatment rules" by Weibin Mo, Zhengling Qi, and Yufeng Liu. We show that it is important to take the curvature of the value function into account when working within the retargeting framework, and we introduce two ways to do so. We also describe more efficient approaches for leveraging calibration data when learning distributionally robust policies.

Related papers

• The Mirage of Optimizing Training Policies: Monotonic Inference Policies as the Real Objective for LLM Reinforcement Learning [65.72688364382331]
  Reinforcement learning (RL) has gained growing attention in large language model (LLM) post-training, yet RL training remains fragile and can suffer from instability or collapse.<n>One vital cause is training-inference mismatch: LLM adopts separate inference and training engines for generation efficiency and training precision.<n>We introduce Monotonic Inference Policy Update (MIPU), a two-step LLM RL framework that constructs sampler-referenced candidate updates and selectively accepts synchronized candidates.
  arXiv  Detail & Related papers  (2026-06-28T17:40:02Z)
• Learning Fair Pareto-Optimal Policies in Multi-Objective Reinforcement Learning [1.3483705601540006]
  We formalize the fair optimization problem in multi-policy MORL.<n>We show that for concave, piecewise-linear welfare functions, fair policies remain in the convex coverage set.<n>We demonstrate that non-stationary policies, augmented with accrued reward histories, and state policies improve fairness by dynamically adapting to historical inequities.
  arXiv  Detail & Related papers  (2026-06-16T16:16:54Z)
• TEACH: Temporal Variance-Driven Curriculum for Reinforcement Learning [8.366600075241847]
  We propose a novel Student-Teacher learning paradigm with a Temporal Variance-Driven Curriculum to accelerate Goal-Conditioned RL.<n>In this framework, the teacher module dynamically prioritizes goals with the highest temporal variance in the policy's confidence score.<n>We demonstrate this through evaluation across 11 diverse robotic manipulation and maze navigation tasks.
  arXiv  Detail & Related papers  (2025-12-28T07:29:29Z)
• Polychromic Objectives for Reinforcement Learning [63.37185057794815]
  Reinforcement learning fine-tuning (RLFT) is a dominant paradigm for improving pretrained policies for downstream tasks.<n>We introduce an objective for policy methods that explicitly enforces the exploration and refinement of diverse generations.<n>We show how proximal policy optimization (PPO) can be adapted to optimize this objective.
  arXiv  Detail & Related papers  (2025-09-29T19:32:11Z)
• Optimal Policy Adaptation under Covariate Shift [15.703626346971182]
  We propose principled approaches for learning the optimal policy in the target domain by leveraging two datasets. We derive the identifiability assumptions for the reward induced by a given policy. We then learn the optimal policy by optimizing the estimated reward.
  arXiv  Detail & Related papers  (2025-01-14T12:33:02Z)
• IOB: Integrating Optimization Transfer and Behavior Transfer for Multi-Policy Reuse [50.90781542323258]
  Reinforcement learning (RL) agents can transfer knowledge from source policies to a related target task. Previous methods introduce additional components, such as hierarchical policies or estimations of source policies' value functions. We propose a novel transfer RL method that selects the source policy without training extra components.
  arXiv  Detail & Related papers  (2023-08-14T09:22:35Z)
• Learning Control Policies for Variable Objectives from Offline Data [2.7174376960271154]
  We introduce a conceptual extension for model-based policy search methods, called variable objective policy (VOP) We demonstrate that by altering the objectives passed as input to the policy, users gain the freedom to adjust its behavior or re-balance optimization targets at runtime.
  arXiv  Detail & Related papers  (2023-08-11T13:33:59Z)
• Counterfactual Explanation Policies in RL [3.674863913115432]
  COUNTERPOL is the first framework to analyze Reinforcement Learning policies using counterfactual explanations. We establish a theoretical connection between Counterpol and widely used trust region-based policy optimization methods in RL.
  arXiv  Detail & Related papers  (2023-07-25T01:14:56Z)
• Fairness in Preference-based Reinforcement Learning [2.3388338598125196]
  We design a new fairness-induced preference-based reinforcement learning or FPbRL. The main idea of FPbRL is to learn vector reward functions associated with multiple objectives via new welfare-based preferences. Experiment studies show that the proposed FPbRL approach can achieve both efficiency and equity for learning effective and fair policies.
  arXiv  Detail & Related papers  (2023-06-16T17:47:36Z)
• Imitating Graph-Based Planning with Goal-Conditioned Policies [72.61631088613048]
  We present a self-imitation scheme which distills a subgoal-conditioned policy into the target-goal-conditioned policy. We empirically show that our method can significantly boost the sample-efficiency of the existing goal-conditioned RL methods.
  arXiv  Detail & Related papers  (2023-03-20T14:51:10Z)
• Goal-Conditioned Q-Learning as Knowledge Distillation [136.79415677706612]
  We explore a connection between off-policy reinforcement learning in goal-conditioned settings and knowledge distillation. We empirically show that this can improve the performance of goal-conditioned off-policy reinforcement learning when the space of goals is high-dimensional. We also show that this technique can be adapted to allow for efficient learning in the case of multiple simultaneous sparse goals.
  arXiv  Detail & Related papers  (2022-08-28T22:01:10Z)
• Variational Empowerment as Representation Learning for Goal-Based Reinforcement Learning [114.07623388322048]
  We discuss how the standard goal-conditioned RL (GCRL) is encapsulated by the objective variational empowerment. Our work lays a novel foundation from which to evaluate, analyze, and develop representation learning techniques in goal-based RL.
  arXiv  Detail & Related papers  (2021-06-02T18:12:26Z)
• Adversarial Intrinsic Motivation for Reinforcement Learning [60.322878138199364]
  We investigate whether the Wasserstein-1 distance between a policy's state visitation distribution and a target distribution can be utilized effectively for reinforcement learning tasks. Our approach, termed Adversarial Intrinsic Motivation (AIM), estimates this Wasserstein-1 distance through its dual objective and uses it to compute a supplemental reward function.
  arXiv  Detail & Related papers  (2021-05-27T17:51:34Z)
• Learn Goal-Conditioned Policy with Intrinsic Motivation for Deep Reinforcement Learning [9.014110264448371]
  We propose a novel unsupervised learning approach named goal-conditioned policy with intrinsic motivation (GPIM) GPIM jointly learns both an abstract-level policy and a goal-conditioned policy. Experiments on various robotic tasks demonstrate the effectiveness and efficiency of our proposed GPIM method.
  arXiv  Detail & Related papers  (2021-04-11T16:26:10Z)
• Efficient Deep Reinforcement Learning via Adaptive Policy Transfer [50.51637231309424]
  Policy Transfer Framework (PTF) is proposed to accelerate Reinforcement Learning (RL) Our framework learns when and which source policy is the best to reuse for the target policy and when to terminate it. Experimental results show it significantly accelerates the learning process and surpasses state-of-the-art policy transfer methods.
  arXiv  Detail & Related papers  (2020-02-19T07:30:57Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.