CueLearner: Bootstrapping and local policy adaptation from relative feedback
- URL: http://arxiv.org/abs/2507.04730v1
- Date: Mon, 07 Jul 2025 07:54:28 GMT
- Title: CueLearner: Bootstrapping and local policy adaptation from relative feedback
- Authors: Giulio Schiavi, Andrei Cramariuc, Lionel Ott, Roland Siegwart,
- Abstract summary: Relative feedback offers balance between usability and information richness.<n>Previous research has shown that relative feedback can be used to enhance policy search methods.<n>We introduce a novel method to learn from relative feedback and combine it with off-policy reinforcement learning.
- Score: 31.015306281489327
- License: http://creativecommons.org/licenses/by/4.0/
- Abstract: Human guidance has emerged as a powerful tool for enhancing reinforcement learning (RL). However, conventional forms of guidance such as demonstrations or binary scalar feedback can be challenging to collect or have low information content, motivating the exploration of other forms of human input. Among these, relative feedback (i.e., feedback on how to improve an action, such as "more to the left") offers a good balance between usability and information richness. Previous research has shown that relative feedback can be used to enhance policy search methods. However, these efforts have been limited to specific policy classes and use feedback inefficiently. In this work, we introduce a novel method to learn from relative feedback and combine it with off-policy reinforcement learning. Through evaluations on two sparse-reward tasks, we demonstrate our method can be used to improve the sample efficiency of reinforcement learning by guiding its exploration process. Additionally, we show it can adapt a policy to changes in the environment or the user's preferences. Finally, we demonstrate real-world applicability by employing our approach to learn a navigation policy in a sparse reward setting.
Related papers
- Vision-Language Models Provide Promptable Representations for Reinforcement Learning [67.40524195671479]
We propose a novel approach that uses the vast amounts of general and indexable world knowledge encoded in vision-language models (VLMs) pre-trained on Internet-scale data for embodied reinforcement learning (RL)
We show that our approach can use chain-of-thought prompting to produce representations of common-sense semantic reasoning, improving policy performance in novel scenes by 1.5 times.
arXiv Detail & Related papers (2024-02-05T00:48:56Z) - RLIF: Interactive Imitation Learning as Reinforcement Learning [56.997263135104504]
We show how off-policy reinforcement learning can enable improved performance under assumptions that are similar but potentially even more practical than those of interactive imitation learning.
Our proposed method uses reinforcement learning with user intervention signals themselves as rewards.
This relaxes the assumption that intervening experts in interactive imitation learning should be near-optimal and enables the algorithm to learn behaviors that improve over the potential suboptimal human expert.
arXiv Detail & Related papers (2023-11-21T21:05:21Z) - Provable Benefits of Policy Learning from Human Preferences in
Contextual Bandit Problems [82.92678837778358]
preference-based methods have demonstrated substantial success in empirical applications such as InstructGPT.
We show how human bias and uncertainty in feedback modelings can affect the theoretical guarantees of these approaches.
arXiv Detail & Related papers (2023-07-24T17:50:24Z) - Representation-Driven Reinforcement Learning [57.44609759155611]
We present a representation-driven framework for reinforcement learning.
By representing policies as estimates of their expected values, we leverage techniques from contextual bandits to guide exploration and exploitation.
We demonstrate the effectiveness of this framework through its application to evolutionary and policy gradient-based approaches.
arXiv Detail & Related papers (2023-05-31T14:59:12Z) - Click-Feedback Retrieval [10.203235400791845]
We study a setting where the feedback is provided through users clicking liked and disliked searching results.
We construct a new benchmark termed click-feedback retrieval based on a large-scale dataset in fashion domain.
arXiv Detail & Related papers (2023-04-28T19:03:03Z) - Sample Efficient Social Navigation Using Inverse Reinforcement Learning [11.764601181046498]
We describe an inverse reinforcement learning based algorithm which learns from human trajectory observations without knowing their specific actions.
We show that our approach yields better performance while also decreasing training time and sample complexity.
arXiv Detail & Related papers (2021-06-18T19:07:41Z) - PEBBLE: Feedback-Efficient Interactive Reinforcement Learning via
Relabeling Experience and Unsupervised Pre-training [94.87393610927812]
We present an off-policy, interactive reinforcement learning algorithm that capitalizes on the strengths of both feedback and off-policy learning.
We demonstrate that our approach is capable of learning tasks of higher complexity than previously considered by human-in-the-loop methods.
arXiv Detail & Related papers (2021-06-09T14:10:50Z) - Generative Inverse Deep Reinforcement Learning for Online Recommendation [62.09946317831129]
We propose a novel inverse reinforcement learning approach, namely InvRec, for online recommendation.
InvRec extracts the reward function from user's behaviors automatically, for online recommendation.
arXiv Detail & Related papers (2020-11-04T12:12:25Z) - Reward-Conditioned Policies [100.64167842905069]
imitation learning requires near-optimal expert data.
Can we learn effective policies via supervised learning without demonstrations?
We show how such an approach can be derived as a principled method for policy search.
arXiv Detail & Related papers (2019-12-31T18:07:43Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.