From Past to Future: Rethinking Eligibility Traces
- URL: http://arxiv.org/abs/2312.12972v1
- Date: Wed, 20 Dec 2023 12:23:30 GMT
- Title: From Past to Future: Rethinking Eligibility Traces
- Abstract summary: We investigate the concept of a novel value function, which we refer to as the emphbidirectional value function.
We derive principled update equations to learn this value function and, through experimentation, demonstrate its efficacy in enhancing the process of policy evaluation.
- Score: 21.737721213359183
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: In this paper, we introduce a fresh perspective on the challenges of credit
assignment and policy evaluation. First, we delve into the nuances of
eligibility traces and explore instances where their updates may result in
unexpected credit assignment to preceding states. From this investigation
emerges the concept of a novel value function, which we refer to as the
\emph{bidirectional value function}. Unlike traditional state value functions,
bidirectional value functions account for both future expected returns (rewards
anticipated from the current state onward) and past expected returns
(cumulative rewards from the episode's start to the present). We derive
principled update equations to learn this value function and, through
experimentation, demonstrate its efficacy in enhancing the process of policy
evaluation. In particular, our results indicate that the proposed learning
approach can, in certain challenging contexts, perform policy evaluation more
rapidly than TD($\lambda$) -- a method that learns forward value functions,
$v^\pi$, \emph{directly}. Overall, our findings present a new perspective on
eligibility traces and potential advantages associated with the novel value
function it inspires, especially for policy evaluation.
Related papers
- Value Function Decomposition in Markov Recommendation Process [19.082512423102855]
We propose an online reinforcement learning framework to improve recommender performance.
We show that these two factors can be separately approximated by decomposing the original temporal difference loss.
The disentangled learning framework can achieve a more accurate estimation with faster learning and improved robustness against action exploration.
arXiv Detail & Related papers (2025-01-29T04:22:29Z) - Policy Optimization via Adv2: Adversarial Learning on Advantage Functions [6.793286055326244]
We revisit the reduction of learning in adversarial Markov decision processes [MDPs] to adversarial learning based on $Q$--values.<n>We discuss the impact of the reduction of learning in adversarial MDPs to adversarial learning in the practical scenarios where transition kernels are unknown.
arXiv Detail & Related papers (2023-10-25T08:53:51Z) - Leveraging Prior Knowledge in Reinforcement Learning via Double-Sided
Bounds on the Value Function [4.48890356952206]
We show how an arbitrary approximation for the value function can be used to derive double-sided bounds on the optimal value function of interest.
We extend the framework with error analysis for continuous state and action spaces.
arXiv Detail & Related papers (2023-02-19T21:47:24Z) - A Generalized Bootstrap Target for Value-Learning, Efficiently Combining
Value and Feature Predictions [39.17511693008055]
Estimating value functions is a core component of reinforcement learning algorithms.
We focus on bootstrapping targets used when estimating value functions.
We propose a new backup target, the $eta$-return mixture.
arXiv Detail & Related papers (2022-01-05T21:54:55Z) - Robust and Adaptive Temporal-Difference Learning Using An Ensemble of
Gaussian Processes [70.80716221080118]
The paper takes a generative perspective on policy evaluation via temporal-difference (TD) learning.
The OS-GPTD approach is developed to estimate the value function for a given policy by observing a sequence of state-reward pairs.
To alleviate the limited expressiveness associated with a single fixed kernel, a weighted ensemble (E) of GP priors is employed to yield an alternative scheme.
arXiv Detail & Related papers (2021-12-01T23:15:09Z) - Offline Reinforcement Learning with Implicit Q-Learning [85.62618088890787]
Current offline reinforcement learning methods need to query the value of unseen actions during training to improve the policy.
We propose an offline RL method that never needs to evaluate actions outside of the dataset.
This method enables the learned policy to improve substantially over the best behavior in the data through generalization.
arXiv Detail & Related papers (2021-10-12T17:05:05Z) - Direct Advantage Estimation [63.52264764099532]
We show that the expected return may depend on the policy in an undesirable way which could slow down learning.
We propose the Direct Advantage Estimation (DAE), a novel method that can model the advantage function and estimate it directly from data.
If desired, value functions can also be seamlessly integrated into DAE and be updated in a similar way to Temporal Difference Learning.
arXiv Detail & Related papers (2021-09-13T16:09:31Z) - Taylor Expansion of Discount Factors [56.46324239692532]
In practical reinforcement learning (RL), the discount factor used for estimating value functions often differs from that used for defining the evaluation objective.
In this work, we study the effect that this discrepancy of discount factors has during learning, and discover a family of objectives that interpolate value functions of two distinct discount factors.
arXiv Detail & Related papers (2021-06-11T05:02:17Z) - Counterfactual Credit Assignment in Model-Free Reinforcement Learning [47.79277857377155]
Credit assignment in reinforcement learning is the problem of measuring an action's influence on future rewards.
We adapt the notion of counterfactuals from causality theory to a model-free RL setup.
We formulate a family of policy algorithms that use future-conditional value functions as baselines or critics, and show that they are provably low variance.
arXiv Detail & Related papers (2020-11-18T18:41:44Z) - Inverse Policy Evaluation for Value-based Sequential Decision-making [10.188967035477217]
Value-based methods for reinforcement learning lack generally applicable ways to derive behavior from a value function.
We show that inverse policy evaluation, combined with an approximate value iteration algorithm, is a feasible method for value-based control.
arXiv Detail & Related papers (2020-08-26T01:31:38Z) - Learning "What-if" Explanations for Sequential Decision-Making [92.8311073739295]
Building interpretable parameterizations of real-world decision-making on the basis of demonstrated behavior is essential.
We propose learning explanations of expert decisions by modeling their reward function in terms of preferences with respect to "what if" outcomes.
We highlight the effectiveness of our batch, counterfactual inverse reinforcement learning approach in recovering accurate and interpretable descriptions of behavior.
arXiv Detail & Related papers (2020-07-02T14:24:17Z) - Estimating Q(s,s') with Deep Deterministic Dynamics Gradients [25.200259376015744]
We introduce a novel form of value function, $Q(s, s')$, that expresses the utility of transitioning from a state $s$ to a neighboring state $s'$.
In order to derive an optimal policy, we develop a forward dynamics model that learns to make next-state predictions that maximize this value.
arXiv Detail & Related papers (2020-02-21T19:05:24Z) - Value-driven Hindsight Modelling [68.658900923595]
Value estimation is a critical component of the reinforcement learning (RL) paradigm.
Model learning can make use of the rich transition structure present in sequences of observations, but this approach is usually not sensitive to the reward function.
We develop an approach for representation learning in RL that sits in between these two extremes.
This provides tractable prediction targets that are directly relevant for a task, and can thus accelerate learning the value function.
arXiv Detail & Related papers (2020-02-19T18:10:20Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.