Predecessor Features

• URL: http://arxiv.org/abs/2206.00303v1
• Date: Wed, 1 Jun 2022 08:05:59 GMT
• Title: Predecessor Features
• Abstract summary: "Predecessor Features" is an algorithm that achieves this richer form of credit assignment. By maintaining a representation that approximates the expected sum of past occupancies, our algorithm allows temporal difference (TD) errors to be propagated accurately.
• Score: 0.0
• License: http://creativecommons.org/licenses/by-sa/4.0/
• Abstract: Any reinforcement learning system must be able to identify which past events contributed to observed outcomes, a problem known as credit assignment. A common solution to this problem is to use an eligibility trace to assign credit to recency-weighted set of experienced events. However, in many realistic tasks, the set of recently experienced events are only one of the many possible action events that could have preceded the current outcome. This suggests that reinforcement learning can be made more efficient by allowing credit assignment to any viable preceding state, rather than only those most recently experienced. Accordingly, we propose "Predecessor Features", an algorithm that achieves this richer form of credit assignment. By maintaining a representation that approximates the expected sum of past occupancies, our algorithm allows temporal difference (TD) errors to be propagated accurately to a larger number of predecessor states than conventional methods, greatly improving learning speed. Our algorithm can also be naturally extended from tabular state representation to feature representations allowing for increased performance on a wide range of environments. We demonstrate several use cases for Predecessor Features and contrast its performance with other similar approaches.

Related papers

• Person Re-Identification via Generalized Class Prototypes [9.600466490978665]
  selecting better class representatives is an underexplored area of research.<n>We propose a generalized selection method that involves choosing representations that are not limited to class centroids.<n>Our approach strikes a balance between accuracy and mean average precision, leading to improvements beyond the state of the art.
  arXiv  Detail & Related papers  (2025-10-19T23:16:57Z)
• Multi-level and Multi-modal Action Anticipation [12.921307214813357]
  Action anticipation, the task of predicting future actions from partially observed videos, is crucial for advancing intelligent systems.<n>We introduce textitMulti-level and Multi-modal Action Anticipation (m&m-Ant), a novel multi-modal action anticipation approach.<n>Experiments on widely used datasets, including Breakfast, 50 Salads, and DARai, demonstrate the effectiveness of our approach.
  arXiv  Detail & Related papers  (2025-06-03T02:39:33Z)
• Incremental Sequence Classification with Temporal Consistency [9.65650774513798]
  We address the problem of incremental sequence classification, where predictions are updated as new elements in the sequence are revealed.<n>We leverage a temporal-consistency condition that successive predictions should satisfy to develop a novel loss function for training incremental sequence classifiers.<n>Our results show that models trained with our method are better able to distinguish promising generations from unpromising ones after observing only a few tokens.
  arXiv  Detail & Related papers  (2025-05-22T11:37:53Z)
• Bidirectional Decoding: Improving Action Chunking via Closed-Loop Resampling [51.38330727868982]
  Bidirectional Decoding (BID) is a test-time inference algorithm that bridges action chunking with closed-loop operations. We show that BID boosts the performance of two state-of-the-art generative policies across seven simulation benchmarks and two real-world tasks.
  arXiv  Detail & Related papers  (2024-08-30T15:39:34Z)
• Generalizing Multi-Step Inverse Models for Representation Learning to Finite-Memory POMDPs [23.584313644411967]
  We study the problem of discovering an informative, or agent-centric, state representation that encodes only the relevant information while discarding the irrelevant. Our results include theory in the deterministic dynamics setting as well as counter-examples for alternative intuitive algorithms. We show that these can be a double-edged sword: making the algorithms more successful when used correctly and causing dramatic failure when used incorrectly.
  arXiv  Detail & Related papers  (2024-04-22T19:46:16Z)
• Predictive Coding beyond Correlations [59.47245250412873]
  We show how one of such algorithms, called predictive coding, is able to perform causal inference tasks. First, we show how a simple change in the inference process of predictive coding enables to compute interventions without the need to mutilate or redefine a causal graph.
  arXiv  Detail & Related papers  (2023-06-27T13:57:16Z)
• Multi-Armed Bandits with Generalized Temporally-Partitioned Rewards [0.4194295877935867]
  In some real-world applications, feedback about a decision is delayed and may arrive via partial rewards that are observed with different delays. We propose a novel problem formulation called multi-armed bandits with generalized temporally-partitioned rewards. We derive a lower bound on the performance of any uniformly efficient algorithm for the considered problem.
  arXiv  Detail & Related papers  (2023-03-01T16:22:22Z)
• Self-Supervised Learning via Maximum Entropy Coding [57.56570417545023]
  We propose Maximum Entropy Coding (MEC) as a principled objective that explicitly optimize on the structure of the representation. MEC learns a more generalizable representation than previous methods based on specific pretext tasks. It achieves state-of-the-art performance consistently on various downstream tasks, including not only ImageNet linear probe, but also semi-supervised classification, object detection, instance segmentation, and object tracking.
  arXiv  Detail & Related papers  (2022-10-20T17:58:30Z)
• Non-Clairvoyant Scheduling with Predictions Revisited [77.86290991564829]
  In non-clairvoyant scheduling, the task is to find an online strategy for scheduling jobs with a priori unknown processing requirements. We revisit this well-studied problem in a recently popular learning-augmented setting that integrates (untrusted) predictions in algorithm design. We show that these predictions have desired properties, admit a natural error measure as well as algorithms with strong performance guarantees.
  arXiv  Detail & Related papers  (2022-02-21T13:18:11Z)
• Improving Experience Replay with Successor Representation [0.0]
  Prioritized experience replay is a reinforcement learning technique shown to speed up learning. Recent work in neuroscience suggests that, in biological organisms, replay is prioritized by both gain and need.
  arXiv  Detail & Related papers  (2021-11-29T05:25:54Z)
• Metalearning Linear Bandits by Prior Update [7.519872646378836]
  Fully Bayesian approaches assume that problem parameters are generated from a known prior, while in practice, such information is often lacking. This problem is exacerbated in decision-making setups with partial information, where using a misspecified prior may lead to poor exploration and inferior performance. In this work we prove, in the context of linear bandits and Gaussian priors, that as long as the prior estimate is sufficiently close to the true prior, the performance of an algorithm that uses the misspecified prior is close to that of an algorithm that uses the true prior.
  arXiv  Detail & Related papers  (2021-07-12T11:17:01Z)
• Expected Eligibility Traces [31.942254205913798]
  Expected traces allow to update states and actions that could have preceded the current state, even if they did not do so on this occasion. We provide a way to smoothly interpolate between instantaneous and expected traces by a mechanism similar to bootstrapping.
  arXiv  Detail & Related papers  (2020-07-03T17:46:16Z)
• Optimizing for the Future in Non-Stationary MDPs [52.373873622008944]
  We present a policy gradient algorithm that maximizes a forecast of future performance. We show that our algorithm, called Prognosticator, is more robust to non-stationarity than two online adaptation techniques.
  arXiv  Detail & Related papers  (2020-05-17T03:41:19Z)
• Ambiguity in Sequential Data: Predicting Uncertain Futures with Recurrent Models [110.82452096672182]
  We propose an extension of the Multiple Hypothesis Prediction (MHP) model to handle ambiguous predictions with sequential data. We also introduce a novel metric for ambiguous problems, which is better suited to account for uncertainties.
  arXiv  Detail & Related papers  (2020-03-10T09:15:42Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.