Expected Eligibility Traces

• URL: http://arxiv.org/abs/2007.01839v2
• Date: Mon, 8 Feb 2021 13:02:30 GMT
• Title: Expected Eligibility Traces
• Abstract summary: Expected traces allow to update states and actions that could have preceded the current state, even if they did not do so on this occasion. We provide a way to smoothly interpolate between instantaneous and expected traces by a mechanism similar to bootstrapping.
• Score: 31.942254205913798
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: The question of how to determine which states and actions are responsible for a certain outcome is known as the credit assignment problem and remains a central research question in reinforcement learning and artificial intelligence. Eligibility traces enable efficient credit assignment to the recent sequence of states and actions experienced by the agent, but not to counterfactual sequences that could also have led to the current state. In this work, we introduce expected eligibility traces. Expected traces allow, with a single update, to update states and actions that could have preceded the current state, even if they did not do so on this occasion. We discuss when expected traces provide benefits over classic (instantaneous) traces in temporal-difference learning, and show that sometimes substantial improvements can be attained. We provide a way to smoothly interpolate between instantaneous and expected traces by a mechanism similar to bootstrapping, which ensures that the resulting algorithm is a strict generalisation of TD($\lambda$). Finally, we discuss possible extensions and connections to related ideas, such as successor features.

Related papers

• World Models for Embodied Intelligence: From Plausible to Controllable to Actionable [100.13060753774657]
  World models connect perception and decision-making in embodied intelligence by maintaining hidden state, anticipating consequences, comparing interventions, and adapting when execution departs from expectations.<n>This raises a central question: which predictive capabilities improve behavior?<n>We introduce three progressively stronger capability levels: Plausible models preserve task-relevant temporal, geometric, or physical structure; Controllable models additionally predict how interventions alter that structure; and Actionable models translate predictions into measurable gains in planning, action, learning, evaluation, verification, recovery, or data selection.
  arXiv  Detail & Related papers  (2026-09-15T06:22:42Z)
• FactCheck: Feasibility-aware Long-term Action Anticipation with Multi-agent Collaboration [29.089100270702613]
  Long-term action anticipation aims to predict an ordered sequence of future verb-noun actions from a partially observed video.<n>We propose FactCheck, a novel multi-agent collaboration framework that improves feasibility through a closed-loop "Observe-Plan-Verify" mechanism.<n>Our work establishes a new paradigm for feasibility-aware long-term action anticipation, effectively closing the loop of action recognition, action prediction and action verification.
  arXiv  Detail & Related papers  (2026-06-10T06:25:09Z)
• Rethinking Video Human-Object Interaction: Set Prediction over Time for Unified Detection and Anticipation [18.693454975393703]
  Video-based human-object interaction (HOI) understanding requires both detecting ongoing interactions and anticipating their future evolution.<n>We introduce DETAnt-HOI and HOI-DA, a pair-centric framework that jointly performs subject-object localization, present HOI detection, and future anticipation.<n>Experiments show consistent improvements in both detection and anticipation, with larger gains at longer horizons.
  arXiv  Detail & Related papers  (2026-04-12T01:07:43Z)
• When Visibility Outpaces Verification: Delayed Verification and Narrative Lock-in in Agentic AI Discourse [2.5424331328233207]
  Agentic AI systems-autonomous entities capable of independent planning and execution-reshape the landscape of human-AI trust.<n>This paper investigates the interplay between social proof and verification timing in online discussions of agentic AI.
  arXiv  Detail & Related papers  (2026-02-11T22:30:12Z)
• Conditionals Based on Selection Functions, Modal Operators and Probabilities [0.0]
  We prove general results concerning the connection between conditionals and their probabilities.<n>This will allow us to characterize the probabilities of certain conditional connectives.<n>We adopt a general perspective that encompasses a large class of conditionals and a wide range of updating methods.
  arXiv  Detail & Related papers  (2025-11-27T12:13:45Z)
• Multi-level and Multi-modal Action Anticipation [12.921307214813357]
  Action anticipation, the task of predicting future actions from partially observed videos, is crucial for advancing intelligent systems.<n>We introduce textitMulti-level and Multi-modal Action Anticipation (m&m-Ant), a novel multi-modal action anticipation approach.<n>Experiments on widely used datasets, including Breakfast, 50 Salads, and DARai, demonstrate the effectiveness of our approach.
  arXiv  Detail & Related papers  (2025-06-03T02:39:33Z)
• Bidirectional Decoding: Improving Action Chunking via Closed-Loop Resampling [51.38330727868982]
  Bidirectional Decoding (BID) is a test-time inference algorithm that bridges action chunking with closed-loop operations. We show that BID boosts the performance of two state-of-the-art generative policies across seven simulation benchmarks and two real-world tasks.
  arXiv  Detail & Related papers  (2024-08-30T15:39:34Z)
• From Past to Future: Rethinking Eligibility Traces [21.737721213359183]
  We investigate the concept of a novel value function, which we refer to as the emphbidirectional value function. We derive principled update equations to learn this value function and, through experimentation, demonstrate its efficacy in enhancing the process of policy evaluation.
  arXiv  Detail & Related papers  (2023-12-20T12:23:30Z)
• Continual Action Assessment via Task-Consistent Score-Discriminative Feature Distribution Modeling [31.696222064667243]
  Action Quality Assessment (AQA) is a task that tries to answer how well an action is carried out. Existing works on AQA assume that all the training data are visible for training at one time, but do not enable continual learning. We propose a unified model to learn AQA tasks sequentially without forgetting.
  arXiv  Detail & Related papers  (2023-09-29T10:06:28Z)
• Sequence Model Imitation Learning with Unobserved Contexts [39.4969161422156]
  We consider imitation learning problems where the expert has access to a per-episode context hidden from the learner. We show that on-policy approaches are able to use history to identify the context while off-policy approaches actually perform worse when given access to history.
  arXiv  Detail & Related papers  (2022-08-03T17:27:44Z)
• Predecessor Features [0.0]
  "Predecessor Features" is an algorithm that achieves this richer form of credit assignment. By maintaining a representation that approximates the expected sum of past occupancies, our algorithm allows temporal difference (TD) errors to be propagated accurately.
  arXiv  Detail & Related papers  (2022-06-01T08:05:59Z)
• Efficient Embedding of Semantic Similarity in Control Policies via Entangled Bisimulation [3.5092955099876266]
  Learning generalizeable policies from visual input in the presence of visual distractions is a challenging problem in reinforcement learning. We propose entangled bisimulation, a bisimulation metric that allows the specification of the distance function between states. We show how entangled bisimulation can meaningfully improve over previous methods on the Distracting Control Suite (DCS)
  arXiv  Detail & Related papers  (2022-01-28T18:06:06Z)
• Self-Regulated Learning for Egocentric Video Activity Anticipation [147.9783215348252]
  Self-Regulated Learning (SRL) aims to regulate the intermediate representation consecutively to produce representation that emphasizes the novel information in the frame of the current time-stamp. SRL sharply outperforms existing state-of-the-art in most cases on two egocentric video datasets and two third-person video datasets.
  arXiv  Detail & Related papers  (2021-11-23T03:29:18Z)
• LSTC: Boosting Atomic Action Detection with Long-Short-Term Context [60.60267767456306]
  We decompose the action recognition pipeline into short-term and long-term reliance. Within our design, a local aggregation branch is utilized to gather dense and informative short-term cues. Both branches independently predict the context-specific actions and the results are merged in the end.
  arXiv  Detail & Related papers  (2021-10-19T10:09:09Z)
• Nested Counterfactual Identification from Arbitrary Surrogate Experiments [95.48089725859298]
  We study the identification of nested counterfactuals from an arbitrary combination of observations and experiments. Specifically, we prove the counterfactual unnesting theorem (CUT), which allows one to map arbitrary nested counterfactuals to unnested ones.
  arXiv  Detail & Related papers  (2021-07-07T12:51:04Z)
• Learning "What-if" Explanations for Sequential Decision-Making [92.8311073739295]
  Building interpretable parameterizations of real-world decision-making on the basis of demonstrated behavior is essential. We propose learning explanations of expert decisions by modeling their reward function in terms of preferences with respect to "what if" outcomes. We highlight the effectiveness of our batch, counterfactual inverse reinforcement learning approach in recovering accurate and interpretable descriptions of behavior.
  arXiv  Detail & Related papers  (2020-07-02T14:24:17Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.