Direct Advantage Estimation for Scalable and Sample-efficient Deep Reinforcement Learning

• URL: http://arxiv.org/abs/2606.20411v1
• Date: Thu, 18 Jun 2026 15:58:48 GMT
• Title: Direct Advantage Estimation for Scalable and Sample-efficient Deep Reinforcement Learning
• Abstract summary: We introduce discrete latent dynamics models that efficiently approximate transition probabilities.<n>We find that DAE scales effectively with function approximator capacity while retaining high sample efficiency.
• Score: 54.916555668983726
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Direct Advantage Estimation (DAE) has been shown to improve the sample efficiency of deep reinforcement learning algorithms. However, its reliance on full environment observability limits its applicability in realistic settings, and its requirement to model transition probabilities incurs substantial computational overhead for high-dimensional observations. In the present work, we address both limitations. First, we extend the theoretical framework of DAE to partially observable domains with minimal modifications. Second, we reduce its computational complexity by introducing discrete latent dynamics models that efficiently approximate transition probabilities. We evaluate our approach on the Arcade Learning Environment and find that DAE scales effectively with function approximator capacity while retaining high sample efficiency.

Related papers

• Reinforcement Learning under Latent Dynamics: Toward Statistical and Algorithmic Modularity [51.40558987254471]
  Real-world applications of reinforcement learning often involve environments where agents operate on complex, high-dimensional observations. This paper addresses the question of reinforcement learning under $textitgeneral$ latent dynamics from a statistical and algorithmic perspective.
  arXiv  Detail & Related papers  (2024-10-23T14:22:49Z)
• On the Sample Efficiency of Abstractions and Potential-Based Reward Shaping in Reinforcement Learning [1.3381647310440739]
  Potential-Based Reward Shaping (PBRS) has shown great promise in the ongoing research effort to tackle sample inefficiency in Reinforcement Learning (RL)<n>We first build some theoretically-grounded intuition on why selecting the potential function as the optimal value function of the task at hand produces performance advantages.<n>We then analyse the bias induced by finite horizons in the context of PBRS producing novel insights.
  arXiv  Detail & Related papers  (2024-04-11T15:09:49Z)
• A Surrogate Data Assimilation Model for the Estimation of Dynamical System in a Limited Area [12.849290207344422]
  We propose a learning-based surrogate data assimilation model for efficient state estimation in a limited area. Our model employs a feedforward neural network for online computation, eliminating the need for integrating high-dimensional limited-area models.
  arXiv  Detail & Related papers  (2023-07-14T06:10:00Z)
• Efficient Model-Free Exploration in Low-Rank MDPs [76.87340323826945]
  Low-Rank Markov Decision Processes offer a simple, yet expressive framework for RL with function approximation. Existing algorithms are either (1) computationally intractable, or (2) reliant upon restrictive statistical assumptions. We propose the first provably sample-efficient algorithm for exploration in Low-Rank MDPs.
  arXiv  Detail & Related papers  (2023-07-08T15:41:48Z)
• Learning Better with Less: Effective Augmentation for Sample-Efficient Visual Reinforcement Learning [57.83232242068982]
  Data augmentation (DA) is a crucial technique for enhancing the sample efficiency of visual reinforcement learning (RL) algorithms. It remains unclear which attributes of DA account for its effectiveness in achieving sample-efficient visual RL. This work conducts comprehensive experiments to assess the impact of DA's attributes on its efficacy.
  arXiv  Detail & Related papers  (2023-05-25T15:46:20Z)
• Posterior Sampling for Deep Reinforcement Learning [0.0]
  This paper introduces Posterior Sampling for Deep Reinforcement Learning (PSDRL), the first truly scalable approximation of Posterior Sampling for Reinforcement Learning. Experiments on the Atari benchmark show that PSDRL significantly outperforms previous state-of-the-art attempts at scaling up posterior sampling.
  arXiv  Detail & Related papers  (2023-04-30T13:23:50Z)
• Latent Variable Representation for Reinforcement Learning [131.03944557979725]
  It remains unclear theoretically and empirically how latent variable models may facilitate learning, planning, and exploration to improve the sample efficiency of model-based reinforcement learning. We provide a representation view of the latent variable models for state-action value functions, which allows both tractable variational learning algorithm and effective implementation of the optimism/pessimism principle. In particular, we propose a computationally efficient planning algorithm with UCB exploration by incorporating kernel embeddings of latent variable models.
  arXiv  Detail & Related papers  (2022-12-17T00:26:31Z)
• On Duality Gap as a Measure for Monitoring GAN Training [2.733700237741334]
  Generative adversarial network (GAN) is among the most popular deep learning models for learning complex data distributions. This paper presents a theoretical understanding of this limitation and proposes a more dependable estimation process for the duality gap.
  arXiv  Detail & Related papers  (2020-12-12T04:32:52Z)
• Extrapolation for Large-batch Training in Deep Learning [72.61259487233214]
  We show that a host of variations can be covered in a unified framework that we propose. We prove the convergence of this novel scheme and rigorously evaluate its empirical performance on ResNet, LSTM, and Transformer.
  arXiv  Detail & Related papers  (2020-06-10T08:22:41Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.