Streaming Deep Reinforcement Learning Finally Works

• URL: http://arxiv.org/abs/2410.14606v1
• Date: Fri, 18 Oct 2024 17:00:29 GMT
• Title: Streaming Deep Reinforcement Learning Finally Works
• Abstract summary: Streaming learning is ideal for resource-limited, communication-constrained, and privacy-sensitive applications. In deep RL, learners almost always use batch updates and replay buffers, making them computationally expensive and incompatible with streaming learning. This paper introduces the stream-x algorithms, the first class of deep RL algorithms to overcome stream barrier for both prediction and control and match sample efficiency of batch RL.
• Score: 6.106447284305318
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Natural intelligence processes experience as a continuous stream, sensing, acting, and learning moment-by-moment in real time. Streaming learning, the modus operandi of classic reinforcement learning (RL) algorithms like Q-learning and TD, mimics natural learning by using the most recent sample without storing it. This approach is also ideal for resource-constrained, communication-limited, and privacy-sensitive applications. However, in deep RL, learners almost always use batch updates and replay buffers, making them computationally expensive and incompatible with streaming learning. Although the prevalence of batch deep RL is often attributed to its sample efficiency, a more critical reason for the absence of streaming deep RL is its frequent instability and failure to learn, which we refer to as stream barrier. This paper introduces the stream-x algorithms, the first class of deep RL algorithms to overcome stream barrier for both prediction and control and match sample efficiency of batch RL. Through experiments in Mujoco Gym, DM Control Suite, and Atari Games, we demonstrate stream barrier in existing algorithms and successful stable learning with our stream-x algorithms: stream Q, stream AC, and stream TD, achieving the best model-free performance in DM Control Dog environments. A set of common techniques underlies the stream-x algorithms, enabling their success with a single set of hyperparameters and allowing for easy extension to other algorithms, thereby reviving streaming RL.

Related papers

• Reversal Q-Learning [57.43956630070019]
  Reversal Q-learning is an off-policy RL algorithm that trains a flow policy based on prior data.<n>We show that RQL leads to the best average offline RL performance compared to state-of-the-art flow-based offline RL algorithms.
  arXiv  Detail & Related papers  (2026-06-16T05:56:10Z)
• Streaming Reinforcement Learning under Partial Observability with Real-Time Recurrent Learning [22.705756039708035]
  Streaming reinforcement learning has emerged as an online learning paradigm that conforms to the restrictions of natural learning agents that process data incrementally.<n>We propose a diagonal recurrent architecture that enables exact RTRL with linear time and memory complexity in the parameter count.<n>We show that our method recovers a substantial fraction of batched performance on masked MuJoCo, despite using no replay buffer or batched updates.
  arXiv  Detail & Related papers  (2026-05-23T19:25:30Z)
• FASTER: Value-Guided Sampling for Fast RL [103.55398181003262]
  FASTER is a method for getting the benefits of sampling-based test-time scaling of diffusion-based policies without the computational cost.<n>FASTER achieves the same performance while substantially reducing training and inference compute requirements.
  arXiv  Detail & Related papers  (2026-04-21T17:52:17Z)
• Towards Batch-to-Streaming Deep Reinforcement Learning for Continuous Control [7.726701007187284]
  State-of-the-art deep reinforcement learning (RL) methods have achieved remarkable performance in continuous control tasks.<n>Streaming deep RL addresses this limitation through purely online updates, achieving strong empirical performance on standard benchmarks.<n>We propose two novel streaming deep RL algorithms, Streaming Soft Actor-Critic (S2AC) and Streaming Deterministic Actor-Critic (SDAC)
  arXiv  Detail & Related papers  (2026-03-09T16:40:06Z)
• PROL : Rehearsal Free Continual Learning in Streaming Data via Prompt Online Learning [17.230781041043823]
  We propose a novel prompt-based method for online continual learning (OCL) that includes 4 main components.<n>Our proposed method achieves significantly higher performance than the current SOTAs in CIFAR100, ImageNet-R, ImageNet-A, and CUB dataset.
  arXiv  Detail & Related papers  (2025-07-16T15:04:46Z)
• Filtering Learning Histories Enhances In-Context Reinforcement Learning [12.697029805927398]
  Transformer models (TMs) have exhibited remarkable in-context reinforcement learning capabilities.<n>We propose a simple yet effective approach, learning history filtering (LHF) to enhance ICRL.<n>LHF is the first approach to avoid source suboptimality by dataset preprocessing.
  arXiv  Detail & Related papers  (2025-05-21T06:00:41Z)
• StreamRL: Scalable, Heterogeneous, and Elastic RL for LLMs with Disaggregated Stream Generation [55.75008325187133]
  Reinforcement learning (RL) has become the core post-training technique for large language models (LLMs) StreamRL is designed with disaggregation from first principles to address two types of performance bottlenecks. Experiments show that StreamRL improves throughput by up to 2.66x compared to existing state-of-the-art systems.
  arXiv  Detail & Related papers  (2025-04-22T14:19:06Z)
• Flow Q-Learning [61.60383927357656]
  We present flow Q-learning (FQL), a simple and performant offline reinforcement learning (RL) method. FQL trains an expressive one-step policy with RL, rather than directly guiding an iterative flow policy to maximize values. We experimentally show that FQL leads to strong performance across 73 challenging state- and pixel-based OGBench and D4RL tasks.
  arXiv  Detail & Related papers  (2025-02-04T18:04:05Z)
• Continuous Control with Coarse-to-fine Reinforcement Learning [15.585706638252441]
  We present a framework that trains RL agents to zoom-into a continuous action space in a coarse-to-fine manner. We introduce a concrete, value-based algorithm within the framework called Coarse-to-fine Q-Network (CQN) CQN robustly learns to solve real-world manipulation tasks within a few minutes of online training.
  arXiv  Detail & Related papers  (2024-07-10T16:04:08Z)
• Simplifying Deep Temporal Difference Learning [3.458933902627673]
  We investigate whether it is possible to accelerate and simplify TD training while maintaining its stability. Our key theoretical result demonstrates for the first time that regularisation techniques such as LayerNorm can yield provably convergent TD algorithms. Motivated by these findings, we propose PQN, our simplified deep online Q-Learning algorithm.
  arXiv  Detail & Related papers  (2024-07-05T18:49:07Z)
• Hybrid Inverse Reinforcement Learning [34.793570631021005]
  inverse reinforcement learning approach to imitation learning is a double-edged sword. We propose using hybrid RL -- training on a mixture of online and expert data -- to curtail unnecessary exploration. We derive both model-free and model-based hybrid inverse RL algorithms with strong policy performance guarantees.
  arXiv  Detail & Related papers  (2024-02-13T23:29:09Z)
• Cold Start Streaming Learning for Deep Networks [13.452510519858995]
  Cold Start Streaming Learning (CSSL) is a simple, end-to-end approach for streaming learning with deep networks. In experiments, we find that CSSL outperforms existing baselines for streaming learning.
  arXiv  Detail & Related papers  (2022-11-09T00:53:19Z)
• Jump-Start Reinforcement Learning [68.82380421479675]
  We present a meta algorithm that can use offline data, demonstrations, or a pre-existing policy to initialize an RL policy. In particular, we propose Jump-Start Reinforcement Learning (JSRL), an algorithm that employs two policies to solve tasks. We show via experiments that JSRL is able to significantly outperform existing imitation and reinforcement learning algorithms.
  arXiv  Detail & Related papers  (2022-04-05T17:25:22Z)
• RvS: What is Essential for Offline RL via Supervised Learning? [77.91045677562802]
  Recent work has shown that supervised learning alone, without temporal difference (TD) learning, can be remarkably effective for offline RL. In every environment suite we consider simply maximizing likelihood with two-layer feedforward is competitive. They also probe the limits of existing RvS methods, which are comparatively weak on random data.
  arXiv  Detail & Related papers  (2021-12-20T18:55:16Z)
• A Workflow for Offline Model-Free Robotic Reinforcement Learning [117.07743713715291]
  offline reinforcement learning (RL) enables learning control policies by utilizing only prior experience, without any online interaction. We develop a practical workflow for using offline RL analogous to the relatively well-understood for supervised learning problems. We demonstrate the efficacy of this workflow in producing effective policies without any online tuning.
  arXiv  Detail & Related papers  (2021-09-22T16:03:29Z)
• Text Generation with Efficient (Soft) Q-Learning [91.47743595382758]
  Reinforcement learning (RL) offers a more flexible solution by allowing users to plug in arbitrary task metrics as reward. We introduce a new RL formulation for text generation from the soft Q-learning perspective. We apply the approach to a wide range of tasks, including learning from noisy/negative examples, adversarial attacks, and prompt generation.
  arXiv  Detail & Related papers  (2021-06-14T18:48:40Z)
• Learning Dexterous Manipulation from Suboptimal Experts [69.8017067648129]
  Relative Entropy Q-Learning (REQ) is a simple policy algorithm that combines ideas from successful offline and conventional RL algorithms. We show how REQ is also effective for general off-policy RL, offline RL, and RL from demonstrations.
  arXiv  Detail & Related papers  (2020-10-16T18:48:49Z)
• Decoupling Representation Learning from Reinforcement Learning [89.82834016009461]
  We introduce an unsupervised learning task called Augmented Temporal Contrast (ATC) ATC trains a convolutional encoder to associate pairs of observations separated by a short time difference. In online RL experiments, we show that training the encoder exclusively using ATC matches or outperforms end-to-end RL.
  arXiv  Detail & Related papers  (2020-09-14T19:11:13Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.