Streaming Reinforcement Learning under Partial Observability with Real-Time Recurrent Learning

• URL: http://arxiv.org/abs/2605.24709v1
• Date: Sat, 23 May 2026 19:25:30 GMT
• Title: Streaming Reinforcement Learning under Partial Observability with Real-Time Recurrent Learning
• Abstract summary: Streaming reinforcement learning has emerged as an online learning paradigm that conforms to the restrictions of natural learning agents that process data incrementally.<n>We propose a diagonal recurrent architecture that enables exact RTRL with linear time and memory complexity in the parameter count.<n>We show that our method recovers a substantial fraction of batched performance on masked MuJoCo, despite using no replay buffer or batched updates.
• Score: 22.705756039708035
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Streaming reinforcement learning has emerged as an online learning paradigm that conforms to the restrictions of natural learning agents that process data incrementally, i.e. with a batch size of 1 and no replay buffer. While streaming RL has recently been shown to scale with deep function approximation with full observability, partially observable settings have remained out of reach. Truncated backpropagation through time collapses to a one-step gradient horizon under the streaming setting, and exact real-time recurrent learning is prohibitively expensive. We close this gap using recurrent trace units, a diagonal recurrent architecture that enables exact RTRL with linear time and memory complexity in the parameter count, and show that they integrate cleanly into existing streaming algorithms across both discrete and continuous control. On a MemoryChain diagnostic with chain lengths from 2 to 128, our method sustains performance where streaming TBPTT(1) baselines using feedforward, GRU, and RTU networks collapse. On five POPGym tasks and on partially observable MuJoCo continuous control, the streaming approach is competitive with batched PPO on POPGym and recovers a substantial fraction of batched performance on masked MuJoCo, despite using no replay buffer or batched updates.

Related papers

• TRaM-VSR: Importance-Aware Token Routing and Merging for One-Step Diffusion Video Super-Resolution [52.96050253456634]
  TRaM-VSR is a framework for adaptive token allocation using context-aware video priors and network-levels.<n>It is shown that TRaM-VSR accelerates inference significantly while preserving state-of-the-art reconstruction quality and robust temporal consistency.
  arXiv  Detail & Related papers  (2026-07-24T11:57:26Z)
• Surprise Forcing: What to Remember, When to Skip in Long Video Generation [65.0022589149937]
  Streaming autoregressive diffusion makes minute-scale video synthesis practical, but its bounded context and fixed denoising schedule allocate resources uniformly.<n>We introduce Surprise Forcing, a training-free framework that treats both limitations as online resource-allocation problems.<n> Experiments on VBench, VBench-Long, and VBench-2.0 show that the proposed allocation strategy improves long-horizon consistency and visual quality while retaining real-time streaming throughput.
  arXiv  Detail & Related papers  (2026-07-20T18:37:58Z)
• Latent Recurrent Transformer: Architecture Exploration, Training Strategies, and Scaling Behavior [107.2098567818173]
  Latent Recurrent Transformer (LRT) is a lightweight augmentation of autoregressive transformers.<n>LRT reuses a high-level source-layer hidden state from the previous token as recurrent memory for the next token.
  arXiv  Detail & Related papers  (2026-05-26T10:10:26Z)
• LC-Flow: Learning Local Continuous Optical Flow and Confidence from events [5.146238489510258]
  Event cameras capture brightness changes asynchronously with microsecond resolution, yet existing optical flow methods fail to exploit this temporal continuity.<n>We propose textbfLC-Flow, the first temporally continuous, learning-based optical flow estimator that operates purely from local events.
  arXiv  Detail & Related papers  (2026-05-23T14:42:24Z)
• RAVEN: Real-time Autoregressive Video Extrapolation with Consistency-model GRPO [53.38929612273108]
  Causal autoregressive video diffusion models support real-time streaming generation by extrapolating future chunks from previously generated content.<n>We introduce the Real-time Autoregressive Video Extrapolation Network (RAVEN), a training-time test framework that repacks each self rollout into an interleaved sequence of clean historical endpoints.
  arXiv  Detail & Related papers  (2026-05-14T17:59:30Z)
• TSN-Affinity: Similarity-Driven Parameter Reuse for Continual Offline Reinforcement Learning [5.680044533158534]
  Continual offline reinforcement learning (CORL) aims to learn a sequence of tasks from datasets collected over time while preserving performance on previously learned tasks.<n>We propose TSN-Affinity, a novel CORL method based on TinySubNetworks and Decision Transformer.<n>We evaluate the approach on benchmarks based on Atari games and simulations of manipulation tasks with the Franka Emika Panda robotic arm.
  arXiv  Detail & Related papers  (2026-04-28T17:41:04Z)
• Squeezing More from the Stream : Learning Representation Online for Streaming Reinforcement Learning [14.799267729619428]
  In streaming Reinforcement Learning (RL), transitions are observed and discarded immediately after a single update.<n>We propose extending Self-Predictive Representations (SPR) to the streaming pipeline to maximize the utility of every observed frame.<n>We show that our method learns significantly richer representations, bridging the performance gap caused by the absence of a replay buffer.
  arXiv  Detail & Related papers  (2026-02-10T04:06:32Z)
• Trust Region Continual Learning as an Implicit Meta-Learner [3.705371747297478]
  We study a hybrid perspective: emphtrust region continual learning that combines generative replay with a Fisher-metric trust region constraint.<n>We show that, under local approximations, the resulting update admits a MAML-style interpretation with a single implicit inner step.<n>This yields an emergent meta-learning property in continual learning.
  arXiv  Detail & Related papers  (2026-02-02T18:19:16Z)
• StreamRL: Scalable, Heterogeneous, and Elastic RL for LLMs with Disaggregated Stream Generation [55.75008325187133]
  Reinforcement learning (RL) has become the core post-training technique for large language models (LLMs)<n>StreamRL is designed with disaggregation from first principles to address two types of performance bottlenecks.<n> Experiments show that StreamRL improves throughput by up to 2.66x compared to existing state-of-the-art systems.
  arXiv  Detail & Related papers  (2025-04-22T14:19:06Z)
• Trajectory Balance with Asynchrony: Decoupling Exploration and Learning for Fast, Scalable LLM Post-Training [71.16258800411696]
  Reinforcement learning (RL) is a critical component of large language model (LLM) post-training.<n>Existing on-policy algorithms used for post-training are inherently incompatible with the use of experience replay buffers.<n>We propose efficiently obtaining this benefit of replay buffers via Trajectory Balance with Asynchrony (TBA)
  arXiv  Detail & Related papers  (2025-03-24T17:51:39Z)
• ResFlow: Fine-tuning Residual Optical Flow for Event-based High Temporal Resolution Motion Estimation [50.80115710105251]
  Event cameras hold significant promise for high-temporal-resolution (HTR) motion estimation.<n>We propose a residual-based paradigm for estimating HTR optical flow with event data.
  arXiv  Detail & Related papers  (2024-12-12T09:35:47Z)
• Streaming Deep Reinforcement Learning Finally Works [6.106447284305318]
  Streaming learning is ideal for resource-limited, communication-constrained, and privacy-sensitive applications.<n>In deep RL, learners almost always use batch updates and replay buffers, making them computationally expensive and incompatible with streaming learning.<n>This paper introduces the stream-x algorithms, the first class of deep RL algorithms to overcome stream barrier for both prediction and control and match sample efficiency of batch RL.
  arXiv  Detail & Related papers  (2024-10-18T17:00:29Z)
• Structured Sparsity Learning for Efficient Video Super-Resolution [99.1632164448236]
  We develop a structured pruning scheme called Structured Sparsity Learning (SSL) according to the properties of video super-resolution (VSR) models. In SSL, we design pruning schemes for several key components in VSR models, including residual blocks, recurrent networks, and upsampling networks.
  arXiv  Detail & Related papers  (2022-06-15T17:36:04Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.