An Adaptive Deep RL Method for Non-Stationary Environments with Piecewise Stable Context

• URL: http://arxiv.org/abs/2212.12735v1
• Date: Sat, 24 Dec 2022 13:43:39 GMT
• Title: An Adaptive Deep RL Method for Non-Stationary Environments with Piecewise Stable Context
• Abstract summary: Existing works on adaptation to unknown environment contexts either assume the contexts are the same for the whole episode or assume the context variables are Markovian. In this paper, we propose a textittextbfSegmented textbfContext textbfBelief textbfAugmented textbfDeep(SeCBAD) RL method. Our method can jointly infer the belief distribution over latent context with the posterior over segment length and perform more accurate belief context inference with observed data within the current
• Score: 109.49663559151377
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: One of the key challenges in deploying RL to real-world applications is to adapt to variations of unknown environment contexts, such as changing terrains in robotic tasks and fluctuated bandwidth in congestion control. Existing works on adaptation to unknown environment contexts either assume the contexts are the same for the whole episode or assume the context variables are Markovian. However, in many real-world applications, the environment context usually stays stable for a stochastic period and then changes in an abrupt and unpredictable manner within an episode, resulting in a segment structure, which existing works fail to address. To leverage the segment structure of piecewise stable context in real-world applications, in this paper, we propose a \textit{\textbf{Se}gmented \textbf{C}ontext \textbf{B}elief \textbf{A}ugmented \textbf{D}eep~(SeCBAD)} RL method. Our method can jointly infer the belief distribution over latent context with the posterior over segment length and perform more accurate belief context inference with observed data within the current context segment. The inferred belief context can be leveraged to augment the state, leading to a policy that can adapt to abrupt variations in context. We demonstrate empirically that SeCBAD can infer context segment length accurately and outperform existing methods on a toy grid world environment and Mujuco tasks with piecewise-stable context.

Related papers

• Dynamic Context Scheduling: Learning Beyond the Static Universe [3.5178680418107464]
  We study dynamic context scheduling as a training instrument for contextual re- inforcement learning.<n>We introduce DYNAMICCARLENV, a framework that wraps contextual environments with pluggable schedule families.<n>Preliminary findings indicate that automatic search for multi-stage curricula can successfully discover schedules that improve generalization.
  arXiv  Detail & Related papers  (2026-08-21T07:17:19Z)
• Information Abundance Paradox: Long-Context Training Undermines Parametric Knowledge [72.54679781634037]
  We study how the context window shapes a model's mode of learning, shifting it between parametric internalization and contextualization.<n>We propose the Information Paradox, which hypothesizes that abundant relevant information in the training context can reduce the incentive to encode that information parametrically.
  arXiv  Detail & Related papers  (2026-08-12T16:13:05Z)
• EnvACE: Internalizing Environment Dynamics via World Rehearsal for Agentic Reinforcement Learning [57.312638079204255]
  Training large language model agents for long-horizon tool use typically relies on interactions with real or synthesized executable environments.<n>We introduce EnvACE, an agentic reinforcement learning method that replaces external environment interaction during training with world rehearsal.
  arXiv  Detail & Related papers  (2026-08-06T15:54:36Z)
• Every Client Is an Environment: Federated De-confounding for Spatio-Temporal Forecasting [39.11762381290371]
  We propose method, a novel federated de-confounding framework that textbfs clients as distinct causal.<n>We derive a theoretical federated de-confounding bound that is linearly controlled by the averaged confounding strength.<n>Experiments demonstrate method consistently outperforms federated baselines, while providing transferable, interpretable, and communication-efficient environmental representations.
  arXiv  Detail & Related papers  (2026-07-27T09:50:08Z)
• ACE: Pluggable Adaptive Context Elasticizer across Agents [65.45147906718154]
  Adaptive Context Elasticizer (ACE) orchestrates historical step information into the agent's context at each decision step.<n>ACE consistently outperforms truncation and summarization baselines.
  arXiv  Detail & Related papers  (2026-06-30T12:20:45Z)
• Agent-BRACE: Decoupling Beliefs from Actions in Long-Horizon Tasks via Verbalized State Uncertainty [70.43119366710778]
  We introduce Agent-BRACE: Agent Belief state Representation via Abstraction and Confidence Estimation.<n>It is a method that decouples an LLM agent into a belief state model and a policy model, jointly optimized via reinforcement learning.<n>It achieves an average absolute improvement of +14.5% across long-horizon, partially observable language environments.
  arXiv  Detail & Related papers  (2026-05-12T02:37:04Z)
• Contextual Safety Reasoning and Grounding for Open-World Robots [79.98924225712668]
  CORE is a safety framework that enables online contextual reasoning, grounding, and enforcement without prior knowledge of the environment.<n>We provide probabilistic safety guarantees for CORE that account for perceptual uncertainty.<n>We demonstrate through simulation and real-world experiments that CORE enforces contextually appropriate behavior in unseen environments.
  arXiv  Detail & Related papers  (2026-02-23T15:51:23Z)
• LOCA-bench: Benchmarking Language Agents Under Controllable and Extreme Context Growth [32.1520194112537]
  Large language models (LLMs) are increasingly capable of carrying out long-running, real-world tasks.<n>As the amount of context grows, their reliability often deteriorates, a phenomenon known as "context rot"
  arXiv  Detail & Related papers  (2026-02-08T13:20:39Z)
• Gated Differentiable Working Memory for Long-Context Language Modeling [80.27483324685434]
  We propose Gdwm (Gated Differentiable Working Memory), a framework that introduces a write controller to gate the consolidation process.<n>Experiments on ZeroSCROLLS and LongBench v2 demonstrate that Gdwm achieves comparable or superior performance with 4$times$ fewer gradient steps than uniform baselines.
  arXiv  Detail & Related papers  (2026-01-19T10:00:33Z)
• Grounding Long-Context Reasoning with Contextual Normalization for Retrieval-Augmented Generation [57.97548022208733]
  We show that seemingly superficial choices in key-value extraction can induce shifts in accuracy and stability.<n>We introduce Contextual Normalization, a strategy that adaptively standardizes context representations before generation.
  arXiv  Detail & Related papers  (2025-10-15T06:28:25Z)
• Dynamics-Aligned Latent Imagination in Contextual World Models for Zero-Shot Generalization [1.6332728502735252]
  We introduce Dynamics-Aligned Latent Imagination (DALI), a framework that infers latent context representations from agent-environment interactions.<n>DALI generates actionable representations conditioning the world model and policy, bridging perception and control.<n>On challenging cMDP benchmarks, DALI achieves significant gains over context-unaware baselines.
  arXiv  Detail & Related papers  (2025-08-27T22:02:56Z)
• Zero-Shot Context Generalization in Reinforcement Learning from Few Training Contexts [1.5020330976600738]
  We introduce context-enhanced Bellman equation (CEBE) to improve generalization when training on a single context.<n>We derive context sample enhancement (CSE) as an efficient data augmentation method for approximating the CEBE in deterministic control environments.
  arXiv  Detail & Related papers  (2025-07-10T00:23:13Z)
• Parallel Key-Value Cache Fusion for Position Invariant RAG [55.9809686190244]
  Large Language Models (LLMs) are sensitive to the position of relevant information within contexts.<n>We introduce a framework that generates consistent outputs for decoder-only models, irrespective of the input context order.
  arXiv  Detail & Related papers  (2025-01-13T17:50:30Z)
• ContextDet: Temporal Action Detection with Adaptive Context Aggregation [47.84334557998388]
  We introduce a single-stage ContextDet framework for temporal action detection (TAD) Our model features a pyramid adaptive context aggragation (ACA) architecture, capturing long context and improving action discriminability. By varying the length of these large kernels across the ACA pyramid, our model provides lightweight yet effective context aggregation and action discrimination.
  arXiv  Detail & Related papers  (2024-10-20T04:28:19Z)
• Spatial Semantic Recurrent Mining for Referring Image Segmentation [63.34997546393106]
  We propose Stextsuperscript2RM to achieve high-quality cross-modality fusion. It follows a working strategy of trilogy: distributing language feature, spatial semantic recurrent coparsing, and parsed-semantic balancing. Our proposed method performs favorably against other state-of-the-art algorithms.
  arXiv  Detail & Related papers  (2024-05-15T00:17:48Z)
• Context is Environment [45.88558331853988]
  Researchers should consider environment as context, and harness the power of learning. Researchers in domains should consider context as environment to better structure data towards adaptive learning.
  arXiv  Detail & Related papers  (2023-09-18T15:51:27Z)
• State Regularized Policy Optimization on Data with Dynamics Shift [25.412472472457324]
  In many real-world scenarios, Reinforcement Learning (RL) algorithms are trained on data with dynamics shift, i.e., with different underlying environment dynamics. In this paper, we find that in many environments with similar structures and different dynamics, optimal policies have similar stationary state distributions. Such distribution is used to regularize the policy trained in a new environment, leading to the SRPO (textbfS textbfRegularized textbfPolicy textbfOptimization) algorithm.
  arXiv  Detail & Related papers  (2023-06-06T10:06:09Z)
• Context-aware Fine-tuning of Self-supervised Speech Models [56.95389222319555]
  We study the use of context, i.e., surrounding segments, during fine-tuning. We propose a new approach called context-aware fine-tuning. We evaluate the proposed approach using the SLUE and Libri-light benchmarks for several downstream tasks.
  arXiv  Detail & Related papers  (2022-12-16T15:46:15Z)
• Dynamics-Adaptive Continual Reinforcement Learning via Progressive Contextualization [29.61829620717385]
  Key challenge of continual reinforcement learning (CRL) in dynamic environments is to promptly adapt the RL agent's behavior as the environment changes over its lifetime. DaCoRL learns a context-conditioned policy using progressive contextualization. DaCoRL features consistent superiority over existing methods in terms of the stability, overall performance and generalization ability.
  arXiv  Detail & Related papers  (2022-09-01T10:26:58Z)
• AACC: Asymmetric Actor-Critic in Contextual Reinforcement Learning [13.167123175701802]
  This paper formalizes the task of adapting to changing environmental dynamics in Reinforcement Learning (RL) We then propose the Asymmetric Actor-Critic in Contextual RL (AACC) as an end-to-end actor-critic method to deal with such generalization tasks. We demonstrate the essential improvements in the performance of AACC over existing baselines experimentally in a range of simulated environments.
  arXiv  Detail & Related papers  (2022-08-03T22:52:26Z)
• RoME: Role-aware Mixture-of-Expert Transformer for Text-to-Video Retrieval [66.2075707179047]
  We propose a novel mixture-of-expert transformer RoME that disentangles the text and the video into three levels. We utilize a transformer-based attention mechanism to fully exploit visual and text embeddings at both global and local levels. Our method outperforms the state-of-the-art methods on the YouCook2 and MSR-VTT datasets.
  arXiv  Detail & Related papers  (2022-06-26T11:12:49Z)
• Reinforcement Learning in Presence of Discrete Markovian Context Evolution [7.467644044726776]
  We consider a context-dependent Reinforcement Learning setting, which is characterized by: a) an unknown finite number of not directly observable contexts; b) abrupt (discontinuous) context changes occurring during an episode; and c) Markovian context evolution. We adapt a sticky Hierarchical Dirichlet Process (HDP) prior for model learning, which is arguably best-suited for Markov process modeling. We argue that the combination of these two components allows to infer the number of contexts from data thus dealing with the context cardinality assumption.
  arXiv  Detail & Related papers  (2022-02-14T08:52:36Z)
• Invariant Causal Prediction for Block MDPs [106.63346115341862]
  Generalization across environments is critical to the successful application of reinforcement learning algorithms to real-world challenges. We propose a method of invariant prediction to learn model-irrelevance state abstractions (MISA) that generalize to novel observations in the multi-environment setting.
  arXiv  Detail & Related papers  (2020-03-12T21:03:01Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.