Towards Control-Centric Representations in Reinforcement Learning from Images

• URL: http://arxiv.org/abs/2310.16655v2
• Date: Fri, 27 Oct 2023 10:14:16 GMT
• Title: Towards Control-Centric Representations in Reinforcement Learning from Images
• Abstract summary: ReBis aims to capture control-centric information by integrating reward-free control information alongside reward-specific knowledge. Empirical studies on two large benchmarks, including Atari games and DeepMind Control Suit, demonstrate that ReBis has superior performance compared to existing methods.
• Score: 43.21376020617722
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Image-based Reinforcement Learning is a practical yet challenging task. A major hurdle lies in extracting control-centric representations while disregarding irrelevant information. While approaches that follow the bisimulation principle exhibit the potential in learning state representations to address this issue, they still grapple with the limited expressive capacity of latent dynamics and the inadaptability to sparse reward environments. To address these limitations, we introduce ReBis, which aims to capture control-centric information by integrating reward-free control information alongside reward-specific knowledge. ReBis utilizes a transformer architecture to implicitly model the dynamics and incorporates block-wise masking to eliminate spatiotemporal redundancy. Moreover, ReBis combines bisimulation-based loss with asymmetric reconstruction loss to prevent feature collapse in environments with sparse rewards. Empirical studies on two large benchmarks, including Atari games and DeepMind Control Suit, demonstrate that ReBis has superior performance compared to existing methods, proving its effectiveness.

Related papers

• ARISE-RL: Agentic Rubric-Grounded Iterative Self-Evolution with Reinforcement Learning [74.19730275669227]
  ARISE-RL is a novel full-cycle self-evolution framework for training open-ended agents.<n>We present Reward-Gated Self-Evolution Distillation (RG-SED), which distills a memory-augmented variant of the same policy back into itself.<n>We also present ECR-Bench, an expert-calibrated rubric benchmark suite covering single-tool deep research and multi-tool travel planning.
  arXiv  Detail & Related papers  (2026-09-01T10:54:13Z)
• Large Vision-Language Models Get Lost in Attention [51.851592109135716]
  We propose a unified framework grounded in information theory and geometry to quantify the geometric and entropic nature of residual updates.<n>Applying this unified framework reveals a fundamental functional decoupling: Attention acts as a subspace-language operator focused on reconfiguration, whereas FFNs serve as subspace-language operators driving semantic innovation.
  arXiv  Detail & Related papers  (2026-05-07T04:45:52Z)
• Agent-Centric Visual Reinforcement Learning under Dynamic Perturbations [24.021326345297535]
  We introduce the Visual Degraded Control Suite (VDCS) to simulate non-stationary real-world perturbations.<n>Experiments on VDCS reveal severe performance degradation in existing methods.<n>We propose Agent-Centric Observations with Mixture-of-Experts (ACO-MoE) to robustify visual RL against perturbations.
  arXiv  Detail & Related papers  (2026-04-27T16:24:40Z)
• ARES: Scalable and Practical Gradient Inversion Attack in Federated Learning through Activation Recovery [73.8181449261685]
  Federated Learning (FL) enables collaborative model training by sharing model updates instead of raw data, aiming to protect user privacy.<n>Recent studies reveal that these shared updates can inadvertently leak sensitive training data through gradient inversion attacks (GIAs)
  arXiv  Detail & Related papers  (2026-03-18T11:40:44Z)
• Rethinking Hebbian Principle: Low-Dimensional Structural Projection for Unsupervised Learning [17.299267108673277]
  Hebbian learning is a biological principle that intuitively describes how neurons adapt their connections through repeated stimuli.<n>We introduce the Structural Projection Hebbian Representation (SPHeRe), a novel unsupervised learning method.<n> Experimental results show that SPHeRe achieves SOTA performance among unsupervised synaptic plasticity approaches.
  arXiv  Detail & Related papers  (2025-10-16T15:47:29Z)
• Single-Reset Divide & Conquer Imitation Learning [49.87201678501027]
  Demonstrations are commonly used to speed up the learning process of Deep Reinforcement Learning algorithms. Some algorithms have been developed to learn from a single demonstration.
  arXiv  Detail & Related papers  (2024-02-14T17:59:47Z)
• Learning Interpretable Policies in Hindsight-Observable POMDPs through Partially Supervised Reinforcement Learning [57.67629402360924]
  We introduce the Partially Supervised Reinforcement Learning (PSRL) framework. At the heart of PSRL is the fusion of both supervised and unsupervised learning. We show that PSRL offers a potent balance, enhancing model interpretability while preserving, and often significantly outperforming, the performance benchmarks set by traditional methods.
  arXiv  Detail & Related papers  (2024-02-14T16:23:23Z)
• Understanding and Addressing the Pitfalls of Bisimulation-based Representations in Offline Reinforcement Learning [34.66035026036424]
  We aim to understand why bisimulation methods succeed in online settings, but falter in offline tasks. Our analysis reveals that missing transitions in the dataset are particularly harmful to the bisimulation principle. We implement these recommendations on two state-of-the-art bisimulation-based algorithms, MICo and SimSR.
  arXiv  Detail & Related papers  (2023-10-26T04:20:55Z)
• Sequential Action-Induced Invariant Representation for Reinforcement Learning [1.2046159151610263]
  How to accurately learn task-relevant state representations from high-dimensional observations with visual distractions is a challenging problem in visual reinforcement learning. We propose a Sequential Action-induced invariant Representation (SAR) method, in which the encoder is optimized by an auxiliary learner to only preserve the components that follow the control signals of sequential actions.
  arXiv  Detail & Related papers  (2023-09-22T05:31:55Z)
• VIBR: Learning View-Invariant Value Functions for Robust Visual Control [3.2307366446033945]
  VIBR (View-Invariant Bellman Residuals) is a method that combines multi-view training and invariant prediction to reduce out-of-distribution gap for RL based visuomotor control. We show that VIBR outperforms existing methods on complex visuo-motor control environment with high visual perturbation.
  arXiv  Detail & Related papers  (2023-06-14T14:37:34Z)
• Combining Reconstruction and Contrastive Methods for Multimodal Representations in RL [16.792949555151978]
  Learning self-supervised representations using reconstruction or contrastive losses improves performance and sample complexity of image-based and multimodal reinforcement learning (RL) Here, different self-supervised loss functions have distinct advantages and limitations depending on the information density of the underlying sensor modality. We propose Contrastive Reconstructive Aggregated representation Learning (CoRAL), a unified framework enabling us to choose the most appropriate self-supervised loss for each sensor modality.
  arXiv  Detail & Related papers  (2023-02-10T15:57:20Z)
• DR3: Value-Based Deep Reinforcement Learning Requires Explicit Regularization [125.5448293005647]
  We discuss how the implicit regularization effect of SGD seen in supervised learning could in fact be harmful in the offline deep RL. Our theoretical analysis shows that when existing models of implicit regularization are applied to temporal difference learning, the resulting derived regularizer favors degenerate solutions. We propose a simple and effective explicit regularizer, called DR3, that counteracts the undesirable effects of this implicit regularizer.
  arXiv  Detail & Related papers  (2021-12-09T06:01:01Z)
• Residual Reinforcement Learning from Demonstrations [51.56457466788513]
  Residual reinforcement learning (RL) has been proposed as a way to solve challenging robotic tasks by adapting control actions from a conventional feedback controller to maximize a reward signal. We extend the residual formulation to learn from visual inputs and sparse rewards using demonstrations. Our experimental evaluation on simulated manipulation tasks on a 6-DoF UR5 arm and a 28-DoF dexterous hand demonstrates that residual RL from demonstrations is able to generalize to unseen environment conditions more flexibly than either behavioral cloning or RL fine-tuning.
  arXiv  Detail & Related papers  (2021-06-15T11:16:49Z)
• Return-Based Contrastive Representation Learning for Reinforcement Learning [126.7440353288838]
  We propose a novel auxiliary task that forces the learnt representations to discriminate state-action pairs with different returns. Our algorithm outperforms strong baselines on complex tasks in Atari games and DeepMind Control suite.
  arXiv  Detail & Related papers  (2021-02-22T13:04:18Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.