Learning to Perceive in Deep Model-Free Reinforcement Learning

• URL: http://arxiv.org/abs/2301.03730v1
• Date: Tue, 10 Jan 2023 00:31:57 GMT
• Title: Learning to Perceive in Deep Model-Free Reinforcement Learning
• Abstract summary: This work proposes a novel model-free Reinforcement Learning (RL) agent that is able to learn how to complete an unknown task having access to only a part of the input observation. We take inspiration from the concepts of visual attention and active perception that are characteristic of humans and tried to apply them to our agent, creating a hard attention mechanism.
• Score: 1.2891210250935146
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: This work proposes a novel model-free Reinforcement Learning (RL) agent that is able to learn how to complete an unknown task having access to only a part of the input observation. We take inspiration from the concepts of visual attention and active perception that are characteristic of humans and tried to apply them to our agent, creating a hard attention mechanism. In this mechanism, the model decides first which region of the input image it should look at, and only after that it has access to the pixels of that region. Current RL agents do not follow this principle and we have not seen these mechanisms applied to the same purpose as this work. In our architecture, we adapt an existing model called recurrent attention model (RAM) and combine it with the proximal policy optimization (PPO) algorithm. We investigate whether a model with these characteristics is capable of achieving similar performance to state-of-the-art model-free RL agents that access the full input observation. This analysis is made in two Atari games, Pong and SpaceInvaders, which have a discrete action space, and in CarRacing, which has a continuous action space. Besides assessing its performance, we also analyze the movement of the attention of our model and compare it with what would be an example of the human behavior. Even with such visual limitation, we show that our model matches the performance of PPO+LSTM in two of the three games tested.

Related papers

• Dissecting model behavior through agent trajectories [15.811597127775812]
  We formalize the intent-execution' gap: the mismatch between what the model intends and what the harness executes, and vice versa.<n>To illustrate the impact of this harness-model alignment, we develop a simple and customizable harness called Simple Strands Agent' (SSA)<n>We make two contributions: (i) we reproduce or improve on the pass@1 performance reported by diverse model-provider families on popular agentic benchmarks, and (ii) building on an analysis of 138k trajectories generated by SSA.
  arXiv  Detail & Related papers  (2026-06-16T03:17:03Z)
• A Lightweight Library for Energy-Based Joint-Embedding Predictive Architectures [58.26804959656713]
  We present EB-JEPA, an open-source library for learning representations and world models using Joint-Embedding Predictive Architectures (JEPAs)<n>JEPAs learn to predict in representation space rather than pixel space, avoiding the pitfalls of generative modeling.<n>We show how these representations can drive action-conditioned world models, achieving a 97% planning success rate on the Two Rooms navigation task.
  arXiv  Detail & Related papers  (2026-02-03T14:56:24Z)
• Combining Pre-Trained Models for Enhanced Feature Representation in Reinforcement Learning [16.04558746520946]
  Reinforcement Learning (RL) focuses on maximizing the cumulative reward obtained via agent's interaction with the environment.<n>We propose Weight Sharing Attention (WSA), a new architecture to combine embeddings of multiple pre-trained models to shape an enriched state representation.
  arXiv  Detail & Related papers  (2025-07-09T18:13:52Z)
• SOLD: Reinforcement Learning with Slot Object-Centric Latent Dynamics [16.020835290802548]
  Slot-Attention for Object-centric Latent Dynamics is a novel algorithm that learns object-centric dynamics models from pixel inputs. We demonstrate that the structured latent space not only improves model interpretability but also provides a valuable input space for behavior models to reason over. Our results show that SOLD outperforms DreamerV3, a state-of-the-art model-based RL algorithm, across a range of benchmark robotic environments.
  arXiv  Detail & Related papers  (2024-10-11T14:03:31Z)
• Early Action Recognition with Action Prototypes [62.826125870298306]
  We propose a novel model that learns a prototypical representation of the full action for each class. We decompose the video into short clips, where a visual encoder extracts features from each clip independently. Later, a decoder aggregates together in an online fashion features from all the clips for the final class prediction.
  arXiv  Detail & Related papers  (2023-12-11T18:31:13Z)
• Contrastive Value Learning: Implicit Models for Simple Offline RL [40.95632543012637]
  We propose Contrastive Value Learning (CVL), which learns an implicit, multi-step model of the environment dynamics. CVL can be learned without access to reward functions, but nonetheless can be used to directly estimate the value of each action. Our experiments demonstrate that CVL outperforms prior offline RL methods on complex continuous control benchmarks.
  arXiv  Detail & Related papers  (2022-11-03T19:10:05Z)
• H-SAUR: Hypothesize, Simulate, Act, Update, and Repeat for Understanding Object Articulations from Interactions [62.510951695174604]
  "Hypothesize, Simulate, Act, Update, and Repeat" (H-SAUR) is a probabilistic generative framework that generates hypotheses about how objects articulate given input observations. We show that the proposed model significantly outperforms the current state-of-the-art articulated object manipulation framework. We further improve the test-time efficiency of H-SAUR by integrating a learned prior from learning-based vision models.
  arXiv  Detail & Related papers  (2022-10-22T18:39:33Z)
• REST: REtrieve & Self-Train for generative action recognition [54.90704746573636]
  We propose to adapt a pre-trained generative Vision & Language (V&L) Foundation Model for video/action recognition. We show that direct fine-tuning of a generative model to produce action classes suffers from severe overfitting. We introduce REST, a training framework consisting of two key components.
  arXiv  Detail & Related papers  (2022-09-29T17:57:01Z)
• Simplifying Model-based RL: Learning Representations, Latent-space Models, and Policies with One Objective [142.36200080384145]
  We propose a single objective which jointly optimize a latent-space model and policy to achieve high returns while remaining self-consistent. We demonstrate that the resulting algorithm matches or improves the sample-efficiency of the best prior model-based and model-free RL methods.
  arXiv  Detail & Related papers  (2022-09-18T03:51:58Z)
• INFOrmation Prioritization through EmPOWERment in Visual Model-Based RL [90.06845886194235]
  We propose a modified objective for model-based reinforcement learning (RL) We integrate a term inspired by variational empowerment into a state-space model based on mutual information. We evaluate the approach on a suite of vision-based robot control tasks with natural video backgrounds.
  arXiv  Detail & Related papers  (2022-04-18T23:09:23Z)
• STAR: Sparse Transformer-based Action Recognition [61.490243467748314]
  This work proposes a novel skeleton-based human action recognition model with sparse attention on the spatial dimension and segmented linear attention on the temporal dimension of data. Experiments show that our model can achieve comparable performance while utilizing much less trainable parameters and achieve high speed in training and inference.
  arXiv  Detail & Related papers  (2021-07-15T02:53:11Z)
• Learning intuitive physics and one-shot imitation using state-action-prediction self-organizing maps [0.0]
  Humans learn by exploration and imitation, build causal models of the world, and use both to flexibly solve new tasks. We suggest a simple but effective unsupervised model which develops such characteristics. We demonstrate its performance on a set of several related, but different one-shot imitation tasks, which the agent flexibly solves in an active inference style.
  arXiv  Detail & Related papers  (2020-07-03T12:29:11Z)
• A Neuromorphic Proto-Object Based Dynamic Visual Saliency Model with an FPGA Implementation [1.2387676601792899]
  We present a neuromorphic, bottom-up, dynamic visual saliency model based on the notion of proto-objects. This model outperforms state-of-the-art dynamic visual saliency models in predicting human eye fixations on a commonly used video dataset. We introduce a Field-Programmable Gate Array implementation of the model on an Opal Kelly 7350 Kintex-7 board.
  arXiv  Detail & Related papers  (2020-02-27T03:31:56Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.