Learning to Represent Action Values as a Hypergraph on the Action Vertices

• URL: http://arxiv.org/abs/2010.14680v2
• Date: Sun, 20 Jun 2021 15:14:12 GMT
• Title: Learning to Represent Action Values as a Hypergraph on the Action Vertices
• Authors: Arash Tavakoli, Mehdi Fatemi, Petar Kormushev
• Abstract summary: Action-value estimation is a critical component of reinforcement learning (RL) methods. We conjecture that leveraging the structure of multi-dimensional action spaces is a key ingredient for learning good representations of action. We show the effectiveness of our approach on a myriad of domains: illustrative prediction problems under minimal confounding effects, Atari 2600 games, and discretised physical control benchmarks.
• Score: 17.811355496708728
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Action-value estimation is a critical component of many reinforcement learning (RL) methods whereby sample complexity relies heavily on how fast a good estimator for action value can be learned. By viewing this problem through the lens of representation learning, good representations of both state and action can facilitate action-value estimation. While advances in deep learning have seamlessly driven progress in learning state representations, given the specificity of the notion of agency to RL, little attention has been paid to learning action representations. We conjecture that leveraging the combinatorial structure of multi-dimensional action spaces is a key ingredient for learning good representations of action. To test this, we set forth the action hypergraph networks framework -- a class of functions for learning action representations in multi-dimensional discrete action spaces with a structural inductive bias. Using this framework we realise an agent class based on a combination with deep Q-networks, which we dub hypergraph Q-networks. We show the effectiveness of our approach on a myriad of domains: illustrative prediction problems under minimal confounding effects, Atari 2600 games, and discretised physical control benchmarks.

Related papers

• Reinforcement Learning with Action Sequence for Data-Efficient Robot Learning [62.3886343725955]
  We introduce a novel RL algorithm that learns a critic network that outputs Q-values over a sequence of actions. By explicitly training the value functions to learn the consequence of executing a series of current and future actions, our algorithm allows for learning useful value functions from noisy trajectories.
  arXiv  Detail & Related papers  (2024-11-19T01:23:52Z)
• An Information Compensation Framework for Zero-Shot Skeleton-based Action Recognition [49.45660055499103]
  Zero-shot human skeleton-based action recognition aims to construct a model that can recognize actions outside the categories seen during training. Previous research has focused on aligning sequences' visual and semantic spatial distributions. We introduce a new loss function sampling method to obtain a tight and robust representation.
  arXiv  Detail & Related papers  (2024-06-02T06:53:01Z)
• Sequential Action-Induced Invariant Representation for Reinforcement Learning [1.2046159151610263]
  How to accurately learn task-relevant state representations from high-dimensional observations with visual distractions is a challenging problem in visual reinforcement learning. We propose a Sequential Action-induced invariant Representation (SAR) method, in which the encoder is optimized by an auxiliary learner to only preserve the components that follow the control signals of sequential actions.
  arXiv  Detail & Related papers  (2023-09-22T05:31:55Z)
• TACO: Temporal Latent Action-Driven Contrastive Loss for Visual Reinforcement Learning [73.53576440536682]
  We introduce TACO: Temporal Action-driven Contrastive Learning, a powerful temporal contrastive learning approach. TACO simultaneously learns a state and an action representation by optimizing the mutual information between representations of current states. For online RL, TACO achieves 40% performance boost after one million environment interaction steps.
  arXiv  Detail & Related papers  (2023-06-22T22:21:53Z)
• Proto-Value Networks: Scaling Representation Learning with Auxiliary Tasks [33.98624423578388]
  Auxiliary tasks improve representations learned by deep reinforcement learning agents. We derive a new family of auxiliary tasks based on the successor measure. We show that proto-value networks produce rich features that may be used to obtain performance comparable to established algorithms.
  arXiv  Detail & Related papers  (2023-04-25T04:25:08Z)
• An Empirical Investigation of Representation Learning for Imitation [76.48784376425911]
  Recent work in vision, reinforcement learning, and NLP has shown that auxiliary representation learning objectives can reduce the need for large amounts of expensive, task-specific data. We propose a modular framework for constructing representation learning algorithms, then use our framework to evaluate the utility of representation learning for imitation.
  arXiv  Detail & Related papers  (2022-05-16T11:23:42Z)
• Visuomotor Control in Multi-Object Scenes Using Object-Aware Representations [25.33452947179541]
  We show the effectiveness of object-aware representation learning techniques for robotic tasks. Our model learns control policies in a sample-efficient manner and outperforms state-of-the-art object techniques.
  arXiv  Detail & Related papers  (2022-05-12T19:48:11Z)
• Task-Induced Representation Learning [14.095897879222672]
  We evaluate the effectiveness of representation learning approaches for decision making in visually complex environments. We find that representation learning generally improves sample efficiency on unseen tasks even in visually complex scenes.
  arXiv  Detail & Related papers  (2022-04-25T17:57:10Z)
• Spot What Matters: Learning Context Using Graph Convolutional Networks for Weakly-Supervised Action Detection [0.0]
  We introduce an architecture based on self-attention and Convolutional Networks to improve human action detection in video. Our model aids explainability by visualizing the learned context as an attention map, even for actions and objects unseen during training. Experimental results show that our contextualized approach outperforms a baseline action detection approach by more than 2 points in Video-mAP.
  arXiv  Detail & Related papers  (2021-07-28T21:37:18Z)
• How Fine-Tuning Allows for Effective Meta-Learning [50.17896588738377]
  We present a theoretical framework for analyzing representations derived from a MAML-like algorithm. We provide risk bounds on the best predictor found by fine-tuning via gradient descent, demonstrating that the algorithm can provably leverage the shared structure. This separation result underscores the benefit of fine-tuning-based methods, such as MAML, over methods with "frozen representation" objectives in few-shot learning.
  arXiv  Detail & Related papers  (2021-05-05T17:56:00Z)
• Semi-Supervised Few-Shot Atomic Action Recognition [59.587738451616495]
  We propose a novel model for semi-supervised few-shot atomic action recognition. Our model features unsupervised and contrastive video embedding, loose action alignment, multi-head feature comparison, and attention-based aggregation. Experiments show that our model can attain high accuracy on representative atomic action datasets outperforming their respective state-of-the-art classification accuracy in full supervision setting.
  arXiv  Detail & Related papers  (2020-11-17T03:59:05Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.