Priors, Hierarchy, and Information Asymmetry for Skill Transfer in Reinforcement Learning

• URL: http://arxiv.org/abs/2201.08115v2
• Date: Tue, 25 Apr 2023 03:26:17 GMT
• Title: Priors, Hierarchy, and Information Asymmetry for Skill Transfer in Reinforcement Learning
• Authors: Sasha Salter, Kristian Hartikainen, Walter Goodwin, Ingmar Posner
• Abstract summary: We show the crucial expressivity-transferability trade-off of skills across sequential tasks controlled by information asymmetry. We introduce Attentive Priors for Expressive and Transferable Skills (APES) Unlike existing approaches, APES automates the choice of asymmetry by learning it in a data-driven, domain-dependent, way.
• Score: 18.865535706610522
• License: http://creativecommons.org/publicdomain/zero/1.0/
• Abstract: The ability to discover behaviours from past experience and transfer them to new tasks is a hallmark of intelligent agents acting sample-efficiently in the real world. Equipping embodied reinforcement learners with the same ability may be crucial for their successful deployment in robotics. While hierarchical and KL-regularized reinforcement learning individually hold promise here, arguably a hybrid approach could combine their respective benefits. Key to these fields is the use of information asymmetry across architectural modules to bias which skills are learnt. While asymmetry choice has a large influence on transferability, existing methods base their choice primarily on intuition in a domain-independent, potentially sub-optimal, manner. In this paper, we theoretically and empirically show the crucial expressivity-transferability trade-off of skills across sequential tasks, controlled by information asymmetry. Given this insight, we introduce Attentive Priors for Expressive and Transferable Skills (APES), a hierarchical KL-regularized method, heavily benefiting from both priors and hierarchy. Unlike existing approaches, APES automates the choice of asymmetry by learning it in a data-driven, domain-dependent, way based on our expressivity-transferability theorems. Experiments over complex transfer domains of varying levels of extrapolation and sparsity, such as robot block stacking, demonstrate the criticality of the correct asymmetric choice, with APES drastically outperforming previous methods.

Related papers

• When Single-Agent with Skills Replace Multi-Agent Systems and When They Fail [40.69885101060645]
  Multi-agent AI systems have proven effective for complex reasoning.<n>Can we achieve similar modularity benefits with a single agent that selects from a library of skills?<n>We investigate the scaling behavior of skill selection and observe a striking pattern.<n>We find evidence that semantic confusability among similar skills, rather than library size alone, plays a central role in this degradation.
  arXiv  Detail & Related papers  (2026-01-08T09:14:26Z)
• Beyond Neural Incompatibility: Easing Cross-Scale Knowledge Transfer in Large Language Models through Latent Semantic Alignment [22.84428628659889]
  Large Language Models (LLMs) encode vast amounts of knowledge in their massive parameters, which is accessible to locate, trace, and analyze.<n>Despite advances in neural interpretability, it is still not clear how to transfer knowledge in a fine-grained manner, namely parametric knowledge transfer (PKT)
  arXiv  Detail & Related papers  (2025-10-28T09:25:40Z)
• Quantifying Dataset Similarity to Guide Transfer Learning [1.6328866317851185]
  Cross-Learning Score ( CLS) measures dataset similarity through bidirectional performance between domains.<n> CLS can reliably predict whether transfer will improve or degrade performance.<n> CLS is efficient and fast to compute as it bypasses the problem of expensive distribution estimation for high-dimensional problems.
  arXiv  Detail & Related papers  (2025-10-13T00:18:35Z)
• FAST: Similarity-based Knowledge Transfer for Efficient Policy Learning [57.4737157531239]
  Transfer Learning offers the potential to accelerate learning by transferring knowledge across tasks.<n>It faces critical challenges such as negative transfer, domain adaptation and inefficiency in selecting solid source policies.<n>In this work we challenge the key issues in TL to improve knowledge transfer, agents performance across tasks and reduce computational costs.
  arXiv  Detail & Related papers  (2025-07-27T22:21:53Z)
• Pretrained Bayesian Non-parametric Knowledge Prior in Robotic Long-Horizon Reinforcement Learning [10.598207472087578]
  Reinforcement learning (RL) methods typically learn new tasks from scratch, often disregarding prior knowledge that could accelerate the learning process. This work introduces a method that models potential primitive skill motions as having non-parametric properties with an unknown number of underlying features. We utilize a non-parametric model, specifically Dirichlet Process Mixtures, enhanced with birth and merge, to pre-train a skill prior that effectively captures the diverse nature of skills.
  arXiv  Detail & Related papers  (2025-03-27T20:43:36Z)
• On Learning Informative Trajectory Embeddings for Imitation, Classification and Regression [19.01804572722833]
  In real-world sequential decision making tasks, learning from observed state-action trajectories is critical for tasks like imitation, classification, and clustering. We propose a novel method for embedding state-action trajectories into a latent space that captures the skills and competencies in the dynamic underlying decision-making processes.
  arXiv  Detail & Related papers  (2025-01-16T06:52:58Z)
• From Novice to Expert: LLM Agent Policy Optimization via Step-wise Reinforcement Learning [62.54484062185869]
  We introduce StepAgent, which utilizes step-wise reward to optimize the agent's reinforcement learning process. We propose implicit-reward and inverse reinforcement learning techniques to facilitate agent reflection and policy adjustment.
  arXiv  Detail & Related papers  (2024-11-06T10:35:11Z)
• Latent-Predictive Empowerment: Measuring Empowerment without a Simulator [56.53777237504011]
  We present Latent-Predictive Empowerment (LPE), an algorithm that can compute empowerment in a more practical manner. LPE learns large skillsets by maximizing an objective that is a principled replacement for the mutual information between skills and states.
  arXiv  Detail & Related papers  (2024-10-15T00:41:18Z)
• Conditional Neural Expert Processes for Learning Movement Primitives from Demonstration [1.9336815376402723]
  Conditional Neural Expert Processes (CNEP) learns to assign demonstrations from different modes to distinct expert networks. CNEP does not require supervision on which mode the trajectories belong to. Our system is capable of on-the-fly adaptation to environmental changes via an online conditioning mechanism.
  arXiv  Detail & Related papers  (2024-02-13T12:52:02Z)
• Self-Optimizing Feature Transformation [33.458785763961004]
  Feature transformation aims to extract a good representation (feature) space by mathematically transforming existing features. Current research focuses on domain knowledge-based feature engineering or learning latent representations. We present a self-optimizing framework for feature transformation.
  arXiv  Detail & Related papers  (2022-09-16T16:50:41Z)
• Learning Transferable Motor Skills with Hierarchical Latent Mixture Policies [37.09286945259353]
  We propose an approach to learn abstract motor skills from data using a hierarchical mixture latent variable model. We demonstrate in manipulation domains that the method can effectively cluster offline data into distinct, executable behaviours.
  arXiv  Detail & Related papers  (2021-12-09T17:37:14Z)
• Hierarchical Few-Shot Imitation with Skill Transition Models [66.81252581083199]
  Few-shot Imitation with Skill Transition Models (FIST) is an algorithm that extracts skills from offline data and utilizes them to generalize to unseen tasks. We show that FIST is capable of generalizing to new tasks and substantially outperforms prior baselines in navigation experiments.
  arXiv  Detail & Related papers  (2021-07-19T15:56:01Z)
• Uniform Priors for Data-Efficient Transfer [65.086680950871]
  We show that features that are most transferable have high uniformity in the embedding space. We evaluate the regularization on its ability to facilitate adaptation to unseen tasks and data.
  arXiv  Detail & Related papers  (2020-06-30T04:39:36Z)
• Task-Feature Collaborative Learning with Application to Personalized Attribute Prediction [166.87111665908333]
  We propose a novel multi-task learning method called Task-Feature Collaborative Learning (TFCL) Specifically, we first propose a base model with a heterogeneous block-diagonal structure regularizer to leverage the collaborative grouping of features and tasks. As a practical extension, we extend the base model by allowing overlapping features and differentiating the hard tasks.
  arXiv  Detail & Related papers  (2020-04-29T02:32:04Z)
• Self-Attention Attribution: Interpreting Information Interactions Inside Transformer [89.21584915290319]
  We propose a self-attention attribution method to interpret the information interactions inside Transformer. We show that the attribution results can be used as adversarial patterns to implement non-targeted attacks towards BERT.
  arXiv  Detail & Related papers  (2020-04-23T14:58:22Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.