A Taxonomy of Similarity Metrics for Markov Decision Processes

• URL: http://arxiv.org/abs/2103.04706v1
• Date: Mon, 8 Mar 2021 12:36:42 GMT
• Title: A Taxonomy of Similarity Metrics for Markov Decision Processes
• Authors: \'Alvaro Vis\'us, Javier Garc\'ia and Fernando Fern\'andez
• Abstract summary: In recent years, transfer learning has succeeded in making Reinforcement Learning (RL) algorithms more efficient. In this paper, we propose a categorization of these metrics and analyze the definitions of similarity proposed so far.
• Score: 62.997667081978825
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Although the notion of task similarity is potentially interesting in a wide range of areas such as curriculum learning or automated planning, it has mostly been tied to transfer learning. Transfer is based on the idea of reusing the knowledge acquired in the learning of a set of source tasks to a new learning process in a target task, assuming that the target and source tasks are close enough. In recent years, transfer learning has succeeded in making Reinforcement Learning (RL) algorithms more efficient (e.g., by reducing the number of samples needed to achieve the (near-)optimal performance). Transfer in RL is based on the core concept of similarity: whenever the tasks are similar, the transferred knowledge can be reused to solve the target task and significantly improve the learning performance. Therefore, the selection of good metrics to measure these similarities is a critical aspect when building transfer RL algorithms, especially when this knowledge is transferred from simulation to the real world. In the literature, there are many metrics to measure the similarity between MDPs, hence, many definitions of similarity or its complement distance have been considered. In this paper, we propose a categorization of these metrics and analyze the definitions of similarity proposed so far, taking into account such categorization. We also follow this taxonomy to survey the existing literature, as well as suggesting future directions for the construction of new metrics.

Related papers

• Transferability Estimation Based On Principal Gradient Expectation [68.97403769157117]
  Cross-task transferability is compatible with transferred results while keeping self-consistency. Existing transferability metrics are estimated on the particular model by conversing source and target tasks. We propose Principal Gradient Expectation (PGE), a simple yet effective method for assessing transferability across tasks.
  arXiv  Detail & Related papers  (2022-11-29T15:33:02Z)
• NEVIS'22: A Stream of 100 Tasks Sampled from 30 Years of Computer Vision Research [96.53307645791179]
  We introduce the Never-Ending VIsual-classification Stream (NEVIS'22), a benchmark consisting of a stream of over 100 visual classification tasks. Despite being limited to classification, the resulting stream has a rich diversity of tasks from OCR, to texture analysis, scene recognition, and so forth. Overall, NEVIS'22 poses an unprecedented challenge for current sequential learning approaches due to the scale and diversity of tasks.
  arXiv  Detail & Related papers  (2022-11-15T18:57:46Z)
• The Curse of Low Task Diversity: On the Failure of Transfer Learning to Outperform MAML and Their Empirical Equivalence [20.965759895300327]
  We propose a novel metric -- the diversity coefficient -- to measure the diversity of tasks in a few-shot learning benchmark. Using the diversity coefficient, we show that the popular MiniImageNet and CIFAR-FS few-shot learning benchmarks have low diversity.
  arXiv  Detail & Related papers  (2022-08-02T15:49:11Z)
• Structural Similarity for Improved Transfer in Reinforcement Learning [0.0]
  We present an algorithm that calculates a state similarity measure for states in two finite MDPs based on previously developed bisimulation metrics. We show that the measure satisfies properties of a distance metric and can be used to improve transfer performance for Q-Learning agents.
  arXiv  Detail & Related papers  (2022-07-27T22:21:38Z)
• On Generalizing Beyond Domains in Cross-Domain Continual Learning [91.56748415975683]
  Deep neural networks often suffer from catastrophic forgetting of previously learned knowledge after learning a new task. Our proposed approach learns new tasks under domain shift with accuracy boosts up to 10% on challenging datasets such as DomainNet and OfficeHome.
  arXiv  Detail & Related papers  (2022-03-08T09:57:48Z)
• Automating Transfer Credit Assessment in Student Mobility -- A Natural Language Processing-based Approach [5.947076788303102]
  This research article focuses on identifying a model that exploits the advancements in the field of Natural Language Processing (NLP) to effectively automate this process. The proposed model uses a clustering-inspired methodology based on knowledge-based semantic similarity measures to assess the taxonomic similarity of learning outcomes (LOs) The similarity between LOs is further aggregated to form course to course similarity.
  arXiv  Detail & Related papers  (2021-04-05T15:14:59Z)
• Dif-MAML: Decentralized Multi-Agent Meta-Learning [54.39661018886268]
  We propose a cooperative multi-agent meta-learning algorithm, referred to as MAML or Dif-MAML. We show that the proposed strategy allows a collection of agents to attain agreement at a linear rate and to converge to a stationary point of the aggregate MAML. Simulation results illustrate the theoretical findings and the superior performance relative to the traditional non-cooperative setting.
  arXiv  Detail & Related papers  (2020-10-06T16:51:09Z)
• Unsupervised Transfer Learning for Spatiotemporal Predictive Networks [90.67309545798224]
  We study how to transfer knowledge from a zoo of unsupervisedly learned models towards another network. Our motivation is that models are expected to understand complex dynamics from different sources. Our approach yields significant improvements on three benchmarks fortemporal prediction, and benefits the target even from less relevant ones.
  arXiv  Detail & Related papers  (2020-09-24T15:40:55Z)
• Learning similarity measures from data [1.4766350834632755]
  Defining similarity measures is a requirement for some machine learning methods. Data sets are typically gathered as part of constructing a CBR or machine learning system. Our objective is to investigate how to apply machine learning to effectively learn a similarity measure.
  arXiv  Detail & Related papers  (2020-01-15T13:29:48Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.