A Systems Theory of Transfer Learning

• URL: http://arxiv.org/abs/2107.01196v1
• Date: Fri, 2 Jul 2021 17:25:42 GMT
• Title: A Systems Theory of Transfer Learning
• Authors: Tyler Cody, Peter A. Beling
• Abstract summary: We use Mesarovician systems theory to define transfer learning as a relation on sets. We then characterize the general nature of transfer learning as a mathematical construct. Despite its formalism, our framework avoids the detailed mathematics of learning theory or machine learning solution methods.
• Score: 3.5281112495479245
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Existing frameworks for transfer learning are incomplete from a systems theoretic perspective. They place emphasis on notions of domain and task, and neglect notions of structure and behavior. In doing so, they limit the extent to which formalism can be carried through into the elaboration of their frameworks. Herein, we use Mesarovician systems theory to define transfer learning as a relation on sets and subsequently characterize the general nature of transfer learning as a mathematical construct. We interpret existing frameworks in terms of ours and go beyond existing frameworks to define notions of transferability, transfer roughness, and transfer distance. Importantly, despite its formalism, our framework avoids the detailed mathematics of learning theory or machine learning solution methods without excluding their consideration. As such, we provide a formal, general systems framework for modeling transfer learning that offers a rigorous foundation for system design and analysis.

Related papers

• Categorical semiotics: Foundations for Knowledge Integration [0.0]
  We tackle the challenging task of developing a comprehensive framework for defining and analyzing deep learning architectures. Our methodology employs graphical structures that resemble Ehresmann's sketches, interpreted within a universe of fuzzy sets. This approach offers a unified theory that elegantly encompasses both deterministic and non-deterministic neural network designs.
  arXiv  Detail & Related papers  (2024-04-01T23:19:01Z)
• Fundamental Components of Deep Learning: A category-theoretic approach [0.0]
  This thesis develops a novel mathematical foundation for deep learning based on the language of category theory. We also systematise many existing approaches, placing many existing constructions and concepts under the same umbrella.
  arXiv  Detail & Related papers  (2024-03-13T01:29:40Z)
• Towards a General Framework for Continual Learning with Pre-training [55.88910947643436]
  We present a general framework for continual learning of sequentially arrived tasks with the use of pre-training. We decompose its objective into three hierarchical components, including within-task prediction, task-identity inference, and task-adaptive prediction. We propose an innovative approach to explicitly optimize these components with parameter-efficient fine-tuning (PEFT) techniques and representation statistics.
  arXiv  Detail & Related papers  (2023-10-21T02:03:38Z)
• Homomorphisms Between Transfer, Multi-Task, and Meta-Learning Systems [0.0]
  This manuscript formalizes transfer learning, multi-task learning, and meta-learning as abstract learning systems. It uses the presented formalism to relate the three concepts of learning in terms of composition, hierarchy, and structural homomorphism.
  arXiv  Detail & Related papers  (2022-08-04T18:13:59Z)
• Mesarovician Abstract Learning Systems [0.0]
  Current approaches to learning hold notions of problem domain and problem task as fundamental precepts. Mesarovician abstract systems theory is used as a super-structure for learning.
  arXiv  Detail & Related papers  (2021-11-29T18:17:32Z)
• Empirically Measuring Transfer Distance for System Design and Operation [2.9864637081333085]
  We show that transfer learning algorithms have little, if any, examples from which to learn. We consider the use of transfer distance in the design of machine rebuild procedures to allow for transferable prognostic models. Practitioners can use the presented methodology to design and operate systems with consideration for the learning theoretic challenges faced by component learning systems.
  arXiv  Detail & Related papers  (2021-07-02T16:45:58Z)
• Formalising Concepts as Grounded Abstractions [68.24080871981869]
  This report shows how representation learning can be used to induce concepts from raw data. The main technical goal of this report is to show how techniques from representation learning can be married with a lattice-theoretic formulation of conceptual spaces.
  arXiv  Detail & Related papers  (2021-01-13T15:22:01Z)
• Developing Constrained Neural Units Over Time [81.19349325749037]
  This paper focuses on an alternative way of defining Neural Networks, that is different from the majority of existing approaches. The structure of the neural architecture is defined by means of a special class of constraints that are extended also to the interaction with data. The proposed theory is cast into the time domain, in which data are presented to the network in an ordered manner.
  arXiv  Detail & Related papers  (2020-09-01T09:07:25Z)
• Concept Learners for Few-Shot Learning [76.08585517480807]
  We propose COMET, a meta-learning method that improves generalization ability by learning to learn along human-interpretable concept dimensions. We evaluate our model on few-shot tasks from diverse domains, including fine-grained image classification, document categorization and cell type annotation.
  arXiv  Detail & Related papers  (2020-07-14T22:04:17Z)
• Learnability with Indirect Supervision Signals [74.39088325025378]
  Learning from indirect supervision signals is important in real-world AI applications when, often, gold labels are missing or too costly. We develop a unified theoretical framework for multi-class classification when the supervision is provided by a variable that contains nonzero mutual information with the gold label. Our framework relaxes assumptions made in the literature, and supports learning with unknown, non-invertible and instance-dependent transitions.
  arXiv  Detail & Related papers  (2020-06-15T21:57:11Z)
• Neuro-symbolic Architectures for Context Understanding [59.899606495602406]
  We propose the use of hybrid AI methodology as a framework for combining the strengths of data-driven and knowledge-driven approaches. Specifically, we inherit the concept of neuro-symbolism as a way of using knowledge-bases to guide the learning progress of deep neural networks.
  arXiv  Detail & Related papers  (2020-03-09T15:04:07Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.