A Framework of Meta Functional Learning for Regularising Knowledge
Transfer
- URL: http://arxiv.org/abs/2203.14840v1
- Date: Mon, 28 Mar 2022 15:24:09 GMT
- Title: A Framework of Meta Functional Learning for Regularising Knowledge
Transfer
- Authors: Pan Li, Yanwei Fu and Shaogang Gong
- Abstract summary: This work proposes a novel framework of Meta Functional Learning (MFL) by meta-learning a generalisable functional model from data-rich tasks.
The MFL computes meta-knowledge on functional regularisation generalisable to different learning tasks by which functional training on limited labelled data promotes more discriminative functions to be learned.
- Score: 89.74127682599898
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: Machine learning classifiers' capability is largely dependent on the scale of
available training data and limited by the model overfitting in data-scarce
learning tasks. To address this problem, this work proposes a novel framework
of Meta Functional Learning (MFL) by meta-learning a generalisable functional
model from data-rich tasks whilst simultaneously regularising knowledge
transfer to data-scarce tasks. The MFL computes meta-knowledge on functional
regularisation generalisable to different learning tasks by which functional
training on limited labelled data promotes more discriminative functions to be
learned. Based on this framework, we formulate three variants of MFL: MFL with
Prototypes (MFL-P) which learns a functional by auxiliary prototypes, Composite
MFL (ComMFL) that transfers knowledge from both functional space and
representational space, and MFL with Iterative Updates (MFL-IU) which improves
knowledge transfer regularisation from MFL by progressively learning the
functional regularisation in knowledge transfer. Moreover, we generalise these
variants for knowledge transfer regularisation from binary classifiers to
multi-class classifiers. Extensive experiments on two few-shot learning
scenarios, Few-Shot Learning (FSL) and Cross-Domain Few-Shot Learning (CD-FSL),
show that meta functional learning for knowledge transfer regularisation can
improve FSL classifiers.
Related papers
- Interactive Continual Learning: Fast and Slow Thinking [19.253164551254734]
This paper presents a novel Interactive Continual Learning framework, enabled by collaborative interactions among models of various sizes.
To improve memory retrieval in System1, we introduce the CL-vMF mechanism, based on the von Mises-Fisher (vMF) distribution.
Comprehensive evaluation of our proposed ICL demonstrates significant resistance to forgetting and superior performance relative to existing methods.
arXiv Detail & Related papers (2024-03-05T03:37:28Z) - SAPT: A Shared Attention Framework for Parameter-Efficient Continual Learning of Large Language Models [71.78800549517298]
Continual learning (CL) ability is vital for deploying large language models (LLMs) in the dynamic world.
Existing methods devise the learning module to acquire task-specific knowledge with parameter-efficient tuning (PET) block and the selection module to pick out the corresponding one for the testing input.
We propose a novel Shared Attention Framework (SAPT) to align the PET learning and selection via the Shared Attentive Learning & Selection module.
arXiv Detail & Related papers (2024-01-16T11:45:03Z) - A Survey on Efficient Federated Learning Methods for Foundation Model
Training [66.19763977571114]
Federated Learning (FL) has become an established technique to facilitate privacy-preserving collaborative training across a multitude of clients.
In the wake of Foundation Models (FM), the reality is different for many deep learning applications.
We discuss the benefits and drawbacks of parameter-efficient fine-tuning (PEFT) for FL applications.
arXiv Detail & Related papers (2024-01-09T10:22:23Z) - Federated Continual Learning via Knowledge Fusion: A Survey [33.74289759536269]
Federated Continual Learning (FCL) is an emerging paradigm to address model learning in both federated and continual learning environments.
The key objective of FCL is to fuse heterogeneous knowledge from different clients and retain knowledge of previous tasks while learning on new ones.
In this work, we delineate federated learning and continual learning first and then discuss their integration, i.e., FCL, and particular FCL via knowledge fusion.
arXiv Detail & Related papers (2023-12-27T08:47:39Z) - Supervised Knowledge Makes Large Language Models Better In-context Learners [94.89301696512776]
Large Language Models (LLMs) exhibit emerging in-context learning abilities through prompt engineering.
The challenge of improving the generalizability and factuality of LLMs in natural language understanding and question answering remains under-explored.
We propose a framework that enhances the reliability of LLMs as it: 1) generalizes out-of-distribution data, 2) elucidates how LLMs benefit from discriminative models, and 3) minimizes hallucinations in generative tasks.
arXiv Detail & Related papers (2023-12-26T07:24:46Z) - Challenges and Opportunities of Using Transformer-Based Multi-Task
Learning in NLP Through ML Lifecycle: A Survey [0.6240603866868214]
Multi-Task Learning (MTL) has emerged as a promising approach to improve efficiency and performance through joint training.
We discuss the challenges and opportunities of using MTL approaches throughout typical machine learning lifecycle phases.
We believe it would be practical to have a model that can handle both MTL and continual learning.
arXiv Detail & Related papers (2023-08-16T09:11:00Z) - Learning without Forgetting for Vision-Language Models [65.49600786387106]
Class-Incremental Learning (CIL) or continual learning is a desired capability in the real world.
Recent advances in Vision-Language Models (VLM) have shown promising capabilities in learning generalizable representations.
We propose PROjectiOn Fusion (PROOF) that enables VLMs to learn without forgetting.
arXiv Detail & Related papers (2023-05-30T17:59:32Z) - Federated Learning and Meta Learning: Approaches, Applications, and
Directions [94.68423258028285]
In this tutorial, we present a comprehensive review of FL, meta learning, and federated meta learning (FedMeta)
Unlike other tutorial papers, our objective is to explore how FL, meta learning, and FedMeta methodologies can be designed, optimized, and evolved, and their applications over wireless networks.
arXiv Detail & Related papers (2022-10-24T10:59:29Z) - ProtoDA: Efficient Transfer Learning for Few-Shot Intent Classification [21.933876113300897]
We adopt an alternative approach by transfer learning on an ensemble of related tasks using prototypical networks under the meta-learning paradigm.
Using intent classification as a case study, we demonstrate that increasing variability in training tasks can significantly improve classification performance.
arXiv Detail & Related papers (2021-01-28T00:19:13Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.