Retrieval-Augmented Meta Learning for Low-Resource Text Classification
- URL: http://arxiv.org/abs/2309.04979v1
- Date: Sun, 10 Sep 2023 10:05:03 GMT
- Title: Retrieval-Augmented Meta Learning for Low-Resource Text Classification
- Authors: Rongsheng Li, Yangning Li, Yinghui Li, Chaiyut Luoyiching, Hai-Tao
Zheng, Nannan Zhou, Hanjing Su
- Abstract summary: We propose a meta-learning based method called Retrieval-Augmented Meta Learning(RAML)
It uses parameterization for inference but also retrieves non-parametric knowledge from an external corpus to make inferences.
RAML significantly outperforms current SOTA low-resource text classification models.
- Score: 22.653220906899612
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: Meta learning have achieved promising performance in low-resource text
classification which aims to identify target classes with knowledge transferred
from source classes with sets of small tasks named episodes. However, due to
the limited training data in the meta-learning scenario and the inherent
properties of parameterized neural networks, poor generalization performance
has become a pressing problem that needs to be addressed. To deal with this
issue, we propose a meta-learning based method called Retrieval-Augmented Meta
Learning(RAML). It not only uses parameterization for inference but also
retrieves non-parametric knowledge from an external corpus to make inferences,
which greatly alleviates the problem of poor generalization performance caused
by the lack of diverse training data in meta-learning. This method differs from
previous models that solely rely on parameters, as it explicitly emphasizes the
importance of non-parametric knowledge, aiming to strike a balance between
parameterized neural networks and non-parametric knowledge. The model is
required to determine which knowledge to access and utilize during inference.
Additionally, our multi-view passages fusion network module can effectively and
efficiently integrate the retrieved information into low-resource
classification task. The extensive experiments demonstrate that RAML
significantly outperforms current SOTA low-resource text classification models.
Related papers
- Seeking Neural Nuggets: Knowledge Transfer in Large Language Models from a Parametric Perspective [106.92016199403042]
We empirically investigate knowledge transfer from larger to smaller models through a parametric perspective.
We employ sensitivity-based techniques to extract and align knowledge-specific parameters between different large language models.
Our findings highlight the critical factors contributing to the process of parametric knowledge transfer.
arXiv Detail & Related papers (2023-10-17T17:58:34Z) - Complementary Learning Subnetworks for Parameter-Efficient
Class-Incremental Learning [40.13416912075668]
We propose a rehearsal-free CIL approach that learns continually via the synergy between two Complementary Learning Subnetworks.
Our method achieves competitive results against state-of-the-art methods, especially in accuracy gain, memory cost, training efficiency, and task-order.
arXiv Detail & Related papers (2023-06-21T01:43:25Z) - Learning to Learn with Indispensable Connections [6.040904021861969]
We propose a novel meta-learning method called Meta-LTH that includes indispensible (necessary) connections.
Our method improves the classification accuracy by approximately 2% (20-way 1-shot task setting) for omniglot dataset.
arXiv Detail & Related papers (2023-04-06T04:53:13Z) - MARS: Meta-Learning as Score Matching in the Function Space [79.73213540203389]
We present a novel approach to extracting inductive biases from a set of related datasets.
We use functional Bayesian neural network inference, which views the prior as a process and performs inference in the function space.
Our approach can seamlessly acquire and represent complex prior knowledge by metalearning the score function of the data-generating process.
arXiv Detail & Related papers (2022-10-24T15:14:26Z) - Improving Meta-learning for Low-resource Text Classification and
Generation via Memory Imitation [87.98063273826702]
We propose a memory imitation meta-learning (MemIML) method that enhances the model's reliance on support sets for task adaptation.
A theoretical analysis is provided to prove the effectiveness of our method.
arXiv Detail & Related papers (2022-03-22T12:41:55Z) - Generating meta-learning tasks to evolve parametric loss for
classification learning [1.1355370218310157]
In existing meta-learning approaches, learning tasks for training meta-models are usually collected from public datasets.
We propose a meta-learning approach based on randomly generated meta-learning tasks to obtain a parametric loss for classification learning based on big data.
arXiv Detail & Related papers (2021-11-20T13:07:55Z) - Knowledge-Aware Meta-learning for Low-Resource Text Classification [87.89624590579903]
This paper studies a low-resource text classification problem and bridges the gap between meta-training and meta-testing tasks.
We propose KGML to introduce additional representation for each sentence learned from the extracted sentence-specific knowledge graph.
arXiv Detail & Related papers (2021-09-10T07:20:43Z) - Learning an Explicit Hyperparameter Prediction Function Conditioned on
Tasks [62.63852372239708]
Meta learning aims to learn the learning methodology for machine learning from observed tasks, so as to generalize to new query tasks.
We interpret such learning methodology as learning an explicit hyper- parameter prediction function shared by all training tasks.
Such setting guarantees that the meta-learned learning methodology is able to flexibly fit diverse query tasks.
arXiv Detail & Related papers (2021-07-06T04:05:08Z) - Learning to Learn Kernels with Variational Random Features [118.09565227041844]
We introduce kernels with random Fourier features in the meta-learning framework to leverage their strong few-shot learning ability.
We formulate the optimization of MetaVRF as a variational inference problem.
We show that MetaVRF delivers much better, or at least competitive, performance compared to existing meta-learning alternatives.
arXiv Detail & Related papers (2020-06-11T18:05:29Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.