MetaIE: Distilling a Meta Model from LLM for All Kinds of Information Extraction Tasks

• URL: http://arxiv.org/abs/2404.00457v1
• Date: Sat, 30 Mar 2024 19:43:45 GMT
• Title: MetaIE: Distilling a Meta Model from LLM for All Kinds of Information Extraction Tasks
• Authors: Letian Peng, Zilong Wang, Feng Yao, Zihan Wang, Jingbo Shang,
• Abstract summary: We propose a novel framework MetaIE to build a small LM as meta-model by learning to extract "important information" Specifically, MetaIE obtains the small LM via a symbolic distillation from an LLM following the label-to-span scheme. We construct the distillation dataset via sampling sentences from language model pre-training datasets. We evaluate the meta-model under the few-shot adaptation setting.
• Score: 40.84745946091173
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Information extraction (IE) is a fundamental area in natural language processing where prompting large language models (LLMs), even with in-context examples, cannot defeat small LMs tuned on very small IE datasets. We observe that IE tasks, such as named entity recognition and relation extraction, all focus on extracting important information, which can be formalized as a label-to-span matching. In this paper, we propose a novel framework MetaIE to build a small LM as meta-model by learning to extract "important information", i.e., the meta-understanding of IE, so that this meta-model can be adapted to all kind of IE tasks effectively and efficiently. Specifically, MetaIE obtains the small LM via a symbolic distillation from an LLM following the label-to-span scheme. We construct the distillation dataset via sampling sentences from language model pre-training datasets (e.g., OpenWebText in our implementation) and prompting an LLM to identify the typed spans of "important information". We evaluate the meta-model under the few-shot adaptation setting. Extensive results on 13 datasets from 6 IE tasks confirm that MetaIE can offer a better starting point for few-shot tuning on IE datasets and outperform other meta-models from (1) vanilla language model pre-training, (2) multi-IE-task pre-training with human annotations, and (3) single-IE-task symbolic distillation from LLM. Moreover, we provide comprehensive analyses of MetaIE, such as the size of the distillation dataset, the meta-model architecture, and the size of the meta-model.

Related papers

• Web-Scale Visual Entity Recognition: An LLM-Driven Data Approach [56.55633052479446]
  Web-scale visual entity recognition presents significant challenges due to the lack of clean, large-scale training data. We propose a novel methodology to curate such a dataset, leveraging a multimodal large language model (LLM) for label verification, metadata generation, and rationale explanation. Experiments demonstrate that models trained on this automatically curated data achieve state-of-the-art performance on web-scale visual entity recognition tasks.
  arXiv  Detail & Related papers  (2024-10-31T06:55:24Z)
• Privacy Challenges in Meta-Learning: An Investigation on Model-Agnostic Meta-Learning [2.8948274245812327]
  In current meta-learning methods, task learners locally learn models from sensitive data, termed support sets. Despite the absence of explicit data sharing, privacy concerns persist. This paper examines potential data leakage in a prominent metalearning algorithm, specifically Model-Agnostic Meta-Learning (MAML)
  arXiv  Detail & Related papers  (2024-06-01T01:10:35Z)
• MAML-en-LLM: Model Agnostic Meta-Training of LLMs for Improved In-Context Learning [43.512739869120125]
  We propose MAML-en-LLM, a novel method for meta-training large language models (LLMs) MAML-en-LLM can learn truly generalizable parameters that not only perform well on disjointed tasks but also adapts to unseen tasks. We demonstrate that MAML-en-LLM outperforms baselines in settings with limited amount of training data on both seen and unseen domains.
  arXiv  Detail & Related papers  (2024-05-19T04:49:42Z)
• ADELIE: Aligning Large Language Models on Information Extraction [55.60192044049083]
  Large language models (LLMs) usually fall short on information extraction tasks. In this paper, we introduce ADELIE, an aligned LLM that effectively solves various IE tasks. We show that our models achieve state-of-the-art (SoTA) performance among open-source models.
  arXiv  Detail & Related papers  (2024-05-08T12:24:52Z)
• Benchmarking Large Language Models with Augmented Instructions for Fine-grained Information Extraction [46.09887436555637]
  This paper introduces a fine-grained IE benchmark dataset tailored for Large Language Models (LLMs) Through extensive evaluations, we observe that encoder-decoder models, particularly T5 and FLAN-T5, perform well in generalizing to unseen information types.
  arXiv  Detail & Related papers  (2023-10-08T09:41:18Z)
• Universal Information Extraction with Meta-Pretrained Self-Retrieval [39.69130086395689]
  Universal Information Extraction(Universal IE) aims to solve different extraction tasks in a uniform text-to-structure generation manner. Retrieving knowledge from external knowledge bases may help models to overcome this problem but it is impossible to construct a knowledge base suitable for various IE tasks. We propose MetaRetriever to retrieve task-specific knowledge from PLMs to enhance universal IE.
  arXiv  Detail & Related papers  (2023-06-18T00:16:00Z)
• Learning to Learn from APIs: Black-Box Data-Free Meta-Learning [95.41441357931397]
  Data-free meta-learning (DFML) aims to enable efficient learning of new tasks by meta-learning from a collection of pre-trained models without access to the training data. Existing DFML work can only meta-learn from (i) white-box and (ii) small-scale pre-trained models. We propose a Bi-level Data-free Meta Knowledge Distillation (BiDf-MKD) framework to transfer more general meta knowledge from a collection of black-box APIs to one single model.
  arXiv  Detail & Related papers  (2023-05-28T18:00:12Z)
• MetaICL: Learning to Learn In Context [87.23056864536613]
  We introduce MetaICL, a new meta-training framework for few-shot learning where a pretrained language model is tuned to do in-context learn-ing on a large set of training tasks. We show that MetaICL approaches (and sometimes beats) the performance of models fully finetuned on the target task training data, and outperforms much bigger models with nearly 8x parameters.
  arXiv  Detail & Related papers  (2021-10-29T17:42:08Z)
• Incremental Meta-Learning via Indirect Discriminant Alignment [118.61152684795178]
  We develop a notion of incremental learning during the meta-training phase of meta-learning. Our approach performs favorably at test time as compared to training a model with the full meta-training set.
  arXiv  Detail & Related papers  (2020-02-11T01:39:12Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.