GoLLIE: Annotation Guidelines improve Zero-Shot Information-Extraction

• URL: http://arxiv.org/abs/2310.03668v5
• Date: Wed, 6 Mar 2024 16:38:03 GMT
• Title: GoLLIE: Annotation Guidelines improve Zero-Shot Information-Extraction
• Authors: Oscar Sainz, Iker Garc\'ia-Ferrero, Rodrigo Agerri, Oier Lopez de Lacalle, German Rigau, Eneko Agirre
• Abstract summary: We propose GoLLIE (Guideline-following Large Language Model for IE), a model able to improve zero-shot results on unseen IE tasks. GoLLIE is able to generalize to and follow unseen guidelines, outperforming previous attempts at zero-shot information extraction.
• Score: 25.028613319081696
• License: http://creativecommons.org/licenses/by-sa/4.0/
• Abstract: Large Language Models (LLMs) combined with instruction tuning have made significant progress when generalizing to unseen tasks. However, they have been less successful in Information Extraction (IE), lagging behind task-specific models. Typically, IE tasks are characterized by complex annotation guidelines that describe the task and give examples to humans. Previous attempts to leverage such information have failed, even with the largest models, as they are not able to follow the guidelines out of the box. In this paper, we propose GoLLIE (Guideline-following Large Language Model for IE), a model able to improve zero-shot results on unseen IE tasks by virtue of being fine-tuned to comply with annotation guidelines. Comprehensive evaluation empirically demonstrates that GoLLIE is able to generalize to and follow unseen guidelines, outperforming previous attempts at zero-shot information extraction. The ablation study shows that detailed guidelines are key for good results.

Related papers

• RUIE: Retrieval-based Unified Information Extraction using Large Language Model [6.788855739199981]
  Unified information extraction aims to complete all information extraction tasks using a single model or framework. We propose RUIE (Retrieval-based Unified Information Extraction), a framework that leverages in-context learning to enable rapid generalization. Experimental results on 8 held-out datasets demonstrate RUIE's effectiveness in generalizing to unseen tasks.
  arXiv  Detail & Related papers  (2024-09-18T03:20:04Z)
• ADELIE: Aligning Large Language Models on Information Extraction [55.60192044049083]
  Large language models (LLMs) usually fall short on information extraction tasks. In this paper, we introduce ADELIE, an aligned LLM that effectively solves various IE tasks. We show that our models achieve state-of-the-art (SoTA) performance among open-source models.
  arXiv  Detail & Related papers  (2024-05-08T12:24:52Z)
• Instruct and Extract: Instruction Tuning for On-Demand Information Extraction [86.29491354355356]
  On-Demand Information Extraction aims to fulfill the personalized demands of real-world users. We present a benchmark named InstructIE, inclusive of both automatically generated training data, as well as the human-annotated test set. Building on InstructIE, we further develop an On-Demand Information Extractor, ODIE.
  arXiv  Detail & Related papers  (2023-10-24T17:54:25Z)
• Benchmarking Large Language Models with Augmented Instructions for Fine-grained Information Extraction [46.09887436555637]
  This paper introduces a fine-grained IE benchmark dataset tailored for Large Language Models (LLMs) Through extensive evaluations, we observe that encoder-decoder models, particularly T5 and FLAN-T5, perform well in generalizing to unseen information types.
  arXiv  Detail & Related papers  (2023-10-08T09:41:18Z)
• InstructIE: A Bilingual Instruction-based Information Extraction Dataset [44.65162892808696]
  Large language models can perform well on general natural language tasks, but their effectiveness is still suboptimal for information extraction (IE) Recent works indicate that the main reason lies in the lack of extensive data on IE instructions. We introduce InstructIE, a bilingual instruction-based IE dataset, which covers 12 diverse domains.
  arXiv  Detail & Related papers  (2023-05-19T08:51:11Z)
• CodeIE: Large Code Generation Models are Better Few-Shot Information Extractors [92.17328076003628]
  Large language models (LLMs) pre-trained on massive corpora have demonstrated impressive few-shot learning ability on many NLP tasks. In this paper, we propose to recast the structured output in the form of code instead of natural language.
  arXiv  Detail & Related papers  (2023-05-09T18:40:31Z)
• Discover, Explanation, Improvement: An Automatic Slice Detection Framework for Natural Language Processing [72.14557106085284]
  slice detection models (SDM) automatically identify underperforming groups of datapoints. This paper proposes a benchmark named "Discover, Explain, improve (DEIM)" for classification NLP tasks. Our evaluation shows that Edisa can accurately select error-prone datapoints with informative semantic features.
  arXiv  Detail & Related papers  (2022-11-08T19:00:00Z)
• Don't Blame the Annotator: Bias Already Starts in the Annotation Instructions [71.5668415104079]
  We study a form of bias, termed instruction bias, in 14 recent NLU benchmarks. We show that instruction examples often exhibit concrete patterns, which are propagated by crowdworkers to the collected data.
  arXiv  Detail & Related papers  (2022-05-01T07:51:22Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.