LLMs Learn Task Heuristics from Demonstrations: A Heuristic-Driven Prompting Strategy for Document-Level Event Argument Extraction

• URL: http://arxiv.org/abs/2311.06555v3
• Date: Thu, 8 Aug 2024 03:20:05 GMT
• Title: LLMs Learn Task Heuristics from Demonstrations: A Heuristic-Driven Prompting Strategy for Document-Level Event Argument Extraction
• Authors: Hanzhang Zhou, Junlang Qian, Zijian Feng, Hui Lu, Zixiao Zhu, Kezhi Mao,
• Abstract summary: We introduce the Heuristic-Driven Link-of- Analogy (HD-LoA) prompting to address the challenge of example selection. Inspired by the analogical reasoning of human, we propose the link-of-analogy prompting, which enables LLMs to process new situations. Experiments show that our method outperforms existing prompting methods and few-shot supervised learning methods on document-level EAE datasets.
• Score: 12.673710691468264
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: In this study, we investigate in-context learning (ICL) in document-level event argument extraction (EAE) to alleviate the dependency on large-scale labeled data for this task. We introduce the Heuristic-Driven Link-of-Analogy (HD-LoA) prompting to address the challenge of example selection and to develop a prompting strategy tailored for EAE. Specifically, we hypothesize and validate that LLMs learn task-specific heuristics from demonstrations via ICL. Building upon this hypothesis, we introduce an explicit heuristic-driven demonstration construction approach, which transforms the haphazard example selection process into a methodical method that emphasizes task heuristics. Additionally, inspired by the analogical reasoning of human, we propose the link-of-analogy prompting, which enables LLMs to process new situations by drawing analogies to known situations, enhancing their performance on unseen classes beyond limited ICL examples. Experiments show that our method outperforms existing prompting methods and few-shot supervised learning methods on document-level EAE datasets. Additionally, the HD-LoA prompting shows effectiveness in diverse tasks like sentiment analysis and natural language inference, demonstrating its broad adaptability.

Related papers

• In-Context Transfer Learning: Demonstration Synthesis by Transferring Similar Tasks [93.46282380831339]
  In-context learning helps large language models adapt to various tasks by providing demonstrations of the target task. We propose In-Context Transfer Learning (ICTL), which synthesizes target task demonstrations by transferring labeled demonstrations from similar source tasks. Experiments on Super-NI show that ICTL outperforms synthesis from scratch by 2.0% on average.
  arXiv  Detail & Related papers  (2024-10-02T13:37:54Z)
• Enhancing Document-level Argument Extraction with Definition-augmented Heuristic-driven Prompting for LLMs [0.0]
  Event Argument Extraction (EAE) is pivotal for extracting structured information from unstructured text. We propose a novel Definition-augmented Heuristic-driven Prompting (DHP) method to enhance the performance of Large Language Models (LLMs) in document-level EAE.
  arXiv  Detail & Related papers  (2024-08-30T19:03:14Z)
• Document-Level Event Extraction with Definition-Driven ICL [0.0]
  We propose an optimization strategy called "Definition-driven Document-level Event Extraction (DDEE)" By adjusting the length of the prompt and enhancing the clarity of prompts, we have significantly improved the event extraction performance of Large Language Models (LLMs) In addition, the introduction of structured methods and strict limiting conditions has improved the precision of event and argument role extraction.
  arXiv  Detail & Related papers  (2024-08-10T14:24:09Z)
• Unraveling the Mechanics of Learning-Based Demonstration Selection for In-Context Learning [43.356895599336504]
  We analyze the working mechanisms of the learning-based demonstration selection methods. We empirically identify two important factors related to similarity measurement. We introduce two effective yet simplified exemplar selection methods catering to task-agnostic and task-specific demands.
  arXiv  Detail & Related papers  (2024-06-14T03:34:02Z)
• Decompose, Enrich, and Extract! Schema-aware Event Extraction using LLMs [45.83950260830323]
  This work focuses on harnessing Large Language Models for automated Event Extraction. It introduces a new method to address hallucination by decomposing the task into Event Detection and Event Argument Extraction.
  arXiv  Detail & Related papers  (2024-06-03T06:55:10Z)
• Prompt Optimization with EASE? Efficient Ordering-aware Automated Selection of Exemplars [66.823588073584]
  Large language models (LLMs) have shown impressive capabilities in real-world applications. The quality of these exemplars in the prompt greatly impacts performance. Existing methods fail to adequately account for the impact of exemplar ordering on the performance.
  arXiv  Detail & Related papers  (2024-05-25T08:23:05Z)
• C-ICL: Contrastive In-context Learning for Information Extraction [54.39470114243744]
  c-ICL is a novel few-shot technique that leverages both correct and incorrect sample constructions to create in-context learning demonstrations. Our experiments on various datasets indicate that c-ICL outperforms previous few-shot in-context learning methods.
  arXiv  Detail & Related papers  (2024-02-17T11:28:08Z)
• Mastering the Task of Open Information Extraction with Large Language Models and Consistent Reasoning Environment [52.592199835286394]
  Open Information Extraction (OIE) aims to extract objective structured knowledge from natural texts. Large language models (LLMs) have exhibited remarkable in-context learning capabilities.
  arXiv  Detail & Related papers  (2023-10-16T17:11:42Z)
• Ambiguity-Aware In-Context Learning with Large Language Models [27.20414960164616]
  In-context learning (ICL) i.e. showing LLMs task-specific demonstrations has led to downstream gains with no task-specific fine-tuning required. This study investigates how to select good demonstrations for ICL. We find that it is beneficial to not only choose semantically similar ICL demonstrations but also to choose those that help resolve the inherent label ambiguity surrounding the test example.
  arXiv  Detail & Related papers  (2023-09-14T17:48:34Z)
• Iterative Forward Tuning Boosts In-Context Learning in Language Models [88.25013390669845]
  In this study, we introduce a novel two-stage framework to boost in-context learning in large language models (LLMs) Specifically, our framework delineates the ICL process into two distinct stages: Deep-Thinking and test stages. The Deep-Thinking stage incorporates a unique attention mechanism, i.e., iterative enhanced attention, which enables multiple rounds of information accumulation.
  arXiv  Detail & Related papers  (2023-05-22T13:18:17Z)
• ICL-D3IE: In-Context Learning with Diverse Demonstrations Updating for Document Information Extraction [56.790794611002106]
  Large language models (LLMs) have demonstrated remarkable results in various natural language processing (NLP) tasks with in-context learning. We propose a simple but effective in-context learning framework called ICL-D3IE. Specifically, we extract the most difficult and distinct segments from hard training documents as hard demonstrations.
  arXiv  Detail & Related papers  (2023-03-09T06:24:50Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.