DocIE@XLLM25: In-Context Learning for Information Extraction using Fully Synthetic Demonstrations

• URL: http://arxiv.org/abs/2507.05997v1
• Date: Tue, 08 Jul 2025 13:55:25 GMT
• Title: DocIE@XLLM25: In-Context Learning for Information Extraction using Fully Synthetic Demonstrations
• Authors: Nicholas Popovič, Ashish Kangen, Tim Schopf, Michael Färber,
• Abstract summary: In this paper, we present a fully automatic, LLM-based pipeline for synthetic data generation and in-context learning.<n>We produce a synthetic dataset of over $5k$ Wikipedia abstracts with approximately $59k$ entities and $30k$ relation triples.<n>We find that in-context joint entity and relation extraction at document-level remains a challenging task, even for state-of-the-art large language models.
• Score: 8.612181075294327
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Large, high-quality annotated corpora remain scarce in document-level entity and relation extraction in zero-shot or few-shot settings. In this paper, we present a fully automatic, LLM-based pipeline for synthetic data generation and in-context learning for document-level entity and relation extraction. In contrast to existing approaches that rely on manually annotated demonstrations or direct zero-shot inference, our method combines synthetic data generation with retrieval-based in-context learning, using a reasoning-optimized language model. This allows us to build a high-quality demonstration database without manual annotation and to dynamically retrieve relevant examples at inference time. Based on our approach we produce a synthetic dataset of over $5k$ Wikipedia abstracts with approximately $59k$ entities and $30k$ relation triples. Finally, we evaluate in-context learning performance on the DocIE shared task, extracting entities and relations from long documents in a zero-shot setting. We find that in-context joint entity and relation extraction at document-level remains a challenging task, even for state-of-the-art large language models.

Related papers

• WildLong: Synthesizing Realistic Long-Context Instruction Data at Scale [86.25450054683172]
  WildLong extracts meta-information from real user queries to produce scalable data.<n>It supports multi-document reasoning, such as cross-document comparison and aggregation.<n>It surpasses existing open-source long-context-optimized models across benchmarks.
  arXiv  Detail & Related papers  (2025-02-23T18:59:09Z)
• Bootstrap Your Own Context Length [74.61148597039248]
  We introduce a bootstrapping approach to train long-context language models by exploiting their short-context capabilities only.<n>The proposed data synthesis workflow requires only a short-context language model, a text retriever, and a document collection.<n>We conduct experiments with the open-source Llama-3 family of models and demonstrate that our method can successfully extend the context length to up to 1M tokens.
  arXiv  Detail & Related papers  (2024-12-25T10:08:54Z)
• Understanding Synthetic Context Extension via Retrieval Heads [51.8869530817334]
  We investigate fine-tuning on synthetic data for three long-context tasks that require retrieval and reasoning.<n>We find that models trained on synthetic data fall short of the real data, but surprisingly, the mismatch can be interpreted.<n>Our results shed light on how to interpret synthetic data fine-tuning performance and how to approach creating better data for learning real-world capabilities over long contexts.
  arXiv  Detail & Related papers  (2024-10-29T17:55:00Z)
• OneNet: A Fine-Tuning Free Framework for Few-Shot Entity Linking via Large Language Model Prompting [49.655711022673046]
  OneNet is an innovative framework that utilizes the few-shot learning capabilities of Large Language Models (LLMs) without the need for fine-tuning. OneNet is structured around three key components prompted by LLMs: (1) an entity reduction processor that simplifies inputs by summarizing and filtering out irrelevant entities, (2) a dual-perspective entity linker that combines contextual cues and prior knowledge for precise entity linking, and (3) an entity consensus judger that employs a unique consistency algorithm to alleviate the hallucination in the entity linking reasoning.
  arXiv  Detail & Related papers  (2024-10-10T02:45:23Z)
• Learning to Extract Structured Entities Using Language Models [52.281701191329]
  Recent advances in machine learning have significantly impacted the field of information extraction. We reformulate the task to be entity-centric, enabling the use of diverse metrics. We contribute to the field by introducing Structured Entity Extraction and proposing the Approximate Entity Set OverlaP metric.
  arXiv  Detail & Related papers  (2024-02-06T22:15:09Z)
• Consistency Guided Knowledge Retrieval and Denoising in LLMs for Zero-shot Document-level Relation Triplet Extraction [43.50683283748675]
  Document-level Relation Triplet Extraction (DocRTE) is a fundamental task in information systems that aims to simultaneously extract entities with semantic relations from a document. Existing methods heavily rely on a substantial amount of fully labeled data. Recent advanced Large Language Models (LLMs), such as ChatGPT and LLaMA, exhibit impressive long-text generation capabilities.
  arXiv  Detail & Related papers  (2024-01-24T17:04:28Z)
• Improving Text Embeddings with Large Language Models [59.930513259982725]
  We introduce a novel and simple method for obtaining high-quality text embeddings using only synthetic data and less than 1k training steps. We leverage proprietary LLMs to generate diverse synthetic data for hundreds of thousands of text embedding tasks across 93 languages. Experiments demonstrate that our method achieves strong performance on highly competitive text embedding benchmarks without using any labeled data.
  arXiv  Detail & Related papers  (2023-12-31T02:13:18Z)
• Semi-automatic Data Enhancement for Document-Level Relation Extraction with Distant Supervision from Large Language Models [26.523153535336725]
  Document-level Relation Extraction (DocRE) aims to extract relations from a long context. We propose a method integrating a large language model (LLM) and a natural language inference (NLI) module to generate relation triples. We demonstrate the effectiveness of our approach by introducing an enhanced dataset known as DocGNRE.
  arXiv  Detail & Related papers  (2023-11-13T13:10:44Z)
• Document-Level In-Context Few-Shot Relation Extraction via Pre-Trained Language Models [29.94694305204144]
  We present a novel framework for document-level in-context few-shot relation extraction. We evaluate our framework using DocRED, the largest publicly available dataset for document-level relation extraction.
  arXiv  Detail & Related papers  (2023-10-17T09:10:27Z)
• AIFB-WebScience at SemEval-2022 Task 12: Relation Extraction First -- Using Relation Extraction to Identify Entities [0.0]
  We present an end-to-end joint entity and relation extraction approach based on transformer-based language models. In contrast to existing approaches, which perform entity and relation extraction in sequence, our system incorporates information from relation extraction into entity extraction.
  arXiv  Detail & Related papers  (2022-03-10T12:19:44Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.