Extracting Dataset Mentions in Forced Displacement and FCV Documents: A Weakly Supervised Framework with LLM-Based Label Refinement

• URL: http://arxiv.org/abs/2609.12107v1
• Date: Thu, 10 Sep 2026 18:34:08 GMT
• Title: Extracting Dataset Mentions in Forced Displacement and FCV Documents: A Weakly Supervised Framework with LLM-Based Label Refinement
• Abstract summary: We present a framework for adapting dataset extraction to forced displacement and Fragile, Conflict, and Violence (FCV) documents.<n>A lightweight model trained on general research literature generates candidate dataset mentions from unlabeled domain documents.<n>We evaluate the resulting model on an independent gold-standard benchmark of 1,706 text passages spanning research, humanitarian, and operational documents.
• Score: 0.0
• License: http://creativecommons.org/licenses/by-sa/4.0/
• Abstract: Development and humanitarian organizations produce and support surveys, administrative registries, and other data resources to inform research, policy, and operations, yet systematically identifying where these datasets are referenced remains difficult. Such references are dispersed across research papers, project documents, humanitarian reports, and other unstructured text, limiting both the ability to trace data use and to identify potential gaps in data availability or dissemination. We present a weakly supervised framework for adapting dataset extraction to forced displacement and Fragile, Conflict, and Violence (FCV) documents without first constructing a large manually labeled training corpus. A lightweight model trained on general research literature generates candidate dataset mentions from unlabeled domain documents, which a frontier large language model (LLM) reviews in context, validating or rejecting candidates and correcting their extraction boundaries. The resulting annotations are supplemented with targeted synthetic and contrastive examples and used to fine-tune the lightweight model for large-scale extraction. We evaluate the resulting model on an independent gold-standard benchmark of 1,706 text passages spanning research, humanitarian, and operational documents. Across the full benchmark, the model achieves 74.1\% precision and 70.5\% recall at the mention level; among passages containing dataset references, precision reaches 89.5\%. At the passage level, the model achieves 88.2\% accuracy and 88.6\% specificity in distinguishing passages with dataset references from those without them. These results demonstrate a practical approach for constructing domain-specific supervision when labeled data are limited, and provide a technical foundation for larger-scale analysis of data use and potential gaps in the displacement data landscape.

Related papers

• Walk Before You Run: The Importance of Data Exploration for Data Analysis Agents [7.140798105546584]
  In current tools and benchmarks, this step is usually left implicit, creating a gap between downstream task performance and dataset understanding.<n>We show through downstream experiments that stronger Data Exploration support improves task performance.<n>Our results show that strong LLMs and data-analysis agents still miss important logical structure even when they read spreadsheet content.
  arXiv  Detail & Related papers  (2026-08-17T03:19:59Z)
• Benchmarking Open-Source Layout Detection Models for Data Snapshot Extraction from Institutional Documents [0.0]
  We introduce a benchmark dataset and evaluation framework for textitdata snapshot extraction.<n>We benchmarked multiple open-source layout detection models and evaluated both detection performance and spatial extraction quality.<n>These findings highlight a persistent gap between generic document layout analysis and operationally useful data snapshot extraction.
  arXiv  Detail & Related papers  (2026-06-04T14:47:40Z)
• AI for Monitoring and Classifying Data Used in Research Literature [0.0]
  No comparable infrastructure exists for monitoring dataset usage in research literature.<n>Traditional NLP approaches struggle with these challenges.<n>We introduce a multitask GLiNER-based framework that jointly performs dataset mention extraction, relation identification, and usage-context classification.
  arXiv  Detail & Related papers  (2026-05-28T21:18:30Z)
• OpenDataArena: A Fair and Open Arena for Benchmarking Post-Training Dataset Value [74.80873109856563]
  OpenDataArena (ODA) is a holistic and open platform designed to benchmark the intrinsic value of post-training data.<n>ODA establishes a comprehensive ecosystem comprising four key pillars: (i) a unified training-evaluation pipeline that ensures fair, open comparisons across diverse models; (ii) a multi-dimensional scoring framework that profiles data quality along tens of distinct axes; and (iii) an interactive data lineage explorer to visualize dataset genealogy and dissect component sources.
  arXiv  Detail & Related papers  (2025-12-16T03:33:24Z)
• Large Language Models and Synthetic Data for Monitoring Dataset Mentions in Research Papers [0.0]
  This paper presents a machine learning framework that automates dataset mention detection across research domains.<n>We employ zero-shot extraction from research papers, an LLM-as-a-Judge for quality assessment, and a reasoning agent for refinement to generate a weakly supervised synthetic dataset.<n>At inference, a ModernBERT-based classifier efficiently filters dataset mentions, reducing computational overhead while maintaining high recall.
  arXiv  Detail & Related papers  (2025-02-14T16:16:02Z)
• Downstream-Pretext Domain Knowledge Traceback for Active Learning [138.02530777915362]
  We propose a downstream-pretext domain knowledge traceback (DOKT) method that traces the data interactions of downstream knowledge and pre-training guidance. DOKT consists of a traceback diversity indicator and a domain-based uncertainty estimator. Experiments conducted on ten datasets show that our model outperforms other state-of-the-art methods.
  arXiv  Detail & Related papers  (2024-07-20T01:34:13Z)
• STAR: Boosting Low-Resource Information Extraction by Structure-to-Text Data Generation with Large Language Models [56.27786433792638]
  STAR is a data generation method that leverages Large Language Models (LLMs) to synthesize data instances. We design fine-grained step-by-step instructions to obtain the initial data instances. Our experiments show that the data generated by STAR significantly improve the performance of low-resource event extraction and relation extraction tasks.
  arXiv  Detail & Related papers  (2023-05-24T12:15:19Z)
• Utilizing Out-Domain Datasets to Enhance Multi-Task Citation Analysis [4.526582372434088]
  Citation sentiment analysis suffers from both data scarcity and tremendous costs for dataset annotation. We explore the impact of out-domain data during training to enhance the model performance. We propose an end-to-end trainable multi-task model that covers the sentiment and intent analysis.
  arXiv  Detail & Related papers  (2022-02-22T13:33:48Z)
• Data-SUITE: Data-centric identification of in-distribution incongruous examples [81.21462458089142]
  Data-SUITE is a data-centric framework to identify incongruous regions of in-distribution (ID) data. We empirically validate Data-SUITE's performance and coverage guarantees.
  arXiv  Detail & Related papers  (2022-02-17T18:58:31Z)
• Partially-Aligned Data-to-Text Generation with Distant Supervision [69.15410325679635]
  We propose a new generation task called Partially-Aligned Data-to-Text Generation (PADTG) It is more practical since it utilizes automatically annotated data for training and thus considerably expands the application domains. Our framework outperforms all baseline models as well as verify the feasibility of utilizing partially-aligned data.
  arXiv  Detail & Related papers  (2020-10-03T03:18:52Z)
• Summary-Source Proposition-level Alignment: Task, Datasets and Supervised Baseline [94.0601799665342]
  Aligning sentences in a reference summary with their counterparts in source documents was shown as a useful auxiliary summarization task. We propose establishing summary-source alignment as an explicit task, while introducing two major novelties. We create a novel training dataset for proposition-level alignment, derived automatically from available summarization evaluation data. We present a supervised proposition alignment baseline model, showing improved alignment-quality over the unsupervised approach.
  arXiv  Detail & Related papers  (2020-09-01T17:27:12Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.