Benchmarking Large Language Models on Reference Extraction and Parsing in the Social Sciences and Humanities

• URL: http://arxiv.org/abs/2603.13651v1
• Date: Fri, 13 Mar 2026 23:25:03 GMT
• Title: Benchmarking Large Language Models on Reference Extraction and Parsing in the Social Sciences and Humanities
• Abstract summary: Most established evaluations focus on clean, English, end-of-document, and therefore underrepresent the Social Sciences and Humanities.<n>We present a unified benchmark that targets these SSH-realistic conditions across three complementary datasets.<n>We evaluate three tasks of increasing difficulty -- reference extraction, reference parsing, and end-to-end document parsing.
• Score: 11.04552942214836
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Bibliographic reference extraction and parsing are foundational for citation indexing, linking, and downstream scholarly knowledge-graph construction. However, most established evaluations focus on clean, English, end-of-document bibliographies, and therefore underrepresent the Social Sciences and Humanities (SSH), where citations are frequently multilingual, embedded in footnotes, abbreviated, and shaped by heterogeneous historical conventions. We present a unified benchmark that targets these SSH-realistic conditions across three complementary datasets: CEX (English journal articles spanning multiple disciplines), EXCITE (German/English documents with end-section, footnote-only, and mixed regimes), and LinkedBooks (humanities references with strong stylistic variation and multilinguality). We evaluate three tasks of increasing difficulty -- reference extraction, reference parsing, and end-to-end document parsing -- under a schema-constrained setup that enables direct comparison between a strong supervised pipeline baseline (GROBID) and contemporary LLMs (DeepSeek-V3.1, Mistral-Small-3.2-24B, Gemma-3-27B-it, and Qwen3-VL (4B-32B variants)). Across datasets, extraction largely saturates beyond a moderate capability threshold, while parsing and end-to-end parsing remain the primary bottlenecks due to structured-output brittleness under noisy layouts. We further show that lightweight LoRA adaptation yields consistent gains -- especially on SSH-heavy benchmarks -- and that segmentation/pipelining can substantially improve robustness. Finally, we argue for hybrid deployment via routing: leveraging GROBID for well-structured, in-distribution PDFs while escalating multilingual and footnote-heavy documents to task-adapted LLMs.

Related papers

• Dataset Scarcity Limits Robust Evaluation of Multilingual Embedding Models: A Case Study of Slavic Languages [3.9407886544063384]
  We propose a two-dimensional framework for analyzing multilingual embedding benchmarks under dataset scarcity.<n>The framework distinguishes between task-specific and cross-task evaluation, while jointly analyzing three complementary aspects.<n>Our analysis reveals severe benchmark sparsity, with many Slavic language-task pairs relying on a single dataset or highly correlated benchmark collections.
  arXiv  Detail & Related papers  (2026-08-25T12:23:31Z)
• MGAL: A Multilingual Granularity-Aware Long-Context Benchmark [24.725846756080788]
  MGAL is constructed from United Nations (UN) reports spanning 8K to 128K tokens across the six official UN languages.<n>It covers four coherent levels of linguistic granularity (word, sentence, paragraph, and document) and further stratifies entries by their position within the document.<n>This design enables systematic diagnosis of multilingual long-context comprehension across different granularities.
  arXiv  Detail & Related papers  (2026-08-21T08:19:00Z)
• G^2C-MT: Graph-Guided Context Selection for Document-Level Machine Translation [7.730467866001466]
  Document-level machine translation (DocMT) requires capturing long-range discourse dependencies.<n>G2C-MT views DocMT context selection as a structured path discovery problem on a lightweight discourse graph.<n>This framework naturally supports multi-path context sampling, which can improve robustness by aggregating diverse translation candidates for discourse-ambiguous inputs.
  arXiv  Detail & Related papers  (2026-06-02T03:09:15Z)
• Job Skill Extraction via LLM-Centric Multi-Module Framework [51.37063712967151]
  We present SRICL, an LLM-centric framework that combines semantic retrieval, in-context learning, and supervised fine-tuning.<n>On six public span-labeled corpora of job-ad sentences, SRICL achieves substantial STRICT-F1 improvements over GPT-3.5 prompting baselines and sharply reduces invalid tags and hallucinated spans.
  arXiv  Detail & Related papers  (2026-04-23T10:46:07Z)
• LEMUR: A Corpus for Robust Fine-Tuning of Multilingual Law Embedding Models for Retrieval [18.46710400838861]
  Large language models (LLMs) are increasingly used to access legal information.<n>Yet, their deployment in multilingual legal settings is constrained by unreliable retrieval and the lack of domain-adapted, open-embedding models.<n>We introduce LEMUR, a large-scale multilingual corpus of EU environmental legislation constructed from 24,953 official EUR-Lex PDF documents covering 25 languages.
  arXiv  Detail & Related papers  (2026-02-10T09:20:24Z)
• Paragraph Segmentation Revisited: Towards a Standard Task for Structuring Speech [61.00008468914252]
  We recast paragraph segmentation as the missing structuring step and fill three gaps at the intersection of speech processing and text segmentation.<n> benchmarks focus on the underexplored speech domain, where paragraph segmentation has traditionally not been part of post-processing.<n>Second, we propose a constrained-decoding formulation that lets large language models insert paragraph breaks while preserving the original transcript.<n>Third, we show that a compact model (MiniSeg) attains state-of-the-art accuracy and, when extended hierarchically, jointly predicts chapters and paragraphs with minimal computational cost.
  arXiv  Detail & Related papers  (2025-12-30T23:29:51Z)
• Beyond Isolated Dots: Benchmarking Structured Table Construction as Deep Knowledge Extraction [80.88654868264645]
  Arranged and Organized Extraction Benchmark designed to evaluate ability of large language models to comprehend fragmented documents.<n>AOE includes 11 carefully crafted tasks across three diverse domains, requiring models to generate context-specific schema tailored to varied input queries.<n>Results show that even the most advanced models struggled significantly.
  arXiv  Detail & Related papers  (2025-07-22T06:37:51Z)
• Hierarchical Lexical Graph for Enhanced Multi-Hop Retrieval [22.33550491040999]
  RAG grounds large language models in external evidence, yet it still falters when answers must be pieced together across semantically distant documents.<n>We build two plug-and-play retrievers: StatementGraphRAG and TopicGraphRAG.<n>Our methods outperform naive chunk-based RAG achieving an average relative improvement of 23.1% in retrieval recall and correctness.
  arXiv  Detail & Related papers  (2025-06-09T17:58:35Z)
• Unstructured Evidence Attribution for Long Context Query Focused Summarization [53.08341620504465]
  We propose to extract unstructured (i.e., spans of any length) evidence in order to acquire more relevant and consistent evidence than in the fixed granularity case.<n>We show how existing systems struggle to copy and properly cite unstructured evidence, which also tends to be "lost-in-the-middle"
  arXiv  Detail & Related papers  (2025-02-20T09:57:42Z)
• Graph-DPEP: Decomposed Plug and Ensemble Play for Few-Shot Document Relation Extraction with Graph-of-Thoughts Reasoning [34.85741925091139]
  Graph-DPEP framework is grounded in the reasoning behind triplet explanation thoughts presented in natural language. We develop "ensemble-play", reapplying generation on the entire type list by leveraging the reasoning thoughts embedded in a sub-graph.
  arXiv  Detail & Related papers  (2024-11-05T07:12:36Z)
• Are the Best Multilingual Document Embeddings simply Based on Sentence Embeddings? [18.968571816913208]
  We provide a systematic comparison of methods to produce document-level representations from sentences based on LASER, LaBSE, and Sentence BERT pre-trained multilingual models. We show that a clever combination of sentence embeddings is usually better than encoding the full document as a single unit.
  arXiv  Detail & Related papers  (2023-04-28T12:11:21Z)
• Advancing Multilingual Pre-training: TRIP Triangular Document-level Pre-training for Multilingual Language Models [107.83158521848372]
  We present textbfTriangular Document-level textbfPre-training (textbfTRIP), which is the first in the field to accelerate the conventional monolingual and bilingual objectives into a trilingual objective with a novel method called Grafting. TRIP achieves several strong state-of-the-art (SOTA) scores on three multilingual document-level machine translation benchmarks and one cross-lingual abstractive summarization benchmark, including consistent improvements by up to 3.11 d-BLEU points and 8.9 ROUGE-L points.
  arXiv  Detail & Related papers  (2022-12-15T12:14:25Z)
• SgSum: Transforming Multi-document Summarization into Sub-graph Selection [27.40759123902261]
  Most existing extractive multi-document summarization (MDS) methods score each sentence individually and extract salient sentences one by one to compose a summary. We propose a novel MDS framework (SgSum) to formulate the MDS task as a sub-graph selection problem. Our model can produce significantly more coherent and informative summaries compared with traditional MDS methods.
  arXiv  Detail & Related papers  (2021-10-25T05:12:10Z)
• Towards Making the Most of Context in Neural Machine Translation [112.9845226123306]
  We argue that previous research did not make a clear use of the global context. We propose a new document-level NMT framework that deliberately models the local context of each sentence.
  arXiv  Detail & Related papers  (2020-02-19T03:30:00Z)
• Multilingual Alignment of Contextual Word Representations [49.42244463346612]
  BERT exhibits significantly improved zero-shot performance on XNLI compared to the base model. We introduce a contextual version of word retrieval and show that it correlates well with downstream zero-shot transfer. These results support contextual alignment as a useful concept for understanding large multilingual pre-trained models.
  arXiv  Detail & Related papers  (2020-02-10T03:27:21Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.