BibTeX Citation Hallucinations in Scientific Publishing Agents: Evaluation and Mitigation

• URL: http://arxiv.org/abs/2604.03159v1
• Date: Fri, 03 Apr 2026 16:30:58 GMT
• Title: BibTeX Citation Hallucinations in Scientific Publishing Agents: Evaluation and Mitigation
• Abstract summary: Prior evaluations tested base models without search, which does not reflect current practice.<n>Three search-enabled frontier models generate Bib entries scored on nine fields and a six-way error taxonomy.<n>Overall accuracy is 83.6%, but only 50.9% of entries are fully correct.
• Score: 34.429649156970015
• License: http://creativecommons.org/publicdomain/zero/1.0/
• Abstract: Large language models with web search are increasingly used in scientific publishing agents, yet they still produce BibTeX entries with pervasive field-level errors. Prior evaluations tested base models without search, which does not reflect current practice. We construct a benchmark of 931 papers across four scientific domains and three citation tiers -- popular, low-citation, and recent post-cutoff -- designed to disentangle parametric memory from search dependence, with version-aware ground truth accounting for multiple citable versions of the same paper. Three search-enabled frontier models (GPT-5, Claude Sonnet-4.6, Gemini-3 Flash) generate BibTeX entries scored on nine fields and a six-way error taxonomy, producing ~23,000 field-level observations. Overall accuracy is 83.6%, but only 50.9% of entries are fully correct; accuracy drops 27.7pp from popular to recent papers, revealing heavy reliance on parametric memory even when search is available. Field-error co-occurrence analysis identifies two failure modes: wholesale entry substitution (identity fields fail together) and isolated field error. We evaluate clibib, an open-source tool for deterministic BibTeX retrieval from the Zotero Translation Server with CrossRef fallback, as a mitigation mechanism. In a two-stage integration where baseline entries are revised against authoritative records, accuracy rises +8.0pp to 91.5%, fully correct entries rise from 50.9% to 78.3%, and regression rate is only 0.8%. An ablation comparing single-stage and two-stage integration shows that separating search from revision yields larger gains and lower regression (0.8% vs. 4.8%), demonstrating that integration architecture matters independently of model capability. We release the benchmark, error taxonomy, and clibib tool to support evaluation and mitigation of citation hallucinations in LLM-based scientific writing.

Related papers

• Biomedical Reference Generation Remains Unreliable across 26 Large Language Models [5.9567656649845135]
  Large language models are increasingly used to help write biomedical text but may fabricate references to nonexistent work.<n>We prompted 26 language models from eight developers to supply a missing reference for each of 69 biomedical passages across ten domains.<n>A reference was considered correct in every evaluated field only when it was verifiable and its journal, year, and listed authors matched those of the cited paper.
  arXiv  Detail & Related papers  (2026-09-14T03:47:08Z)
• Automated Proving of Shannon-Type Entropy Inequalities via Fine-Tuned Language Models and Guided Tree Search [50.16356451328644]
  Proving Shannon-type entropy inequalities is a fundamental task in information theory.<n>We investigate whether small-scale large language models, fine-tuned on atomic proof steps, can automate this process.<n>We show that GPT-5.5 solves 1.7% samples under zero-shot prompting while Psitip solves 33.3% samples.
  arXiv  Detail & Related papers  (2026-06-04T05:43:12Z)
• Amplifying, Not Learning: Fine-Tuned AI Text Detectors Amplify a Pretrained Direction [51.56484100374058]
  Text detectors amplify a pretrained typicality axis; they do not construct an AI-vs-human boundary.<n>On raw encoders before any task supervision, projecting onto centroid(AI)-centroid(HC3) achieves NYT-vs-HC3 AUROC 0.806/0.944/0.834 across three architectures.<n>On RoBERTa-base, raw projection exceeds fine-tuning; on RoBERTa-base, full fine-tuning reduces discrimination below raw on both fluent-formal populations tested.
  arXiv  Detail & Related papers  (2026-05-20T19:08:38Z)
• PBT-Bench: Benchmarking AI Agents on Property-Based Testing [29.035258104995204]
  We introduce PBT-Bench, a benchmark of 100 curated property-based testing problems across 40 real Python libraries.<n>Each problem injects one or more semantic bugs (365 in total, mean 3.65 per problem) designed so that default-strategy random inputs almost never trigger them.<n>Bug recall under the PBT-guided prompt ranges from 42.1% to 83.4% across models; under the open-ended baseline, from 31.4% to 76.7%.
  arXiv  Detail & Related papers  (2026-05-13T18:01:05Z)
• How Far Is Document Parsing from Solved? PureDocBench: A Source-TraceableBenchmark across Clean, Degraded, and Real-World Settings [56.70440596502351]
  The past year has seen over 20 open document parsing models, yet benchmarks almost exclusively on OmniDocBench.<n>We present PureDocBench, a benchmark thatrenders document images from HTML/CSS, covering 10 domains, 66 pages, and 1,475 pages.
  arXiv  Detail & Related papers  (2026-05-08T09:30:31Z)
• When Correct Isn't Usable: Improving Structured Output Reliability in Small Language Models [2.064923532131528]
  Deployed language models must produce outputs that are both correct and format-compliant.<n>We study this structured-output reliability gap using two mathematical benchmarks -- GSM8K and MATH.<n>We develop AloLab, an iterative system-prompt requiring only black-box API access to the target model.
  arXiv  Detail & Related papers  (2026-05-04T09:07:44Z)
• Semantic Layers for Reliable LLM-Powered Data Analytics: A Paired Benchmark of Accuracy and Hallucination Across Three Frontier Models [0.0]
  LLMs deployed for natural-language querying of analytical databases suffer from two intertwined failures.<n>We benchmark three frontier LLMs on 100 natural-language questions over the Cleaned Contoso Retail dataset in ClickHouse.
  arXiv  Detail & Related papers  (2026-04-28T02:53:23Z)
• How LLMs Cite and Why It Matters: A Cross-Model Audit of Reference Fabrication in AI-Assisted Academic Writing and Methods to Detect Phantom Citations [1.0829694003408499]
  Large language models (LLMs) have been noted to fabricate scholarly citations, yet the scope of this behavior remains poorly quantified.<n>We present one of the largest hallucination citation audits to date, in which 10 commercially deployed LLMs were prompted across four academic domains.<n>Our results show that the observed hallucination rates span a fivefold range (between 11.4% and 56.8%) and are strongly shaped by model, domain, and prompt framing.
  arXiv  Detail & Related papers  (2026-02-07T00:14:00Z)
• GhostCite: A Large-Scale Analysis of Citation Validity in the Age of Large Language Models [22.147294042024836]
  Citations provide the basis for trusting scientific claims; when they are invalid or fabricated, this trust collapses.<n>With the advent of Large Language Models (LLMs), this risk has intensified.<n>We develop CiteVerifier, an open-source framework for large-scale citation verification.
  arXiv  Detail & Related papers  (2026-02-06T14:08:34Z)
• Solver-in-the-Loop: MDP-Based Benchmarks for Self-Correction and Behavioral Rationality in Operations Research [19.31559944205485]
  Operations Research practitioners routinely debug infeasible models through an iterative process.<n>We introduce two benchmarks that place the textbfsolver in the evaluation loop<n>We find that domain-specific RLVR training enables an 8B model to surpass frontier APIs.
  arXiv  Detail & Related papers  (2026-01-28T20:02:44Z)
• Assessing LLM Reliability on Temporally Recent Open-Domain Questions [15.456770184839726]
  Large Language Models (LLMs) are increasingly deployed for open-domain question answering.<n>We investigate how four open-source LLMs respond to 15,000 recent Reddit questions.<n>All models achieve over 99% cosine similarity with references despite less than 8% BLEU-1 overlap.
  arXiv  Detail & Related papers  (2026-01-17T21:33:27Z)
• CorBenchX: Large-Scale Chest X-Ray Error Dataset and Vision-Language Model Benchmark for Report Error Correction [11.731590131260424]
  CorBenchX is a suite for automated error detection and correction in chest X-ray reports.<n>We first synthesize a large-scale dataset of 26,326 chest X-ray error reports.<n>We benchmark both open- and closed-source vision-language models.
  arXiv  Detail & Related papers  (2025-05-17T15:39:39Z)
• Benchmarking Reasoning Robustness in Large Language Models [76.79744000300363]
  We find significant performance degradation on novel or incomplete data.<n>These findings highlight the reliance on recall over rigorous logical inference.<n>This paper introduces a novel benchmark, termed as Math-RoB, that exploits hallucinations triggered by missing information to expose reasoning gaps.
  arXiv  Detail & Related papers  (2025-03-06T15:36:06Z)
• Preference Optimization for Reasoning with Pseudo Feedback [100.62603571434167]
  We introduce a novel approach to generate pseudo feedback for reasoning tasks by framing the labeling of solutions as an evaluation against associated test cases.<n>We conduct experiments on both mathematical reasoning and coding tasks using pseudo feedback for preference optimization, and observe improvements across both tasks.
  arXiv  Detail & Related papers  (2024-11-25T12:44:02Z)
• Large Language Monkeys: Scaling Inference Compute with Repeated Sampling [81.34900892130929]
  We explore inference compute as another axis for scaling, using the simple technique of repeatedly sampling candidate solutions from a model.<n>Across multiple tasks and models, we observe that coverage scales with the number of samples over four orders of magnitude.<n>In domains like coding and formal proofs, where answers can be automatically verified, these increases in coverage directly translate into improved performance.
  arXiv  Detail & Related papers  (2024-07-31T17:57:25Z)
• TACRED Revisited: A Thorough Evaluation of the TACRED Relation Extraction Task [80.38130122127882]
  TACRED is one of the largest, most widely used crowdsourced datasets in Relation Extraction (RE) In this paper, we investigate the questions: Have we reached a performance ceiling or is there still room for improvement? We find that label errors account for 8% absolute F1 test error, and that more than 50% of the examples need to be relabeled.
  arXiv  Detail & Related papers  (2020-04-30T15:07:37Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.