Biomedical Reference Generation Remains Unreliable across 26 Large Language Models

• URL: http://arxiv.org/abs/2609.14988v1
• Date: Mon, 14 Sep 2026 03:47:08 GMT
• Title: Biomedical Reference Generation Remains Unreliable across 26 Large Language Models
• Abstract summary: Large language models are increasingly used to help write biomedical text but may fabricate references to nonexistent work.<n>We prompted 26 language models from eight developers to supply a missing reference for each of 69 biomedical passages across ten domains.<n>A reference was considered correct in every evaluated field only when it was verifiable and its journal, year, and listed authors matched those of the cited paper.
• Score: 5.9567656649845135
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Background. Large language models are increasingly used to help write biomedical text but may fabricate references to nonexistent work. How often large language models do so is not well characterized. Methods. We prompted 26 language models from eight developers (2023 to 2026) to supply a missing reference for each of 69 biomedical passages across ten domains. References were classified as verifiable (real paper with a resolving identifier), partial matches (real paper without a resolving identifier), fabricated (no matching indexed paper), or declined (the model refused to supply a reference). A reference was considered correct in every evaluated bibliographic field only when it was verifiable and its journal, year, and listed authors matched those of the cited paper. Results. Fabrication ranged from 10.2% (Claude Opus 4.8, which declined 52.1% of prompts) to 98.4% (Ministral 3B, which produced no verifiable reference). Claude Opus 4.6 and Claude Sonnet 4.5 produced similar proportions of verifiable references (77.6% and 76.6%) but named authors correctly in 78.7% and 28.7% of author-evaluable verifiable references, respectively, and were correct in every evaluated field in 54.6% and 19.9% of responses. GPT-5.5 was correct in every field in 48.1%. Across all models, 55.4% of responses were fabricated and 14.9% were correct in every field. Among the five tested models first released in 2026, the corresponding proportions were 35.3% and 31.8%, respectively. Conclusions. Fabrication remained common, and no model was correct in every evaluated bibliographic field in more than 54.6% of responses. Models that identify real papers may still misstate their metadata, so references produced with model assistance require verification before use.

Related papers

• IdeaAMBIG: Benchmarking Implementation-Critical Gaps in Research-Idea Specifications [52.570108663867046]
  A research idea may be novel, coherent, and scientifically plausible, yet its proposed method may remain insufficiently specified for faithful implementation.<n>We study the codification readiness of implementation-facing research-method specifications, defined by whether they provide sufficient methodological information for a competent implementer or coding agent to construct the intended method without unsupported assumptions.<n>We introduce IdeaAMBIG, a benchmark of 660 evidence-grounded instances: 163 real-world gaps from reports and GitHub issues, and 497 controlled synthetic gaps injected into codification-ready references.
  arXiv  Detail & Related papers  (2026-09-09T17:59:04Z)
• Correction as Annotation: Bootstrapping a Dependency Parser for Documentary Medieval Latin [0.0]
  Medieval documentary sources remain inadequately served by natural language processing tools.<n>None of the five readily available Latin treebank models attains usable performance on a collection of 160 inventories compiled in Marseille between 1258 and 1446.
  arXiv  Detail & Related papers  (2026-09-05T21:23:35Z)
• When AI Writes, Who Gets Cited? Evidence of Citation Monoculture Across Language Models [45.91367912142692]
  We show that different models may still select the same narrow subset, producing citation monoculture without any single citation being wrong.<n>Eleven models from three vendors choose at most ten papers from uniformly random panels of thirty, with real titles and abstracts but fabricated authors, reassigned years, and hidden venues and citation counts.
  arXiv  Detail & Related papers  (2026-08-03T14:27:46Z)
• Do large language models scrutinise what they review? A multimodal audit of scoring calibration, error detection, and author-identity effects [0.0]
  Large language models (LLMs) are increasingly used to generate peer reviews.<n>This study evaluates two multimodal LLMs as reviewers across 165 submissions to the 2026 International Conference on Learning Representations.
  arXiv  Detail & Related papers  (2026-07-31T12:12:33Z)
• How Far Is Document Parsing from Solved? PureDocBench: A Source-TraceableBenchmark across Clean, Degraded, and Real-World Settings [56.70440596502351]
  The past year has seen over 20 open document parsing models, yet benchmarks almost exclusively on OmniDocBench.<n>We present PureDocBench, a benchmark thatrenders document images from HTML/CSS, covering 10 domains, 66 pages, and 1,475 pages.
  arXiv  Detail & Related papers  (2026-05-08T09:30:31Z)
• BibTeX Citation Hallucinations in Scientific Publishing Agents: Evaluation and Mitigation [34.429649156970015]
  Prior evaluations tested base models without search, which does not reflect current practice.<n>Three search-enabled frontier models generate Bib entries scored on nine fields and a six-way error taxonomy.<n>Overall accuracy is 83.6%, but only 50.9% of entries are fully correct.
  arXiv  Detail & Related papers  (2026-04-03T16:30:58Z)
• PanCanBench: A Comprehensive Benchmark for Evaluating Large Language Models in Pancreatic Oncology [48.732366302949515]
  Large language models (LLMs) have achieved expert-level performance on standardized examinations, yet multiple-choice accuracy poorly reflects real-world clinical utility and safety.<n>We developed a human-in-the-loop pipeline to create expert rubrics for de-identified patient questions.<n>We evaluated 22 proprietary and open-source LLMs using an LLM-as-a-judge framework, measuring clinical completeness, factual accuracy, and web-search integration.
  arXiv  Detail & Related papers  (2026-03-02T00:50:39Z)
• Beyond Correctness: Evaluating Subjective Writing Preferences Across Cultures [87.75098311090642]
  Current preference learning methods achieve high accuracy on standard benchmarks but exhibit significant performance degradation when objective quality signals are removed.<n>We introduce WritingPreferenceBench, a dataset of 1,800 human-annotated preference pairs (1,200 English, 600 Chinese) across 8 creative writing genres.
  arXiv  Detail & Related papers  (2025-10-16T12:23:13Z)
• Instruct-Tuning Pretrained Causal Language Models for Ancient Greek Papyrology and Epigraphy [0.0]
  This article presents an experiment in fine-tuning a pretrained causal language model to restore missing or illegible characters in ancient Greek inscriptions and documentary papyri. Benchmarked against the state-of-the-art model (Ithaca), the instruction-tuned models excelled in text restoration. Results suggest that fine-tuning larger pretrained causal language models using instruction templates for emendations and conjectures holds promise.
  arXiv  Detail & Related papers  (2024-09-20T19:49:45Z)
• PART: Pre-trained Authorship Representation Transformer [52.623051272843426]
  Authors writing documents imprint identifying information within their texts.<n>Previous works use hand-crafted features or classification tasks to train their authorship models.<n>We propose a contrastively trained model fit to learn textbfauthorship embeddings instead of semantics.
  arXiv  Detail & Related papers  (2022-09-30T11:08:39Z)
• Document Classification for COVID-19 Literature [15.458071120159307]
  We provide an analysis of several multi-label document classification models on the LitCovid dataset. We find that pre-trained language models fine-tuned on this dataset outperform all other baselines. We also explore 50 errors made by the best performing models on LitCovid documents.
  arXiv  Detail & Related papers  (2020-06-15T20:03:28Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.