Source or It Didn't Happen: A Multi-Agent Framework for Citation Hallucination Detection

• URL: http://arxiv.org/abs/2605.08583v1
• Date: Sat, 09 May 2026 00:53:24 GMT
• Title: Source or It Didn't Happen: A Multi-Agent Framework for Citation Hallucination Detection
• Abstract summary: Existing detectors reduce to binary found/not-found decisions, offering little field-level signal to auditors.<n>We introduce a 12-code taxonomy spanning Real, Potential, and Hallucinated citations.<n>CiteTracer extracts citations from PDF and Bib, fetches evidence through lookup URL, scholar connectors, and web search.<n>It reaches 97.1% accuracy on the synthetic benchmark, with class-level F1 scores of 97.0, 95.8, and 98.5 for Real, Potential, and Hallucinated, respectively, and fabricates hallucinations on the real-world set without abstaining.
• Score: 30.352741291341843
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Large language models are increasingly used in scientific writing, yet they can fabricate citation-shaped references that appear plausible but fail bibliographic verification. Existing detectors often reduce verification to binary found/not-found decisions and rely on brittle parsing or incomplete retrieval, offering little field-level signal to auditors. We reframe citation hallucination detection as taxonomy-aligned field-level adjudication and introduce a 12-code taxonomy spanning Real, Potential, and Hallucinated citations. Based on this taxonomy, we build CiteTracer, a cascading multi-agent detector that extracts structured citations from PDF and BibTeX, retrieves evidence through cache lookup, URL fetch, scholar connectors, and web search, applies deterministic field matching, and routes ambiguous cases to class-specialist judgers. We release a benchmark of 2,450 synthetic citations built from real seeds with controlled LLM mutations, paired with 957 real-world fabricated citations drawn from ICLR 2026 and an anonymous conference desk-rejected submissions. CiteTracer reaches 97.1% accuracy on the synthetic benchmark, with class-level F1 scores of 97.0, 95.8, and 98.5 for Real, Potential, and Hallucinated, respectively, and detects 97.1% of fabrications on the real-world set without abstaining. Code: https://github.com/aaFrostnova/CiteTracer.

Related papers

• Who is the Agent to Blame? Localizing Faithfulness and Citation Mistakes in Agentic Deep Research [75.8770212601511]
  Deep research (DR) systems produce long-form cited reports by orchestrating multiple agents that search and synthesize information from the web.<n> Citations are the primary mechanism for evaluating the faithfulness of these reports, yet current DR systems exhibit poor citation recall.<n>We propose an evaluation method that pinpoints which agent introduced each error by locally testing agent invocations for faithfulness and verifiability relative to their own inputs.
  arXiv  Detail & Related papers  (2026-08-25T09:34:18Z)
• Fantastic Adaptive Taxonomies and How to Use Them [100.20614173157708]
  AdaMAST converts execution traces into a compact, evidence-grounded failure taxonomy.<n>No code is hand-authored, no trace human-annotated.<n>In agent-system search, taxonomy-coded diagnoses of failed candidates outperform free-form.
  arXiv  Detail & Related papers  (2026-07-17T17:41:16Z)
• Phantom References: Hallucinated Citations That Survive Peer Review at Top-Tier Conferences [4.766369673059568]
  We apply RefChecker to accepted camera-ready papers from ICLR, ICML, NeurIPS, and USENIX Security.<n>We measure citation hallucination in peer-reviewed proceedings using a conservative definition.<n>Results suggest peer review alone does not reliably enforce citation integrity, yet auditing is tractable.
  arXiv  Detail & Related papers  (2026-07-01T10:21:07Z)
• CiteCheck: Retrieval-Grounded Detection of LLM Citation Hallucinations in Scientific Text [39.146761527401424]
  We introduce CiteCheck, a framework for citation hallucination detection.<n>CiteCheck retrieves candidate publications from external scholarly sources, compares the citation against the retrieved candidate using a structured LLM verifier, and maps verifier scores into three labels: Exact, Minor, and Major.<n>On the held-out test set, CiteCheck achieves 88.7 macro-F1 and 88.9% accuracy, outperforming GPT, Claude, and Gemini baselines, including web-search and few-shot variants.
  arXiv  Detail & Related papers  (2026-05-26T21:20:40Z)
• HalluCiteChecker: A Lightweight Toolkit for Hallucinated Citation Detection and Verification in the Era of AI Scientists [58.87954687016989]
  HalluCiteChecker is a toolkit for detecting and verifying hallucinated citations in scientific papers.<n>Our code is released under the Apache 2.0 license on GitHub and is distributed as an installable package via PyPI.
  arXiv  Detail & Related papers  (2026-04-29T16:01:42Z)
• How unique are hallucinated citations offered by generative Artificial Intelligence models? [0.0]
  This paper investigates how generative AI produces and propagates hallucinated academic references.<n>It focuses on the recurring non-existent citation 'Education Governance and Datafication' attributed to Ben Williamson and Nelli Piattoeva.<n>It shows that hallucinated citations are not random inventions but patterned recombinations of real authors, journals, dates, and keywords.
  arXiv  Detail & Related papers  (2026-03-31T21:24:34Z)
• CiteAudit: You Cited It, But Did You Read It? A Benchmark for Verifying Scientific References in the LLM Era [51.63024682584688]
  Large language models (LLMs) introduce a new risk: fabricated references that appear plausible but correspond to no real publications.<n>We present the first comprehensive benchmark and detection framework for hallucinated citations in scientific writing.<n>Our framework significantly outperforms prior methods in both accuracy and interpretability.
  arXiv  Detail & Related papers  (2026-02-26T19:17:39Z)
• How LLMs Cite and Why It Matters: A Cross-Model Audit of Reference Fabrication in AI-Assisted Academic Writing and Methods to Detect Phantom Citations [1.0829694003408499]
  Large language models (LLMs) have been noted to fabricate scholarly citations, yet the scope of this behavior remains poorly quantified.<n>We present one of the largest hallucination citation audits to date, in which 10 commercially deployed LLMs were prompted across four academic domains.<n>Our results show that the observed hallucination rates span a fivefold range (between 11.4% and 56.8%) and are strongly shaped by model, domain, and prompt framing.
  arXiv  Detail & Related papers  (2026-02-07T00:14:00Z)
• BibAgent: An Agentic Framework for Traceable Miscitation Detection in Scientific Literature [21.872874595027824]
  BibAgent is a scalable, end-to-end agentic framework for automated citation verification.<n>It integrates retrieval, reasoning, and adaptive evidence aggregation, applying strategies for accessible and paywalled sources.<n>Our results demonstrate that BibAgent outperforms state-of-the-art Large Language Model (LLM) baselines in citation verification accuracy and interpretability.
  arXiv  Detail & Related papers  (2026-01-12T16:30:45Z)
• Hallucinations in Bibliographic Recommendation: Citation Frequency as a Proxy for Training Data Redundancy [0.0]
  Large hallucination models (LLMs) have been increasingly applied to a wide range of tasks.<n>This study hypothesizes that an LLM's ability to correctly produce information depends on whether the underlying knowledge is generated or memorized.
  arXiv  Detail & Related papers  (2025-10-29T10:51:35Z)
• DeepTRACE: Auditing Deep Research AI Systems for Tracking Reliability Across Citations and Evidence [50.97612134791782]
  Generative search engines and deep research LLM agents promise trustworthy, source-grounded synthesis, yet users regularly encounter overconfidence, weak sourcing, and confusing citation practices.<n>We introduce DeepTRACE, a novel sociotechnically grounded audit framework that turns prior community-identified failure cases into eight measurable dimensions spanning answer text, sources, and citations.
  arXiv  Detail & Related papers  (2025-09-02T00:32:38Z)
• Deep Graph Learning for Anomalous Citation Detection [55.81334139806342]
  We propose a novel deep graph learning model, namely GLAD (Graph Learning for Anomaly Detection), to identify anomalies in citation networks. Within the GLAD framework, we propose an algorithm called CPU (Citation PUrpose) to discover the purpose of citation based on citation texts.
  arXiv  Detail & Related papers  (2022-02-23T09:05:28Z)
• Towards generating citation sentences for multiple references with intent control [86.53829532976303]
  We build a novel generation model with the Fusion-in-Decoder approach to cope with multiple long inputs. Experiments demonstrate that the proposed approaches provide much more comprehensive features for generating citation sentences.
  arXiv  Detail & Related papers  (2021-12-02T15:32:24Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.