Testing Our Foundations: Citation Trends, Errors, and Emerging Hallucinations in the Computing Education Literature

• URL: http://arxiv.org/abs/2609.16574v1
• Date: Tue, 15 Sep 2026 03:18:20 GMT
• Title: Testing Our Foundations: Citation Trends, Errors, and Emerging Hallucinations in the Computing Education Literature
• Abstract summary: We analyze referencing trends across 24,751 computing education papers and compare them with the broader ACM corpus of more than 723,000 papers and 15 million references.<n>In 2025, hallucinated references appeared across five SIGCSE-sponsored or LLM-cooperation venues.<n>Although still relatively rare for now, this growth poses an integrity risk our community should not ignore.
• Score: 3.299067446425981
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Accurate references are foundational to scholarly work, enabling verification, attribution, and systematic review. However, the rapid adoption of large language models has introduced a serious integrity concern: plausible-looking but fabricated citations. Although hallucinated references are widely discussed, their visibility within specific research communities remains unclear. We address this gap by examining reference integrity at key computing education venues using ACM Digital Library data. We analyze referencing trends across 24,751 computing education papers and compare them with the broader ACM corpus of more than 723,000 papers and 15 million references. We then examine reference lists from these venues, classify common bibliographic errors, and manually identify LLM-generated hallucinations containing verifiably false information, including impossible page ranges, invented titles, and misattributed authors. In 2025, hallucinated references appeared across five SIGCSE-sponsored or in-cooperation venues. At the Technical Symposium alone, verified hallucinated references increased from 3 in 2025 to 17 in 2026, appearing in 2.3\% of 2026 proceedings papers. Although still relatively rare for now, this growth poses an integrity risk our community should not ignore.

Related papers

• Deep Academic Survey: Stateful Agentic Closed-Loop Paradigm for Academic Survey Automation [60.151050016855834]
  DAS is a stateful agentic framework for generating publication-oriented academic surveys.<n>Its key idea is to separate reusable paper analysis from topic-specific manuscript construction.<n>DAS achieves the highest average in all four dimensions, with an overall score of 4.34 compared with 4.03 for the strongest competitor.
  arXiv  Detail & Related papers  (2026-08-18T17:29:00Z)
• Phantom References: Hallucinated Citations That Survive Peer Review at Top-Tier Conferences [4.766369673059568]
  We apply RefChecker to accepted camera-ready papers from ICLR, ICML, NeurIPS, and USENIX Security.<n>We measure citation hallucination in peer-reviewed proceedings using a conservative definition.<n>Results suggest peer review alone does not reliably enforce citation integrity, yet auditing is tractable.
  arXiv  Detail & Related papers  (2026-07-01T10:21:07Z)
• LLM hallucinations in the wild: Large-scale evidence from non-existent citations [5.051133029117994]
  We audit 111 million references across 2.5 million papers in arXiv, bioRxiv, SSRN, and PubMed Central.<n>We find a sharp rise in non-existent references following widespread large language models adoption.<n>These errors are diffusely embedded across many papers but especially pronounced in fields with rapid AI uptake.
  arXiv  Detail & Related papers  (2026-05-08T13:26:41Z)
• CiteAudit: You Cited It, But Did You Read It? A Benchmark for Verifying Scientific References in the LLM Era [51.63024682584688]
  Large language models (LLMs) introduce a new risk: fabricated references that appear plausible but correspond to no real publications.<n>We present the first comprehensive benchmark and detection framework for hallucinated citations in scientific writing.<n>Our framework significantly outperforms prior methods in both accuracy and interpretability.
  arXiv  Detail & Related papers  (2026-02-26T19:17:39Z)
• GhostCite: A Large-Scale Analysis of Citation Validity in the Age of Large Language Models [22.147294042024836]
  Citations provide the basis for trusting scientific claims; when they are invalid or fabricated, this trust collapses.<n>With the advent of Large Language Models (LLMs), this risk has intensified.<n>We develop CiteVerifier, an open-source framework for large-scale citation verification.
  arXiv  Detail & Related papers  (2026-02-06T14:08:34Z)
• HalluCitation Matters: Revealing the Impact of Hallucinated References with 300 Hallucinated Papers in ACL Conferences [58.87954687016989]
  We analyze all papers published at ACL, NAACL, and EMNLP in 2024 and 2025.<n>Half of these papers were identified at EMNLP 2025, indicating that this issue is rapidly increasing.<n>More than 100 such papers were accepted as main conference and Findings papers at EMNLP 2025, affecting the credibility.
  arXiv  Detail & Related papers  (2026-01-26T17:48:23Z)
• Hallucinate or Memorize? The Two Sides of Probabilistic Learning in Large Language Models [0.0]
  Large language models (LLMs) have been increasingly applied to a wide range of tasks, from natural language understanding to code generation.<n>While they have also been used to assist in citation recommendation, the hallucination of non-existent papers remains a major issue.<n>This study hypothesizes that an LLM's ability to correctly produce records depends on whether the underlying knowledge is generated or memorized.
  arXiv  Detail & Related papers  (2025-11-12T01:29:12Z)
• Hallucinations in Bibliographic Recommendation: Citation Frequency as a Proxy for Training Data Redundancy [0.0]
  Large hallucination models (LLMs) have been increasingly applied to a wide range of tasks.<n>This study hypothesizes that an LLM's ability to correctly produce information depends on whether the underlying knowledge is generated or memorized.
  arXiv  Detail & Related papers  (2025-10-29T10:51:35Z)
• Mapping the Increasing Use of LLMs in Scientific Papers [99.67983375899719]
  We conduct the first systematic, large-scale analysis across 950,965 papers published between January 2020 and February 2024 on the arXiv, bioRxiv, and Nature portfolio journals. Our findings reveal a steady increase in LLM usage, with the largest and fastest growth observed in Computer Science papers.
  arXiv  Detail & Related papers  (2024-04-01T17:45:15Z)
• Monitoring AI-Modified Content at Scale: A Case Study on the Impact of ChatGPT on AI Conference Peer Reviews [51.453135368388686]
  We present an approach for estimating the fraction of text in a large corpus which is likely to be substantially modified or produced by a large language model (LLM) Our maximum likelihood model leverages expert-written and AI-generated reference texts to accurately and efficiently examine real-world LLM-use at the corpus level.
  arXiv  Detail & Related papers  (2024-03-11T21:51:39Z)
• Forgotten Knowledge: Examining the Citational Amnesia in NLP [63.13508571014673]
  We show how far back in time do we tend to go to cite papers? How has that changed over time, and what factors correlate with this citational attention/amnesia? We show that around 62% of cited papers are from the immediate five years prior to publication, whereas only about 17% are more than ten years old. We show that the median age and age diversity of cited papers were steadily increasing from 1990 to 2014, but since then, the trend has reversed, and current NLP papers have an all-time low temporal citation diversity.
  arXiv  Detail & Related papers  (2023-05-29T18:30:34Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.