Not All That Is Fluent Is Factual: Investigating Hallucinations of Large Language Models in Academic Writing

• URL: http://arxiv.org/abs/2605.04171v1
• Date: Tue, 05 May 2026 18:08:15 GMT
• Title: Not All That Is Fluent Is Factual: Investigating Hallucinations of Large Language Models in Academic Writing
• Abstract summary: Large Language models (LLMs) show extraordinary abilities, but they are still prone to hallucinations.<n>We have investigated four popular LLMs, ChatGPT, Grok, Gemini, and Copilot for hallucinations specifically for academic writing.
• Score: 0.6783367407525908
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: Large Language models (LLMs) show extraordinary abilities, but they are still prone to hallucinations, especially when we use them for generating Academic content. We have investigated four popular LLMs, ChatGPT, Grok, Gemini, and Copilot for hallucinations specifically for academic writing. We have designed 80 prompts across four categories, namely, reference generation, factual explanation, abstract generation, and writing improvement. We evaluated the model using a 0-5 rubric score, which checks factual accuracy, reference validity, coherence, style consistency, and academic tone. A novel weighted metric, Hallucination Index (HI), was introduced to measure hallucination in the responses generated by the models. Some of the most widely used evaluation metrics often fail to check errors which alter sentiment in machine-translated text. We found that Grok and Copilot perform better on reference generation tasks, but they often struggle with abstract or stylistic prompts, with HI values of 0.67 and 0.70, respectively. Whereas, Gemini and ChatGPT have done well with having stronger tone control, but they lack in writing factual tasks and higher hallucination risk with HI scores of 0.53 and 0.57, respectively. Our study found that hallucination behavior does not depend solely on model architecture but also on the type of task and the prompting conditions we are providing. We propose that our work opens new research dimensions for future researchers.

Related papers

• HausaNLP at SemEval-2025 Task 3: Towards a Fine-Grained Model-Aware Hallucination Detection [1.8230982862848586]
  We aim to provide a nuanced, model-aware understanding of hallucination occurrences and severity in English.<n>We used natural language inference and fine-tuned a ModernBERT model using a synthetic dataset of 400 samples.<n>Results indicate a moderately positive correlation between the model's confidence scores and the actual presence of hallucinations.
  arXiv  Detail & Related papers  (2025-03-25T13:40:22Z)
• Trust Me, I'm Wrong: LLMs Hallucinate with Certainty Despite Knowing the Answer [51.7407540261676]
  We investigate a distinct type of hallucination, where a model can consistently answer a question correctly, but a seemingly trivial perturbation causes it to produce a hallucinated response with high certainty.<n>This phenomenon is particularly concerning in high-stakes domains such as medicine or law, where model certainty is often used as a proxy for reliability.<n>We show that CHOKE examples are consistent across prompts, occur in different models and datasets, and are fundamentally distinct from other hallucinations.
  arXiv  Detail & Related papers  (2025-02-18T15:46:31Z)
• Can Hallucinations Help? Boosting LLMs for Drug Discovery [8.960425754918974]
  Hallucinations in large language models (LLMs) are often viewed as undesirable.<n>We find that hallucinations significantly improve predictive accuracy for some models.<n>We categorize over 18,000 beneficial hallucinations, with structural misdescriptions emerging as the most impactful type.
  arXiv  Detail & Related papers  (2025-01-23T16:45:51Z)
• HALoGEN: Fantastic LLM Hallucinations and Where to Find Them [39.678012380996854]
  We release HALoGEN, a comprehensive hallucination benchmark consisting of 10,923 prompts for generative models spanning nine domains.<n>We use this framework to evaluate 150,000 generations from 14 language models, finding that even the best-performing models are riddled with hallucinations.
  arXiv  Detail & Related papers  (2025-01-14T18:13:08Z)
• Training Language Models on the Knowledge Graph: Insights on Hallucinations and Their Detectability [83.0884072598828]
  Hallucinations come in many forms, and there is no universally accepted definition. We focus on studying only those hallucinations where a correct answer appears verbatim in the training set. We find that for a fixed dataset, larger and longer-trained LMs hallucinate less. While we see detector size improves performance on fixed LM's outputs, we find an inverse relationship between the scale of the LM and the detectability of its hallucinations.
  arXiv  Detail & Related papers  (2024-08-14T23:34:28Z)
• Knowledge Overshadowing Causes Amalgamated Hallucination in Large Language Models [65.32990889402927]
  We coin this phenomenon as knowledge overshadowing'' We show that the hallucination rate grows with both the imbalance ratio and the length of dominant condition description. We propose to utilize overshadowing conditions as a signal to catch hallucination before it is produced.
  arXiv  Detail & Related papers  (2024-07-10T20:37:42Z)
• On Large Language Models' Hallucination with Regard to Known Facts [74.96789694959894]
  Large language models are successful in answering factoid questions but are also prone to hallucination. We investigate the phenomenon of LLMs possessing correct answer knowledge yet still hallucinating from the perspective of inference dynamics. Our study shed light on understanding the reasons for LLMs' hallucinations on their known facts, and more importantly, on accurately predicting when they are hallucinating.
  arXiv  Detail & Related papers  (2024-03-29T06:48:30Z)
• Fine-grained Hallucination Detection and Editing for Language Models [109.56911670376932]
  Large language models (LMs) are prone to generate factual errors, which are often called hallucinations. We introduce a comprehensive taxonomy of hallucinations and argue that hallucinations manifest in diverse forms. We propose a novel task of automatic fine-grained hallucination detection and construct a new evaluation benchmark, FavaBench.
  arXiv  Detail & Related papers  (2024-01-12T19:02:48Z)
• Factored Verification: Detecting and Reducing Hallucination in Summaries of Academic Papers [1.7100359620532977]
  We use Factored Verification to detect hallucinations in abstractive summaries. We estimate how often language models hallucinate when summarizing across multiple academic papers. The hallucinations we find are often subtle, so we advise caution when using models to synthesize academic papers.
  arXiv  Detail & Related papers  (2023-10-16T17:51:17Z)
• Evaluating Hallucinations in Chinese Large Language Models [65.4771562909392]
  We establish a benchmark named HalluQA (Chinese Hallucination Question-Answering) to measure the hallucination phenomenon in Chinese large language models. We consider two types of hallucinations: imitative falsehoods and factual errors, and we construct adversarial samples based on GLM-130B and ChatGPT. For evaluation, we design an automated evaluation method using GPT-4 to judge whether a model output is hallucinated.
  arXiv  Detail & Related papers  (2023-10-05T07:57:09Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.