Mitigating Hallucinations in Zero-Shot Scientific Summarisation: A Pilot Study

• URL: http://arxiv.org/abs/2512.00931v1
• Date: Sun, 30 Nov 2025 15:19:41 GMT
• Title: Mitigating Hallucinations in Zero-Shot Scientific Summarisation: A Pilot Study
• Abstract summary: Large language models (LLMs) produce context inconsistency hallucinations.<n>This research project investigates whether prompt engineering methods can mitigate context inconsistency in zero-shot scientific summarisation tasks.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Large language models (LLMs) produce context inconsistency hallucinations, which are LLM generated outputs that are misaligned with the user prompt. This research project investigates whether prompt engineering (PE) methods can mitigate context inconsistency hallucinations in zero-shot LLM summarisation of scientific texts, where zero-shot indicates that the LLM relies purely on its pre-training data. Across eight yeast biotechnology research paper abstracts, six instruction-tuned LLMs were prompted with seven methods: a base- line prompt, two levels of increasing instruction complexity (PE-1 and PE-2), two levels of context repetition (CR-K1 and CR-K2), and two levels of random addition (RA-K1 and RA-K2). Context repetition involved the identification and repetition of K key sentences from the abstract, whereas random addition involved the repetition of K randomly selected sentences from the abstract, where K is 1 or 2. A total of 336 LLM-generated summaries were evaluated using six metrics: ROUGE-1, ROUGE-2, ROUGE-L, BERTScore, METEOR, and cosine similarity, which were used to compute the lexical and semantic alignment be- tween the summaries and the abstracts. Four hypotheses on the effects of prompt methods on summary alignment with the reference text were tested. Statistical analysis on 3744 collected datapoints was performed using bias-corrected and accelerated (BCa) bootstrap confidence intervals and Wilcoxon signed-rank tests with Bonferroni-Holm correction. The results demonstrated that CR and RA significantly improve the lexical alignment of LLM-generated summaries with the abstracts. These findings indicate that prompt engineering has the potential to impact hallucinations in zero-shot scientific summarisation tasks.

Related papers

• Production and Perception in LLMs: A Token Probability Approach [0.3226162580622704]
  We operationalize the production-perception distinction through direct token probability measurements.<n>We show the effect is specific to communicative framing rather than prompt surface variation.<n>These findings suggest that prompt framing alone induces a production-perception distinction in LLM probability distributions.
  arXiv  Detail & Related papers  (2026-07-13T15:33:45Z)
• Influence-Guided Symbolic Regression: Scientific Discovery via LLM-Driven Equation Search with Granular Feedback [56.69850045068714]
  We introduce textitInfluence-Guided Regression (IGSR), a method that frames equation discovery as an iterative two-step process.<n>We demonstrate IGSRs effectiveness on a diverse suite of benchmarks, including LLM-SRBench, pharmacological PKPD models, an epidemiological simulation, and real-world genomic data.
  arXiv  Detail & Related papers  (2026-05-27T23:48:01Z)
• Saying More Than They Know: A Framework for Quantifying Epistemic-Rhetorical Miscalibration in Large Language Models [0.30458514384586394]
  Large language models (LLMs) exhibit systematic miscalibration with rhetorical intensity not proportionate to epistemic grounding.<n>This study proposes a framework for quantifying this decoupling by designing a triadic epistemic-rhetorical marker (ERM) taxonomy.<n> applied to 225 argumentative texts spanning approximately 0.6 Million tokens across human expert, human non-expert, and LLM-generated sub-corpora.
  arXiv  Detail & Related papers  (2026-03-27T05:33:10Z)
• Evaluating LLMs for Zeolite Synthesis Event Extraction (ZSEE): A Systematic Analysis of Prompting Strategies [1.3986052226424095]
  This work addresses a fundamental question: what is the efficacy of different prompting strategies when applying Large Language Models?<n>We focus on four key subtasks: event type classification, trigger text identification, argument role extraction, and argument text extraction.<n>We evaluate four prompting strategies - zero-shot, few-shot, event-specific, and reflection-based - across six state-of-the-art LLMs.
  arXiv  Detail & Related papers  (2025-12-17T11:02:31Z)
• Language Bottleneck Models: A Framework for Interpretable Knowledge Tracing and Beyond [55.984684518346924]
  We recast Knowledge Tracing as an inverse problem: learning the minimum natural-language summary that makes past answers explainable and future answers predictable.<n>Our Language Bottleneck Model (LBM) consists of an encoder LLM that writes an interpretable knowledge summary and a frozen decoder LLM that must reconstruct and predict student responses using only that summary text.<n> Experiments on synthetic arithmetic benchmarks and the large-scale Eedi dataset show that LBMs rival the accuracy of state-of-the-art KT and direct LLM methods while requiring orders-of-magnitude fewer student trajectories.
  arXiv  Detail & Related papers  (2025-06-20T13:21:14Z)
• CLATTER: Comprehensive Entailment Reasoning for Hallucination Detection [60.98964268961243]
  We propose that guiding models to perform a systematic and comprehensive reasoning process allows models to execute much finer-grained and accurate entailment decisions.<n>We define a 3-step reasoning process, consisting of (i) claim decomposition, (ii) sub-claim attribution and entailment classification, and (iii) aggregated classification, showing that such guided reasoning indeed yields improved hallucination detection.
  arXiv  Detail & Related papers  (2025-06-05T17:02:52Z)
• Span-Level Hallucination Detection for LLM-Generated Answers [0.0]
  This paper presents a span-level hallucination detection framework for the SemEval-2025 Shared Task, focusing on English and Arabic texts.<n>Our approach integrates Semantic Role Labeling (SRL) to decompose the answer into atomic roles, which are then compared with a retrieved reference context.<n>Using a DeBERTa-based textual entailment model, we evaluate each role semantic alignment with the retrieved context.
  arXiv  Detail & Related papers  (2025-04-25T18:35:25Z)
• TRACT: Regression-Aware Fine-tuning Meets Chain-of-Thought Reasoning for LLM-as-a-Judge [59.57934574562651]
  TRACT (Two-stage Regression-Aware fine-tuning with CoT) is a method combining CoT reasoning with regression-aware training.<n>Experiments across four LLM-as-a-judge datasets and two LLMs show that TRACT significantly outperforms existing methods.
  arXiv  Detail & Related papers  (2025-03-06T12:33:20Z)
• Evaluating Human Alignment and Model Faithfulness of LLM Rationale [66.75309523854476]
  We study how well large language models (LLMs) explain their generations through rationales. We show that prompting-based methods are less "faithful" than attribution-based explanations.
  arXiv  Detail & Related papers  (2024-06-28T20:06:30Z)
• Assessing LLMs for Zero-shot Abstractive Summarization Through the Lens of Relevance Paraphrasing [37.400757839157116]
  Large Language Models (LLMs) have achieved state-of-the-art performance at zero-shot generation of abstractive summaries for given articles.<n>We propose relevance paraphrasing, a simple strategy that can be used to measure the robustness of LLMs as summarizers.
  arXiv  Detail & Related papers  (2024-06-06T12:08:43Z)
• FactPICO: Factuality Evaluation for Plain Language Summarization of Medical Evidence [46.71469172542448]
  This paper presents FactPICO, a factuality benchmark for plain language summarization of medical texts. It consists of 345 plain language summaries of abstracts generated from three randomized controlled trials (RCTs) We assess the factuality of critical elements of RCTs in those summaries, as well as the reported findings concerning these.
  arXiv  Detail & Related papers  (2024-02-18T04:45:01Z)
• "Knowing When You Don't Know": A Multilingual Relevance Assessment Dataset for Robust Retrieval-Augmented Generation [90.09260023184932]
  Retrieval-Augmented Generation (RAG) grounds Large Language Model (LLM) output by leveraging external knowledge sources to reduce factual hallucinations. NoMIRACL is a human-annotated dataset for evaluating LLM robustness in RAG across 18 typologically diverse languages. We measure relevance assessment using: (i) hallucination rate, measuring model tendency to hallucinate, when the answer is not present in passages in the non-relevant subset, and (ii) error rate, measuring model inaccuracy to recognize relevant passages in the relevant subset.
  arXiv  Detail & Related papers  (2023-12-18T17:18:04Z)
• Improving Factual Consistency of News Summarization by Contrastive Preference Optimization [65.11227166319546]
  Large language models (LLMs) generate summaries that are factually inconsistent with original articles.<n>These hallucinations are challenging to detect through traditional methods.<n>We propose Contrastive Preference Optimization (CPO) to disentangle the LLMs' propensities to generate faithful and fake content.
  arXiv  Detail & Related papers  (2023-10-30T08:40:16Z)
• BooookScore: A systematic exploration of book-length summarization in the era of LLMs [53.42917858142565]
  We develop an automatic metric, BooookScore, that measures the proportion of sentences in a summary that do not contain any of the identified error types. We find that closed-source LLMs such as GPT-4 and 2 produce summaries with higher BooookScore than those generated by open-source models.
  arXiv  Detail & Related papers  (2023-10-01T20:46:44Z)
• Evaluating Factual Consistency of Summaries with Large Language Models [24.416837319515896]
  We explore evaluating factual consistency of summaries by directly prompting large language models (LLMs) Our experiments demonstrate that prompting LLMs is able to outperform the previous best factuality systems in all settings.
  arXiv  Detail & Related papers  (2023-05-23T13:48:32Z)
• Z-ICL: Zero-Shot In-Context Learning with Pseudo-Demonstrations [97.41375480696972]
  We introduce Z-ICL, a new zero-shot method that closes the gap by constructing pseudo-demonstrations for a given test input. evaluation on nine classification datasets shows that Z-ICL outperforms previous zero-shot methods by a significant margin.
  arXiv  Detail & Related papers  (2022-12-19T21:34:26Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.