Expanding before Inferring: Enhancing Factuality in Large Language Models through Premature Layers Interpolation

• URL: http://arxiv.org/abs/2506.02973v1
• Date: Tue, 03 Jun 2025 15:07:13 GMT
• Title: Expanding before Inferring: Enhancing Factuality in Large Language Models through Premature Layers Interpolation
• Authors: Dingwei Chen, Ziqiang Liu, Feiteng Fang, Chak Tou Leong, Shiwen Ni, Ahmadreza Argha, Hamid Alinejad-Rokny, Min Yang, Chengming Li,
• Abstract summary: Large Language Models (LLMs) produce factually inconsistent outputs, commonly referred to as ''hallucinations''<n>We propose PLI (Premature Layers Interpolation), a novel, training-free, and plug-and-play intervention designed to enhance factuality.<n>Experiments on four publicly available datasets demonstrate that PLI effectively reduces hallucinations while outperforming existing baselines in most cases.
• Score: 18.976879901235574
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Large Language Models (LLMs) demonstrate remarkable capabilities in text understanding and generation. However, their tendency to produce factually inconsistent outputs, commonly referred to as ''hallucinations'', remains a critical challenge. Existing approaches, such as retrieval-based and inference-time correction methods, primarily address this issue at the input or output level, often overlooking the intrinsic information refinement process and the role of premature layers. Meanwhile, alignment- and fine-tuning-based methods are resource-intensive. In this paper, we propose PLI (Premature Layers Interpolation), a novel, training-free, and plug-and-play intervention designed to enhance factuality. PLI mitigates hallucinations by inserting premature layers formed through mathematical interpolation with adjacent layers. Inspired by stable diffusion and sampling steps, PLI extends the depth of information processing and transmission in LLMs, improving factual coherence. Experiments on four publicly available datasets demonstrate that PLI effectively reduces hallucinations while outperforming existing baselines in most cases. Further analysis suggests that the success of layer interpolation is closely linked to LLMs' internal mechanisms. To promote reproducibility, we will release our code and data upon acceptance.

Related papers

• Extracting Visual Facts from Intermediate Layers for Mitigating Hallucinations in Multimodal Large Language Models [22.091545786344994]
  We introduce Decoding by Extracting Visual Facts (EVA), a training-free method that dynamically selects intermediate layers with the most significant visual factual information.<n>EVA significantly reduces hallucinations rates compared to baseline methods, underscoring its effectiveness in mitigating hallucinations.
  arXiv  Detail & Related papers  (2025-07-21T14:15:34Z)
• Mitigating Hallucinations via Inter-Layer Consistency Aggregation in Large Vision-Language Models [3.9464481148889354]
  We propose a novel decoding mechanism, Decoding with Inter-layer Consistency via Layer Aggregation (DCLA)<n>Our approach constructs a dynamic semantic reference by aggregating representations from previous layers, and corrects semantically deviated layers to enforce inter-layer consistency.<n> Experiments on hallucination benchmarks such as MME and POPE demonstrate that DCLA effectively reduces hallucinations while enhancing the reliability and performance of LVLMs.
  arXiv  Detail & Related papers  (2025-05-18T10:15:42Z)
• Invoke Interfaces Only When Needed: Adaptive Invocation for Large Language Models in Question Answering [5.100085108873068]
  We propose a practical invocation evaluation metric called AttenHScore.<n>It calculates the accumulation and propagation of hallucinations during the generation process of small LMs.<n>By dynamically adjusting the detection threshold, we achieve more accurate real-time invocation of large LMs.
  arXiv  Detail & Related papers  (2025-05-05T01:45:56Z)
• END: Early Noise Dropping for Efficient and Effective Context Denoising [60.24648712022382]
  Large Language Models (LLMs) have demonstrated remarkable performance across a wide range of natural language processing tasks.<n>They are often distracted by irrelevant or noisy context in input sequences that degrades output quality.<n>We introduce Early Noise Dropping (textscEND), a novel approach to mitigate this issue without requiring fine-tuning the LLMs.
  arXiv  Detail & Related papers  (2025-02-26T08:07:17Z)
• Mitigating Hallucination for Large Vision Language Model by Inter-Modality Correlation Calibration Decoding [66.06337890279839]
  Large vision-language models (LVLMs) have shown remarkable capabilities in visual-language understanding for downstream multi-modal tasks.<n>LVLMs still suffer from generating hallucinations in complex generation tasks, leading to inconsistencies between visual inputs and generated content.<n>We propose an Inter-Modality Correlation Decoding (IMCCD) method to mitigate hallucinations in LVLMs in a training-free manner.
  arXiv  Detail & Related papers  (2025-01-03T17:56:28Z)
• Iter-AHMCL: Alleviate Hallucination for Large Language Model via Iterative Model-level Contrastive Learning [16.883679810267342]
  Iterative Model-level Contrastive Learning (Iter-AHMCL) to address hallucination. This paper introduces a novel approach called Iterative Model-level Contrastive Learning (Iter-AHMCL) to address hallucination.
  arXiv  Detail & Related papers  (2024-10-16T00:15:40Z)
• Lower Layer Matters: Alleviating Hallucination via Multi-Layer Fusion Contrastive Decoding with Truthfulness Refocused [44.37155553647802]
  Large Language Models (LLMs) have demonstrated exceptional performance across various natural language processing tasks. They occasionally yield content that factually inaccurate or discordant with the expected output. Recent works have investigated contrastive decoding between the original model and an amateur model with induced hallucination. We introduce a novel contrastive decoding framework termed LOL (LOwer Layer Matters)
  arXiv  Detail & Related papers  (2024-08-16T14:23:59Z)
• On-Policy Self-Alignment with Fine-grained Knowledge Feedback for Hallucination Mitigation [47.35777964373532]
  Hallucination occurs when large language models exhibit behavior that deviates from the boundaries of their knowledge during response generation.<n>Previous learning-based methods attempt to finetune models but are limited by off-policy sampling and coarse-grained feedback.<n>We present RLFH, an on-policy self-alignment approach that enables LLMs to actively explore their knowledge boundaries and self-correct generation behavior.
  arXiv  Detail & Related papers  (2024-06-18T02:43:49Z)
• Entropy Guided Extrapolative Decoding to Improve Factuality in Large Language Models [55.45444773200529]
  Large language models (LLMs) exhibit impressive natural language capabilities but suffer from hallucination. Recent work has focused on decoding techniques to improve factuality during inference.
  arXiv  Detail & Related papers  (2024-04-14T19:45:35Z)
• Debiasing Multimodal Large Language Models [61.6896704217147]
  Large Vision-Language Models (LVLMs) have become indispensable tools in computer vision and natural language processing. Our investigation reveals a noteworthy bias in the generated content, where the output is primarily influenced by the underlying Large Language Models (LLMs) prior to the input image. To rectify these biases and redirect the model's focus toward vision information, we introduce two simple, training-free strategies.
  arXiv  Detail & Related papers  (2024-03-08T12:35:07Z)
• Improving Factual Consistency of News Summarization by Contrastive Preference Optimization [65.11227166319546]
  Large language models (LLMs) generate summaries that are factually inconsistent with original articles.<n>These hallucinations are challenging to detect through traditional methods.<n>We propose Contrastive Preference Optimization (CPO) to disentangle the LLMs' propensities to generate faithful and fake content.
  arXiv  Detail & Related papers  (2023-10-30T08:40:16Z)
• Accelerating LLaMA Inference by Enabling Intermediate Layer Decoding via Instruction Tuning with LITE [62.13435256279566]
  Large Language Models (LLMs) have achieved remarkable performance across a wide variety of natural language tasks. However, their large size makes their inference slow and computationally expensive. We show that it enables these layers to acquire 'good' generation ability without affecting the generation ability of the final layer.
  arXiv  Detail & Related papers  (2023-10-28T04:07:58Z)
• DoLa: Decoding by Contrasting Layers Improves Factuality in Large Language Models [79.01926242857613]
  Large language models (LLMs) are prone to hallucinations, generating content that deviates from facts seen during pretraining. We propose a simple decoding strategy for reducing hallucinations with pretrained LLMs. We find that this Decoding by Contrasting Layers (DoLa) approach is able to better surface factual knowledge and reduce the generation of incorrect facts.
  arXiv  Detail & Related papers  (2023-09-07T17:45:31Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.