Know the Unknown: An Uncertainty-Sensitive Method for LLM Instruction Tuning

• URL: http://arxiv.org/abs/2406.10099v2
• Date: Sat, 12 Oct 2024 14:52:00 GMT
• Title: Know the Unknown: An Uncertainty-Sensitive Method for LLM Instruction Tuning
• Authors: Jiaqi Li, Yixuan Tang, Yi Yang,
• Abstract summary: Large language models (LLMs) have demonstrated remarkable capabilities but still face challenges such as hallucinations. We propose a novel approach called uncertainty-sensitive tuning to improve models' capability to recognize the boundaries of their knowledge. Our experimental results demonstrate that our proposed uncertainty-sensitive tuning method enhance the model's ability to identify areas of uncertainty.
• Score: 18.283963879468466
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Large language models (LLMs) have demonstrated remarkable capabilities but still face challenges such as hallucinations. One potential reason for hallucinations is the lack of relevant knowledge or context. Thus, a promising solution involves instructing LLMs to respond with "I do not know" when a question falls outside their knowledge domain or the provided context. However, in this work, we observed that LLMs struggle to admit their lack of knowledge, primarily due to existing instruction datasets designed to encourage specific answers. To improve models' capability to recognize the boundaries of their knowledge, we propose a novel approach called uncertainty-sensitive tuning. This method involves two-stage training designed for uncertainty recognition and prompt-sensitive activation. In the first stage, we guide the LLM to reject unknown questions. In the second stage, we force the model to follow the instructions by incorporating designed causal instructions. The experimental results demonstrate that our proposed uncertainty-sensitive tuning method enhance the model's ability to identify areas of uncertainty. Specifically, it achieves a substantial improvement of up to 34.7% in handling questions involving knowledge gaps compared to the original model. Moreover, our finetuned models even outperform GPT-4, exhibiting an overall performance improvement of up to 4.2%.

Related papers

• Uncertainty-Aware Attention Heads: Efficient Unsupervised Uncertainty Quantification for LLMs [114.46334319795785]
  Large language models (LLMs) exhibit impressive fluency, but often produce critical errors known as "hallucinations"<n>We propose RAUQ (Recurrent Attention-based Uncertainty Quantification), an unsupervised approach that leverages intrinsic attention patterns in transformers to detect hallucinations efficiently.<n> Experiments across 4 LLMs and 12 question answering, summarization, and translation tasks demonstrate that RAUQ yields excellent results.
  arXiv  Detail & Related papers  (2025-05-26T14:28:37Z)
• Are LLMs Really Not Knowledgable? Mining the Submerged Knowledge in LLMs' Memory [15.986679553468989]
  Large language models (LLMs) have shown promise as potential knowledge bases.<n>LLMs often struggle with question-answering tasks and are prone to hallucinations.<n>We develop SkipUnsure, a method to improve answer accuracy by leveraging detected but unexpressed knowledge.
  arXiv  Detail & Related papers  (2024-12-30T10:29:18Z)
• Disentangling Memory and Reasoning Ability in Large Language Models [97.26827060106581]
  We propose a new inference paradigm that decomposes the complex inference process into two distinct and clear actions. Our experiment results show that this decomposition improves model performance and enhances the interpretability of the inference process.
  arXiv  Detail & Related papers  (2024-11-20T17:55:38Z)
• Gradual Learning: Optimizing Fine-Tuning with Partially Mastered Knowledge in Large Language Models [51.20499954955646]
  Large language models (LLMs) acquire vast amounts of knowledge from extensive text corpora during the pretraining phase. In later stages such as fine-tuning and inference, the model may encounter knowledge not covered in the initial training. We propose a two-stage fine-tuning strategy to improve the model's overall test accuracy and knowledge retention.
  arXiv  Detail & Related papers  (2024-10-08T08:35:16Z)
• UNLEARN Efficient Removal of Knowledge in Large Language Models [1.9797215742507548]
  This paper proposes a novel method to achieve this objective called UNLEARN. The approach builds upon subspace methods to identify and specifically target the removal of knowledge without adversely affecting other knowledge in the LLM. Results demonstrate 96% of targeted knowledge can be forgotten while maintaining performance on other knowledge within 2.5% of the original model.
  arXiv  Detail & Related papers  (2024-08-08T00:53:31Z)
• Uncertainty-Based Abstention in LLMs Improves Safety and Reduces Hallucinations [63.330182403615886]
  A major barrier towards the practical deployment of large language models (LLMs) is their lack of reliability. Three situations where this is particularly apparent are correctness, hallucinations when given unanswerable questions, and safety. In all three cases, models should ideally abstain from responding, much like humans, whose ability to understand uncertainty makes us refrain from answering questions we don't know.
  arXiv  Detail & Related papers  (2024-04-16T23:56:38Z)
• Rejection Improves Reliability: Training LLMs to Refuse Unknown Questions Using RL from Knowledge Feedback [14.120154004011084]
  Large Language Models (LLMs) often generate erroneous outputs, known as hallucinations. We present a novel alignment framework called Reinforcement Learning from Knowledge Feedback (RLKF)
  arXiv  Detail & Related papers  (2024-03-27T08:39:56Z)
• KnowTuning: Knowledge-aware Fine-tuning for Large Language Models [83.5849717262019]
  We propose a knowledge-aware fine-tuning (KnowTuning) method to improve fine-grained and coarse-grained knowledge awareness of LLMs. KnowTuning generates more facts with less factual error rate under fine-grained facts evaluation.
  arXiv  Detail & Related papers  (2024-02-17T02:54:32Z)
• Uncertainty Quantification for In-Context Learning of Large Language Models [52.891205009620364]
  In-context learning has emerged as a groundbreaking ability of Large Language Models (LLMs) We propose a novel formulation and corresponding estimation method to quantify both types of uncertainties. The proposed method offers an unsupervised way to understand the prediction of in-context learning in a plug-and-play fashion.
  arXiv  Detail & Related papers  (2024-02-15T18:46:24Z)
• Examining LLMs' Uncertainty Expression Towards Questions Outside Parametric Knowledge [35.067234242461545]
  Large language models (LLMs) express uncertainty in situations where they lack sufficient parametric knowledge to generate reasonable responses. This work aims to systematically investigate LLMs' behaviors in such situations, emphasizing the trade-off between honesty and helpfulness.
  arXiv  Detail & Related papers  (2023-11-16T10:02:40Z)
• R-Tuning: Instructing Large Language Models to Say `I Don't Know' [66.11375475253007]
  Large language models (LLMs) have revolutionized numerous domains with their impressive performance but still face their challenges. Previous instruction tuning methods force the model to complete a sentence no matter whether the model knows the knowledge or not. We present a new approach called Refusal-Aware Instruction Tuning (R-Tuning) Experimental results demonstrate R-Tuning effectively improves a model's ability to answer known questions and refrain from answering unknown questions.
  arXiv  Detail & Related papers  (2023-11-16T08:45:44Z)
• Self-Knowledge Guided Retrieval Augmentation for Large Language Models [59.771098292611846]
  Large language models (LLMs) have shown superior performance without task-specific fine-tuning. Retrieval-based methods can offer non-parametric world knowledge and improve the performance on tasks such as question answering. Self-Knowledge guided Retrieval augmentation (SKR) is a simple yet effective method which can let LLMs refer to the questions they have previously encountered.
  arXiv  Detail & Related papers  (2023-10-08T04:22:33Z)
• Improving the Reliability of Large Language Models by Leveraging Uncertainty-Aware In-Context Learning [76.98542249776257]
  Large-scale language models often face the challenge of "hallucination" We introduce an uncertainty-aware in-context learning framework to empower the model to enhance or reject its output in response to uncertainty.
  arXiv  Detail & Related papers  (2023-10-07T12:06:53Z)
• Knowledge of Knowledge: Exploring Known-Unknowns Uncertainty with Large Language Models [44.117620571329596]
  We focus on addressing known-unknown questions, characterized by high uncertainty due to the absence of definitive answers. To facilitate our study, we collect a new dataset with Known-Unknown Questions (KUQ) We examine the performance of open-source LLMs, fine-tuned using this dataset, in distinguishing between known and unknown queries.
  arXiv  Detail & Related papers  (2023-05-23T05:59:21Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.