KnowRL: Teaching Language Models to Know What They Know

• URL: http://arxiv.org/abs/2510.11407v1
• Date: Mon, 13 Oct 2025 13:47:14 GMT
• Title: KnowRL: Teaching Language Models to Know What They Know
• Authors: Sahil Kale, Devendra Singh Dhami,
• Abstract summary: We present a simple but powerful framework KnowRL that strengthens a model's internal understanding of its own feasibility boundaries.<n>Our framework combines two components: (i) introspection, where the model generates and classifies tasks it judges feasible or infeasible, and (ii) consensus-based rewarding.<n>With nothing more than a small seed set and no external supervision, our method drove gains as high as 28% in accuracy and 12% in F1.
• Score: 9.341830361844337
• License: http://creativecommons.org/licenses/by-sa/4.0/
• Abstract: Truly reliable AI requires more than simply scaling up knowledge; it demands the ability to know what it knows and when it does not. Yet recent research shows that even the best LLMs misjudge their own competence in more than one in five cases, making any response born of such internal uncertainty impossible to fully trust. Inspired by self-improvement reinforcement learning techniques that require minimal data, we present a simple but powerful framework KnowRL that strengthens a model's internal understanding of its own feasibility boundaries, enabling safer and more responsible behaviour. Our framework combines two components: (i) introspection, where the model generates and classifies tasks it judges feasible or infeasible, and (ii) consensus-based rewarding, where stability of self-knowledge assessment is reinforced through internal agreement. By using internally generated data, this design strengthens consistency in self-knowledge and entirely avoids costly external supervision. In experiments on LLaMA-3.1-8B and Qwen-2.5-7B, KnowRL steadily improved self-knowledge, validated by both intrinsic self-consistency and extrinsic benchmarking. With nothing more than a small seed set and no external supervision, our method drove gains as high as 28% in accuracy and 12% in F1, outperforming baselines in just a few iterations. Our framework essentially unlocks the untapped capacity of LLMs to self-improve their knowledge awareness, opening the door to reliable, more accountable AI and safer deployment in critical applications. Owing to its simplicity and independence from external effort, we encourage applying this reliability-enhancing process to all future models.

Related papers

• Know More, Know Clearer: A Meta-Cognitive Framework for Knowledge Augmentation in Large Language Models [80.21037538996553]
  We propose a novel meta-cognitive framework for reliable knowledge augmentation via differentiated intervention and alignment.<n>Our approach leverages internal cognitive signals to partition the knowledge space into mastered, confused, and missing regions, guiding targeted knowledge expansion.<n>Our framework consistently outperforms strong baselines, validating its rationality in not only enhancing knowledge capabilities but also fostering cognitive behaviors that better distinguish knowns from unknowns.
  arXiv  Detail & Related papers  (2026-02-13T15:07:35Z)
• Epistemic Context Learning: Building Trust the Right Way in LLM-Based Multi-Agent Systems [94.9141394384021]
  Individual agents in multi-agent systems often lack robustness, tending to blindly conform to misleading peers.<n>We show this weakness stems from both sycophancy and inadequate ability to evaluate peer reliability.<n>We first formalize the learning problem of history-aware reference, introducing the historical interactions of peers as additional input.<n>We then develop Epistemic Context Learning (ECL), a reasoning framework that conditions predictions on explicitly-built peer profiles from history.
  arXiv  Detail & Related papers  (2026-01-29T13:59:32Z)
• Rewarding the Journey, Not Just the Destination: A Composite Path and Answer Self-Scoring Reward Mechanism for Test-Time Reinforcement Learning [29.778703252962092]
  Reinforcement Learning (RL) has emerged as a powerful paradigm for advancing Large Language Models (LLMs)<n>We develop a novel test-time reward mechanism that operates without external supervision.
  arXiv  Detail & Related papers  (2025-10-20T07:53:51Z)
• Trust, But Verify: A Self-Verification Approach to Reinforcement Learning with Verifiable Rewards [67.86091419220816]
  Large Language Models (LLMs) show great promise in complex reasoning.<n>A prevalent issue is superficial self-reflection'', where models fail to robustly verify their own outputs.<n>We introduce RISE (Reinforcing Reasoning with Self-Verification), a novel online RL framework designed to tackle this.
  arXiv  Detail & Related papers  (2025-05-19T17:59:31Z)
• Line of Duty: Evaluating LLM Self-Knowledge via Consistency in Feasibility Boundaries [0.0]
  This study aims to obtain intrinsic insights into different types of LLM self-knowledge with a novel methodology.<n>We find that even frontier models like GPT-4o and Mistral Large are not sure of their own capabilities more than 80% of the time.
  arXiv  Detail & Related papers  (2025-03-14T10:07:07Z)
• To Trust or Not to Trust? Enhancing Large Language Models' Situated Faithfulness to External Contexts [10.748768620243982]
  Large Language Models (LLMs) are often augmented with external contexts, such as those used in retrieval-augmented generation (RAG)<n>We show that when provided with both correct and incorrect contexts, both open-source and proprietary models tend to overly rely on external information.<n>We propose two approaches: Self-Guided Confidence Reasoning (SCR) and Rule-Based Confidence Reasoning (RCR)
  arXiv  Detail & Related papers  (2024-10-18T17:59:47Z)
• How Reliable are LLMs as Knowledge Bases? Re-thinking Facutality and Consistency [60.25969380388974]
  Large Language Models (LLMs) are increasingly explored as knowledge bases (KBs)<n>Current evaluation methods focus too narrowly on knowledge retention, overlooking other crucial criteria for reliable performance.<n>We propose new criteria and metrics to quantify factuality and consistency, leading to a final reliability score.
  arXiv  Detail & Related papers  (2024-07-18T15:20:18Z)
• Confidence Under the Hood: An Investigation into the Confidence-Probability Alignment in Large Language Models [14.5291643644017]
  We introduce the concept of Confidence-Probability Alignment. We probe the alignment between models' internal and expressed confidence. Among the models analyzed, OpenAI's GPT-4 showed the strongest confidence-probability alignment.
  arXiv  Detail & Related papers  (2024-05-25T15:42:04Z)
• Self-Alignment for Factuality: Mitigating Hallucinations in LLMs via Self-Evaluation [71.91287418249688]
  Large language models (LLMs) often struggle with factual inaccuracies, even when they hold relevant knowledge. We leverage the self-evaluation capability of an LLM to provide training signals that steer the model towards factuality. We show that the proposed self-alignment approach substantially enhances factual accuracy over Llama family models across three key knowledge-intensive tasks.
  arXiv  Detail & Related papers  (2024-02-14T15:52:42Z)
• Learning to Trust Your Feelings: Leveraging Self-awareness in LLMs for Hallucination Mitigation [9.730412606588335]
  We evaluate the ability of Large Language Models (LLMs) to discern and express their internal knowledge state. We propose a Reinforcement Learning from Knowledge Feedback (RLKF) training framework, leveraging reinforcement learning to enhance the factuality and honesty of LLMs.
  arXiv  Detail & Related papers  (2024-01-27T16:19:30Z)
• Do Large Language Models Know What They Don't Know? [74.65014158544011]
  Large language models (LLMs) have a wealth of knowledge that allows them to excel in various Natural Language Processing (NLP) tasks. Despite their vast knowledge, LLMs are still limited by the amount of information they can accommodate and comprehend. This study aims to evaluate LLMs' self-knowledge by assessing their ability to identify unanswerable or unknowable questions.
  arXiv  Detail & Related papers  (2023-05-29T15:30:13Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.