No Answer Needed: Predicting LLM Answer Accuracy from Question-Only Linear Probes

• URL: http://arxiv.org/abs/2509.10625v1
• Date: Fri, 12 Sep 2025 18:09:55 GMT
• Title: No Answer Needed: Predicting LLM Answer Accuracy from Question-Only Linear Probes
• Abstract summary: We train linear probes to predict whether a model's forthcoming answer will be correct.<n>Across three open-source model families, projections on this "in-advance correctness direction" trained on generic trivia questions predict success in distribution.<n>For models responding "I don't know", doing so strongly correlates with the probe score, indicating that the same direction also captures confidence.
• Score: 2.6550928535945872
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Do large language models (LLMs) anticipate when they will answer correctly? To study this, we extract activations after a question is read but before any tokens are generated, and train linear probes to predict whether the model's forthcoming answer will be correct. Across three open-source model families ranging from 7 to 70 billion parameters, projections on this "in-advance correctness direction" trained on generic trivia questions predict success in distribution and on diverse out-of-distribution knowledge datasets, outperforming black-box baselines and verbalised predicted confidence. Predictive power saturates in intermediate layers, suggesting that self-assessment emerges mid-computation. Notably, generalisation falters on questions requiring mathematical reasoning. Moreover, for models responding "I don't know", doing so strongly correlates with the probe score, indicating that the same direction also captures confidence. By complementing previous results on truthfulness and other behaviours obtained with probes and sparse auto-encoders, our work contributes essential findings to elucidate LLM internals.

Related papers

• What LLM Forecasters Know but Don't Say: Probing Internal Representations for Calibration and Faithfulness [3.4789212936038116]
  We train representation-pooling probes on intermediate activations and find they achieve substantially better calibration.<n>We assess chain-of-thought faithfulness through evidence ablation and diversionary injection.<n>A single pre-reasoning pass recovers the committed answer and confidence, and routing questions by the spread of this pre-set answer distribution saves 30-47% of generated tokens.
  arXiv  Detail & Related papers  (2026-07-09T01:52:09Z)
• More Yap Less Meaning: Uncovering Self-Improvement Behavior in SLMs [3.373200015661364]
  This study examines the self-correction capabilities of small language models (SLMs)<n>We propose a minimal three-step self-correction pipeline that collects initial SLM answers, prompts the same model to generate hints for its incorrect responses.<n>We evaluate a variety of instruction-tuned and reasoning SLMs in this experimental setup on arithmetic and logical reasoning benchmarks.
  arXiv  Detail & Related papers  (2026-06-07T06:27:37Z)
• Don't Miss the Forest for the Trees: In-Depth Confidence Estimation for LLMs via Reasoning over the Answer Space [16.679707332912255]
  We demonstrate that predicting a verbalized probability distribution can effectively encourage in-depth reasoning for confidence estimation.<n>This method shows an advantage across different models and various tasks, regardless of whether the answer space is known.
  arXiv  Detail & Related papers  (2025-11-18T09:09:23Z)
• Verifying Large Language Models' Reasoning Paths via Correlation Matrix Rank [71.09032766271493]
  Large language models (LLMs) are prone to errors and hallucinations.<n>How to check their outputs effectively and efficiently has become a critical problem in their applications.
  arXiv  Detail & Related papers  (2025-10-28T11:01:10Z)
• Generalized Correctness Models: Learning Calibrated and Model-Agnostic Correctness Predictors from Historical Patterns [67.24756301536617]
  We propose a Generalized Correctness Model (GCM) to generate accurate and calibrated confidence estimates.<n>We first show that GCMs can be trained on the correctness data from many LLMs.<n>We then use CMs as a lens for studying the source of correctness prediction ability and its generalization.
  arXiv  Detail & Related papers  (2025-09-29T16:19:01Z)
• Mitigating Spurious Correlations Between Question and Answer via Chain-of-Thought Correctness Perception Distillation [25.195244084313114]
  Chain-of-Thought Correctness Perception Distillation (CoPeD) aims to improve the reasoning quality of the student model.<n>CoPeD encourages the student model to predict answers based on correct rationales and revise them when they are incorrect.
  arXiv  Detail & Related papers  (2025-09-06T05:33:17Z)
• Are LLMs Really Not Knowledgable? Mining the Submerged Knowledge in LLMs' Memory [15.986679553468989]
  Large language models (LLMs) have shown promise as potential knowledge bases.<n>LLMs often struggle with question-answering tasks and are prone to hallucinations.<n>We develop SkipUnsure, a method to improve answer accuracy by leveraging detected but unexpressed knowledge.
  arXiv  Detail & Related papers  (2024-12-30T10:29:18Z)
• Lachesis: Predicting LLM Inference Accuracy using Structural Properties of Reasoning Paths [12.377041655669728]
  We introduce Lachesis, a predictive model for self-consistency based LLM inferences.<n>We empirically evaluate it using AutoFL, a recently proposed LLM-based fault localisation technique.<n>Results suggest that Lachesis can predict the correctness of answers with a precision of up to 0.8136.
  arXiv  Detail & Related papers  (2024-12-11T10:56:47Z)
• Graph-based Confidence Calibration for Large Language Models [22.394717844099684]
  We propose using an auxiliary learning model to assess response correctness based on the self-consistency of multiple outputs generated by the large language models.<n>Our method builds a consistency graph to represent the agreement among multiple responses and uses a graph neural network (GNN) to estimate the likelihood that each response is correct.
  arXiv  Detail & Related papers  (2024-11-03T20:36:44Z)
• LACIE: Listener-Aware Finetuning for Confidence Calibration in Large Language Models [69.68379406317682]
  We introduce a listener-aware finetuning method (LACIE) to calibrate implicit and explicit confidence markers. We show that LACIE models the listener, considering not only whether an answer is right, but whether it will be accepted by a listener. We find that training with LACIE results in 47% fewer incorrect answers being accepted while maintaining the same level of acceptance for correct answers.
  arXiv  Detail & Related papers  (2024-05-31T17:16:38Z)
• Eliciting Latent Knowledge from Quirky Language Models [1.8035046415192353]
  Eliciting Latent Knowledge aims to find patterns in a capable neural network's activations that robustly track the true state of the world. We introduce 12 datasets and a suite of "quirky" language models (LMs) that are finetuned to make systematic errors when answering questions. We find that, especially in middle layers, linear probes usually report an LM's knowledge independently of what the LM outputs.
  arXiv  Detail & Related papers  (2023-12-02T05:47:22Z)
• R-Tuning: Instructing Large Language Models to Say `I Don't Know' [66.11375475253007]
  Large language models (LLMs) have revolutionized numerous domains with their impressive performance but still face their challenges. Previous instruction tuning methods force the model to complete a sentence no matter whether the model knows the knowledge or not. We present a new approach called Refusal-Aware Instruction Tuning (R-Tuning) Experimental results demonstrate R-Tuning effectively improves a model's ability to answer known questions and refrain from answering unknown questions.
  arXiv  Detail & Related papers  (2023-11-16T08:45:44Z)
• Making Pre-trained Language Models both Task-solvers and Self-calibrators [52.98858650625623]
  Pre-trained language models (PLMs) serve as backbones for various real-world systems. Previous work shows that introducing an extra calibration task can mitigate this issue. We propose a training algorithm LM-TOAST to tackle the challenges.
  arXiv  Detail & Related papers  (2023-07-21T02:51:41Z)
• Realistic Conversational Question Answering with Answer Selection based on Calibrated Confidence and Uncertainty Measurement [54.55643652781891]
  Conversational Question Answering (ConvQA) models aim at answering a question with its relevant paragraph and previous question-answer pairs that occurred during conversation multiple times. We propose to filter out inaccurate answers in the conversation history based on their estimated confidences and uncertainties from the ConvQA model. We validate our models, Answer Selection-based realistic Conversation Question Answering, on two standard ConvQA datasets.
  arXiv  Detail & Related papers  (2023-02-10T09:42:07Z)
• Language Models (Mostly) Know What They Know [10.836210010868932]
  We study whether language models can evaluate the validity of their own claims and predict which questions they will be able to answer correctly. We investigate whether models can be trained to predict "P(IK)", the probability that "I know" the answer to a question, without reference to any particular proposed answer.
  arXiv  Detail & Related papers  (2022-07-11T22:59:39Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.