Latent Self-Consistency for Reliable Majority-Set Selection in Short- and Long-Answer Reasoning

• URL: http://arxiv.org/abs/2508.18395v1
• Date: Mon, 25 Aug 2025 18:36:28 GMT
• Title: Latent Self-Consistency for Reliable Majority-Set Selection in Short- and Long-Answer Reasoning
• Abstract summary: We introduce Latent Self-Consistency, which selects the most semantically consistent response using learnable token embeddings.<n>A lightweight forward generation of summary tokens increases inference time by less than 1% and requires no changes to the model architecture.
• Score: 17.045726739741813
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Probabilistic decoding in Large Language Models (LLMs) often yields inconsistent outputs, particularly on complex or long-form questions. Self-Consistency (SC) mitigates this for short-form QA by majority voting over exact strings, whereas Universal Self-Consistency (USC) and Weighted Unigram Consistency Score (WUCS) extend to long-form responses but lose accuracy on short-form benchmarks. We introduce Latent Self-Consistency (LSC), which selects the most semantically consistent response using learnable token embeddings. A lightweight forward generation of summary tokens increases inference time by less than 1% and requires no changes to the model architecture. Across 6 short-form and 5 long-form reasoning benchmarks (e.g., MATH, MMLU, TruthfulQA), LSC surpasses SC, USC and WUCS on all short-form and long-form ones on average, while maintaining negligible computational overhead. These results position LSC as a practical consistency-selection method that works reliably across answer formats. Additionally, LSC provides well-calibrated confidence estimates, maintaining low Expected Calibration Error across both answer formats.

Related papers

• Boosting Self-Consistency with Ranking [56.38798757709555]
  Self-consistency improves large language models by sampling multiple reasoning paths and selecting the most frequent answer.<n>We address this limitation with Ranking-Improved Self-Consistency (RISC), which reformulates answer selection in self-consistency as a ranking problem.
  arXiv  Detail & Related papers  (2026-06-03T16:12:30Z)
• Multimodal Learning on Low-Quality Data with Conformal Predictive Self-Calibration [72.0672328514289]
  Multimodal learning often grapples with the challenge of low-quality data.<n>We propose a unified framework, termed Conformal Predictive Self-Calibration.<n>Our framework consistently outperforms existing state-of-the-art methods.
  arXiv  Detail & Related papers  (2026-05-05T14:48:52Z)
• Training-free Context-adaptive Attention for Efficient Long Context Modeling [57.703159205740185]
  Training-free Context-adaptive Attention (TCA-Attention) is a training-free sparse attention mechanism that selectively attends to only the informative tokens for efficient long-context inference.<n>TCA-Attention achieves a 2.8$times$ speedup and reduces KV cache by 61% at 128K context length while maintaining performance comparable to full attention.
  arXiv  Detail & Related papers  (2025-12-10T01:54:57Z)
• CCQA: Generating Question from Solution Can Improve Inference-Time Reasoning in SLMs [14.97707719362011]
  We propose textbfCycle-textbfConsistency in textbfQuestion textbfAnswering (CCQA)<n>Inspired by cycle consistency, CCQA generates a question from each reasoning path and answer, evaluates each by its similarity to the original question, and then selects the candidate solution with the highest similarity score as the final response.<n>It is verified that CCQA consistently outperforms existing state-of-the-art (SOTA) methods across eight models on mathematical and commonsense reasoning benchmarks.
  arXiv  Detail & Related papers  (2025-09-23T02:01:03Z)
• Mind the Gap: A Closer Look at Tokenization for Multiple-Choice Question Answering with LLMs [16.357595595062946]
  There is no consensus on how to tokenize the space following the colon, often overlooked as a trivial choice.<n>Surprisingly, we are able to recommend one specific strategy -- tokenizing the space together with the answer letter.<n>Our findings underscore the importance of careful evaluation design and highlight the need for standardized, transparent evaluation protocols.
  arXiv  Detail & Related papers  (2025-09-18T14:47:58Z)
• Learning to Refine: Self-Refinement of Parallel Reasoning in LLMs [102.48588475875749]
  We introduce Generative Self-Refinement (GSR), a novel parallel test-time scaling framework.<n>GSR generates a set of candidate responses in parallel and then performs self-refinement to synthesize a new superior solution.<n>We show that our method achieves state-of-the-art performance across five mathematical benchmarks.
  arXiv  Detail & Related papers  (2025-08-27T06:51:48Z)
• UNCLE: Uncertainty Expressions in Long-Form Generation [48.7696074873262]
  Large Language Models (LLMs) are prone to hallucination, particularly in long-form generations.<n>We introduce UNCLE, a benchmark designed to evaluate uncertainty expression in both long- and short-form question answering (QA)<n>Our dataset is the first to directly bridge short- and long-form QA with paired questions and gold-standard answers.
  arXiv  Detail & Related papers  (2025-05-22T17:16:08Z)
• Scalable Best-of-N Selection for Large Language Models via Self-Certainty [65.31658824274894]
  Best-of-N selection is a key technique for improving the reasoning performance of Large Language Models.<n>We propose self-certainty, a novel and efficient metric to estimate response quality without requiring external reward models.<n>Our findings establish self-certainty as a practical and efficient way for improving LLM reasoning capabilities.
  arXiv  Detail & Related papers  (2025-02-25T19:08:07Z)
• Revisiting the Test-Time Scaling of o1-like Models: Do they Truly Possess Test-Time Scaling Capabilities? [61.85289698610747]
  We study whether o1-like large language models (LLMs) truly possess test-time scaling capabilities.<n>We find that longer CoTs of these o1-like models do not consistently enhance accuracy.<n>We propose Shortest Majority Vote, a method that combines parallel scaling strategies with CoT length characteristics.
  arXiv  Detail & Related papers  (2025-02-17T07:21:11Z)
• Confidence Improves Self-Consistency in LLMs [9.764747744761085]
  We introduce Confidence-Informed Self-Consistency (CISC)<n>CISC performs a weighted majority vote based on confidence scores obtained directly from the model.<n>When tested on nine models and four datasets, CISC outperforms self-consistency in nearly all configurations.
  arXiv  Detail & Related papers  (2025-02-10T08:10:29Z)
• Reasoning Aware Self-Consistency: Leveraging Reasoning Paths for Efficient LLM Sampling [9.44858963874474]
  Self-Consistency mitigates hallucinations in Large Language Models (LLMs) by sampling multiple reasoning paths.<n>We introduce Reasoning-Aware Self-Consistency (RASC), a novel framework that enhances sampling efficiency and reasoning faithfulness.
  arXiv  Detail & Related papers  (2024-08-30T05:14:59Z)
• Atomic Self-Consistency for Better Long Form Generations [12.753854064540636]
  Atomic Self-Consistency (ASC) is a technique for improving the recall of relevant information in a long-form response. ASC follows recent work, Universal Self-Consistency (USC) in using multiple samples to improve the long-form response. Through extensive experiments and ablations, we show that merging relevant subparts of multiple samples performs significantly better than picking a single sample.
  arXiv  Detail & Related papers  (2024-05-21T18:05:44Z)
• Soft Self-Consistency Improves Language Model Agents [57.66282463340297]
  Current "sample and select" methods rely on majority voting to score answers. Soft Self-Consistency (SOFT-SC) replaces SC's discontinuous scoring with a continuous score computed from model likelihoods. For a fixed number of samples, SOFT-SC leads to a 1.3% increase over SC in absolute success rate on writing bash programs, a 6.6% increase on online shopping, and a 4.7% increase for an interactive household game.
  arXiv  Detail & Related papers  (2024-02-20T18:22:38Z)
• Universal Self-Consistency for Large Language Model Generation [72.6761480346095]
  Self-consistency with chain-of-thought prompting (CoT) has demonstrated remarkable performance gains on challenging tasks. We propose Universal Self-Consistency (USC), which leverages large language models (LLMs) to select the most consistent answer.
  arXiv  Detail & Related papers  (2023-11-29T02:07:09Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.