Evaluating Reasoning Models for Queries with Presuppositions

• URL: http://arxiv.org/abs/2605.03050v1
• Date: Mon, 04 May 2026 18:15:28 GMT
• Title: Evaluating Reasoning Models for Queries with Presuppositions
• Abstract summary: We evaluate whether large reasoning models (LRMs) can reason about the underlying assumptions and respond to user queries appropriately.<n>We construct queries with varying degrees of presuppositions spanning health, science, and general knowledge.<n>We find that reasoning models achieve a slightly higher accuracy (2-11%), but they still fail to challenge a large fraction (26-42%) of false presuppositions.
• Score: 13.688612326499117
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Millions of users turn to AI models for their information needs. It is conceivable that a large number of user queries contain assumptions that may be factually inaccurate. Prior work notes that large language models (LLMs) often fail to challenge such erroneous assumptions, and can reinforce users' misinformed opinions. However, given the recent advances, especially in model's reasoning capabilities, we revisit whether large reasoning models (LRMs) can reason about the underlying assumptions and respond to user queries appropriately. We construct queries with varying degrees of presuppositions spanning health, science, and general knowledge, and use it to evaluate several widely-deployed models When compared to non-reasoning models, we find that reasoning models achieve a slightly higher accuracy (2-11%), but they still fail to challenge a large fraction (26-42%) of false presuppositions. Further, reasoning models remain susceptible to how strongly the presupposition is expressed.

Related papers

• TopBench: A Benchmark for Implicit Prediction and Reasoning over Tabular Question Answering [80.93487993878836]
  A common class of real-world queries is implicitly predictive, requiring the inference of unobserved answers from historical patterns rather than mere retrieval.<n>These queries introduce two challenges: recognizing latent intent and reliable predictive reasoning over massive tables.<n>We introduce TopBench, a benchmark consisting of 779 samples across four sub-tasks, ranging from single-point prediction to decision making, treatment effect analysis, and complex reasoning.
  arXiv  Detail & Related papers  (2026-04-30T16:22:51Z)
• To Think or Not To Think, That is The Question for Large Reasoning Models in Theory of Mind Tasks [56.11584171938381]
  Theory of Mind (ToM) assesses whether models can infer hidden mental states such as beliefs, desires, and intentions.<n>Recent progress in Large Reasoning Models (LRMs) has boosted step-by-step inference in mathematics and coding.<n>We present a systematic study of nine advanced Large Language Models (LLMs) comparing reasoning models with non-reasoning models.
  arXiv  Detail & Related papers  (2026-02-11T08:16:13Z)
• Linear Model Extraction via Factual and Counterfactual Queries [2.6499018693213316]
  In model extraction attacks, the goal is to reveal the parameters of a black-box machine learning model.<n>We consider linear models and three types of queries: factual, counterfactual, and robust counterfactual.
  arXiv  Detail & Related papers  (2026-02-10T12:57:53Z)
• From Black-box to Causal-box: Towards Building More Interpretable Models [57.23201263629627]
  We introduce the notion of causal interpretability, which formalizes when counterfactual queries can be evaluated from a specific class of models.<n>We derive a complete graphical criterion that determines whether a given model architecture supports a given counterfactual query.
  arXiv  Detail & Related papers  (2025-10-24T20:03:18Z)
• Generics and Default Reasoning in Large Language Models [0.0]
  This paper evaluates the capabilities of 28 large language models to reason with 20 defeasible reasoning patterns involving generic generalizations.<n>We find that while several frontier models handle many default reasoning problems well, performance varies widely across models and prompting styles.
  arXiv  Detail & Related papers  (2025-08-19T10:28:53Z)
• Reasoning about Uncertainty: Do Reasoning Models Know When They Don't Know? [7.423494663010787]
  Reasoning language models have set state-of-the-art (SOTA) records on many challenging benchmarks.<n>Like previous language models, reasoning models are prone to generating confident, plausible responses that are incorrect.<n>Knowing when and how much to trust these models is critical to the safe deployment of reasoning models in real-world applications.
  arXiv  Detail & Related papers  (2025-06-22T21:46:42Z)
• Surface Fairness, Deep Bias: A Comparative Study of Bias in Language Models [45.41676783204022]
  We investigate various proxy measures of bias in large language models (LLMs)<n>We find that evaluating models with pre-prompted personae on a multi-subject benchmark (MMLU) leads to negligible and mostly random differences in scores.<n>With the recent trend for LLM assistant memory and personalization, these problems open up from a different angle.
  arXiv  Detail & Related papers  (2025-06-12T08:47:40Z)
• Self-supervised Analogical Learning using Language Models [59.64260218737556]
  We propose SAL, a self-supervised analogical learning framework.<n> SAL mimics the human analogy process and trains models to explicitly transfer high-quality symbolic solutions.<n>We show that the resulting models outperform base language models on a wide range of reasoning benchmarks.
  arXiv  Detail & Related papers  (2025-02-03T02:31:26Z)
• Large Language Models Must Be Taught to Know What They Don't Know [97.90008709512921]
  We show that fine-tuning on a small dataset of correct and incorrect answers can create an uncertainty estimate with good generalization and small computational overhead.<n>We also investigate the mechanisms that enable reliable uncertainty estimation, finding that many models can be used as general-purpose uncertainty estimators.
  arXiv  Detail & Related papers  (2024-06-12T16:41:31Z)
• Getting MoRE out of Mixture of Language Model Reasoning Experts [71.61176122960464]
  We propose a Mixture-of-Reasoning-Experts (MoRE) framework that ensembles diverse specialized language models. We specialize the backbone language model with prompts optimized for different reasoning categories, including factual, multihop, mathematical, and commonsense reasoning. Our human study confirms that presenting expert predictions and the answer selection process helps annotators more accurately calibrate when to trust the system's output.
  arXiv  Detail & Related papers  (2023-05-24T02:00:51Z)
• Roses Are Red, Violets Are Blue... but Should Vqa Expect Them To? [0.0]
  We argue that the standard evaluation metric, which consists in measuring the overall in-domain accuracy, is misleading. We propose the GQA-OOD benchmark designed to overcome these concerns.
  arXiv  Detail & Related papers  (2020-06-09T08:50:39Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.