Evaluating Pragmatic Reasoning in Large Language Models: Evidence from Scalar Diversity

• URL: http://arxiv.org/abs/2605.09042v1
• Date: Sat, 09 May 2026 16:28:50 GMT
• Title: Evaluating Pragmatic Reasoning in Large Language Models: Evidence from Scalar Diversity
• Abstract summary: This study compares direct probability measurement and metalinguistic prompting across multiple models and experimental settings.<n>The results show that neither evaluation method consistently outperforms the other and that pragmatic behavior varies substantially across model families, prompting strategies, and task structures.
• Score: 2.538209532048867
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Evaluating pragmatic reasoning in large language models (LLMs) remains challenging because model behavior can vary depending on evaluation methods. Previous studies suggest that prompt-based judgments may diverge from models' internal probability distributions, raising questions about whether observed performance reflects underlying competence or task-induced behavior. This study examines this issue using scalar diversity as a graded diagnostic for pragmatic inference. Following Hu & Levy (2023), this study compares direct probability measurement and metalinguistic prompting across multiple models and experimental settings. The results show that neither evaluation method consistently outperforms the other and that pragmatic behavior varies substantially across model families, prompting strategies, and task structures. Moreover, scalar diversity gradients emerge only in specific model-condition combinations, suggesting that pragmatic reasoning in LLMs reflects an interaction between internal probabilistic representations and task-induced prompting behavior rather than a stable competence captured by a single evaluation paradigm. These findings highlight the central role of evaluation design in interpreting pragmatic abilities in LLMs.

Related papers

• Modeling Pathology-Like Behavioral Patterns in Language Models Through Behavioral Fine-Tuning [0.0]
  Large language models are increasingly used as computational tools for modeling human-like behavior.<n>We introduce a behavioral induction framework that modifies model policies through fine-tuning on structured decision-making tasks.<n>We then test whether this behavioral optimization produces systematic changes in generative distributions.
  arXiv  Detail & Related papers  (2026-05-21T11:42:38Z)
• Inference-Time Reasoning Selectively Reduces Implicit Social Bias in Large Language Models [0.0]
  We study how reasoning-enabled inference affects implicit bias in large language models (LLMs)<n>We find that enabling reasoning significantly reduces measured implicit bias on an IAT-style evaluation for some model classes across fifteen stereotype topics.<n>This work highlights how theory from cognitive science and psychology can complement AI evaluation research by providing methodological and interpretive frameworks.
  arXiv  Detail & Related papers  (2026-02-04T16:44:23Z)
• Reasoning as State Transition: A Representational Analysis of Reasoning Evolution in Large Language Models [50.39102836928242]
  We introduce a representational perspective to investigate the dynamics of the model's internal states.<n>We discover that post-training yields only limited improvement in static initial representation quality.
  arXiv  Detail & Related papers  (2026-01-31T15:23:33Z)
• D-Models and E-Models: Diversity-Stability Trade-offs in the Sampling Behavior of Large Language Models [91.21455683212224]
  In large language models (LLMs), the probability of relevance for the next piece of information is linked to the probability of relevance for the next product.<n>But whether fine-grained sampling probabilities faithfully align with task requirements remains an open question.<n>We identify two model types: D-models, whose P_token exhibits large step-to-step variability and poor alignment with P_task; and E-models, whose P_token is more stable and better aligned with P_task.
  arXiv  Detail & Related papers  (2026-01-25T14:59:09Z)
• Internal Causal Mechanisms Robustly Predict Language Model Out-of-Distribution Behaviors [61.92704516732144]
  We show that the most robust features for correctness prediction are those that play a distinctive causal role in the model's behavior.<n>We propose two methods that leverage causal mechanisms to predict the correctness of model outputs.
  arXiv  Detail & Related papers  (2025-05-17T00:31:39Z)
• Confidence in Large Language Model Evaluation: A Bayesian Approach to Limited-Sample Challenges [13.526258635654882]
  This study introduces a Bayesian approach for large language models (LLMs) capability assessment.<n>We treat model capabilities as latent variables and leverage a curated query set to induce discriminative responses.<n> Experimental evaluations with GPT-series models demonstrate that the proposed method achieves superior discrimination compared to conventional evaluation methods.
  arXiv  Detail & Related papers  (2025-04-30T04:24:50Z)
• Causality can systematically address the monsters under the bench(marks) [64.36592889550431]
  Benchmarks are plagued by various biases, artifacts, or leakage.<n>Models may behave unreliably due to poorly explored failure modes.<n> causality offers an ideal framework to systematically address these challenges.
  arXiv  Detail & Related papers  (2025-02-07T17:01:37Z)
• On the Reasoning Capacity of AI Models and How to Quantify It [0.0]
  Large Language Models (LLMs) have intensified the debate surrounding the fundamental nature of their reasoning capabilities.<n>While achieving high performance on benchmarks such as GPQA and MMLU, these models exhibit limitations in more complex reasoning tasks.<n>We propose a novel phenomenological approach that goes beyond traditional accuracy metrics to probe the underlying mechanisms of model behavior.
  arXiv  Detail & Related papers  (2025-01-23T16:58:18Z)
• Evaluating Interventional Reasoning Capabilities of Large Language Models [58.52919374786108]
  Large language models (LLMs) are used to automate decision-making tasks.<n>In this paper, we evaluate whether LLMs can accurately update their knowledge of a data-generating process in response to an intervention.<n>We create benchmarks that span diverse causal graphs (e.g., confounding, mediation) and variable types.<n>These benchmarks allow us to isolate the ability of LLMs to accurately predict changes resulting from their ability to memorize facts or find other shortcuts.
  arXiv  Detail & Related papers  (2024-04-08T14:15:56Z)
• Revisiting Demonstration Selection Strategies in In-Context Learning [66.11652803887284]
  Large language models (LLMs) have shown an impressive ability to perform a wide range of tasks using in-context learning (ICL) In this work, we first revisit the factors contributing to this variance from both data and model aspects, and find that the choice of demonstration is both data- and model-dependent. We propose a data- and model-dependent demonstration selection method, textbfTopK + ConE, based on the assumption that textitthe performance of a demonstration positively correlates with its contribution to the model's understanding of the test samples.
  arXiv  Detail & Related papers  (2024-01-22T16:25:27Z)
• Competence-Based Analysis of Language Models [21.43498764977656]
  CALM (Competence-based Analysis of Language Models) is designed to investigate LLM competence in the context of specific tasks.<n>We develop a new approach for performing causal probing interventions using gradient-based adversarial attacks.<n>We carry out a case study of CALM using these interventions to analyze and compare LLM competence across a variety of lexical inference tasks.
  arXiv  Detail & Related papers  (2023-03-01T08:53:36Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.