The Imperfective Paradox Is Not Necessarily in Large Language Models: A Benchmark Failure Before a Model Failure

• URL: http://arxiv.org/abs/2608.25005v1
• Date: Tue, 25 Aug 2026 18:00:18 GMT
• Title: The Imperfective Paradox Is Not Necessarily in Large Language Models: A Benchmark Failure Before a Model Failure
• Abstract summary: We examine the benchmark and conclusions and show that it is substantially affected by conceptual and evaluation mis-specifications.<n>Under a strict NLI standard, 76% of Group A instances do not explicitly rule out culmination.<n>We formulate event-semantic NLI as a Multi-step Reasoning Problem and assess both intermediate semantic decisions and final predictions.
• Score: 42.25959731143407
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: The imperfective paradox provides a useful test of compositional semantic analysis. Recent work constructs an NLI benchmark and reports that models frequently infer completed telic events from progressive descriptions, attributing this behavior to a Teleological Bias. It further argues that prompting interventions cause a Calibration Crisis. We reexamine the benchmark and conclusions and show that it is substantially affected by conceptual and evaluation mis-specifications. We identify three conceptual mis-specifications. In particular, Aspectual Reduction affects the benchmark construction, analysis, experiments, and conclusions. Under a strict NLI standard, 76% of Group A instances do not explicitly rule out culmination. In our native-speaker annotation, 38% of Group A examples and 29% of the Group C examples were judged to permit an alternative interpretation. To control these issues and lexical variation, we construct Lexically Matched Minimal Pairs. At the evaluation level, we formulate event-semantic NLI as a Multi-step Reasoning Problem and assess both intermediate semantic decisions and final predictions. Our results show that models often do not affirm culmination but nevertheless accept the corresponding simple-past hypothesis, a pattern we characterize as Sufficiency Bias. We further show that prompting interventions produce a Decision Shift among labels without reliably improving the underlying semantic understanding and reasoning. Intermediate and oracle-guided analyses identify two additional failure modes: errors in compositional aspectual classification and Surface-form Attraction toward surface-associated answers. Our experiments on Qwen-7B with suitable prompts, GPT-5.4, and Qwen-72B provide initial evidence for the context sensitivity of aspectual classification and suggest that these models can achieve performance comparable to that of human annotators.

Related papers

• Do Large Language Models Know What They Don't Know II? A Fully Behavioral, Non-Cognitive Measure of Epistemic Honesty [8.148357447770893]
  Large Language Models (LLMs) are frequently confident, eloquent, and well versed.<n>Do they know what they don't know?<n>We develop a novel metric to evaluate whether an LLM appropriately acknowledges the boundaries of its knowledge.
  arXiv  Detail & Related papers  (2026-09-07T18:40:08Z)
• Before the Action: Benchmarking LLMs on Prospective Hypothesis Discovery [95.44432473367836]
  Large language models (LLMs) excel at answering pre-specified questions, yet their ability to navigate the open-ended, pre-conclusion stage of discovery remains largely unmeasured.<n>We introduce Prospective Hypothesis Discovery (PHD), which asks models to autonomously construct grounded, discnative, and testable hypothesis spaces from inconclusive evidence.<n>We introduce HypoArena, comprising HypoData, a benchmark of 988 cases across six scientific and analytical domains, and HypoEval, an evaluation framework for open-ended hypothesis sets.
  arXiv  Detail & Related papers  (2026-07-17T08:56:43Z)
• Aligning LLM Uncertainty with Human Disagreement in Subjectivity Analysis [57.78407973423517]
  We advocate uncertainty-aware subjectivity analysis, where models are expected to make predictions while expressing uncertainty that reflects human disagreement.<n>To operationalize this perspective, we propose a two-phase Disagreement Perception and Uncertainty Alignment framework.<n> Experiments on three subjectivity analysis tasks show that DPUA preserves task performance while better aligning model uncertainty with human disagreement.
  arXiv  Detail & Related papers  (2026-05-11T11:52:58Z)
• Deconfounding Scores and Representation Learning for Causal Effect Estimation with Weak Overlap [75.41420283374737]
  Overlap is a key condition for causal treatment effect estimation.<n>We propose a class of feature representations called deconfounding scores.<n>We show that prognostic scores are overlap-optimal within this class.
  arXiv  Detail & Related papers  (2026-04-01T12:19:42Z)
• Measuring Language Model Hallucinations Through Distributional Correctness [7.106986689736826]
  A novel evaluation metric, the Distributional Correctness Score (DCS), is introduced to solve this problem.<n>DCS distinguishes between harmful overconfidence in wrong answers and uncertainty expressed through abstention, providing scores in an interpretable default range.<n>DCS is demonstrated to offer a more nuanced and aligned evaluation paradigm that incentivises models to express genuine uncertainty rather than guessing.
  arXiv  Detail & Related papers  (2025-10-05T17:50:42Z)
• A Closer Look at Bias and Chain-of-Thought Faithfulness of Large (Vision) Language Models [58.32070787537946]
  Chain-of-thought (CoT) reasoning enhances performance of large language models.<n>We present the first comprehensive study of CoT faithfulness in large vision-language models.
  arXiv  Detail & Related papers  (2025-05-29T18:55:05Z)
• Estimating the Causal Effects of Natural Logic Features in Transformer-Based NLI Models [16.328341121232484]
  We apply causal effect estimation strategies to measure the effect of context interventions. We investigate robustness to irrelevant changes and sensitivity to impactful changes of Transformers.
  arXiv  Detail & Related papers  (2024-04-03T10:22:35Z)
• HANS, are you clever? Clever Hans Effect Analysis of Neural Systems [1.6267479602370545]
  Large Language Models (It-LLMs) have been exhibiting outstanding abilities to reason around cognitive states, intentions, and reactions of all people involved, letting humans guide and comprehend day-to-day social interactions effectively. Several multiple-choice questions (MCQ) benchmarks have been proposed to construct solid assessments of the models' abilities. However, earlier works are demonstrating the presence of inherent "order bias" in It-LLMs, posing challenges to the appropriate evaluation.
  arXiv  Detail & Related papers  (2023-09-21T20:52:18Z)
• Delving into Identify-Emphasize Paradigm for Combating Unknown Bias [52.76758938921129]
  We propose an effective bias-conflicting scoring method (ECS) to boost the identification accuracy. We also propose gradient alignment (GA) to balance the contributions of the mined bias-aligned and bias-conflicting samples. Experiments are conducted on multiple datasets in various settings, demonstrating that the proposed solution can mitigate the impact of unknown biases.
  arXiv  Detail & Related papers  (2023-02-22T14:50:24Z)
• Achieving Equalized Odds by Resampling Sensitive Attributes [13.114114427206678]
  We present a flexible framework for learning predictive models that approximately satisfy the equalized odds notion of fairness. This differentiable functional is used as a penalty driving the model parameters towards equalized odds. We develop a formal hypothesis test to detect whether a prediction rule violates this property, the first such test in the literature.
  arXiv  Detail & Related papers  (2020-06-08T00:18:34Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.