When Certainty Is an Artifact: Keyword Lexicon Blindness and the (Mis)Measurement of Rhetorical Stance

• URL: http://arxiv.org/abs/2606.26062v1
• Date: Wed, 24 Jun 2026 17:36:39 GMT
• Title: When Certainty Is an Artifact: Keyword Lexicon Blindness and the (Mis)Measurement of Rhetorical Stance
• Abstract summary: We find a robust negative-affect/emphatic-certainty lexical co-occurrence pattern under keyword-based scoring.<n>We argue that keyword lexicons measure a universal lexical co-occurrence tendency.
• Score: 4.093009863847175
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Can a statistically significant, large-effect-size finding in computational social science be entirely an artifact of the measurement instrument? We present a case where the answer appears to be yes. Analyzing 85 interviews across four public intellectuals (2016--2026), we find a robust negative-affect/emphatic-certainty lexical co-occurrence pattern under keyword-based scoring ($r = 0.72$--$0.93$, $p < 0.01$ for all four speakers). Replacing keyword counting with LLM-based zero-shot semantic classification on the complete diarized corpus (32,625 sentences) dramatically reduces this correlation: Dalio's $r = 0.851$ drops to $r = 0.206$, with two speakers showing negative $r(\text{neg}, \text{emphatic})$ and one showing null. In contrast, the LLM reveals a strong negative-hedging coupling across speakers -- Rogoff's $r(\text{neg}, \text{hedged}) = 0.875$ ($p = 0.001$) and Zeihan's $r(\text{neg}, \text{hedged}) = 0.722$ ($p = 0.008$) -- consistent with the conventional expectation that pessimistic discourse attracts hedging, not certainty. Sentence-level error analysis traces this discrepancy to three structural failure modes in keyword lexicons -- syntactic blindness, polysemy blindness, and categorical absence -- illustrated through cases where keyword counting inverts semantic meaning (e.g., ''never absolutely totally confident'' scored as high-certainty). We argue that keyword lexicons measure a universal lexical co-occurrence tendency -- negative discourse naturally attracts emphatic vocabulary -- that is orthogonal to, and can systematically invert, rhetorical stance. Treating keyword counts as measurements of epistemic certainty is a category error: a finding that appears to be about a speaker's psychology may be entirely about the counting of words.

Related papers

• From a Word-Level Dictionary to Sentence-Level Semantics: Multilingual Grievance Labelling with Contextual Models [3.5005774502836995]
  Grievance is one of the warning signs analysts look for when assessing threats of violence.<n>It is increasingly measured at scale from online text, most often with word-level lexicons that score by matching weighted terms.<n>We show that grievance is measured more faithfully by reading the surrounding context.
  arXiv  Detail & Related papers  (2026-07-23T05:56:04Z)
• Attribute-Based Diagnosis of LLM Alignment with Hate Speech Annotations [48.69228180369574]
  Hate speech annotation is costly, subjective, and prone to annotator disagreement.<n>We analyze how well large language models (LLMs) align with human judgments.<n>We reconstruct continuous hate speech scores from the Measuring Hate Speech corpus.
  arXiv  Detail & Related papers  (2026-05-26T13:44:48Z)
• Sparse Autoencoders Map Brain-LLM Alignment onto Cortical Semantic Topography [13.891522069967507]
  We show that semantic features alone recover 94% of peak encoding performance.<n>We also show that SAE-discovered features recapitulate known cortical semantic organization.<n>Results generalize across English, Chinese, and French.
  arXiv  Detail & Related papers  (2026-05-21T21:00:24Z)
• Scale-Sensitive Shattering: Learnability and Evaluability at Optimal Scale [54.65053906803857]
  We study the optimal scale at which real-valued function classes exhibit uniform convergence and learnability.<n>Our main result establishes a scale-sensitive generalization of the fundamental theorem of PAC learning.<n>We also highlight several open questions on quantitative sample complexity and evaluability.
  arXiv  Detail & Related papers  (2026-05-13T15:41:30Z)
• Beyond Cosine Similarity: Taming Semantic Drift and Antonym Intrusion in a 15-Million Node Turkish Synonym Graph [0.0]
  We build a large-scale semantic clustering system specifically designed to tackle this problem head on.<n>We introduce a labeled dataset of 843,000 concept pairs spanning synonymy, antonymy, and co-hyponymy.<n>Second, we propose a specialized three-way semantic relation discriminator that achieves 90% macro-F1.<n>Third, we introduce a novel soft-to-hard clustering algorithm that mitigates semantic drift while simultaneously resolving polysemy.
  arXiv  Detail & Related papers  (2026-01-19T17:37:25Z)
• Causal Estimation of Tokenisation Bias [58.20086589761273]
  We quantify the effect of including or not a subword in a tokeniser's vocabulary on the probability a trained model assigns to the corresponding characters.<n>We find that tokenisation consistently affects models' outputs across scales, vocabularies, and tokenisers.<n> Notably, a subword's presence in a small model's vocabulary may increase its characters' probability by up to 17 times.
  arXiv  Detail & Related papers  (2025-06-03T17:59:47Z)
• Calibrating Verbal Uncertainty as a Linear Feature to Reduce Hallucinations [51.92795774118647]
  We find that verbal uncertainty'' is governed by a single linear feature in the representation space of LLMs.<n>We show that this has only moderate correlation with the actual semantic uncertainty'' of the model.
  arXiv  Detail & Related papers  (2025-03-18T17:51:04Z)
• Leading Whitespaces of Language Models' Subword Vocabulary Pose a Confound for Calculating Word Probabilities [15.073507986272027]
  We argue that there is a confound posed by the most common method of aggregating subword probabilities into word probabilities. This is due to the fact that tokens in the subword vocabulary of most language models have leading whitespaces. We present a simple decoding technique to reaccount the probability of the trailing whitespace into that of the current word.
  arXiv  Detail & Related papers  (2024-06-16T08:44:56Z)
• Understanding and Mitigating Spurious Correlations in Text Classification with Neighborhood Analysis [69.07674653828565]
  Machine learning models have a tendency to leverage spurious correlations that exist in the training set but may not hold true in general circumstances. In this paper, we examine the implications of spurious correlations through a novel perspective called neighborhood analysis. We propose a family of regularization methods, NFL (doN't Forget your Language) to mitigate spurious correlations in text classification.
  arXiv  Detail & Related papers  (2023-05-23T03:55:50Z)
• Speakers Fill Lexical Semantic Gaps with Context [65.08205006886591]
  We operationalise the lexical ambiguity of a word as the entropy of meanings it can take. We find significant correlations between our estimate of ambiguity and the number of synonyms a word has in WordNet. This suggests that, in the presence of ambiguity, speakers compensate by making contexts more informative.
  arXiv  Detail & Related papers  (2020-10-05T17:19:10Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.